A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval
Authors: Hao Li
Jingkuan Song
Lianli Gao
Pengpeng Zeng
Haonan Zhang
Gongfu Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: cross-modal retrieval
probabilistic embedding
image-text matching
multiplicity
metric learning
robust

Abstract: Cross-modal retrieval aims to build correspondence between multiple modalities by learning a common representation space. Typically, an image can match multiple texts semantically and vice versa, which significantly increases the difficulty of this task. To address this problem, probabilistic embedding is proposed to quantify these many-to-many relationships. However, existing datasets (e.g., MS-COCO) and metrics (e.g., Recall@K) cannot fully represent these diversity correspondences due to non-exhaustive annotations. Based on this observation, we utilize semantic correlation computed by CIDEr to find the potential correspondences. Then we present an effective metric, named Average Semantic Precision (ASP), which can measure the ranking precision of semantic correlation for retrieval sets. Additionally, we introduce a novel and concise objective, coined Differentiable ASP Approximation (DAA). Concretely, DAA can optimize ASP directly by making the ranking function of ASP differentiable through a sigmoid function. To verify the effectiveness of our approach, extensive experiments are conducted on MS-COCO, CUB Captions, and Flickr30K, which are commonly used in cross-modal retrieval. The results show that our approach obtains superior performance over the state-of-the-art approaches on all metrics. The code and trained models are released at https://github.com/leolee99/2022-NeurIPS-DAA.

A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval
Hao Li 18th. www.eelleee@ll.....mmm. com
Jingkuan Song!"
Lianli Gao
j lingkuan. song@gmail.... com
lianli. www.li.gguuesttoomm edu. cn
Pengpeng Zeng!
Haonan Zhang! zchiowa1@gmail. com com
Gongfu Li² gongfuli@tencent. com
is pengpengzenn@@maill com
'Center for Future Media, University Electronic Science and Technology of China
TTencent Wechat Group, Shenzhen, China.
Abstract
Cross-modal retrieval aims to build correspondence between multiple modalities by learning a common representation space. Typically, an image can match multiple texts semantically and vice versa, which significantly increases the difficulty of this task. To address this problem, probabilistic embedding is proposed quantify these many-to-many relationships. However, existing datasets (e.g.. MS-COCO) and metrics (e.g., Recall @ K) cannot fully represent these diversity correspondences due to non-exhaustive annotations. Based on this observation, we utilize semantic correlation computed by CIDEr to find the potential correspondences. Then we present effective metric, named Average Semantic Precision (ASP), which measure the ranking precision of semantic correlation for retrieval sets. Addi- tionally, introduce novel and concise objective, coined Differentiable ASP Approximation (DAA). Concretely, DAA can optimize ASP directly by making the ranking function of ASP differentiable through sigmoid funetion. To verify the effectiveness our approach, extensive experiments are conducted on MS-CÓCO, CUB Captions, and Flickr30K, which are commonly used in cross-modal retrieval. The results show that our approach obtains superior performance over the state- of-the-art approaches all metrics. The code and trained models released https:/ /gi thub. www.w//oeee/////////// 20022-2222222222 Neur IPS-DAA,
Introduction
Vision-language approaches have demonstrated success in various multimodal tasks, such as video
question answering [57], image captioning [56], and text-to-image generation [51]. As one of the most fundamental multimodal tasks, image-text retrieval in great demand due to the explosive increase of multimedia data in social activities. Given an image or text query, it aims to retrieve the most relevant items in the database of other modalities. Although significant progress has been made [21, 12, 43, 37, 55, 26, 25], this task remains challenging since the heterogeneity across different modalities.
tackle this problem, most existing methods map the query and retrieval items
from distinct modalíties into common embedding space, which allows computing the similarity between different modalities.
Deep neural networks have achieved great success in various applications [58, 53J. In cross-modal retrieval, a line of methods [21, 39, 12] extracts representations of images and texts by constructing a
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Ground trath
Image
CIDEY
extinging husolla
run extinlng hourba hot hull Solf)
boseball fieit A baker is leiiging of
eveging g
hater scraping paach the lorathill
extiglng hosehal has nauther fosics (bul1))
cunegfig
soohall slong
hune suleging ar hoseall
A hanur ostaging ar plich during hoseball gaina
Teunnrree ane viacling sling fince
toschall player
heuchull
hauchell zan playings ei leschall game hochal (0,300)
Rese ayer sleiing bouchell sobul 9000
(b)
Text Query: A mun cutling a slice of cake with the knile.
Baseling CIIDE
0:653
0.335
01015
0:011
0:073
0.11%
8.643)
B4522
11225 0.138 0.125 0.1MM 0.212 0L134 0.242 BUSS
Figure 1: (a) An example of the image and its five ground truth captions. The CIDEr list gives 10 captions, followed by their CIDEr scores in the bracket with ground truth marked green, which are computed with all ground truth shown on the left. The captions with a CIDEr score greater than 0.3 almost describe the image correctly (the last three captions in the CIDEr list). (b) The number of image-text pairs greater than the CIDEr threshold. There are a large number of image-text pairs with high semantic correspondence besides the ground truth. The number of potential positive pairs is about 4 times the number of annotated positive pairs, while CIDEr score is greater than 0.3. (c) The top 100 items of retrieved sets predicted by two approaches. Below each image is the semantic similarity computed by CIDEr. Both retrieved sets obtain the same score by AP or Recall @ K (only the first retrieved item is in ground truth). Nonetheless, compared with the first retrieved set, the second one predicted by the model using DAA is more semantically related to the text query. It shows that the similarities computed by CIDEr are generally consistent with the real semantic similarities between the text queries and the images.
two-stream deep neural network to learn the correspondence of inter-modalities well. Another thread of work [43, 37, 54] focuses the relationship within the intra-modaligy Recent studies [26, 25, 29] have shown that vision-language pre-training is effective to learn generic representations from massive image-text pairs. These methods have achieved impressive results on existing benchmarks. However, most of the prior works [21, 39, 12, 43, 10, 37, 54] use projection functions to map a sample as deterministic embedding, which does not definitely account for the potential many- to-many relationships in the existing datasets (e.g.. MS-COCO) with non-exhaustive annotations. Concretely, as shown in the second row Figure (c), given query caption such as "A man cutting a slice of cake with the knife," only one annotated image (the first image) is considered as positive, whereas some other images are also correct the semantic level. To tackle this problem, Chun al. [7] introduce probabilistic embedding for cross modal retrieval, which is powerful to represent many-to-many relationships, by converting the samples into various representations according to distribution.
Although probabilistic models successfully capture potential many-to-many relationships, the diversity of probabilistic embedding heavily depends on the well-matched datasets, which are expensive to annotate manually. Furthermore, commonly used datasets for the cross-modal retrieval task suffer from non-ex non-exhaustive annotations, which miss several positive pairs. These missing positive pairs (MPPs) enormously limit the multiplicity expressed in training and evaluation processes. To excavate these MPPs, we propose a semantics-based many-to-mant correspondence mining strategy, which represents the relevance between images and captions through semantic similarity. Specifically, the semantic metric CIDEr [45] is used to calculate semantic similarity between images and captions, i.e., image-text pairs with CIDEr scores greater than the threshold are regarded as positives. Then, we observe that caption with CIDEr score greater than 0.3 can typically describe images correctly (see Figure (a)). To figure out the data scale of MPPs in commonly used datasets, the number of potential positives with different CIDEr thresholds in the MS-COCO 5K test set is measured in

---

Figure/ (b). It can be seen that 128,050 image-text pairs with CIDEr scores greater than 0.3, which means that each image equally matches 25.61 captions (only 5 annotated captions) on the semantic. The results powerfully demonstrate large number of potential positives are wrongly regarded as negatives, severely impeding diversity training and evaluation processes.
Nonetheless, most optimization methods [12, 15, 13, 47] and metries revolve around the accurate prediction of positives in the ground truth, ignoring the correlations between other retrieved items and the query. As the situation in Figure (c), the retrieved set at the bottom is more in line with our expectations than the one above, even though they obtain the same score by the metries only focusing on the ground truth, such as AP or Recall @ K. It means that ground-truth based metrics are hard to fully represent the multiplicity for cross- modal retrieval. To address these problems, we carry out the following work.
In this work, we give solution alleviate the hindrance of MPPs multiplicity for probabilistic representations in the training process. Firstly, to evaluate with non-ground-truth relevant items, we propose new evaluation metric, named Average Semantic Precision (ASP). ASP computes the average deviation between prediction ranking and semantic ranking. It indicates the semantic ranking precision of each retrieved item. We utilize CIDEr as the criterion for judging the semantic correlations of captions. Secondly, we propose a concise and novel Differentiable ASP Approximation (DAA) method. which optimizes ASP directly by relaxing the discrete ranking function in the non- differentiable ASP with a sigmoid function. Eventually, our approach achieves state-of-thee -hetthe-rt performance on several multiplicity metrics.
The main contributions and novelties of this paper can be summarized below. i) We count the numher of image-text pairs with strong semantic correlations on the most widely used dataset MS-COCO. The result shows incompletely annotating MPPs is crucial factor restricting many-to-many mapping for probabilistic embedding. Thus, we propose an effective evaluation metric named ASP instead of the existing ones, which are powerless to reflect the correlations between each retrieved item and the query. ii) To eliminate the adverse effect of MPPs in the training process, we propose novel and concise metric learning method DAA, which approximates and optimizes ASP directly. iii) To verify the effectiveness of our method, we conduct experiments on two challenging benchmarks and multiple sensible evaluation metrics.
2 Related Work
In this section, we briefly introduce some recent developments in cross-modal retrieval and metric learning.
2.1 Cross-modal Retrieval
Cross-modal retrieval requires a coordinated representation [3] that allows computing the similarity between the query and the retrieved items from distinct modalities. Early works [15] adopt canonical correlation analysis (CCA) to learn linear projection function and maximize the correspondences between positive images and texts. Faghri et al. [12] first uses a triplet loss, which only samples the hardest negative boost the matching performance. Nonetheless, one of the drawbacks of these approaches is their neglect to capture the correspondences between two modalities at a fine- grained level. Some prior works [23, 24, 10] capture fine-grained local semantic alignments between visual regions and words. SCAN [23] proposes learning to extract region features by an object detector pre-trained reasoning on the region proposals generated by Faster-RCNN [36] detector. It gains significant performance improvements with the large increase in computing expenses. SGRAF [10] constructs a similarity graph to identify complex matching patterns and adopts a similarity attention filtration module eliminate the interferences of no-meaningful alignments. However, the aforementioned approaches ignore the potential many-to-man correspondences in the datasets with non-exhaustive annotations. Parekh et al. [32] construct an extended MS-COCO dataset, i.e., CrissCrossed Caption (CxC) dataset, which provides denser annotations to excavate more potential positive relationships between unassociated images and captions. Recently, Chun el al. [7] use probabilistic embedding in cross-modal retrieval to obtain many-to-many alignments between images and captions. A transition of R-Precision incorporating plausible matches by class-based similarity is adopted to calculate the multiplicity score. Additionally, some works [59, 42] explore the realistic distribution to
Visual Genomes [22]. VSRN [24] adopts graph neural network for semantic
3

---

make the probabilistic models express more accurate saany-to-man--rlaiiinn progress [32. 41, 7] has been made in capturing potential correspondences, MPPs still seriously impede the multiplicity performance of cross-modal models. To tackle this problem, we employ semantic metrics on representing the semantic correlations between images and captions to excavate the potential positive pairs and evaluate the multiplicity of cross-modal models.
Although some
2.2 Metric Learning
Metric Learning maps imput data into an embedding space that distance metrics can be applied easily. A great body of work [2, 1, 48, 44, 40, 6, 49] has focused on metric learning. For instance, contrastive [6] loss and triplet [49] loss are two of the most widely used methods. The former forces all positive instances closer, while negative pairs are only separated by fixed margin. In the latter, positive pairs are pushed closer than negative pairs. Although impressive success has been achieved, the motivation to only minimize distance is limited. The aforementioned methods ignore the ranking orders, which are essential for rank-based metries such Average Precision (AP). Brown al. [5] propose a differentiable AP approximation loss that can optimize AP directly in image retrieval. Compared with triplet loss, it shows better performance in AP, ranking-based metric. Impressive progress [44, 40, 11, 18, 17, 33, 38] in optimizing AP has been achieved, but AP is limited metric, which only focuses on the ground truth. In our work, ranking- based metric ASP is proposed to exploit potential semantic information. Besides, we propose a Differentiable ASP Approximation (DAA) to optimize ASP directly. To our knowledge. DAA is the first work that uses direct metric optimization for cross cross-modal retrieval.
3 Preliminaries
Image B
Text B
A man jumping in the ar while su seiignn leanis racket
match
match
matzh
Image
Text A
Probahilittic Fncoder
Probebilstic Excoder
person that playing in a lecals game.
Shared space
Caption
Image
mam firting timmis hall from the back the court in the daytime.
match
non-melch
C
Image
Figure 2: An example of probabilistic embedding to represent multiplicity. For each positive pair (e.g., image A and text A), it will generate two distributions through a probabilistic encoder. For the distribution coming from image A (marked red in shared space), captions within the distribution (e.g., Text B and C) all match image A. Similarly, Image and C can match Text A. The image and text on the intersection of two distributions (marked purple in shared space) will match each other (e.g., Image B and Text B), while non-match on the outside.
Probabilistic embedding. Chun et al. [7] have a previous attempt at Probabilistic Trosl-Modal Embedding (PCME) to address the multiplicity of cross-mode correlation. PCME models each sample as a normal distribution (See Figure 2), which could be described by mean and variance together. Therefore, the key to generating probabilistic embedding is determining the mean embedding and the variance matrix. Given an image v E V and a caption / E T as inputs, probabilistic model will learn their normal distributions
((E(v)))) (h) (fv), hi (diag(f.))
pf (((())))) (1, (ft). NG (diag(ft))
where f is the feature map extracted from the visual encoder and ft the feature sequence extracted from the textual encoder. E(-) is function mapping input to probabilistic embedding. h" and h"

---

are two head modules used to compute the mean embedding and the variance matrix in PCME [7], respectively. To learn the joint probabilistic embedding, the soft cross-modal contrastive loss is used [14], which can be defined as:
log po (m/u,,
if t is match
(0)
log(1 - (m(m/ t))
otherwise
where 0 (0v,8r) Br) are the parameters for the visual and the textual encoders, respectively. Given an image-text pair (v) t) as input, po pm(m)uu v, t) is the match probability from Euclidean distances can be similarly denoted as:
(o 1) f -
E(L) 1|2 bb,
where E(r)² and E(t)" are samples from the embedding distribution p (b)()))) and p p(E(t))t via Monte-Carlo estimator [31], respectively. (a, b) are learnable scalars and ob-) is sigmoid function.
4 Approach
In this section, elaborate on the proposed method. In Section 4.1, first delineate novel eval- uation metric, i.e., Average Semantic Precision (ASP), which can calculate the semantic correlation scores of the retrieved easily and precisely, even in noisy datasets. In Section 4.2, detail how to optimize ASP directly in probabilistic embedding by using Differentiable ASP Approximation (DAA).
4.1 Average Semantic Precision (ASP)
Existing diversity metries [7, 32] for not well-aligned datasets are hard fully represent the semantic ranking accuracy of retrieved sets limited by incomplete correlation labels, such as PMRP [7] or CxC [32]. The former treats samples in MS-COCO [27] whose class label difference numbers are less than 2 as positives, but it will make lot of false matches. The latter manually annotates the testing set of MS-COCO, but still misses many potential positive pairs. Meanwhile, both of them are merely applicable to MS-COCO, i.e., they are not universal. To solve these problems, we propose Average Semantic Precision (ASP). Given a query set = (Q,, 0, n1), query E can simply calculate the semantic correlation scores of the retrieved set Ω (G,, i 0, ..., m) in various image-text retrieval datasets. Mathematically,
1 ASP,
Σ SEN
R(1,
(4)
Ω
Mar(R(i, R(,
where R(i, C) and R(i, S) refer to the rankings of the instance i in semantic list C (defined as Eq. 6) computed by CIDEr [45], and similarity list S (defined Eq. 5) predicted by the model. In Eq. ASP utilizes the overall gap between the semantic ranking and the similarity ranking to represent the precision of retrieved set.
| |E(i] ) i= 0, 1.......mml m}.
E(q)
E(i)
S = [s+
(5)
C= (= (i.(i.)))) q), i 0,1,..,m
(6)
The SEM (i, q) the semantic correlation score between query q and retrieved item i. When g and refer to an image and a text, we will select all positive texts (5 texts in MS-COCO) corresponding to the image, and calculate their semantic similarities with through the CIDEr metric, respectively. Eventually, the mean of these semantic similarities will represent the semantic similarity between q and Conversely, the positive texts corresponding to image i will be selected calculate the semantic correlation scores with text q. The specific equation is follows:
P
(i,, Pos(a)s)
if q is an image
P
SEM(i, =
(Pos(()),
is text
j=1
where Pos(g) the positive texts of g, P is the number of texts in Pos(g), & stands for the CIDEr metric.
5

---

4.2 Differentiable ASP Approximation (DAA)
Contrastive [6] and triplet [49] losses are widely used metric learning methods, which are driven by minimizing distance. They both ignore the ranking order, which is important for ranking- based metrics. Thus, the ranking-bass based metrics will be indirectly optimized in these methods. ASP is ranking-based metric that is neither differentiable nor decomposable. Consequently, we propose Differentiable ASP Approximation (DAA) to optimize ASP directly.
Given the ranking-basee metric ASP, the reason why it is non-differentiable is that the derivative of ranking function R. for instance defined in Eq. is either gradient zero discontinuous [34], resulting in the inability to be optimized with gradient-based approaches.
R(i,U)=
E 1[(u - uz) 00:
(8)
jEUJ
where 1{+} serves Indicator function, which select I while (uy < O, otherwise select [34]. U is the CIDEr score set C' or the similarity score set S. The difference matrix of D E mR can be computed as:
W1
Um
"1
D
(9)
U1
tmm
tm
The ASP for the query q can be converted from Eq. 4 to:
1 > 0f. E jjn 11D > 0}) 1 Mar( 1 > 0}, Eje.iii 1( > 0)) IEN
1+
ASP, = Ω
Σ
(10)
Inspired by [5], we utilize sigmoid function 6(-) instead of the Indicator function 1(:} to make ASP differentiable, which could achieve direct optimization of ASP. After substituting (6-) into Eq. the approximation of ASP is denoted as:
+ Min(E Jen (DD)), Ejen (DD))) 1 Max(M jen &0 (D Ejea Eg),Eeea øl (Di)))
ASP,
(11)
S2
which is differentiable and could be optimized with gradient- based methods. Therefore, the final loss function is defined as:
CASP
Σ (1-AAPP)) ASP)),
(12)
V
kEW
Experiment
In this section, conduct experiments verify the effectiveness of DAA, which could optimize
ASP directly. We describe the evaluation protocol, including datasets and metrics, as well as the
implementation details. Then report experimental results on MS-COCO [27], CUB Captions [7], and Flickr30K [52] for the cross-modal retrieval task. After that, we carry out a series of ablation studies to verify the effectiveness of DAA in improving multiplicity. Meanwhile, we present experimental analysis the effectiveness of DAA.
5.1
Evaluation Protocol
5.1.1 Datasets
We conduct experiments MS-COCO [27] and CUB Captions [7] to show the performance of probabilistic embedding using DAA for cross-man tross-modal retrieval. Although our method is proposed to improve the multiplicity of the probabilistic model, it seems that the proposed model can be integrated into any tross-modal retrieval method. Thus, we also employ DAA in non-probabiliitic embedding on MS-COCO and Flickr30K [52] to explore the generalization of our method.
MS-COCO is a widely used dataset the cross-modal retrieval task. contains 123,287 images with five captions per image. We follow the data partition in [20]. which consists of 113.287 images

---

for training, 5,000 images for validation, and 5,000 images for testing. We report our results on both 5K and IK testing sets (the average over 5-folds of 1K testing images).
CUB Captions [7, 35, 46] is a benchmark proposed to reduce the impact of false matching for probabilistic emhedding in MS-COCO. It contains 11,788 images of 200 fine-grained bird categories with 10 captions per image. All the image-texi pairs belonging to the same class are considered positive pairs, false negatives rarely exist. Meanwhile, the homogeneity of image-text pairs in the same class could greatly suppress false positives. We follow the class splits in [50], which consists of 150 classes for training and validation, and the remaining 50 for testing.
Flickr30K [52] is a widely used dataset for the cross-modal retrieval task. It contains 31783 images with 5 captions per image. We follow the split in [13], which consists of 1000 images for validation, 1000 images for testing, and the rest for training.
5.1.2 Metrics
We take Recall@K K (R@K), a widely used metric in cross-modal retrieval, measurement in both benchmarks. We report R@ 1, R@,, and @@@ 10 for a comprehensive evaluation. However, R@ K only focuses on the position of the first relevant item according the ground truth, reflecting limited information.
R-Precision (R-P) is alternative proposed by Musgrave et al. [30]. For each query, R-P computes the ratio of matched items in the top-r retrieved items, where is the number of ground-truth matches. In short, for all matched items, the fewer negatives before, the higher the R-P score is. Compared with @@K K. R-P could better evaluate the one-to-many matching for tross-modal retrieval. We use R-P evaluate the performance of DAA CUB Captions.
As MS-COCO, we take Plausible Match R-Precision (PMRP) [7] metric as measurement, which utilizes the class labels for MS-COCO to find out all hidden positive image-text pairs. Concretely, PMRP regards the pair (v, t), whose binary label vectors y", y² E {0, 1)dtoat differ at most at positions (S {0, 1, 2}), positive matches.
The above metries can only reflect the precision of ground-truth items, ignoring the correlations between other non-matee matched retrieved items and queries. Average Semantic Precision (ASP) is metric proposed reflect the models' multiplicity by evaluating the retrieved sequence's 's semantic relevance. Therefore, we refer to ASP (Eq. 4) as an evaluation measurement for both benchmarks.
5.2 Implementation Details
For the probabilistic model, we use ResNet [16] and the basic version of the pre-trained BERT [8] as visual and textual encoders. There are three phases in the training proceeds: The first is a pre-training phase, where only the head modules in the visual encoder and all parameters in the textual encoder are trained. A warm-up phase lasting three epochs is used in the pre-training phase. Finally, all parameters will be trained in the men-ttning phase. Concretely, for MS-COCO we use R5s backbone with embedding dimension D 1024. For CUB, we use ResNet-50 backbone with embedding dimension = 512. Meanwhile, We utilize AdamP Optimizer [19] for model training with the batch size of 128 and 1.50 for MS-COCE and CUB, respectively. For both datasets, we set the probabilistic sample number as and the initial learning rate as 0.00015 with the cosine learning rate scheduler [28]. Besides, models are always trained single RTX A6000 GPU with Cutout [9] and random caption dropping [4] augmentation strategies with 0.2 and 0. crasing ratios, respectively.
For the poopbbobbbillittcc model, we take the publicly released code of SGRAF [10] to employ our method. There are two independent modules in the SGRAF network (SGR and SAF). Consistent settings are used for the training of SGR and SAF as follows. We conduct all experiments on 4 RTX A6000 GPUs with the mini-batch size of 128. Besides, the Adam optimizer [19] is employed for model training with the initial learning rate of 0.0006 for the first 30 (40) epochs and decays it to 0.00006 for the last 10 epochs on MS-COCO [27] (Flickr30K [52]). In the evaluation stage, the performance of SGRAF is predicted by averaging the prediction similarities of SGR and SAF.
7

---

Table 1: Comparison on MS-COCO
IK Test Images
5K Test Images
Method
i2t R@1
t2
i2t
t2i
PMRP
ASP
PMRP
@@l
ASP
PMRP
R @
ASP
PMRP
R@1
ASP
VSE++
64.6
52.0
41.3
30.3
PVSE K=1
40.3
66.7
41.8
53.5
29.3
41.7
30.1
30.6
PVSE K-2
42.8
69.2
43.6
55.2
31.8
45.2
32.0
32.4
PCME
45.0
68.8
54.7
46.0
54.6
55.3
34.1
44.2
54.5
34.4
31.9
55.0
PCME-bert
44.3
69.2
54.4
45.3
56.2
54.4
33.0
45.6
54.2
33.4
33.2
54.2
+DAA
45.9
71.2
58.4
46.7
57.8
57.2
34.6
48.2
58.3
34.9
35,1
57.0
SAF
46.2
77.1
54.7
47.3
62.1
54.8
34.7
55.1
54.5
35.3
40.5
54.5
+DAA
47.1
78.0
67.2
48.7
62.8
61.6
35.7
56.2
67.0
36.6
40.5
61.4
SGR
44.0
77.2
53.9
46.0
61.9
54.6
32.7
55.8
53.7
34.2
40.1
54.4
+DAA
46.4
78.0
68.5
48.6
62.6
62.8
35.3
56.5
68.4
36.9
40.8
62.6
SGRAF
47.1
79.4
54.5
48.0
63.9
55.0
35.5
59.0
54.3
35.9
42.6
54.8
+DAA
48.1
80.2
68.3
49.6
65.0
62.7
36.6
60.0
68.2
37.5
43.5
62.5
5.3 Comparisions with State of The Arts
In this section, compare method with previous works using ResNet as the visual encoder on the two datasets. The comparison methods include VSEO [12], VSE++ [12], PVSE [41], and PCME [7]. Besides, we utilize our method 4 typically baselines in image-text retrieval including PCME-bert [7], SAF [10], SGR [10], and SGRAF [10]. PCME-bert is variety of PCME using the basic version of BERT as the textual encoder. It achieves state-of-the-art performance in probabilistic approaches. For nonpprobabilistic approaches, we select three commonly used baselines with
impressive performance, including SAF, SGR, and SGRAF
Results on MS-COCO. Table shows the quantitative results of MS-COCO 1K and 5K testing sets for image-to-text (i2t) and text-to-
Table 2: Comparison on CUB Captions
image (t2i) retrieval. We observe that ap- proach obtains a consistent improvement in per- formance all metrics of each baseline. Ex- actly, PCME-bert using implementation strategy Section 5.2 emerges better performance than PCME on @@! 1. Then, DAA is employed in the training process of PCMEEbert, resulting in sigmificant boosts on R@ 1, PMRP, and ASP. As for MS-COCO (IK), compared with PCME, we achieve absolute boosts on PMRP, R @@1 1, and ASP in both i2t and t2i, which are 0.9%, 2.4%, 3.7% and 0.7%, 3.2%, 1.9%, respectively. As for MS-COCO (5K), compared with PCME, the boosts on PMRP, R and ASP are 0.5%, 4.0%, 3.8% and 0.5%, 3.2%, 2.0%. Besides, for non-probabilistic approaches, AG using DAA achieves ttate-of-the-ar performance with R@ 1 of 80.2% (60.0%) and 65.0% (43,5%) for text and image retrieval MS-COCO IK (5K), separately. Meanwhile, the impressive improvement in PMRP and ASP means that SGRAF has achieved higher multiplicity performance even than the probabilistic model. The above results show that DAA could effectively enhance the scores of multiple metrics on various cross- modal models, which indicates that semantics useful to boost models' performance in cross-modal retrieval.
Text-to-image R-P R@1 1 ASP R-P R@ ASP
Image-to-text
Method
VSEO
22.4
44.2
22.6
32.7
PVSE K=1
22.3
40.9
20.5
31.7
PVSE K-2
19.7
47.3
21.2
28.0
PCME
26.3
46.9
53.1
26.8
35.2
56.8
PCME-bert +DAA
27.2
49.2
52.9
27.8
36.3
56.0
28.2
53.2
53.4
28.5
37.7
57.6
Results on CUB Captions. To demonstrate the robustness of the proposed approach for probabilistic models, we conduct experiments on CUB Captions proposed to evaluate the multiplicity of proba- bilistic models. Compared with MS-COCO, R-P P is more suitable for CUB due to rare false pairs. The results on CUB are presented Table can be seen that PCME--ber using DAA outperforms all the state-of-the the-tarrt methods. Especially for image-to-text retrieval, using DAA surpasses the

---

previous best method on R-P. R@ I, and ASP with an increase of 1.9%, 6.3%, and 0.3%, respectively. Besides, for text-to-image retrieval, the model using DAA also outperforms the best counterpart by significant margin (1.7% for R-P, 2.5% for R R@1 1, and 0.8% for ASP).
Results on Flickr30K. To further demonstrate the robustness of DAA for non-probabilistic models, we employ SAF, SGR, and SGRAF trained with the proposed method DAA on Flickr30K. The results are shown in Table We can observe that DAA still improves the performance of all models on both @@) and ASP. Additionally, SGRAF trained with DAA achieves state-of-the-ar performance with a 0.9% increase for @@] and a 16.4% increase for ASP on i2t, as well as 1.4% for R@ 1 and 8.6% for ASP on t2i. The above results prove that DAA has strong generalization for improving the performance of cross-modal models and demonstrates the effectiveness and importance of exploiting semantic information in cross- moosl retrieval.
5.4 Ablation Studies
In this section, we perform several ablation stud- ies MS-COCO 5K explore the superiority of DAA over the optimization method only focusing on ground truth (e.g., Smooth-AP [5)), the probabilis- tic model.
Table 3: Comparison on Flickr30K
i2t
t2i
Method
R
ASP
@@1
ASP
To explore the effect of DAA and Smooth-AP, we test the impact of using these two methods on multiple metrics. The results are shown in Table 4. First, evaluate PCME-bert as the baseline, where the sam- ple number is 5. Subsequently,
SAF
73.8
49.0
56.7
50.0
+DAA
73.9
65.0
56.9
58.4
SGR
73.7
54.6
55.8
55.1
+DAA
73.8
65.5
56.6
59.0
train model with Smooth-AP and find that there are only a few improvements on PMRP, R@ 1, and ASP. Finally, we employ the D to the baseline, resulting in consid- erable boosts on all metries, especially @ (2.6%
SGRAF
77.1
49.4
58.5
50.6
+DAA
78.0
65.8
59.9
59.2
for i2t, 1.9% for (2i) and ASP (4.1% % for i2t, 2.8% for 12i). The significant improvement of ASP shows that DAA shifts retrieved items with high relevance forward overall. These results implicate that excavating semantic correlation can not only improve the multiplicity of probabilistic models but also improve the traditional ground-trute trad--ttuuthheeeee based metrics, such as R@K K. The visualization of top 100 retrieved results is shown in Figure (c).
The results in Table 4 prove that DAA can increase the semantic similarity in the whole retrieval set, which can be regarded as a set with large CIDEr score variance. To validate the effect of DAA for subsets with different CIDEr score variance, in descending order and sample the top N to compute ASP. We draw the contrastive curves of ASP in Baseline, DAA, and Smooth-AP [5] with N from 0 to the length of the whole retrieval set in Figure 3, We can observe that DAA always performs better than others, and the gap becomes larger along with the increase of N. The results indicate that DAA could enhance diversity performance any retrieval set. Meanwhile, the curves of Smooth-AP and baseline are almost consistent, which shows that only focusing on ground truth is limited improving multiplicity.
rank the retrieved set according the CIDEr score
Table Effect of Smooth-AP and DAA on MS-COCO 5K
Image-to-ten R@1
Text-to-image
Method
PMRP
ASP
PMRP
R@1 I ASP
Baseline (B)
33.0
45.6
54.2
33.4
33.2
54.2
B+
33.0
45.4
54.2
33.5
33.3
54.2
B+DAA
34.6
48.2
58.3
34.9
35.1
57.0
To make clear the impacts of different methods on multiplicity in the training process, we test ASP in each epoch for baseline, Smooth-AP, and DAA. The results are shown in Figure 4, which provides a quantitative analysis of ASP against epochs. We can observe that with the increase of training rounds, the curve of the training strategy using DAA grows constantly. It substantially verifies that DAA effectively captures complex semantic correspondences and enhances multiplicity in the training process. The ASP curves of baseline and Smooth-AP strategics are almost invariant, while R@K K

---

(a) Imag ett-tee-ttett
(b) Text-to-image
Baseline
+StoothhAA
+SmasthwAP
+DAA
+DAA
57
56
56
55
55
54
54
0
1000
2000
3000
4000
5000
5000
10000
15000
20000
25,00
Sample Number N Ordered CIDEr List
Sample Number of Ordered CIDEr List
Figure 3: Ablation studies of the retrieval subsets with different CIDEr score variance on ASP metric. Given a retrieval set descending order according to CIDEr score, we compute the ASP top-N items. The variance will increase with the increase of N.
Image-to-text
Text-to-lmage
58
Biseline
Baseline
+DAA
ASP (88)
56
54
20
Epoch
Epach
Figure 4: Ablation studies of different training strategies during training on ASP metric. The training process consists of 70 epochs. The first epochs are the warm-up phase, the 27 epochs are the pre-training phase, and the last 40 epochs are the fine-tuning phase.
improves constantly. This means that boosting the prediction accuracy of the items in ground truth may not improve multiplicity. All of these ablation studies strongly demonstrate that DAA is superior to the optimization method focusing only on ground truth in improving multiplicity.
Conclusion
This paper attempts to address a critical problem probabilistic embedding for cross-modal retrieval, ie., the not well-aligned datasets and existing metrics can hardly train and evaluate the multiplicity performance the model. solve this problem, we introduce Average Semantic Precision (ASP) to evaluate the performance of multiplicity in these not well-aligned datasets. Moreover, novel Differentiable ASP Approximation (DAA), which could make the model diversified even training in these noisy datasets. We have demonstrated the effectiveness of DAA for boosting the multiplicity of probabilistic embedding for cross- modal retrieval by outperforming the state-of-the-an models.
propose
Acknowledgements
This study supported by grants from National Key R&D Program of China 200XTTC00000//222XX2220000 the National Natural Science Foundation of China (Grant No. 62122018, No. 62020106008, No. 61772116, No. 61872064).
References
[1] Relja Arandjelovic and Andrew Zisserman. "All About VLAD". In: CVPR. 2013, pp. 1578- 1585.
[2] Relja Arandjelovic et al. "NetVLAD: CNN Architecture for Weakly Supervised Place Recog-
nition". In: IEEE Trans. Pattern Anal. Mach. Intell. 40.6 (2018), pp. 1437-1451.
10

---

[3] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philigee Morency. "Multimodal Machine
Learning: Survey and Taxonomy". In: IEEE Trans. Pattern Anal. Mach. Intell. 41.2 (2019),
pp.
[4] Samuel R. Bowman et al. "Generating Sentences from Continuous Space". In: CoNLL
Pp. 10-21.
[5] Andrew Brown et al. "Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval", In: ECCV. Vol. 12354. 2020, pp. 677-694.
[6] Sumit Chopra, Raia Hadsell, and Yann LeCun. "Learning a Similarity Metric Discriminatively,
with Application to Face Verification". . In: CVPR. 2005, pp. 539-546.
[7] Sanghyuk Chun et al. "Probabilisticc Embeddings for Cross-Modal Retrieval". In: CVPR. 2021,
pp. 1424-.424.
[8] Jacob Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". In: NAACL-HLT. 2019, pp. 4171-4188
[9] Terrance Devries and Graham W. Taylor. "Improved Regularization Convolutional Neural Networks with Cutout". In: CoRR 005/1708.04555 (2017).
[10] Haiwen Diao et al. "Similarity Reasoning and Filtration for Image-Text Matching". In: AAAI
2021, pp. 1218-1226.
[11] Martin Engilberge et al. "SoDeep: A Sorting Deep Net to Learn Ranking Loss Surrogates", In: CVPR. 2019, pp. 10792-10801.
[12] Fartash Faghri et al. "VSE++: Improving Visual-Semantic Embeddings with Hard Negatives".
In: BMVC. 2018, p. 12.
[13] Andrea Frome al. "DeViSE: A Deep Visual-Semantic Embedding Model". In: NIPS. 2013,
Pp. 2121-2129.
[14] Raia Hadsell, Sumit Chopra, and Yann LeCun. "Dimensionality Reduction by Learning an Invariant Mapping". In: CVPR. 2006, pp. 1735-1742.
[15] David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. "Canonical Correlation Analysis:
An Overview with Application to Learning Methods". In: Neural Comput. 16. 12.12 (2004), pp. 2639-2664.
[16] Kaiming He et al. "Deep Residual Learning for Image Recognition". In: CVPR. 2016, pp. 770- 778.
[17] Kun He, Yan Lu, and Stan Sclaroff. "Local Descriptors Optimized for Average Precision". In: CVPR. 2018, pp. 596-605.
[18]
Kun He et "Hashing as Tie-Aware Learning Rank". In: CVPR. 2018, pp. 4023-4032. Byeongho Heo et al. "AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights". In: ICLR. 2021.
[19]
[20] Andrej Karpathy and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions".. In: CVPR. 2015, pp. 3128-3137.
[21] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. "Unifying Visual-Semantic Embed- dings with Multimodal Neural Language Models". In: CoRR abs/1411. 1.2539 (2014).
[22] Ranjay Krishna et al. "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations". In: Int. J. Comput. Vis. 123.1 (2017), pp. 32-73.
[23] Kuang-Huei Lee et al. "Stacked Cross Attention for Image-Text Matching". In: ECCV. Vol. 11208. 2018, pp. 212-228.
[24] Kunpeng Li et al. "Visual Semantic Reasoning for image-Text Matching". In: ICCV. 2019,
pp. 4653-4661.
[25] Wei Li et al. "UNIMO: Towards Unified-Modal Understanding and Generation via Cross-
Modal Contrastive Learning". In: ACL. 2021, pp. 2592-2607.
[26] Xiujun Li al. "Oscar: Object-Semantics Aligned Pre-training for Vision-Languee Tasks".
In: ECCV. Vol. 12375. 2020, pp. 121-137.
[27] Tsung- Yi Lin et al. "Microsoft COCO: Common Objects in Context". In: ECCV. Vol. 8693.
2014, pp. 740-755.
[28] Ilya Loshchilov and Frank Hutter. "SGDR: Stochastic Gradient Descent with Warm Restarts".
In: ICLR. 2017.

---

[29] Xiaopeng Lu, Tiancheng Zhao, and Kyusong Lee. "VisualSparta: An Embarrassingly Simple Approach Large-scale Text-to-Image Search with Weighted ofg-of-words" In: ACL. 2021,
pp. 5020-5029,
[30] Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. "A Metric Learning Reality Check". In: ECCV. Vol. 12370. 2020, pp. 681-699.
[31] Scong Joon Oh et al. "Modeling Uncertainty with Hedged Instance Embeddings". In: ICLR. 2019.
[32] Zarana Parekh et al. "Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO". In: EACL. 2021, pp. 2855-2870.
[33] Sebastian Prillo and Julian Eisenschlos. "SoftSort: A Continuous Relaxation for the argsort Operator". In: JCML. Vol. 1 119. 2020, pp. 7793-7802.
[34] Tao Qin, Tie- Yan Liu, and Hang Li. "A general approximation framework for direct optimiza- tion of information retrieval measures". In: Inf. Retr. 13.4 (2010), pp. 375-397.
[35] Scott E. Reed et al. "Learning Deep Representations of Fine-Grained Visual Descriptions", In:
CVPR. 2016, pp. 49-58.
[36] Shaoqing Ren al. "Faster R-CNN: Towards Real-Time Object Detection with Region
Proposal Networks". In: IEEE Trans. Pattern Anal. Mach. Intell. 39.6 (2017), pp. 1137-1149.
[37] Shuhuai Ren et al. "Learning Relation Alignment for Calibrated Cross-modal Retrieval". In:
ACL. Ed. by Chengqing Zong et al. 2021, pp. 514-524.
[38] Jérôme Revaud et al. "Learning With Average Precision: Training Image Retrieval With a Listwise Loss". In: ICCV. 2019, pp. 5106-5115.
[39] Hwanjun Song et al. "Learning from Noisy Labels with Deep Neural Networks: A Survey". In:
CoRR abs/2007.088 011220007.19999 199 (2020).
[40] Hyun Oh Song et al. "Deep Metric Learning via Lifted Structured Feature Embedding". In: CVPR. 2016, pp. 4004-4012.
[41] Yale Song and Mohammad Soleymani. "Polysemous Visual-Semantic Embedding for Cross- Modal Retrieval". In: CVPR. 2019, pp. 1979-1988.
[42] Yuhta Takida al. "SQ- VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization". In: ICML. Vol. 162. 2022, pp. 20987-21012.
[43] Christopher Thomas and Adriana Kovashka. "Preserving Semantic Neighborhoods for Robust Cross-Modal Retrieval". In: ECCV. Ed. by Andrea Vedaldi et al. Vol. 12363. 2020, pp. 317- 335.
[44] Evgeniya Ustinova and Victor S. Lempitsky. "Learning Deep Embeddings with Histogram Loss". In: NIPS. 2016, pp. 4170-4178.
[45] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. "CIDEr: bonsensus-based image description evaluation". In: CVPR. 2015, pp. 4566-4575. Catherine Wah et al. "The caltech-ucsd birds-200-201 dataset". In: (2011).
[46]
[47] Liwei Wang, Yin Li, and Svetlana Lazebnik. "Learning Deep Strueture-Preservingg Preserving Image- Text Embeddings". In: CVPR. 2016, pp. 5005-501
[48] Xinshao Wang et al. "Ranked List Loss for Deep Metric Learning". In: CVPR. 2019, pp. 5207- 5216.
[49] Kilian Q. Weinberger, John Blitzer, and Lawrence K. Saul. "Distance Metric Learning for Large Margin Nearest Neighbor Classification". In: NIPS. 2005, pp. 1473-1480.
[50] Yongqin Xian, Bernt Schiele, and Zeynep Akata. "Zero-Shot Learning The Good, the Bad and the Ugly". In: CVPR. 2017, pp. 3077-3086.
[51] Tao Xu et al. " "AttGAA: Fine-Grained Text to Image Generation With Attentional Generative Adversarial Networks". In: CVPR. 2018, pp. 1316-1324.
[52] Peter Young et al. "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions".
In: Trans. Assoc. Comput. Linguistics 2 (2014),
pp. 67-78.
[53] Xingrui Yu et al. "How does Disagreement Help Generalization against Label Corruption?" In:
ICML. Vol. 97. 2019, pp. 7164-7173.
[54] Xin Yuan al. "Multimodal Contrastive Training for Visual Representation Learning". In:
CVPR. 2021, pp. 6995-7004.
12

---

[55] Pengpeng Zeng et al. "Conceptual and Syntactical Cross-modal Alignment with Cross-level Consistency for Image-Text Matching". In: ACM MM. 2021, pp. 2205-2213,
[56] Pengpeng Zeng et al. "Progressive Tree-Structured Prototype Network for End-to-End Image Captioning". In: ACM MM. 2022.
[57] Pengpeng Zeng al. "Video Question Answering With Prior Knowledge and Object-Sensitive Learning".. In: IEEE Trans. Image Process. 31 (2022), pp. 5936-5948.
[58] Ji Zhang et al. "Progressive Meta-Learning With Curriculum". In: IEEE Trans. Circuits Syst. Video Technol. 32.9 (2022), pp. 5916-5930.
[59] Xiaosu Zhu et al. "Unified Multivariate Gaussian Mixture for Efficient Neural Image Com- pression". In: CVPR. 2022, pp. 17612-17621.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1
(b) Did you describe the limitations of your work? [No] It need be explored.
(c) Did you discuss any potential negative societal impacts of your work? [No] No explicit negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 4
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See abstract, as URL
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 5
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes) See Section
(b) Did you mention the license of the assets? [Yes] See Section 5
(c) Did you include any assets either in the supplemental material URL? [Yes] See abstract, as a URL
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "kWMZ_0PHSCL", "reviewer": "Reviewer_CS9A", "summary": "The multiplicity of correspondences between images and texts in cross-model retrieval greatly increases the difficulty of this task. To tackle this problem, probabilistic embeddings are proposed to quantify many-to-many relationships, but nonexhaustive annotations limit their performance. Based on this observation, they utilize semantic correlation computed by CIDEr to provide a pseudo annotation. Then they propose an effective metric, named Average Semantic Precision (ASP), to measure the ranking precision of semantic correlation for retrieval sets. \nAdditionally, they introduce a novel and concise objective, coined Differentiable ASP Approximation (DAA) to optimize ASP directly by making the ranking function of ASP differentiable through a sigmoid function. To verify the effectiveness of their approach, extensive experiments are conducted on MS-COCO and CUB Captions.", "questions": "From reading the paper, it is very evident that the authors do not come from ML but from a CV background. Some of the concepts are used in a weird name before definition, e.g., 'multiplicity of correspondences between images and texts', 'semantic correlation', 'many-to-many relationships.' in the abstract.\n\nMethods:\nIt is not convincing that ASP is a valid evaluation metric. More descriptions and experiments are needed.\nWhat is the benefit of these evaluation metrics, and what are the shortcomings?\nIt is unclear to me why ASP is stuck to CIDEr?\n\nMy major concern is about the experiments. The experiment part has three issues:\nFew experiments: I think it is not enough to make general claims from having only two experiment datasets. Especially as the datasets are pretty small, i.e., the two datasets are relatively small, with 123,287 images and 11,788 images. Therefore, different and more datasets would enormously benefit the paper's claims.\n\nLack of quantitative comparisons. The proposed method improves existing SOTAin what cases? and when it fails? Why can it benefit both ASP and R@1?\n\nMissing comparison with alternative approaches. This paper compares with VSE0 [11], PVSE[38], and PCME[7], which were published in 2017, 2019, and 2021.  There is some substantial literature on cross-modal retrieval recently.  Therefore, the paper should experimentally evaluate how these models compare to the proposed approach.", "limitations": "See above", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work nicely combines a novel evaluation metric for cross-model retrieval and deep learning model in a novel way and provides an effective approach. The paper is also well-motivated and well-organized.\nMajor comments:\ni) They find that incompletely annotating MPPs is a crucial factor restricting many-to-many mapping for probabilistic embedding. Thus, we propose an effective evaluation metric named ASP, instead of the existing ones, which are powerless to reflect the correlations between each retrieved item and the query.\nii) To eliminate the adverse effect of MPPs in the training process, they propose a novel and concise metric learning method DAA, which approximates and optimizes ASP directly. This seems to be one of the major contributions of this paper.\niii) Promising experimental results.\n\nI believe the paper has some merits. However, it also faces serval weaknesses that need to be addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "f4LLjo7ZVPY", "reviewer": "Reviewer_YuA1", "summary": "This paper focuses on addressing a critical problem in probabilistic embedding for cross-modal retrieval, i.e., the multiplicity performance of the model is not well studied due to the not-well-aligned datasets and the current widely-used metrics. To address this problem, the metric Average Semantic Precision (ASP) is defined to evaluate the performance of multiplicity in these not-well-aligned datasets. Moreover, a Differentiable ASP Approximation (DAA) method is proposed to make the model diversified during model training over these noisy datasets. Experiments are conducted to show the effectiveness of our DAA for boosting the multiplicity of probabilistic embedding for cross-modal retrieval. ", "questions": "See my questions in weaknesses.", "limitations": "The limitations of this paper are not discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Average Semantic Precision (ASP) is defined to evaluate the performance of multiplicity.\n+ A Differentiable ASP Approximation (DAA) method is proposed for model training.\n\nWeaknesses:\n- The comparison in Table 1 is far from sufficient. Other baselines should be included to obtain more solid results. See some examples below:\\\n[a] Learning fragment self-attention embeddings for image-text matching, ACM-MM 2019.\\\n[b] IMRAM: Iterative Matching With Recurrent Attention Memory for for cross-modal image-text retrieval, CVPR 2020.\\\n[c] Similarity reasoning and filtration for image-text matching, AAAI 2021. \n- The study on multiplicity in this paper is rather shallow. What is the multiplicity performance of an image-text pre-training model for cross-modal retrieval? When a cross-modal model is pre-trained with a very large dataset, can its multiplicity performance still be boosted by the proposed method?\n- The convergence property of min L_{ASP} is not well studied. Considering the complexity of this loss function, is the training process via the proposed method stable? Multiple trials with different random seeds should be provided. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Ld7ypoLe-4e", "reviewer": "Reviewer_mhZH", "summary": "This paper addresses the multiplicity of correspondences between images and texts in cross-modal retrieval. Firstly, they utilize semantic correlation computed by CIDEr to construct the potential correspondence. Then they propose a new metric, named Average Semantic Precision (ASP) to measure the ranking precision of semantic correlation for retrieval sets. Finally, they propose to directly optimize ASP in their objective function directly by making the ranking function of ASP differentiable through a sigmoid function. Experimental results show the promising performance of the proposed method.", "questions": "Please see Weaknesses", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work seems to be technically sound; key modeling assumptions are clearly stated, claims are nicely demonstrated and supported by the results.\n1) Unlike existing cross-modal retrieval methods, which treat the multiplicity of correspondences between images and texts in cross-modal retrieval task as a multi-label problem, this paper considers it as a ranking problem, which is quite interesting, and more reasonable.\n2) They propose a new evaluation metric named Average Semantic Precision (ASP) to measure the ranking precision of semantic correlation for retrieval sets.\n3) It is novel to directly optimize the ASP in their designed network.\n4) The proposed method outperforms SOTA on ASP, and also outperforms them on R@1 (3.0% for i2t and 1.2% for t2i on MS-COCO).\n\nI believe that this paper will be a valuable contribution to the field, and I recommend acceptance. Below I have a few questions and comments for the authors.\n1) Why do you build your method based on 'Probabilistic Embedding'? It seems that the proposed model can be integrated into any cross-modal retrieval method. Therefore, it is necessary to evaluate the proposed model on other cross modal methods instead of 'Probabilistic Embedding'.\n2) Similarly, this model utilizes semantic correlation computed by CIDEr to construct the potential correspondence. It is also necessary to evaluate this model based on other methods e.g., Bert instead of CIDEr.\n3) Writing is a weak point of this paper. There are lots of typos and grammar errors. \n->LINE 284. there are only few improvements on R@1\n->LINE 302. Meanwhile, the curves of Smooth-AP and baseline are almost consistent, which shows that only focusing on ground truth is limited to 302 improving multiplicity.\n->The texts in Fig.3 and 4 are too small\n-> The references format is inconsistent\n4) Another major concern is about the proposed ASP metric. Since it is based on ranking, what is the difference between ASP and existing ranking-based evaluation metrics?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Nd1bd6TDCB", "reviewer": "Reviewer_vghc", "summary": "This paper focuses on the many-to-many relationships in the image-text retrieval task. The authors notice that text pairs with high CIDEr scores are more likely relevant, and thus the corresponding image-text pairs should be considered as positive. On top of this, they propose a metric named Average Semantic Precision (ASP) to capture this information. Then a differentiable surrogate loss of ASP is proposed to directly optimize ASP.", "questions": "Please refer to the weaknesses part for my main concerns. Besides, I still have some minor concerns:\n1. The example in Fig.1(a) shows that the ground truth captions might have lower CIDEr scores. Is it reasonable to punish the model when a ground truth caption ranked higher than captions with higher CIDEr scores?\n2. According to the motivation of this paper, some proper captions might be missed in the annotations. This phenomenon also occurs in the test data, i.e., these missing captions are not considered in the metrics like PMRP and R@1. Could you please explain why the proposed method can still achieve performance improvements on these inconsistent metrics?\n", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is the observation that CIDEr scores are useful to measure the many-to-many relationships. It could be helpful for image-text retrieval, especially in the self-supervised setting.\n\nHowever, the paper could be further improved in the following aspects:\n1. Although the proposed ASP loss looks reasonable to align the prediction with the CIDEr scores, the motivation of the specific formulation (Eq.(4)) should be clarified.\n2. The probabilistic embedding part is exactly the same as the work of Chun et al. It seems that Sec.3 has nothing to do with the proposed ASP. Moreover, it is unclear how these two components are combined in this paper.\n3. Some notations are confusing:\n- In Eq.(8), the $U$ in $R(i, U)$ is an index set, but both $S$ and $C$ in Eq.(4) are score sets;\n- In Eq.(11), the sigmoid function is applied to a Boolean variable: $\\phi(D_{ij}^C > 0)$, should it be $\\phi(-D_{ij}^C)$? Also, should $I{D_{ij}^C > 0}$ be $I{D_{ij}^C < 0}$?\n4. The provided code repository is empty.\n5. All results of competitors are directly copied from other papers, but some implementation details in this paper are different from the original papers of competitors: the training process of PCME consists of two phases including a warmup phase (only train the head modules) for 30 epochs and a finetune phase (train all parameters) for 30 epochs. While the proposed method is trained for three phases including a warmup phase for 3 epochs, a pre-training phase (train the head module of the visual encoder and all parameters of the extual encoder) for 40 epochs, and a finetune phase for 27 epochs. It looks unfair to compare these results under different training settings. Moreover, the result of VSE++ in MS-COCO 5K T2I should be 30.3 instead of 31.3.\n6. The details of baseline in ablation studies are unclear. The baseline in Tab.3 outperforms competitors in Tab.1, could you please explain the reasons for the performance gain?\n\n=============================================\n\nAfter rebuttal, most of my concerns have been addressed, and I'd like to raise my rating to 5. However, I still feel that the introduction of the probabilistic embedding is unnecessary since the proposed method can be directly applied to other cross-modal methods. On the contrary, the motivation of Eq.(4) is more important and interesting. Therefore, it might be better to state the motivation in the main paper instead of the probabilistic embedding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept