Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The multiplicity of correspondences between images and texts in cross-model retrieval greatly increases the difficulty of this task. To tackle this problem, probabilistic embeddings are proposed to quantify many-to-many relationships, but nonexhaustive annotations limit their performance. Based on this observation, they utilize semantic correlation computed by CIDEr to provide a pseudo annotation. Then they propose an effective metric, named Average Semantic Precision (ASP), to measure the ranking precision of semantic correlation for retrieval sets. 
Additionally, they introduce a novel and concise objective, coined Differentiable ASP Approximation (DAA) to optimize ASP directly by making the ranking function of ASP differentiable through a sigmoid function. To verify the effectiveness of their approach, extensive experiments are conducted on MS-COCO and CUB Captions.
strengths_and_weaknesses: This work nicely combines a novel evaluation metric for cross-model retrieval and deep learning model in a novel way and provides an effective approach. The paper is also well-motivated and well-organized.
Major comments:
i) They find that incompletely annotating MPPs is a crucial factor restricting many-to-many mapping for probabilistic embedding. Thus, we propose an effective evaluation metric named ASP, instead of the existing ones, which are powerless to reflect the correlations between each retrieved item and the query.
ii) To eliminate the adverse effect of MPPs in the training process, they propose a novel and concise metric learning method DAA, which approximates and optimizes ASP directly. This seems to be one of the major contributions of this paper.
iii) Promising experimental results.

I believe the paper has some merits. However, it also faces serval weaknesses that need to be addressed.
questions: From reading the paper, it is very evident that the authors do not come from ML but from a CV background. Some of the concepts are used in a weird name before definition, e.g., 'multiplicity of correspondences between images and texts', 'semantic correlation', 'many-to-many relationships.' in the abstract.

Methods:
It is not convincing that ASP is a valid evaluation metric. More descriptions and experiments are needed.
What is the benefit of these evaluation metrics, and what are the shortcomings?
It is unclear to me why ASP is stuck to CIDEr?

My major concern is about the experiments. The experiment part has three issues:
Few experiments: I think it is not enough to make general claims from having only two experiment datasets. Especially as the datasets are pretty small, i.e., the two datasets are relatively small, with 123,287 images and 11,788 images. Therefore, different and more datasets would enormously benefit the paper's claims.

Lack of quantitative comparisons. The proposed method improves existing SOTAin what cases? and when it fails? Why can it benefit both ASP and R@1?

Missing comparison with alternative approaches. This paper compares with VSE0 [11], PVSE[38], and PCME[7], which were published in 2017, 2019, and 2021.  There is some substantial literature on cross-modal retrieval recently.  Therefore, the paper should experimentally evaluate how these models compare to the proposed approach.
limitations: See above
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the multiplicity of correspondences between images and texts in cross-modal retrieval. Firstly, they utilize semantic correlation computed by CIDEr to construct the potential correspondence. Then they propose a new metric, named Average Semantic Precision (ASP) to measure the ranking precision of semantic correlation for retrieval sets. Finally, they propose to directly optimize ASP in their objective function directly by making the ranking function of ASP differentiable through a sigmoid function. Experimental results show the promising performance of the proposed method.
strengths_and_weaknesses: The work seems to be technically sound; key modeling assumptions are clearly stated, claims are nicely demonstrated and supported by the results.
1) Unlike existing cross-modal retrieval methods, which treat the multiplicity of correspondences between images and texts in cross-modal retrieval task as a multi-label problem, this paper considers it as a ranking problem, which is quite interesting, and more reasonable.
2) They propose a new evaluation metric named Average Semantic Precision (ASP) to measure the ranking precision of semantic correlation for retrieval sets.
3) It is novel to directly optimize the ASP in their designed network.
4) The proposed method outperforms SOTA on ASP, and also outperforms them on R@1 (3.0% for i2t and 1.2% for t2i on MS-COCO).

I believe that this paper will be a valuable contribution to the field, and I recommend acceptance. Below I have a few questions and comments for the authors.
1) Why do you build your method based on 'Probabilistic Embedding'? It seems that the proposed model can be integrated into any cross-modal retrieval method. Therefore, it is necessary to evaluate the proposed model on other cross modal methods instead of 'Probabilistic Embedding'.
2) Similarly, this model utilizes semantic correlation computed by CIDEr to construct the potential correspondence. It is also necessary to evaluate this model based on other methods e.g., Bert instead of CIDEr.
3) Writing is a weak point of this paper. There are lots of typos and grammar errors. 
->LINE 284. there are only few improvements on R@1
->LINE 302. Meanwhile, the curves of Smooth-AP and baseline are almost consistent, which shows that only focusing on ground truth is limited to 302 improving multiplicity.
->The texts in Fig.3 and 4 are too small
-> The references format is inconsistent
4) Another major concern is about the proposed ASP metric. Since it is based on ranking, what is the difference between ASP and existing ranking-based evaluation metrics?
questions: Please see Weaknesses
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on the many-to-many relationships in the image-text retrieval task. The authors notice that text pairs with high CIDEr scores are more likely relevant, and thus the corresponding image-text pairs should be considered as positive. On top of this, they propose a metric named Average Semantic Precision (ASP) to capture this information. Then a differentiable surrogate loss of ASP is proposed to directly optimize ASP.
strengths_and_weaknesses: The main strength of this paper is the observation that CIDEr scores are useful to measure the many-to-many relationships. It could be helpful for image-text retrieval, especially in the self-supervised setting.

However, the paper could be further improved in the following aspects:
1. Although the proposed ASP loss looks reasonable to align the prediction with the CIDEr scores, the motivation of the specific formulation (Eq.(4)) should be clarified.
2. The probabilistic embedding part is exactly the same as the work of Chun et al. It seems that Sec.3 has nothing to do with the proposed ASP. Moreover, it is unclear how these two components are combined in this paper.
3. Some notations are confusing:
- In Eq.(8), the $U$ in $R(i, U)$ is an index set, but both $S$ and $C$ in Eq.(4) are score sets;
- In Eq.(11), the sigmoid function is applied to a Boolean variable: $\phi(D_{ij}^C > 0)$, should it be $\phi(-D_{ij}^C)$? Also, should $I{D_{ij}^C > 0}$ be $I{D_{ij}^C < 0}$?
4. The provided code repository is empty.
5. All results of competitors are directly copied from other papers, but some implementation details in this paper are different from the original papers of competitors: the training process of PCME consists of two phases including a warmup phase (only train the head modules) for 30 epochs and a finetune phase (train all parameters) for 30 epochs. While the proposed method is trained for three phases including a warmup phase for 3 epochs, a pre-training phase (train the head module of the visual encoder and all parameters of the extual encoder) for 40 epochs, and a finetune phase for 27 epochs. It looks unfair to compare these results under different training settings. Moreover, the result of VSE++ in MS-COCO 5K T2I should be 30.3 instead of 31.3.
6. The details of baseline in ablation studies are unclear. The baseline in Tab.3 outperforms competitors in Tab.1, could you please explain the reasons for the performance gain?

=============================================

After rebuttal, most of my concerns have been addressed, and I'd like to raise my rating to 5. However, I still feel that the introduction of the probabilistic embedding is unnecessary since the proposed method can be directly applied to other cross-modal methods. On the contrary, the motivation of Eq.(4) is more important and interesting. Therefore, it might be better to state the motivation in the main paper instead of the probabilistic embedding.
questions: Please refer to the weaknesses part for my main concerns. Besides, I still have some minor concerns:
1. The example in Fig.1(a) shows that the ground truth captions might have lower CIDEr scores. Is it reasonable to punish the model when a ground truth caption ranked higher than captions with higher CIDEr scores?
2. According to the motivation of this paper, some proper captions might be missed in the annotations. This phenomenon also occurs in the test data, i.e., these missing captions are not considered in the metrics like PMRP and R@1. Could you please explain why the proposed method can still achieve performance improvements on these inconsistent metrics?

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on addressing a critical problem in probabilistic embedding for cross-modal retrieval, i.e., the multiplicity performance of the model is not well studied due to the not-well-aligned datasets and the current widely-used metrics. To address this problem, the metric Average Semantic Precision (ASP) is defined to evaluate the performance of multiplicity in these not-well-aligned datasets. Moreover, a Differentiable ASP Approximation (DAA) method is proposed to make the model diversified during model training over these noisy datasets. Experiments are conducted to show the effectiveness of our DAA for boosting the multiplicity of probabilistic embedding for cross-modal retrieval. 
strengths_and_weaknesses: Strengths:
+ Average Semantic Precision (ASP) is defined to evaluate the performance of multiplicity.
+ A Differentiable ASP Approximation (DAA) method is proposed for model training.

Weaknesses:
- The comparison in Table 1 is far from sufficient. Other baselines should be included to obtain more solid results. See some examples below:\
[a] Learning fragment self-attention embeddings for image-text matching, ACM-MM 2019.\
[b] IMRAM: Iterative Matching With Recurrent Attention Memory for for cross-modal image-text retrieval, CVPR 2020.\
[c] Similarity reasoning and filtration for image-text matching, AAAI 2021. 
- The study on multiplicity in this paper is rather shallow. What is the multiplicity performance of an image-text pre-training model for cross-modal retrieval? When a cross-modal model is pre-trained with a very large dataset, can its multiplicity performance still be boosted by the proposed method?
- The convergence property of min L_{ASP} is not well studied. Considering the complexity of this loss function, is the training process via the proposed method stable? Multiple trials with different random seeds should be provided. 

questions: See my questions in weaknesses.
limitations: The limitations of this paper are not discussed.
review_title: null

Meta-review: metareview: This paper investigates the multiple correspondence issue of cross-modal retrieval and proposes a method to improve and evaluate the multiplicity of probabilistic embedding. Sufficient experiments are carried out to prove the effectiveness of the proposed method. In addition, the rebuttal successfully addressed the major concerns and, in the end, there is a general consensus about accepting the paper.
confidence: Certain
Decision: Accept