Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the unpaired image and text retrieval task. Concretely, traditional image text retrieval model are learnt via paired and aligned image and text data. However, it is pretty common that the text is not following the same distribution as the text in the training set. Therefore, it is interesting and important to learn the model from unpaired image and text. This paper proposed an interesting model called MACK, which uses image patches to form prototypes for each word/phrase in the given text. Given a new image, it first detects all the objects in the image. Then it aligned the detected object with the prototypes. As the prototypes and the object representation live in the same space. Each prototype also links to one word. It can improve the image and text retrieval performance and conduct alignment across unpaired image and text.
strengths_and_weaknesses: Pros:
I love this idea a lot! This is an elegant way to tackle the image and text retrieval task. The main challenging in this task is the semantic ambiguity in the text side. For example, all images of 'apple' with different appearance is aligned with the text 'apple'. How to align the text 'apple' with multiple 'apple' objects is a challenging task. This paper uses the prototypes to link the text and image worlds is very elegant.

Cons:
I have two high-level questions for the authors:
1. How to handle the out-of-vocabulary issue. For example, if the text has not been seen in the training set, how to handle this case? How to generate/obtain the prototypes for this OOV word/text?
2. How to handle the data imbalance problem? For example, some words might contain less images linked with it. This would lower the quality of the prototype representation. How to handle this case?
3. Correct me if I was wrong. To my understanding, the prototypes are the corresponding images representation of nouns. But how would the model distinguish between the following two cases: 'dog chases cat' vs. 'cat chases dog'. The prototypes for the 'dog' and 'cat' are all the same. How to use the prototypes to distinguish them?

Missing related work: 
Zhang, B., Hu, H., Qiu, L., Shaw, P., & Sha, F. (2021). Visually Grounded Concept Composition. Findings of EMNLP. has a similar idea of the proposed approach.  
questions: I like this idea a lot. Please address three questions mentioned above.
limitations: I don't see any negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: First of all, this paper pays attention to an important and challenging problem -- unpaired image/text retrieval and designs a framework that can be easily plugged into existing image-text matching models for re-ranking.
strengths_and_weaknesses: S1: The proposed model is simple and can be easily plugged into existing image-text matching models to boost the performance.

S2: Extensive experiments are conducted and the results show that using the proposed MACK for re-ranking, the performance can be improved by around 1%.

W1: The proposed model actually employs human-annotated data to obtain prototypical region representations and prior knowledge and the dataset is composed of paired annotations, so "unpaired image-text matching" in the title is misleading.

W2: The pretrained general knowledge is highly related to the training data. In the paper, Visual Genome is used, but some other datasets can also be used, such as MSCOCO and ImageNet. I do not see any ablation studies on the pretrained general knowledge.
questions: see weeknesses.
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes Multimodal Aligned Conceptual Knowledge (MACK) for image-text matching without requiring paired image-text for finetuning the pretrained model. In addition, the proposed approach can also be extended as a re-ranking method to improve the performance of current image-text matching approaches. It uses the word-region aligned datasets to get the pretrained general knowledge, and then finetune the model on unpaired image-text datasets to obtain the finetuned domain knowledge. During inference, image-text similarity scores are calculated by aggregating the word-region similarities. For the experiments, the pretrained general knowledge are obtained from Visual Genome dataset, and image-text matching experiments are conducted on COCO and Flickr30k datasets with both unpaired image-text setting and re-ranking setting.
strengths_and_weaknesses: Strengths:
1) This work proposes a new setup for image-text matching. It proposes to train or fine-tune image-text matching models without paired image text data. 
2) It uses a pretrained detection model to provide the correspondences between visual and word features as domain knowledge.
3) Finetuning the pretrained domain knowledge on unpaired images and texts is effective for improving the image-text matching performance.
4) The proposed approach is not only validated on unpaired image-text matching settings, but also can be used as a re-ranking approach to improve the image-text matching performance of existing models such as CLIP [25] and ALBEF [16].

Weaknesses:
1) I am doubtful about whether the setup of unpaired image-text matching and the proposed solution is realistic or not. Firstly, although the training does not require ground-truth image-text correspondences, it requires the word-region alignment from the Visual Genome dataset to pretrained the detection model, and the ground-truth of VG dataset is more difficult and costly to annotate than image-text alignments. Secondly, the authors claim that they do not need paired image-text data for training, but the experiments are still conducted on COCO and Flickr30k image captioning datasets. Although the image-text correspondences are not used during training, there is an implicit prior that there exist one-to-one correspondences between images and text. It is unclear whether the proposed approach still works well on totally unpaired images and captions, e.g., Flickr images and COCO captions.
2) Why is the aggregation function in Eq.(4) of Section 3.3 defined as max-mean pooling? The ablation study experimented with mean-mean pooling, but what if it is max-max pooling or mean-max pooling?
3) The performance of unpaired image-text matching (Table 1) is quite low. And the authors do not provide the qualitative results on unpaired image-text matching. I am wondering if the retrieval results are reasonable with the unpaired setup.
4) No performance comparison is conducted. The authors could compare it to other unpaired matching or re-ranking approaches. If there are no previous approaches, at least it should be compared to some naive baselines, for example, training the unpaired image-text matching with cycle-consistency loss.
5) The cross-dataset re-ranking in Table 3 only provides marginal performance gain (less than 1%).
6) The re-ranking hyperparameters $k$ and $\lambda$ are not experimented in the ablation study.

questions: Please address my concerns in the weaknesses section.
limitations: The authors mentioned the limitations but do not address the potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a framework for performing unpaired image-text matching. It leverages the Visual Genome dataset and Faster-RCNN to form a region-based method that matches words and regions of images and promises to yield robust image-sentence-level descriptors for pairing. A cycle consistency loss is proposed to improve performance on specific datasets. Since the proposed method is expected to handle region-word correspondences that current image-text matching models such as CLIP cannot solve well, it can be integrated into current systems and improve their matching performance. The authors tested the proposed method on two common benchmarks, Flickr30k and MSCOCO, and conducted image-text pairing, re-ranking matching results, and cross-dataset matching experiments. Results shown the strength of the proposed method.
strengths_and_weaknesses: The studied direction is very important and novel that trying to pair the image and text data according to region-based information. As the author mentioned in the introduction, images usually contain information beyond the info delivered by words. The proposed region-based method hopefully shows more robust performance in conducting image text alignment. All the used components are technically sound, including using Visual Genome, the pre-trained Faster RCNN, and the cycle-consistency loss. Overall the paper is easy to follow.

The author conducted experiments on two common datasets, Flickr30k and MSCOCO, and adopted various baselines, including CLIP, ALBEF, VSRN, and SAEM.  Although the proposed method itself performance is much worse than the models trained on pairs of image-text datasets, it can further improve the performance of those models as shown in Table 2. Also, the reviewer pretty much likes the Cross-dataset experiments, where both the used baseline + MACK did not see the target dataset during training. 

According to the gap between Table 1 and Table 2, it is clear that the current method which leverages VG dataset and pre-trained Faster R-CNN is not powerful enough to conduct image-text alignment. The author did not ablate the used hyper-parameters in re-ranking experiments. The current minor improvements shown in Table 2 might come from carefully hyper-parameter searching. So does Figure 2.
questions: The current hyperparameter used in the re-ranking experiments, $k=15$ and $\lambda =0.1$, seems are set very specific. The reviewer is interesting if the author can do ablations on $k$.

What is the computation cost of the fine-tuned domain knowledge stage? $F$ and $M$ seem too large to make this step practical. If so, the proposed way which involves a transformation matrix is impractical. Does it possible to introduce a NN approximates $W$ but much more efficiently?

Considering the performance of the "pretrained general knowledge", can the author provide the results which only use half the data of Visual Genome? Since annotating Visual Genome is pretty expensive, it is important to see if the proposed approach is sensitive to the sample numbers of the used extra dataset.
limitations: Clearly, the current method is limited to the visual properties annotated in Visual Genome and the ability of the pre-trained Faster-RCNN. If we want to study fine-grained structures within objects, the proposed method will fail. Also, the method will fail if no object associated with the word is detected.

Considering the proposed framework: (1) mainly relies on the extra fine-grained human-annotated dataset and pre-trained detectors; (2) has very poor results itself; (3) has minor improvements when integrating with the current SOTA models via carefully selected hyper-parameters; (4) the proposed fine-tuned stage is probably impractical, it is more like a proof-of-concept paper. The following researchers are hard to scale up the used fine-grained dataset and find stronger pre-trained detectors, and also, there are web-scale paired image-text datasets. The reviewer is not sure if this paper meets the bar of NeurIPS.

If the mentioned points got addressed to some levels, the reviewer is willing to adjust the scores accordingly.

----------------------------------
After rebuttal: the author has well addressed my concerns and questions. The paper now has ablation studies around the hyperparameters and related discussions about the computation resource and the dataset scale. I raised my scores from 4 to 6.
review_title: null

Meta-review: metareview: This paper proposes MACK for unpaired image-text matching, which can also be extended as a re-ranking method to improve the performance of current image-text matching approaches. It uses the word-region aligned datasets to get the pretrained general knowledge, and then finetune the model on unpaired image-text datasets to obtain the finetuned domain knowledge. During inference, image-text similarity scores are calculated by aggregating the word-region similarities. 

The authors have done a good job during rebuttal. After rebuttal, the paper received scores of 5567, which is in general positive. This work proposes a new setup for image-text matching, and the studied direction seems novel. The proposed approach is not only validated on unpaired image-text matching settings, but also as a re-ranking approach to improve the performance of existing models such as CLIP and ALBEF.

On the other hand, clear limitations also exist, such as (1) the current method is limited to the visual properties annotated in Visual Genome and the ability of the pre-trained Faster-RCNN; (2) whether the setup of unpaired image-text matching and the proposed solution is realistic or not; and (3) the performance of unpaired image-text matching itself (Table 1) is quite low.

On balance, the merits slightly outweigh the flaws, and the AC would like to recommend acceptance of the paper. 
confidence: Certain
Decision: Accept