MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching
Authors: Yan Huang
Yuming Wang
Yunan Zeng
Liang Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: image-text matching
multimodal knowledge

Abstract: Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which are trained on millions or billions of paired images and texts. Different from them, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed to be unavailable during model training. To deal with this, we propose a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK), which is inspired by the knowledge use in human brain. It can be directly used as general knowledge to correlate images and texts even without model training, or further fine-tuned based on unpaired images and texts to better generalize to certain datasets. In addition, we extend it as a re-ranking method, which can be easily combined with existing image-text matching models to substantially improve their performance.

MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching
Yan Huang
Yuming Wang Yunan Zeng
Liang Wang
Center for Research on Intelligent Perception and Computing (CRIPAC), State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences (CASIA) School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) Chinese Academy of Sciences Artificial Intelligence Research (CAS-AIR)
(yhuang, wangliang ) 8nlpr conmpr.ia.ane iaa.aaa.. on, { fyuming. wang, yunan zeng @cripac. ia.ac.cn
Abstract
Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which are trained on millions billions of paired images and texts. Different from them, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed be unavailable during model training. To deal with this, we propose simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK), which is inspired by the knowledge use in human brain. It can be directly used as general knowledge to correlate images and texts even without model training, or further fine-tuned based unpaired images and texts better generalize to certain datasets, In addition, we extend as a re-ranking method, which can be easily combined with existing image-text matching models to substantially improve their performance.
Introduction
Image-text matching is one of the most representative techniques in the field of vision and language understanding, which has wide applications such online shopping, human-robot interaction and autonomous driving. Its major challenge lies in how to accurately measure the cross-mode similarity between images and texts. Recently, by training images and texts, various multimodal pretrained models [23 20 7 alleviate the challenge a supervised learning manner. Although the accuracy of image-text matching is greatly improved, collecting and annotating such large-scale data in real world applications is time-consuming and expensive.
very large-scale (millions or billions) paired
Different from them, this work attempts to study the image-text matching the context new scenario, namely unpaired image-text matching, in which paired images and texts assumed to be unavailable during model training. is motivated by the fact that the human brain can well correlate arbitrary images with texts while does not need to learn from such large-scale paired images and texts. Instead, stores semantic knowledge about objects, actions, attributes, etc., which is multimodal
aligned and can be used to correlate visual and linguistic information 28 2 Inspired by these
neuroscience evidences, this work tries to deal with the unpaired image-text matching by modeling human brain-like knowledge.
Unimodal visual or linguistic knowledge has been widely used for vision and language understanding such image captioning [39], visual reasoning 45 and visual question answering [31] By directly combining these two types of knowledge together, multimodal knowledge shows more complementary
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

advantages 46 37. However, existing knowledge- based methods are always designed to corporate with paired data during model training, which cannot well handle the unpaired image-text matching. Another alternative is multimodal knowledge graphs [47, most of which are extended from existing linguistic knowledge graphs by linking semantically related images to corresponding words. However, the image- image-word alignment might have the following issues: 1) images usually include redundant or word-unrelated contents, so the images and words carry unequivalent semantic information, and 2) word-related objects (or attributes) usually have diverse appearances in different images, so the alignment one-to-many rather than one-to-one. In fact, obtaining well-aligned multimodal knowledge is a great challenge.
In this work, we propose a new method namely Multimodal Aligned Conceptual Knowledge (MACK) for unpaired image-text matching. To remove the word-unrelated content in images, it focuses on semantic concepts and collects pairs of words and semantically related image regions from public available datasets. Then it computes prototypical region representations and aligns them to the words, with the goal alleviate the appearance variant. Based on the aligned conceptual knowledge (i.e., word-prototypica region pairs), the MACK can bridge images and texts in the same feature space to measure their cross-modal similarities. To make the pre-computed general knowledge better suit certain datasets, we can further fine-tune it with the principle of region-level cycle-consistency, which does not need paired images and texts. Since the proposed MACK is simple yet effective, it can well corporate with existing image-text matching models re-ranking method to further improve their performance.
Our contributions are summarized as follows. 1) We study rarely investigated problem as unpaired image-text matching, which is under great demand in practical applications. 2) We accordingly propose simple yet effective method to model multimodal aligned conceptual knowledge, which can either perform test directly or perform training based on unpaired data. 3) The proposed method can be easily adopted by existing image-text matching models to re-rank their results to obtain better performance.
2 Related Work
2.1 Image-Tex: Text Matching
Image-text matching draws much attention recently, and many effective models trained on paired images and texts have been proposed. Socher et al. (32] and Frome et al. [6] propose the early framework of Tisual-Semantic Embedding (VSE) to correlate images and their labels. Kiros et al. [6] extend the VSE from image-label matching to maae-text matching, which is later improved from various aspects such as adding intra-modal constraint (36), mining hardest negative sample [5]. analyzing canonical correlation [1], matching local instance [I]] [3). improving sentence representation 29 19 and reasoning visual relation (2 14. In addition to the VSE, another important framework is Cross-Modal Attention (CMA) proposed by Lee el al. [18) which uses pretrained object-level features []] and then generates cross-modal attended representations for similarity measurement. The CMA is later improved from the directions of cross-modal memory 10. 91, cross-modal message passing (38. hybrid learning (8] context modeling [44], iterative based multimodal
matching [3], and graph structure 25 Later, many more models 4 20 27 23
versions of Transffrmmrr [35] are proposed, which rely on millions or billions of paired images and texts for model training and achieve very good performance. Recently, Huang et al. [12] make the early attempt to study how to improve the efficiency of such large image-text matching models, which achieves promising results. In contrast to the existing models, we attempt to study new scenario of unpaired image-text matching that assumes paired images and texts are unavailable for model training, which is more challenging than conventional image-text matching.
2.2 Multimodal Knowledge
There are only a few works that use multimodal knowledge for vision and language understanding tasks. For example, Zhu et al. [46] build heterogeneous graphs corresponding to visual, semantic and factual features for visual question answering. Wang el al. [37 combine visual and textual knowledge to find discriminative parts of objects and improve the performance of few-shot learning. Zhang et al. 43 propose a concept-relation graph and then leverage it for visually grounded concept proposed, most of
learning. In addition, different multimodal knowledge graphs 47 26 33

---

Word and related regions
Prototypical region representation
Pre sired generaal knowledga
Unpaired images and certain catasets
Fine-tumed domaln knowledde
Man
Man
Dog
Dog
say parsure
Bird
andruce
Bird
Know cdge modal bridge
young couple
couple with knneeling vander reading book of mango
Ineellng vendor
reading book
Global similarity
Crock-modal smilarity matri
Visual representations both and words
Inputt image and text
words
regions
Figure 1: The proposed Multimodal Aligned Conceptual Knowledge (MACK) for unpaired image- text matching. The top figures illustrate how to obtain the knowledge in three steps: 1) collecting set of conceptual words and their related image regions, 2) computing prototypical region representations to get the pretrained general knowledge, and 3) training on unpaired images and texts to obtain the fine-tuned domain knowledge. The bottom figures illustrate how to use the knowledge for unpaired image-text matching in three steps: 1) obtaining parsed words and detected regions from the input image and text, 2) aggregating related words and representing them by prototypical region representations in the knowledge, and 3) comparing word and region representations in the same feature space and pooling their cross-modal matrix to obtain global similarity score.
which extend linguistic knowledge graphs by aligning images to words. Such cross-modal alignment could be quite noisy because images usually include redundant or word-unrelated contents. In fact, the multimodal knowledge used in these methods either cross-modal unaligned or noisy aligned, which usually needs paired images and texts strong supervisions for model training. Different from them, our multimodal aligned conceptual knowledge establishes more accurate one-to-one alignment in the fine-grainee concept-level, which can handle the unpaired image-text matching without using strong supervisions.
Multimodal Aligned Conceptual Knowledge
As shown in Figure [ we propose the Multimodal Aligned Conceptual Knowledge (MACK) for unpaired image-text matching. To obtain the knowledge, we first collect a set of conceptual words and their semantically related image regions from public datasets. For each word, obtain prototypical region representation by averaging the representations of all related regions. Thus, we obtain the pretrained general knowledge containing pairwise words and prototypical region representations. To make the knowledge better adapt to certain datasets, we then fine-tune the pretrained general knowledge by modeling the concept-level cycle consistency based on unpaired images and texts.
Give an image and text, we first perform text parsing to obtain a word graph, and use the knowledge cross- mooss- bridge to obtain we perform object detection on the image to obtain another set of region representations. Thus, the representations of words and regions are in the same feature space, in which they be directly compared to get a cross-modal matrix. After pooling the values in row and column dimensions based on heuristic rules, we can finally obtain the global similarity score between the image and text.
of prototypical region representations for all the words. Then,
3.1 Pretrained General Knowledge
The studied knowledge this work has two major properties as follows. 1) Concept-level knowledge: the knowledge is built upon two major types of semantic concepts, which correspond to objects and attributes in images, and nouns and adjectives texts. more fine-grained than existing multimodal knowledge graphs that roughly link global images to words. 2) One-to-one alignment:
3

---

rather than align word to multiple possible images in one-to-man manner, the knowledge focuses on eross-modal one-to-one alignment, with the goal to alleviate the appearance variations of objects and attributes.
We define the knowledge as a set of semantic concepts having paired multimodal representations {WW,, b=)].==,....K where Wik E N and 02 E R are the one-hot word representation and real- valued region representation of k-th semantic concept, respectively, and K is the total number of semantic concepts. To maximize the number of concepts, we collect all the words and their related regions in the Visual Genome (VG) dataset [7]. We could alternatively use other datasets, but the VG has more diverse content that is useful for improving the generalization ability of the knowledge. For each word, we can easily obtain its semantically related image regions in the dataset, and different regions usually have diverse visual appearances. To avoid the impact of appearance variations, we compute a prototypical region representation v/ for each word wi by averaging all its related region representations (ry} j=)........
Uk Jj j=1
(I)
where each region representation ry is obtained by feeding bounding box and an image into the pretrained object detection model Faattr-RRNNy [.] Note that the numbers of regions for different words {Jæ)k=1,..... are quite imbalanced, which vary from tens to tens of thousands. Intuitively, frequently appeared words such as "man" and "dog" will have more related regions, and their corresponding prototypical region representations might be more robust than those of few-shot words such as "otter" and "nun". However, the few-shot words can still compute initial prototypical region representations with very limited numbers (e.g., < 20) of regions and then refine them when more related regions are encountered.
3.2 Fine-tuned Domain Knowledge
To better adapt the pretrained general knowledge certain datasets, we could further fine-tune the prototypical region representations based on the principle of concept-level cycle consistency, if unpaired images and texts in those datasets are available. In particular, given batch of unpaired images and texts, we first obtain a set of detected regions using the Bester-RCNN above and a set of tokenized words. For the set of words, duplicated ones are removed and the rest words are M. Then all the words can be represented by the corresponding prototypical region
{wm hm I representations as (Um m or V E RFxM For the set of regions, we compute similarity scores between the prototypical region representations (in the pretrained general knowledge) and the set of region representations, and then remove the redundant regions belonging to same concepts in a similar way as Maximum Suppression (NMS). The resulting regions are denoted as {rnJm=i,..., or R E FxN.
To fine-tune the prototypical region representations to better suit for certain datasets, we design a cycle consistent loss learn parametric transformation matrix W E FxF, The main idea is to first measure similarities between each word and all regions, and use the similarities as weights to combine all regions to obtain reconstructed word
V
W Wo S = (WV)TWR
where S E R MMN is the similarity matrix between transformed word and region representations, (-)) is the softmax operation along the column dimension, and V E FxM contains the reconstructed word representations. Then, each original word and its reconstructed one are compared to generate a label indicating whether they are the same or not. By minimizing the cross-entropy loss L based on the predicted label and groundtruth one, we can optimize the W:
L -= - - m=1 WIT log(ūm), y = o( (WV)))
(3)
where ÿ E M includes the predicted labels, in which each column is denoted as 9m. and ym is the groundtruth label in which the m-th value is one and the rest values are zeros. After the training, we can use the W to transform all prototypical region representations to obtain the fine-tuned domain knowledge, denoted as [[w,, 2x.) k= K. where un = W E RR.
'We could alternatively use other pretrained models to detect regions. We select this model because it has been widely used and demonstrated to he effective for image-text matching.
In fact, we similarly combine words to obtain reconstructed regions in the reverse direction.

---

3.3 Knowledge-based Unpaired Matching
To decide whether an image and a text is matched or not, we use the knowledgd measure the cross-modal similarity. For the image, we use the Faster-RCNN to obtain set of detected region representations (ri).=I..... or R E FR. For the text, we use the knowledge as a cross-modal bridge to represent all the words as set of prototypical region representations (u, (u,)j=1.....) Even though these two set of representations are the same feature space, they have different amount of semantic information, directly comparing their similarity might be sub-optimal. In fact, each uj is concept-level corresponding a word, while each 7; usually describes not only an object but also its attribute. To make better comparison, word representations before the similarity measurement.
infer the relation among words and aggregate related
In particular, for the words in the text, we use the Natural Language Toolkit (NLTK) to predict their types and parse their dependencies. Then, we take each noun as an anchor and combine its representation with dependent adjective representation in averaged manner. Thus, the set of word representations [=.))....... are re-computed as {ut ( or U E RRLL, Then we can obtain the
desired similarity score s for the image and text
p(U R)
where p(-) is the max-mean pooling that first performs max pooling on the column dimension and then mean pooling on the row dimension on the input matrix. Given multiple images and texts, we can similarly obtain their similarity scores for unpaired image-text matching.
3.4 Re-ranking Existing Models
Note that the knowledge in this work is inspired by human-like knowledge, which is quite different from the pretrained model knowledge learnt from paired data. So the proposed MACK tends to have complementary properties when combining with existing pretrained models. To demonstrate this idea, we try to use the knowledge to re-rank the results by existing models to improve their performance.
Taking the text-driven image retrieval as an example, given a text query and a set of G images as gallery, an existing model can produce a similarity vector s E GR By sorting all the scores of in the descending order, the model is usually able to rank the matched image in top-k. Then we can tune the top-k scores by adding the corresponding scores predicted by the knowledge with a balancing factor X, and then re-rank the top-k images to improve the rank of matched image.
3.5 Discussion
Note that the proposed MACK defines a general framework for adaptive knowledge representation, organization, and reasoning. For the knowledge representation, it uses the prototype learning obtain pairwise prototypical region representations and semantic words. Then, it potentially organizes the knowledge in a graph-like manner, in which the dependency relation among different knowledge pairs are explored. At last, it hierarchically reasons the knowledge in three levels, including donain-level by fine-tuning on certain datasets, sample-level by using the pretrained models, and instance-level by performing the max-mean pooling.
4 Experimental Results
To demonstrate the effectiveness of the proposed method, we perform extensive experiments of image-text matching on two publicly available datasets.
4.1 Datasets and Protocols
The details of datasets and protocols are follows. 1) Flickr30k 42 consists of 31783 images collected from the Flickr website. Each image has 5 human annotated texts. We use the public training, validation and testing splits, which contain 29000, 1000 and 1000 images, respectively. 2)
'The knowledge here could the pretrained general knowledge or fine-tuned domain knowledge. "nttss//www.nltt..rg.
5

---

Table 1: Unpaired image-text matching by ablation models of MACK the Flickr30k dataset.
Details about these ablation models are explained in Section 4.3
Flickr30k dataset
Method
Image Annotation
Image Retrieval
Rs
R@1
R@5
R@10
R@
R@5
R@10
Pretrained general knowledge
10.8
26.6
35.8
4.1
10.8
18.0
106.2
w/o region prototypes
1.5
6.1
9.3
1.1
4.5
9.0
31.7
w/o adjectives
9.8
25.0
34.8
4.2
10.9
16.8
101.5
w/o max- mean pooling
4.2
13.4
20.4
2.9
8.4
12.8
62.2
Fine-tuned domain knowledge
12.7
30.4
40.8
10.3
25.1
34.0
153.5
w/o implicit prior
11.5
30.1
40.6
10.5
24.5
33.9
151.1
w/o knowledge
0.1
0.5
0.9
0.0
0.6
1.1
3.2
MSCOCO 124 consists of 123287 images, each of which is associated with 5 texts. We use the public training, validation and testing splits, with 113287. 1000 and 5000 images, respectively.
The image-text matching usually includes two sub-tasks in terms of: 1) image annotation: retrieving related texts given images, and 2) image retrieval: retrieving related images given texts. The commonly used evaluation criterions are "R@1", "R@5" and "R "R010", i.e., recall rates at the top-1, 5 and 10 results. Following other methods, we also use an additional criterion "Rs" by summing all the recall rates to evaluate the overall performance.
4.2 Implementation Details
In the pretrained general knowledge, we collect all the words in the Visual Genome dataset, so the total number semantic concepts is 27801. For each image, we use the pretrained RaNN to extract region representations, the number of detected regions is / = 36 and the dimension of 2048. In the fine-tuned domain knowledge, we perform
(prototypical) region representation the model training in minibatch manner. Each minibatch includes unpaired 128 images and 128 texts that are randomly sampled from the whole training set. We use the Adam algorithm [[5] to optimize the only parameter matrix W with a learning rate of 2e for 30 epochs. When re-ranking existing models, we empirically set k 15 and A 0.1. The knowledge-beee matching is very efficient, which does not need GPUs for acceleration.
unpaired image-text
4.3 Unpaired Image-text Matching
Since existing models all proposed for paired image-text matching, we compare several ablation models of the proposed MACK to verify their effectiveness for unpaired image-text matching. The first ablation model is pretrained general knowledge, which does not need model training. Then design its three model variants including: I) replacing the prototypical region representation with random region representation, similar existing multimodal knowledge graphs, 2) removing the adjectives when re-computing the word representations, and 3) replacing the max-mean pooling with mean-mean pooling. We also present another ablation model fine-tuned domain knowledge, which additionally performs training stage unpaired images and texts using the concept-level cycle consistent loss. We also design its two model variants including: 1) using Flickr30k images and MSCOCO texts for the fine-tuning rather than Flickr30k images and texts, and 2) directly optimizing the sece-coonistency loss without using the knowledge.
We compare the performance of unpaired image-text matching by all these models on the Flickr30k dataset in Table It is reasonable that the pretrained general knowledge achieves much worse performance than existing image-text matching models trained on paired images and texts. The unpaired image-text matching still needs more research and there is still lot of room for perfor- mance improvement. By removing the region prototypes, adjectives, and max-mean pooling"] the performance all becomes worse. And we find that the region prototypical representation makes the largest contribution to the performance of pretrained general knowledge. By training on the unpaired
In addition to the max-meat pooling, we also performed experiments of mean-man max pooling, max-max pooling, and mean mean pooling, but found they achieve much worse performance.

---

Table 2: Image-text matching by re-ranking two state-of-the-ar models the Flickr30k and
MSCOCO datasets. "+" denotes the re-ranking.
Flickr30k dataset
MSCOCO dataset
Method
Image Annotation
Image Retrieval
Image Annotation
Image Retrieval
Rs
Rs
R@I
R@5
R@10
R@I
R@5
R@10
R@I
R@5
R@10
R@1
R@5
R@10
CLIP 30
65.4
87.2
91.7
85,4
97.1
98.7
525.6
35.3
60.0
70.1
55.2
78.7
86.7
386.1
MACK
66.8
88.2
92.6
86.2
97.2
98.9
530.0
36.9
61.6
71.7
55.7
79.6
87.1
392.8
ALBEF 21
59.9
84.8
90.5
78.2
95.5
97.9
506.9
40.2
68.4
78.9
62.4
85.9
92.1
428.3
MACK
61.8
85.8
91.4
80.1
96.4
97.7
513.3
41.0
69.0
79.4
62.4
86.1
92.7
430.9
Query toxt
Returned top-3 Images by CUP
Query text
Returned top images by CUP
group of people
shirt and red
yellow hat pink
are sitting on the
outside of rustic building
apron holding
food in kitchen
people blue shirts are outside with bullhorn
wearing sunglas- sit halfway water
Figure 2: Examples of retrieved top-3 images based on text queries by the CLIP. Groundtruth matched images are marked by red bounding boxes, which can be re-ranked higher by the MACK.
images and texts, the fine-tumed domain knowledge shows significant performance improvements, especially on the sub-task of image retrieval. By additionally removing the implicit prior that there exist one-to-one correspondences between images and texts, the results become slightly worse but are still much better than pretrained knowledge. The results are very bad if we do not use the knowledge as the cross-modal bridge to correlate images with texts. In the following experiments, we use the pretrained general knowledge as the default MACK due its simplicity.
4.4 Re-ranking State-of-the--at Models
The recent state-of-the-art models for image-text matching are variants of multimodal Transformers, all of which are pretrained on very large-scale datasets containing millions or billions of paired images and texts. To demonstrate that our knowledge-driven MACK is complementary with existing data-driven models, we use the MACK to re-rank the retrieval results two state-of-the-an models including CLIP 33 and ALBEF (1 The original and re-ranked performance of these two models on the Flickr30k and MSCOCO datasets are shown in Table 2 It should be noted that, we use the zero-shot versions of these two models with the goal to make fair comparisons with the proposed MACK. The zero-shot means that they are pretrained on large-scale external datasets while directly perform test the Flickr30k and MSCOCO datasets. From the table, can that although their original accuracies are very high, re-ranking with our proposed MACK can further improve them. For the CLIP, the re-ranking can lead 1.4% and 1.6% improvements in R@I of image annotation on the two datasets, respectively. For the ALBEF, the re-ranking can lead to overall 6.4% improvement in Rs the Flickr30k dataset.
These evidences demonstrate that the MACK does show complementary properties when combining with existing models. To further explain this, we show text queries by the CLIP in F These examples are selected based on the following rules: 1) ranks of groundtruth matched images (marked by red bounding boxes in the figure) by the CLIP not 1op-1, and 2) our proposed MACK can re-rank these matched images higher. In the first example, it seems that the CLIP cannot well understand the "yellow hat", "red apron" "pink shirt" in the query text, since the first retrieved image does not contain either of these semantic concepts but ranks
examples of retrieved images based on
'Note that the original ALBEF is a hybrid model that includes both dual stream architecture and single-stream stream architecture. Here we only use the dtal-stream one with the goal to make fair comparison with the two-stream CLIP.
7

---

Table 3: Cross-dataset image-text matching by re-ranking existing models the Flickr30k and MSCOCO datasets. "+" denotes the re-ranking. MSCOCO -> Flickr30k: training existing models on the MSCOCO dataset and testing them the Flickr30k dataset. Flickr30k MSCOCO: training existing models on the Flickr30k dataset and testing them on the MSCOCO dataset.
MSCOCO >> Flickr30k
Flickr30k MSCOCO
Method
Image Annotation
Image Retrieval
Image Annotation
Image Retrieval
Rs
Rs
R@I
R@5
R@10
R@I R@5
R@10
R@I
R@5
R@10
R@I
R@5
R@10
VSRN 30
42.3
69.3
78.1
53.1
79.5
87.5
409.9
14.0
31.7
42.2
20.4
40.0
50.0
198.4
MACK
42.6
69.8
78.5
53.3
79.7
87.7
411.7
14.4
32.6
43.1
20.5
40.5
50.2
201.4
SAEM 440
41.4
70.2
80.0
53.4
80.9
89.6
415.5
14.8
34.0
45.0
23.2
45.4
57.4
219.8
+ MACK
41.8
70.7
80.0
54.2
81.2
89.9
417.9
15.4
34.9
45.9
23.6
46.0
57.7
223.4
higher than the groundtruth one. While our MACK especially focuses these semantic concepts and can improve the corresponding similarity between the matched image and query text.
4.5 Cross-dataset Image-text Matching
We also try to re-rank existing image-text matching models when applying them to the scenario of cross-dataset image-text matching. The experimental settings are explained as follows: I) selecting two representative image-text matching models (VSRN 30 and SAEM 440) and training them on source dataset (e.g., Flickr30k MSCOCO), 2) performing model test on a different target dataset (e.g., MSCOCO or Flickr30k), and 3) using the proposed MACK to re-rank their results on the target dataset. Note that either our MACK or the two models do not use the target dataset for model training.
In Table 3 the results of two directions of cross-dataset generalization, i.e. MSCOCO >> Flickr30k and Flickr30k -> MSCOCO, are both presented. We find that using the MACK to re-rank the VSRN and SAEM can improve their generalization abilities in the cross-dataset scenario. Although the relative improvements might be little bit small, they are substantial in all evaluation criterions. Comparing Table 3 with Table 2 it seems that the performance improvements of re-ranking partially depend on the performance base models. When the base models are more accurate, the performance improvements of re-ranking are larger.
4.6 Prototypical Representation Analysis
In the proposed MACK, one of the most im- portant part is the prototypical region represen- tation. Even the same object could have com- pletely different appearances in images, so using how many regions compute each region proto- type is crucial. We count the number of regions for each word in the Visual Genome dataset and show the statistical result in Figure 3, in which the horizontal axis is word index and vertical axis is logarithmic transformed number of re- gions. From the figure, we can see that about
Loganthmic number of regions
10
8-
Most only
related regions to computer their prototypical region representations
6-
80% words have very small numbers of regions, i.e. less than 20 (denoted by the dashed line). Therefore, for most semantic concepts in the knowledge, computing their prototypical region representations is data-efficient.
5000 10000 15000 20000 25000
Word index (sorted by frequency)
Figure Logarithmic number of regions v.s. word indexes. (best viewed in colors).
To verify whether using small numbers of re- gions has large impact on the performance, we define threshold f that controls the maximum number of regions. We use the ALBEF MACK as the base model and select from (20, 50, 100, oo J, where is the default setting of ALBEF MACK. The experimental results on the Flickr30k and MSCOCO datasets are shown in Table 4 in which p the percentage of how many word-region annotations of Visual Genome dataset used. We can see that as the becomes larger

---

Table 4: Image-texi matching by ALBEF + MACK on the Flickr30k and MSCOCO datasets. is the maximum number of used regions when computing a prototypical region presentation. p is the percentage of how many word-region annotations of Visual Genome dataset are used.
Flickr30k dataset
MSCOCO dataset
f
p
Image Annotation
Image Retrieval
Image Annotation
Image Retrieval
R@i
Rā5
R@10
R
R
R@10
R@I
R@5
R@
R&1
R@5
R@10
20
15%
60.0
85.1
91.2
79.7
96.0
97.7
509.8
39.7
68.0
79.1
62.2
85.6
92.7
427.5
50
37%
60.2
85.3
91.1
79.9
96.0
97.7
510.3
39.7
68.1
79.1
62.2
85.7
92.7
427.8
100
73%
60.7
85.3
91.2
80.1
95.9
97.7
510.9
39.9
68.3
79.2
62.5
86.0
92.7
428.8
100%
61.8
85.8
91.4
80.1
96.4
97.7
513.3
41.0
69.0
79.4
62.4
86.1
92.7
430.9
Table 5: Image-text matching by re-ranking CLIP using different l and À on the Flickr30k dataset. k the length of re-ranked images/texts. is the balancing factor between knowledge and CLIP.
Image Annotation
Image Retrieval
Image Annotation
Image Retrieval
k
Rs
X
Rs
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@S
R@10
10
66.8
88.2
91.7
86.2
97.2
98.7
528.8
0.1
66.8
88.3
92.7
86.2
97.2
98.9
530.1
15
66.8
88.3
92.7
86.2
97.2
98.9
530.1
0,2
66.6
88.6
92.9
86.4
97.1
98.8
530.5
20
66.8
88.3
92.7
86.2
97.2
98.9
530.1
0.3
66.6
88.4
93.0
83.5
97.0
98.7
527.1
the performance becomes slightly better. It seems that different numbers of regions do not have a significant impact the performance. It is reasonable since computing the mean might not need too many samples. This is also a good property for the knowledge, since if we want to add a new semantic concept, only have annotate a limited number of regions.
4.7 Hyperparameter Analysis
In Ta we perform the experiment of image-text matching by re-ranking CLIP using different and À on the Flickr30k dataset, to study the impact of these two hyperparameters on the final performance. In the table, k is the length of re-ranked images/texts, and is the balancing factor between knowledge and CLIP. In the table, we vary k E {10, 15, 20} and E {0.1, 0.3). and find that the final performance is not very sensitive to the hyperparameters.
4.8 Knowledge Visualization
At last, we visualize the pretrained general knowledge and fine-tuned domain knowledge (trained on the Flickr30k dataset) in Figure 4 to illustrate major changes after the training. In the figure, there are two low-dimensional word distributions (surrounded by solid boxes) corresponding to the pretrained general knowledge and fine-tuned domain knowledge, respectively. The two word distributions are obtained by using the t-SNE method [34] to embed two set of prototypical region presentations (i.e., {vu or [u }. : K). For clear illustration, we set K 100 and only select the most frequently used 100 semantic concepts the Flickr30k dataset. For each semantic concept, we show the corresponding word in the two-dimensional coordinate. We also group semantically related words and show them with different colors.
In the left word distribution, we can see that semantically related words stay nearby. For example, the words in the three groups (marked by dash lines) are about animals, indoor objects and transports, respectively. In the center area, there are many words (marked by the green color) about colors. However, there are still some related words that have remote distances. For example, in the bottom there are also several words about animals such as bird, dog, bear and cat, which are far from the words in the first group. After fine-tuning the knowledge with unpaired data, find the word distribution is semantically more compact. We can see that all the words about animals are very close staying together the first group. In the second group, more words about transports such as plane and boat are added. These evidences indicate that by training with the concept-leve cycle consistency, related semantic concepts are able leverage their potential relations to make the prototypical region representations more discriminative.

---

DAPES
pare
ROhe
beilding
dacc
fence
Fres
handle
shadow
prronos
7adle
Pretrained general knowledge
tuned knowledge
Figure 4: Visualization of pretrained general knowledge and fine-tuned domain knowledge. Each word indicates its corresponding prototypical region representation embedded by the t-SNE method. In the two word distributions, we divide words into semantically related groups and represent them using different colors. We also mark three representative groups by dashed lines, which contain words about animals, indoor objects and transports, respectively.
5 Conclusion and Future Work
This work has studied practically important but seldom investigated problem as unpaired image-text matching, in which paired images and texts are not available during model training. To deal with this problem, inspired by the knowledge used in human brain, we have proposed a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK). Compared with existing multimodal knowledge methods, the MACK has two major properties terms of conceptual level and one-to-one alignment. It can either perform test directly perform training based unpaired data, as well as re-rank existing image-text matching models to improve their performance. We have demonstrated its effectiveness with extensive experiments.
To the best of our knowledge, this might be the first work to study the unpaired image-text matching. So the goal of this initial work is not pursuing high performance but to explore the modeling of knowledge. In the future, we will study more effective strategies to improve the relatively low accuracy. In addition, will also consider to mine more useful information from the knowledge from the aspects of dynamic knowledge updating and advanced knowledge reasoning.
Acknowledgements
This work was jointly supported by National Key Research and Development Program of China Grant No. 2018AAA01004000 National Natural Science Foundation of China (62236010, 62276261, 61721004, and U1803261), Key Research Program of Frontier Sciences CAS Grant No. ZDBS-LY- JSC032, Beijing Nova Program (Z201 (Z00000000))000))) and CAS-AIR.
10

---

References
[1] P. Anderson, X. He, C. Buchler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In IEEE Conference Computer Vision and Pattern Recognition. pages 6077-6086 2018.
[2] Y. Bi. Dual coding of knowledge in the human brain. Trends in Cognitive Sciences, 25(10):883-85, 2021.
[3] H. Chen, G. Ding. X. Liu, Z. Lin, J. Liu, and J. Han. Imram: Iterative matching with recurrent attention memory for cross modal maagettext retrieval. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2265-12263,, 2020.
[4] Y.-C Chen, L. Li, L. Yu, A. EI Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal imagc-text representation learning. In European Conference on Computer Vision, pages 104-120 2020.
[5] F. Faghri. J. Fleet, J. R. Kiros, and S. Fidler. Vse++: Improved visual-semantic embeddiiggs. arXiv
preprint 10xiv:7777006122 2017.
[6] A. Frome, G. S. Corrado, Shlens, Bengio, J. Dean, T. Mikolov, et al. Devise: deep visual-semantic
embedding model. In Neural Inforration Processing Systems, pages 2121 2013.
[7] G. Geigle, J. Pfeiffer. N. Reimers, I. Vulié, and I. Gurevych. Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross-modal retrioval. arXiv preprint arXiv:2103. 1920, 2021.
[8] Y. Huang, Y. Long, and L. Wang. Few-shot image and sentence matching via gated visual-semante embedding. In AAAI Conference on Artificial Intelligence, volume 33, pages 8489-8496, 2019.
[9] Y. Huang, J. Wang, and I., Wang. Few-shot image and sentence matching via aligned cross-modal memory.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6))2968-2999)
2021.
[10] Y. Huang and L. Wang. Acmm: Aligned cross- modal memory for few-shot image and sentence matching. In IEEE International Conference on Computer Vision. pages 5774-5783 2019.
[11] Y. Huang. W. Wang, and I., Wang. Instance-aware image and sentence matching with selective multimodal
Istm. In JEEE Conference on Computer Vision and Pattern Recognition, pages 2310-23101 2017. [12] Y. Huang, Y. Wang, and L.. Wang. Efficient image and sentence matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[13] Y. Huang, Q. Wu, and L. Wang. Learning semantic concepts and order for image and sentence matching.
In JEEE Conference on Computer Vision and Pattern Recognition, pages 6163 6171, 2018.
[14] Y. Huang, Q. Wu, W. Wang, and L. Wang. Image and sentence matching via semantic concepts and order learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(3):636-650, 2018.
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
[16] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantoe embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics, 2015. [17] R. Krishna, Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Sharma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.
International Journal of Computer Vision, 123(1): 73, 2017.
[18] K.H.. Lee, X. Chen, G. Hua, H. Hu, and X. He. Stacked cross attention for image-lext matching. In European Conference on Compuder Vision, pages 21 212-228, 2018.
[19] G. Lov, G. Sadch, Klein, and L. Wolf. Rnn fisher vectors for action recognition and image annotation.
In European Conference on Computer Vision, pages 833-850, 2016.
[20] G. Li, N. Duan. Y. Fang, M. Gong. D. Jiang, and M. Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre- training. In AAAI Conference on Artificial Intelligence, pages 11336- 11344, 2020.
[21] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Neural Information Processing Systems, 34, 2021. [22] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu. Visual semantic reasoning for image-text matching. In IEFE International Conference on Computer Vision, pages 4654-4662, 2019.
[23] X. Li. X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision,
pages 121-137, 2020.
[24] T.-Y. Lin. M. Maire, S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755, 2014. [25] C. Liu, 7. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang. Graph structured network for image-text matching, In IEEF/CVP Conference on Computer Vision and Pattern Recognition, pages 10921-10930, 2020.
[26] Y. Liu, H. A. Garcia-Duran, M. Niepert, D. Onoro-Rubio, and D. S. Rosenblum. Mmkg: multi-modal knowledge graphs. In European Semantic Web Conference, pages 459-474, 2019.
[27] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee, 12-in-l: 1: Multi-task vision and language representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10437- 10446, 2020.
11

---

[28] K. Patterson, P. J. Nestor, and T. T. Rogers. Where do you know what you know? the representation of semantic knowledge in the human brain. Nature Reviews Neuroscience, 8(12):76-987, 2007.
[29] F. Perronnin and C. Dance. Fisher kemelss on visual vocabularies for image categorization. In IEEE
Conference Computer Vision and Pattern Recognition, pages 1-8, 2007.
[30] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International
Conference on Machine Learning, pages 8748-8763, 2021.
[31] S. Shah, A. Mishra, N. Yadati, and P. P. Talukdar. Kvqa: Knowledge-aware visual question answering. In AAAI Conference on Artificial Intelligence, pages 8876-8884, 2019.
[32] R. Socher, M. Ganjoo. C. D. Manning, and A. Ng. Zero-shot learning through cross cross-modal transfer. Neural Information Processing Systems. 26, 2013.
[33] R. Sun, X. Cao, Y. Zhao, J. Wan, K. Zhou, F. Zhang. Z. Wang, and K. Zheng. Multi-modal knowledge graphs for recommender systems. In ACM International Conference on Information & Knowledge Management, pages 1414, 2020.
[34] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
[35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E.. Kaiser, and I. Polosukhin. Attention is all you need. Neural Information Processing Systems, 30, 2017.
[36] L. Wang. Y. Li, and S. Lazebnik. Learning deep structure- preserving image-text embeddings. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5005-5013, 2016.
[37] S. Wang. J. Yue, J. Liu, Q. Tian, and M. Wang. Large-scale few-shot learning via multi-modal knowledge
discovery. In European Conference on Computer Vision, pages 718-734, 2020.
[38] Z. Wang, X. Liu, H. Li, L. Sheng, J. Yan, X. Wang. and J. Shao. Camp: Cross-modal adaptive message passing for text-image retrieval. JEEE/CVF International Conference on Computer Vision, pages
5764-5773, 2019.
[39] Q. Wu, C. Shen, L. Liu, A. Dick, and A. Van Den Hengel. What value do explicit high level concepts have in vision to language problems? In IEEE conference on computer vision and pattern recognition, pages
203-212 2016.
[40] Y. Wu, S. Wang, G. Song, and Q. Huang. Learning fragment self-attention embeddings for image-text
matching. ACM International Conference on Multimedia, pages 2088-2096, 2019.
[41] F. Yan and K. Mikolajezyk. Deep correlation for matching images and text. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3441-3450, 2015.
[42] P. Young. A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Asxociation for Computational Linguistics, 2:67-78. 2014.
[43] B. Zhang, H. Hu, T.. Qiu, P. Shaw, and F. Sha. Visually grounded concept composition. arXiv preprint
arXiv:2 109. 14115, 2021.
[44] Q. Zhang. Z. Lei. Z. Zhang, and S. Z. Li. Context- attention network for image-text retrieval. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1336-3545,, 2020.
[45] Y. Zhang, M. Jiang, and Q. Zhao. Explicit knowledge incorporation for visual reasoning. In IEEF/CVF
Conference on Computer Vision and Pattern Recognition, pages 1356-1365, 2021.
[46] X. Zhu, Z. Li, X. Wang, X. Jiang. P. Sun, X. Wang. Y. Xiao, and N. J. Yuan. Multi-modal knowledge graph construction and application: A survey, arXiv preprint arXiv:2202.00 05786, 2022.
[47] Y. Zhu, C. Zhang, C. Ré, and L. Fei-Fei, Building a large-scale multimodal knowledge base system for answering visual queries. arXiv preprint
2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [No] This work does not
have potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] In the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Yes] See Sections 4.1 and 4.2
(c) Did you report
bars (e.g.. with respect the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e-g., type of GPUs,
internal cluster, or cloud provider)? [Yes] See Section
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes] Section 4.1
(h) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're re us- ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "z1h5-fudPD", "reviewer": "Reviewer_Ypgg", "summary": "First of all, this paper pays attention to an important and challenging problem -- unpaired image/text retrieval and designs a framework that can be easily plugged into existing image-text matching models for re-ranking.", "questions": "see weeknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The proposed model is simple and can be easily plugged into existing image-text matching models to boost the performance.\n\nS2: Extensive experiments are conducted and the results show that using the proposed MACK for re-ranking, the performance can be improved by around 1%.\n\nW1: The proposed model actually employs human-annotated data to obtain prototypical region representations and prior knowledge and the dataset is composed of paired annotations, so \"unpaired image-text matching\" in the title is misleading.\n\nW2: The pretrained general knowledge is highly related to the training data. In the paper, Visual Genome is used, but some other datasets can also be used, such as MSCOCO and ImageNet. I do not see any ablation studies on the pretrained general knowledge.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "mMCm9K3BylU", "reviewer": "Reviewer_MQX3", "summary": "This paper proposes Multimodal Aligned Conceptual Knowledge (MACK) for image-text matching without requiring paired image-text for finetuning the pretrained model. In addition, the proposed approach can also be extended as a re-ranking method to improve the performance of current image-text matching approaches. It uses the word-region aligned datasets to get the pretrained general knowledge, and then finetune the model on unpaired image-text datasets to obtain the finetuned domain knowledge. During inference, image-text similarity scores are calculated by aggregating the word-region similarities. For the experiments, the pretrained general knowledge are obtained from Visual Genome dataset, and image-text matching experiments are conducted on COCO and Flickr30k datasets with both unpaired image-text setting and re-ranking setting.", "questions": "Please address my concerns in the weaknesses section.", "limitations": "The authors mentioned the limitations but do not address the potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This work proposes a new setup for image-text matching. It proposes to train or fine-tune image-text matching models without paired image text data. \n2) It uses a pretrained detection model to provide the correspondences between visual and word features as domain knowledge.\n3) Finetuning the pretrained domain knowledge on unpaired images and texts is effective for improving the image-text matching performance.\n4) The proposed approach is not only validated on unpaired image-text matching settings, but also can be used as a re-ranking approach to improve the image-text matching performance of existing models such as CLIP [25] and ALBEF [16].\n\nWeaknesses:\n1) I am doubtful about whether the setup of unpaired image-text matching and the proposed solution is realistic or not. Firstly, although the training does not require ground-truth image-text correspondences, it requires the word-region alignment from the Visual Genome dataset to pretrained the detection model, and the ground-truth of VG dataset is more difficult and costly to annotate than image-text alignments. Secondly, the authors claim that they do not need paired image-text data for training, but the experiments are still conducted on COCO and Flickr30k image captioning datasets. Although the image-text correspondences are not used during training, there is an implicit prior that there exist one-to-one correspondences between images and text. It is unclear whether the proposed approach still works well on totally unpaired images and captions, e.g., Flickr images and COCO captions.\n2) Why is the aggregation function in Eq.(4) of Section 3.3 defined as max-mean pooling? The ablation study experimented with mean-mean pooling, but what if it is max-max pooling or mean-max pooling?\n3) The performance of unpaired image-text matching (Table 1) is quite low. And the authors do not provide the qualitative results on unpaired image-text matching. I am wondering if the retrieval results are reasonable with the unpaired setup.\n4) No performance comparison is conducted. The authors could compare it to other unpaired matching or re-ranking approaches. If there are no previous approaches, at least it should be compared to some naive baselines, for example, training the unpaired image-text matching with cycle-consistency loss.\n5) The cross-dataset re-ranking in Table 3 only provides marginal performance gain (less than 1%).\n6) The re-ranking hyperparameters $k$ and $\\lambda$ are not experimented in the ablation study.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c7N04ifyj_", "reviewer": "Reviewer_iP3D", "summary": "This paper studies the unpaired image and text retrieval task. Concretely, traditional image text retrieval model are learnt via paired and aligned image and text data. However, it is pretty common that the text is not following the same distribution as the text in the training set. Therefore, it is interesting and important to learn the model from unpaired image and text. This paper proposed an interesting model called MACK, which uses image patches to form prototypes for each word/phrase in the given text. Given a new image, it first detects all the objects in the image. Then it aligned the detected object with the prototypes. As the prototypes and the object representation live in the same space. Each prototype also links to one word. It can improve the image and text retrieval performance and conduct alignment across unpaired image and text.", "questions": "I like this idea a lot. Please address three questions mentioned above.", "limitations": "I don't see any negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nI love this idea a lot! This is an elegant way to tackle the image and text retrieval task. The main challenging in this task is the semantic ambiguity in the text side. For example, all images of 'apple' with different appearance is aligned with the text 'apple'. How to align the text 'apple' with multiple 'apple' objects is a challenging task. This paper uses the prototypes to link the text and image worlds is very elegant.\n\nCons:\nI have two high-level questions for the authors:\n1. How to handle the out-of-vocabulary issue. For example, if the text has not been seen in the training set, how to handle this case? How to generate/obtain the prototypes for this OOV word/text?\n2. How to handle the data imbalance problem? For example, some words might contain less images linked with it. This would lower the quality of the prototype representation. How to handle this case?\n3. Correct me if I was wrong. To my understanding, the prototypes are the corresponding images representation of nouns. But how would the model distinguish between the following two cases: 'dog chases cat' vs. 'cat chases dog'. The prototypes for the 'dog' and 'cat' are all the same. How to use the prototypes to distinguish them?\n\nMissing related work: \nZhang, B., Hu, H., Qiu, L., Shaw, P., & Sha, F. (2021). Visually Grounded Concept Composition. Findings of EMNLP. has a similar idea of the proposed approach.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HUchqKalzmJ", "reviewer": "Reviewer_NfVu", "summary": "This paper presents a framework for performing unpaired image-text matching. It leverages the Visual Genome dataset and Faster-RCNN to form a region-based method that matches words and regions of images and promises to yield robust image-sentence-level descriptors for pairing. A cycle consistency loss is proposed to improve performance on specific datasets. Since the proposed method is expected to handle region-word correspondences that current image-text matching models such as CLIP cannot solve well, it can be integrated into current systems and improve their matching performance. The authors tested the proposed method on two common benchmarks, Flickr30k and MSCOCO, and conducted image-text pairing, re-ranking matching results, and cross-dataset matching experiments. Results shown the strength of the proposed method.", "questions": "The current hyperparameter used in the re-ranking experiments, $k=15$ and $\\lambda =0.1$, seems are set very specific. The reviewer is interesting if the author can do ablations on $k$.\n\nWhat is the computation cost of the fine-tuned domain knowledge stage? $F$ and $M$ seem too large to make this step practical. If so, the proposed way which involves a transformation matrix is impractical. Does it possible to introduce a NN approximates $W$ but much more efficiently?\n\nConsidering the performance of the \"pretrained general knowledge\", can the author provide the results which only use half the data of Visual Genome? Since annotating Visual Genome is pretty expensive, it is important to see if the proposed approach is sensitive to the sample numbers of the used extra dataset.", "limitations": "Clearly, the current method is limited to the visual properties annotated in Visual Genome and the ability of the pre-trained Faster-RCNN. If we want to study fine-grained structures within objects, the proposed method will fail. Also, the method will fail if no object associated with the word is detected.\n\nConsidering the proposed framework: (1) mainly relies on the extra fine-grained human-annotated dataset and pre-trained detectors; (2) has very poor results itself; (3) has minor improvements when integrating with the current SOTA models via carefully selected hyper-parameters; (4) the proposed fine-tuned stage is probably impractical, it is more like a proof-of-concept paper. The following researchers are hard to scale up the used fine-grained dataset and find stronger pre-trained detectors, and also, there are web-scale paired image-text datasets. The reviewer is not sure if this paper meets the bar of NeurIPS.\n\nIf the mentioned points got addressed to some levels, the reviewer is willing to adjust the scores accordingly.\n\n----------------------------------\nAfter rebuttal: the author has well addressed my concerns and questions. The paper now has ablation studies around the hyperparameters and related discussions about the computation resource and the dataset scale. I raised my scores from 4 to 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The studied direction is very important and novel that trying to pair the image and text data according to region-based information. As the author mentioned in the introduction, images usually contain information beyond the info delivered by words. The proposed region-based method hopefully shows more robust performance in conducting image text alignment. All the used components are technically sound, including using Visual Genome, the pre-trained Faster RCNN, and the cycle-consistency loss. Overall the paper is easy to follow.\n\nThe author conducted experiments on two common datasets, Flickr30k and MSCOCO, and adopted various baselines, including CLIP, ALBEF, VSRN, and SAEM.  Although the proposed method itself performance is much worse than the models trained on pairs of image-text datasets, it can further improve the performance of those models as shown in Table 2. Also, the reviewer pretty much likes the Cross-dataset experiments, where both the used baseline + MACK did not see the target dataset during training. \n\nAccording to the gap between Table 1 and Table 2, it is clear that the current method which leverages VG dataset and pre-trained Faster R-CNN is not powerful enough to conduct image-text alignment. The author did not ablate the used hyper-parameters in re-ranking experiments. The current minor improvements shown in Table 2 might come from carefully hyper-parameter searching. So does Figure 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept