Multimodal Aligned Semantic Knowledge for Unpaired Image-text Matching
Authors: Laiguo Yin
Yixin Zhang
YUQING SUN
Lizhen Cui
Venue: ICLR 2026 Oral (2026)
Keywords: Unpaired Image-text Matching
Out-of-Distribution Word
Multimodal Aligned Semantic Knowledge
Prototype
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose multimodal aligned semantic knowledge, which leverages word embeddings as bridges to associate words with prototypes, capturing semantic relationships between words and further utilizing information from OOD words.

Abstract: While existing approaches address unpaired image-text matching by constructing cross-modal aligned knowledge, they often fail to identify semantically corresponding visual representations for Out-of-Distribution (OOD) words. Moreover, the distributional variance of visual representations associated with different words varies significantly, which negatively impacts matching accuracy. To address these issues, we propose a novel method namely Multimodal Aligned Semantic Knowledge (MASK), which leverages word embeddings as bridges to associate words with their corresponding prototypes, thereby enabling semantic knowledge alignment between the image and text modalities. For OOD words, the representative prototypes are constructed by leveraging the semantic relationships encoded in word embeddings. Beyond that, we introduce a prototype consistency contrastive loss to structurally regularize the feature space, effectively mitigating the adverse effects of variance. Experimental results on the Flickr30K and MSCOCO datasets demonstrate that MASK achieves superior performance in unpaired matching.

Published as a conference paper at ICLR 2026
MULTIMODAL ALIGNED SEMANTIC KNOWLEDGE FOR UNPAIRED IMAGE-TEXT MATCHING
Laiguo Yin', Yixin Zhang', Yuging Sun Lizhen Cui'*
'Shandong University 'Nanyang Technological University lgyinemail. sdu. edu. en, zhangy wttpyyxxnn6110@mmee com, {sun.yuging, cl dsdu..euu.......
ABSTRACT
While existing approaches address unpaired image-text matching by construct- ing cross-modal aligned knowledge, they often fail to identify semantically cor- responding visual representations for Duttot-Distribution (OOD) words. More- over, the distributional variance of visual representations associated with differ- ent words varies significantly, which negatively impacts matching accuracy. To address these issues, we propose a novel method namely Multimodal Aligned Semantic Knowledge (MASK), which leverages word embeddings as bridges associate words with their corresponding prototypes. thereby enabling semantic knowledge alignment between the image and text modalities. For OOD words, the representative prototypes are constructed by leveraging the semantic relationships encoded in word embeddings. Beyond that, we introduce a prototype consistency contrastive learning loss to structurally regularize the feature space, effectively mitigating the adverse effects of variance. Experimental results on the Flickr30K and MSCOCO datasets demonstrate that MASK achieves superior performance in unpaired matching.
INTRODUCTION
Image-text matching has become an essential technique for various applications, such as visual question answering Ozdemir & Akagündüz] (2024); Lerner et al. (2024), image captioning Fu et al. (2024); Wang et al. (2024a), cross-modal retrieval Wang et al. (202bb); Li et al. (2024b) and so forth. Due to the heterogeneous representations and asymmetry of information between images and texts, accurately learning cross-modal semantic correspondences remains challenging prob- lem. Although training large-scale paired image-text data has substantially improved matching accuracy, collecting and annotating such data at scale is often impractical in real-world scenarios.
Viuud prasstyp spaze
Werd enahadding space
AMD
(a) Existing unpaired image-text matching
(b) Multimodal aligned semantic knowledge
Figure 1: Comparison between existing matching paradigms and our proposed unpaired framework.
To reduce the dependence of image-text matching on paired data, the unpaired image-text matching paradigm Huang et al. (2022) was first proposed, in which domain-specific paired images and texts are assumed to be unavailable during model training. Inspired by the fact that the human brain can correlate arbitrary images and texts well while does not need learn from large-scale paired
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
images and texts, the unpaired image-text matching is implemented by modeling human brain-like like knowledge, which is multimodal aligned and used to associate visual and linguistic information.
Note that unimodal visual or linguistic knowledge has been widely used for vision and language understanding task [Chen & Zhao (202)). There are also some works [i et al. (2025); Gao et al. (222)) directly combining these two types of knowledge together, but the resulting knowledge is not multimodal aligned. Another alternative is Multimodal Aligned Conceptual Knowledge Huang al. (2024a), which establishes correspondences between prototypical region representations and words, as shown in Figure (a). However, these knowledge- methods still face the following issues: 1) OOD words have not been thoroughly investigated. Existing knowledge-basse based meth- ods fail to leverage the underlying semantic structure to transfer the visual prototypes of known words OOD words; 2) The influence of distributional variance has been largely overlooked. The region representations corresponding to different words exhibit substantial appearance varia- tions. Consequently, certain instances that deviate substantially from the distributional mean may be prone to misclassification into other words; 3) The raw region representation is insufficient in effectively capturing the semantic relationships between words. The raw region representation is predominantly influenced by the co-occurrence relationships among regions. However, there is inherent relationship between semantic relevance and co-occurrence patterns. For instance, while 'human' and 'hat' often co-occur in visual contexts, 'human' and "gentleman" may exhibit higher semantic similarity.
To address these issues, propose method namely Multimodal Aligned Semantic Knowl- edge for unpaired image-text matching, which establishes semantic alignment between prototypical region representations and word embeddings, as shown in Figure [ (b). We summarize our key contributions as follows:
We propose a novel cross-modal semantic alignment method, MASK, which constructs representative prototypes for OOD words by exploiting the intrinsic relationships among word embeddings, thereby enhancing the model's generalization ability in unpaired image- text matching.
We introduce prototype consistency contrastive learning loss structurally regularize the feature space, which explicitly encourages region representations associated with the same word align closely with their prototype, thereby mitigating the adverse impact of distributional variance.
We incorporate external knowledge from pre-trained word vectors as auxiliary supervision signals, which establishes relation-presee preserrinneeeegggeee equivariant mapping between region repre- sentations and word embeddings, enabling the region representations to effectively capture semantic relationships among words.
2
RELATED WORK
2.1 MODEL- MATCHING
Extensive model-based matching works have been made on measuring the semantic correlation between vision and language. To our knowledge, Socher el al. Socher et al. (2013) might propose the first framework of Visual-Semantic Embedding (VSE) to correlate images and their class labels in two-stream manner. Lee et al. Lee al. (2018) propose a Stacked Cross Attention (SCAA))(SCAN) to discover all latent alignments by using regions of the image and words in a sentence as context. The SCAN has been extensively studied from various aspects such as memory modeling |Huang al. (021), context modeling Zhang et al. (2020) and graph structure Liu et al. (2020). Later, by using millions or billions of paired images and texts for supervised model learning, many models Li et al. 2020); Pan et al. (2023); Wu et al. (2024); Li et al. (2024a); Pham et al. (2024); Ge al. (2024) based on multimodal versions of Transformer have been proposed and have achieved remarkable results. However, while these existing methods achieve relatively strong performance, they rely heavily on extensive paired image-text datasets for supervised training, which significantly restricts their applicability.
2

---

Published as a conference paper at ICLR 2026
2.2 KNOWLEDGE-BASSE MATCHING
Knowledge-basee matching has been explored in a few vision and language understanding tasks. Feng et al. Feng al. (2019) use visual concept detector to encourage generated captions to be semantically consistent with visual concepts. This work focuses more on the visual concepts while paying less attention to visual relations. Gu et al. Gu et al. (019) propose to align two scene graphs of images and texts, by mapping one to the other one, and vice versa. The performance of this work depends more the accuracy of predicted scene graphs that contain both visual concepts and relations. Huang et al. |Huang et al. 2022) introduce Multimodal Aligned Conceptual Knowl- edge (MACK) for unpaired image-text matching, effectively addressing challenges associated with diverse appearances. Subsequently, Huang et al. Huang et al. (2024b) further extend MACK to broaden applicability. However, these approaches still face certain limitations, particularly in their inability to leverage the underlying semantic structure to transfer the prototype representations of known words OOD words.
METHOD
This section illustrates the pipeline of obtaining the proposed MASK for unpaired image-text match- ing, shown in Figure For each region, the image embedding branch is utilized extract region representations characterized by high cohesion and low coupling. For each word, the text embed- ding branch is employed to generate its corresponding word embedding. Consequently, a knowledge set in which each word is aligned with its corresponding prototypical region repre- sentation. This knowledge serves bridge the knowledge-basse enabling the association between domain-specifee images and texts and thereby supporting unpaired matching. It is worth noting that the knowledge can be fine-tuned better adapt to specific do- main (Appendix H. However, the fine-tuning step is optional according to whether unpaired data in certain domain are given or not.
obtain
image-text matching module,
Region
Pretraine Fastor-RCN
Image Embedding Branch
Text Embedding Braneh
c Erem-medal
alggeeent bas
Pretralned word vectoos
prah
Prestot pe consitunes contrautive bos L.,
Predicted word embedding space
Weed curbelding
Knowledge-based Image-text Matching
Matcimdal Alianed Semantic Kaowledae
Global smmitait
repeesentations
reginn
Tokeniizaiion
Ganeian
distrihation
whlle dog and restel
bedid her.
recamstrcted
region
vecsor
represeraaiion:
Finstee RCCNNCCNNN
Protraiinel
peik
Chaad pardoge suen
reperssacation
protstypical (prstotypc)
Figure 2: The Multimodal Aligned Semantic Knowledge (MASK) for unpaired image-text match- ing. The top figures illustrate how to obtain the knowledge and the bottom figures illustrate how to use the knowledge for unpaired image-text matching.

---

Published as a conference paper at ICLR 2026
3.1 MULTIMODAL ALIGNED SEMANTIC KNOWLEDGE
In addition to the semantic concepts, the studied knowledge also has another important property of cross-modal one-to-one alignment. For each word, its semantically related objects in different regions often exhibit diverse visual appearances, which could easily lead to confusion in practice. Therefore, rather than align each word multiple related regions in an one-to-many manner, the MASK aligns each word to single prototypical region, with the goal to alleviate the issue of appearance variation. In particular, we formulate the knowledge as a set of semantic concepts having paired multimodal representations {(w,, Uk) [=))k= where wy and vy the word embedding and prototypical region representation of the k-th semantic concept, respectively, and is the total number of semantic concepts.
As shown in Figure 2 for each word, we compute the word embeddings Wi by using pre-trained word vectors. For each region, first extract the raw region representations r 1,..., .., J.)) by feeding a bounding box and an image into the pre-trained object detection model Faster-RCNN. Then extract region representations μ4 by utilizing the Prototype-Aware Encoder (PAE) h, with ry as the input:
Uy = h(ry; On).
(1)
where θ₁ is the parameters of h and oj represents the variance of distribution. Finally, we compute the prototypical region representations Uk by averaging all related region representations (ugls=1....dd
2k
(2)
Jk
where the Js indicates the number of regions for the k-th semantic concept.
3.2 IMAGE EMBEDDING BRANCII
Given a batch B of paired regions and words, we first obtain raw region representations R fra (rabx=1....B (R E R))MM, where M is the dimension of Tu- We then extract region repre- sentations j using the PAE model h, which takes R as input and consists of a Fully Connected (FC) layer followed by three self-attention layers:
H. h(R; On).
where the mean I BBZ and variance E Rx2 are used preserve the information of the raw region representations R by using the Feature Restoration Module (F (FM g comprising a self-attention layer and two FC layers:
R' 1(, o, 2; Og),
where the is random vector sampled from standard normal distribution and O is the parameters of g. The Z represents the feature dimension of latent space. As shown in Figure the PAE model and F FRM model are trained jointly using the information retention loss function C..:
Eir DKL( (O,
1) ((r..)))((RRe)))
[]]r r 12)
(5)
where the D 01 N 1) implies that the data distribution in the latent space gradually approaches the standard normal distribution. The ((r,.))) [[|r, r/ 2 measures the dif- ference between the reconstructed raw region representations R' and the raw region representations R. The loss Ltr ensures that the mean μ retains a significant amount of information from the raw region representations R.
Inspired by clustering theory and contrastive learning, we design a prototype consistency contrastive learning loss Lat to reduce the influence of distributional variance between prototypes and their re- lated region representations. The loss La employs prototypes as class centers, maximizing the similarity between region representations and their corresponding prototypes while minimizing similarity with other prototypes, thereby achieving intra-class aggregation and inter-class separa- tion. Compared to traditional instance-to-instane contrastive learning. Lat introduces prototypes as global semantic representatives, explicitly aggregating instances of the same class around their cor- responding prototypes. This process constructs more structured and discriminative representation
4

---

Published as a conference paper at ICLR 2026
space, enabling the model to capture clearer semantic boundaries. The loss Lat is defined as follows (Appendix G
exp(uk (4/5)
Ld
log Σ exp('% -Pn/T)
(6)
B
where the μ4, refers to the region representations associated with the prototypical region representa- tions 2k2, i.e., positive examples. The hyperparameter 7 regulates the capacity of the model to distin- guish between negative examples. The loss La encourages all region representations corresponding to the same word to be closer each other, while driving the region representations corresponding to different words farther apart, which effectively mitigates the impact of variance among different words on the similarity computation.
3.3 TEXT EMBEDDING BRANCH
Given batch B of paired regions and words, we obtain word embeddings V (ww. h (V RBBxN by utilizing the pretrained word vectors, where N is the dimension of w2. Pre-trained word embeddings typically exhibit well-structured semantic properties, where semantically related words are mapped to vectors that are close to each other in the embedding space. To enable region representations to effectively capture semantic correlations between words, we utilize a Modality Transfer Model (MTM) with three self-attention layers and three FC layers that can map the mean y output by the PAF model h into the word embedding space:
V f(u5 θy),
where the , is the parameters of the MTM model f and V E BBN represents the predicted word embeddings. The model is relation- relation--reservinng equivariant mapping that lays the foundation for constructing prototypical region representations corresponding to OOD words. Formally, for any two region representations μ4 and pz, the function f should satisfy (Appendix) B):
d θÿ), f(uy; θy)) x (p My),
where the distance metric d, captures the pairwise relations between representations within each modality. The PAE model h and MT M model f are trained jointly using the gross-modal align- ment loss function fem (Appendix and F
Con E[(1 cos(
Wi
w
))] + [E[((cos(
W/ [w]ll2 [|will22
cos(
Mi
)))
(9)
| 122
1/ms/12" ||my112
where w/, ut/ (i + and wy E V. The loss function feo enforces the predicted word embed- dings V to gradually converge toward the word embeddings V. while simultaneously ensuring that the region representations effectively capture the semantic relationships between words.
3.4 KNOWLEDGE-BASED IMAGE-TEXT MATCHING
To decide whether a given image and a text are matched or not, we first obtain a set of raw region representations R = (ri).=i,..... (R R(XM using the Faster-RCNN above and set of parsed words through tokenization operation implemented via NLTK [
as shown in Figure Then, we use the knowledge as a cross-min modal bridge to represent all the words into the corresponding prototypical region representations U {uy]j=...... (U E Rx22.. For the set of regions R, we extract region representations μ R!x by utilizing the PAE model h. Finally, we obtain the desired global similarity score s for the given image and text as:
pl p. UT)
(10)
where p(-) denotes the max-mean pooling operation, which first performs max pooling along the column dimension and then mean pooling along the row dimension of the input matrix.
However, the scope of knowledge inherently limited and heavily reliant on the volume of paired data available in public datasets. The vocabulary size supported by the pre-trained word vectors
https//www.nltk.ogg
5

---

Published as a conference paper at ICLR 2026
significantly surpasses the scale of the existing knowledge. Therefore, for OOD words relative to the knowledge, their corresponding word embeddings can typically be obtained by leveraging pre-trained word vectors. To fully utilize these OOD words, first sample paired multimodal representations {(my ) from the knowledge. Then, we calculate the similarity scores {8t { between the m word embeddings {wy ) and the word embedding Wont
{ = so [max(Wool {we) (w=))==))
(11)
By utilizing the sampled prototypical region representations {1g) base vectors and the sim- ilarity scores {= we can obtain the prototypical region representation Vout corresponding to word embedding Wout!
Vouat 2 Sq Uq-
(12)
In unpaired image-text matching, constructing prototypical region representations based on semantic similarities between words enables the effective utilization of information from OOD words.
To ensure the semantic quality of the visual prototypes constructed for OOD words, we select the top-m paired multimodal representations from the knowledge whose word embeddings are most relevant to OOD words. This selection strategy is motivated by the local linearity property of word embeddings on the semantic manifold. Semantically related words lie close each other in the embedding space and approximately reside in locally linear subspace. Consequently, the top- m neighbors provide the most informative directions for reconstructing the corresponding visual representations. Moreover, the fem constrains local alignment between the word embedding space and the visual prototype space, making nearest neighbors in the embedding space more likely to preserve geometric relationships in the prototype space, thereby reducing reconstruction bias. In this way, the top-m neighbors effectively capture the most salient semantic and structural information needed for accurate and robust prototype estimation. To obtain these top-m semantic neighbors for OOD words, we first normalize all word embeddings {un}g= in the knowledge, and denote the normalized embeddings as {
[ jk
9=1- Similarly, the normalized embedding of OOD words as d #had and ( wn |
Mont The similarity between Woot
Tw,, 1q=1 are computed as:
K
Wont [
{ Wg 1K 1q=1"
(13)
The top-m nearest neighbors visual prototype construction.
{ are then selected based on { to support subsequent
3.5
MODEL TRAINING
The studied knowledge mainly contains dadeseetindeeeedent semantic concepts, with the goal to be generally applicable to different scenarios. The semantic concepts multimodal, which includes objects and attributes images, and nouns and adjectives texts. To obtain them, we resort to publicly available dataset Visual Genome (VG) Krishna et al. (211)) P and collect corresponding words and regions. For the textual knowledge, we obtain various words from synsets in the dataset. For the visual knowledge, detect regions from images and then associate them with the words.
After collecting set of words and their semantically related image regions from publicly available datasets, we train our model on these paired data to construct MASK. The loss of the entire training process is expressed as C:
Lir + Ae + Azhoet
(14)
where A and X2 are trade-off factors for balancing different losses. By optimizing the loss L, the region representations exhibit properties of high cohesion and low coupling, indicating that representations corresponding to the same word become more compact and semantically consistent.
3.6 RE-RANKING EXTENSION
The proposed MASK a knowledge-basee approach, which differs significantly from existing data- driven models. Due to this distinction, it is expected to exhibit complementary properties when
hhtp://visualgeeomm.rr//

---

Published as a conference paper at ICLR 2026
combined with existing models. We extend MASK into re-ranking method to re-rank the initial results produced by existing multimodal models.
Taking the sub-task of image retricval an example, given text query and gallery of L images, an existing model can compute similarity scores and produce a similarity vector E R 124 By sorting the values of in descending order, the model ranks the images and identifies the top-k candidates, denoted as & E R For the text query and the top-k retrieved images, we then compute an additional similarity vector s R using MASK in an unpaired image-text matching setting. Finally, the two similarity vectors are combined using a balancing factor o::
g* = ZS(3*) +a ZS(s*),
(15)
where ZS represents the Z-Score normalization and s is the similarity vector that can be used to re-rank the top-k images to improve the rank of matched images.
4 EXPERIMENTS
4.1 DATASETS AND METRICS
We test the performance of MASK on two standard datasets: Flickr30k and MSCOCO. The com- monly used evaluation criterions are "R@1 1", "R "5" and "R@ 10", i.e., recall rates at the top-1, and 10 results. Following existing works Ge et al. (2024), we use an additional criterion of "Rs" by summing all the recall rates evaluate the overall performance. Experimental details are provided in the Appendix
4.2 UNPAIRED IMAGE-TEXT MATCHING
We design several experiments comparing state-of-the-are model-based matching methods (e.g., CHAN Pan et al. (2023), DSRLN Wu et al. (2024). CORA (hamm et al. (2024), BOOM Li al. (2024a). and 3 of Ge et al. (2024)) and knowledge-basee matching methods (c.g.. MACK Huang al. 2022) and MACK K- Huang al. (2024b)) verify the effectiveness for un- paired image-text matching. We compare the performance of unpaired image-text matching in Ta- ble [ Model-based matching and knowledge-ber baseedge-based matching exhibit comparable performance the Flickr30k dataset, whereas their performance diverges significantly on the MSCOCO dataset. Compared to Flickr30k, MSCOCO exhibits greater sample diversity, with images typically contain- ing multiple target objects and semantic regions, resulting in more complex visual structures. The knowledge-basee matching constructs explicit multimodal-aligned knowledge as a bridge between regions and words, facilitating more accurate modeling of local visual-semantic relationships in complex visual scenes.
Table 1: Performance comparison between model-based matching and knowledge- matching on the Flickr30k and MSCOCO datasets for the unpaired image-text matching.
Flickr30k datasel
MSCOCO dulaset
Mathed
RS1
Image Reiniraal
RG3
ICGTU
RB
Imagee Armotatio
KG53
Rs
RGT
Image Reineval
RG3
IGG
RE]
Imuge Amnotation
RG3
KGTT0
Rs
CHHNN
24.5
2.1
DSRLN 2004
model-fuood
maiching
BOOMam
CORA2
3.5
3.9
12.6
12.0
23.8
22.9
8.3
8.3
22.6
22.7
35.2
103.4
100.4
5.8
5.5
19.2
18.7
32.6
32.5
12.9
12.4
29.8
29.3
45.4
45.2
145.7
143.6
3.8
12.2
23.1
8.1
22.0
34.3
103.5
6.0
19.7
33.4
13.2
29.6
47.8
149.7
knowledge based
muiching
MACK
MaCK
901
3.8
3.0
9.9
15.4
10.1
24.6
32.3
95.3
7.2
25.9
40.6
21.8
46.2
60.0
201.7
205.2
7.6
safo region protatypes
MACKMO-BE
MACCK
1.7
7.0
10.9
5.3
16.2
23.4
64.5
4.6
4.6
16.9
29.2
29.6
13.3
31.4
45.3
139D
141.1
MASK
2.5
9.1
14.5
5.6
17.1
23.9
72.7
5.2
18.9
31.7
13.6
33.5
45.6
148.5
M.ACK
2.5
9.0
14.3
1.7
5.6
8.6
41.7
6.3
21.
11.3
3.0
11.6
17.6
94.1
MACKVO-K
paoling
2004
MASK
22.1
8.0
The region prototypes and max-mean pooling have a significant impact on knowledge-based un- paired matching methods. However, MASK consistently outperforms existing methods. This is
primarily because the MASK exhibits strong intra-class cohesion among region representations, ie., the variance between any region representation and the prototypical region representation is relatively small. Consequently, replacing the prototypical region representation with randomly
7

---

Published as a conference paper at ICLR 2026
selected region representation has minimal impact overall performance. Furthermore, incorpo- rating semantic relationships between word embeddings reduces the coupling among region repre- sentations across words. Therefore, substituting max-mean pooling with global mean has a minor effect on overall performance.
4.3 ZERO-SHOT IMAGE-TEXT MATCHING
To evaluate the effectiveness the MASK in complementing with pre-trained models for zero-shot image-text matching, we compare several state-of-the-an re-ranking strategies, including M ACC Huang et al. (2022). LeaPRR Qu al. (2023), M ACCK M Huang et al. (2024b) and F Wei et al. (022)) The original and re-ranked performance are compared in Table 2 We can see that although the original accuracies are already high, further improvements can be achieved by applying re-ranking strategies to these pre-trained models. Among them, MASK yields more substantial performance gains. This can be attributed to the high cohesion and low coupling of the region representations, which ensure that each region representation remains closest to its corresponding prototypical region representation while maintaining substantial spatial separation from those of non-corresponding terms. Consequently, MASK reduces the risk of region representations being misclassified in zero-shot image-text matching. These evidences demonstrate that the MASK can be well combined with existing models to further improve their performance.
Table 2: Zero-shot image-text matching by re-ranking two state-of-the-art models on the Flickr30k and MSCOCO datasets.
Flickr30k dalaset
MSCOCO datanel
Method
R38
Inigge Retrieval
R 10
R:6
Image Ammxiattin
R.005
R 10)
R5
R
Imagee Retheval
R
R 10)
R
Image Aumolation
R
R
Rs
CLIP
65.4
87.2
91.7
85.4
97.1
98.7
525.5
35.3
60.0
70.1
55.2
78.7
86.7
386.0
CLiP- - MAKKaa
66.8
88.2
92.6
86.2
97.2
98.9
529.0
36.9
61.6
71.7
55.7
79.5
87.1
392.6
CLIP- LMMga
65.9
87.9
92.8
86.2
97.3
98.1
528.2 2
36.2
61.9
70.8
55.9
80.2
85.4
390.4
CLIP MACKYG- 2004
92.8
99.0
71.8
$7.1
CLIP Ranes
66.4
$3.4
93.1
86.8
97.3
99.1
$31.1
36.7
62.1
72.4
56.2
80.3
87.8
395.5
CLIP MASK
67.3
$8.7
93.9
87.9
97.6
98.9
534.3
37.6
62.7
73.9
56.7
$0.8
$8.7
400.4
ALBEF
$9.9
K4.8
90.5
78.2
95.5
97.9
506.9
40.2
68.4
78.9
62.4
85.9
92.1
428.1
ALBEF: MACK
91.3
80.1
96.4
97.7
69.0
79.4
86.1
92.7
ALBEF LeaPRRops
61.4
35.9
91.6
79.3
96.2
97.5
511.9
40.7
69.1
79.6
62.8
86.4
91.8
430.4
ALBEF MACK
61.8
86.1
91.6
80.5
95.5
97.9
513.4
41.6
69.5
79.7
63.0
86.0
92.4
432.2
ALBEF F'ooss
91.8
96.2
97.6
ALBEF MASK
63.1
86.6
92.0
81.6
96.9
97.9
518.1
42.1
70.5
80.8
86.9
92.3
4.4 KNOWLEDGE VISUALIZATION
To qualitatively illustrate major differences between MACK and MASK, we visualize two low- dimensional word distributions in Figure 3 The words in the four numbered groups (marked by dashed lines in different colors) are about animals, transports, faces and humans, respectively. In the left distribution corresponding to MACK, there are still some related words that have remote distances. contrast, the right distribution generated by MASK semantically more compact. The underlying mechanism is that semantic relationships between word embeddings are incorporated during model training, ensuring that the corresponding prototypical region representations also ex- hibit semantic associations. These evidences indicate that the MASK can make their prototypical region representations more discriminative.
4.5 OOD WORDS ANALYSIS AND LOSS ABLATION ANALYSIS
To evaluate the impact of OOD words on image-text matching accuracy, we conduct compara- tive experiment in Table 3 We observe that image-text matching accuracy significantly improves in both image retrieval and image annotation tasks when OOD words are incorporated, and this improvement is consistently validated across different datasets. Therefore, leveraging the semantic relationships between OOD words and known words to construct corresponding prototypical region representations for OOD words is effective approach. This phenomenon can be attributed to the colation-preserviie equivariant mapping, as shown in Eq 8 As a result, region representations inherit the semantic structure encoded in the word embeddings, allowing the relationships between regions to reflect semantic distances and similarities, thereby enhancing the generalization ability of the matching process.
8

---

Published as a conference paper at ICLR 2026
MACK
(b) MASK
Figure 3: Visualization of prototypical region representations from MACK and MASK. Each word indicates its corresponding prototypical region representation embedded by t-SNE. In the word distributions, we group semantically related words by using same-colored dashed lines.
Table 3: Ablation of the overall loss and the impact of OOD words on unpaired image-text matching.
Flickr30k dataset
MSCOCO dutaset
Method
Retrieval
Image
Retrival
Insaee
R@l
RGP
R
R@
R
R
RGR
RGP
R I0
R
R
R 10
MASK
AS AS Do/taaaaa
4,5
13.6
20.5
11.8
28.5
37.3
116.2
1.2
24.3
38.2
22.2
44.6
56.4
193.
MASK
22.0
39.0
122.8
48.5
209.5
MASK
3.1
10.5
15.2
10.2
26.6
35.4
101.0
5.2
16.5
24.9
17.2
37.6
49.4
150.8
MASK Ca
2.6
8.3
13.7
9.1
23.9
34.8
92.4
4.5
13.6
20.9
13.3
31.0
40.2
123.5
To testify the contribution of each component of loss L to overall performance, we compare the performance of various losses in Table It can be observed that the performance of the overall loss function L deteriorates when certain components are omitted. Among them, the prototype consis- tency contrastive learning loss La makes the largest contribution the performance. is reasonable since the loss Cat constrains all region representations related the same word to be close to each other during model training. Additionally, the semantic relationships between word embeddings in loss f.em serve only as reference for determining the degree of separation between region representa- tions. Therefore, the loss Ld can make the prototypical region representations more discriminative, which obviously affects the accuracy of matching.
4.6 HYPERPARAMETES ANALYSIS
The proposed MASK involves two trade-off parameters a and X in Eq. [4). To evaluate the impact of different hyperparameters on matching accuracy, we design three controlled experiments in Table The results indicate that MASK achieves the best performance in unpaired image-text matching A 0.3 achieves better performance, with improvements of approximately 3.5% and 10.3% in Rs on Flickr30k and MSCOCO, respectively. These findings suggest that the cross-modal alignment loss and prototype consistency contrastive loss complementary, each contributing distinct yet essential benefits to the overall performance. By jointly optimizing these two losses in a balanced manner, we mitigate the risk overfitting to single loss term, thereby improving the model's generalization capability.
Å₁ A2. Compared to X 3.0 (i.e., A1 X2), the di
Table 4: Unpaired image-text matching by MASK using different \1/A2 on the Flickr30k and MSCOCO datasets.
Flickr30k datuset
MSCOCO dataset
A1
Image Retrieval
Image Annotation
Image Retrieval
Image Annotation
Rs
Rs
R@1
R@5
R
RØ
R@5
R 10
R@1
K@5
R 10
RØ
R@5
R 10
0.3
4.2
13.0
19.2
11.6
27.9
36.8
112.7
22.9
35.5
20.9
43.2
1.0
4.8
14.8
22.0
12.1
30.1
39.0
122.8
26.7
41.8
22.7
48.5
62.2
209.5
3.0
3.9
12.2
18.0
11.2
27.5
36.4
109.2
6.3
21.0
32.3
19.8
41.5
53.9
174.8

---

Published as a conference paper at ICLR 2026
4.7 ANALYSIS OF SAMPLING SIZE
The sampling size m for selecting paired multimodal representations critically affects the semantic quality of the constructed prototypes. To evaluate the impact of different sampling sizes on matching accuracy, we perform the experiment of unpaired image-text matching by MASK using different sampling sizes
the Flickr30k and MSCOCO datasets in Table Experimental results show that the matching accuracy follows rise-then- trend as the sampling size m increases, achieving its optimum around m 50. This phenomenon can be explained as follows. When the sampling size is too small, the constructed visual prototype relies excessively on only a few nearest neighbors. Although this preserves strong local semantic characteristics, also makes the prototype highly sensitive to noise and outliers in the word embedding space. As the sampling size increases moderate level, more semantically relevant neighbors contribute their visual information, thereby enhancing robustness and discriminability. However, when the sampling size becomes too large, semantically weak or marginal neighbors begin to dominate. Their less relevant visual cues dilute the contributions of the core semantic neighbors, ultimately reducing matching accuracy.
Table 5: Unpaired image-text matching by MASK using different sampling sizes on the Flickr30k and MSCOCO datasets.
Flickr30k dataset
MSCOCO dataset
Image Retrieval
Image Annotation
Image Retrieval
Image Annotation
Rs
Rs
R
R03
R@
R@
R@S
RØ
10
R@ I
R@3
RØ
10
R@
R@S
R 10
4.6
13.9
20.9
11.9
28.9
37.7
117.9
7.3
25.1
39.1
22.3
45.6
57.9
197.3
10
4.8
14.8
22.0
12.1
30.1
39.0
122.8
7.6
26.7
41.8
22.7
48.5
62.2
209.5
50
5.0
15.8
23.2
12.3
31.4
40.4
128.1
7.9
28.5
44.7
23.1
$1.6
66.8
222.6
100
4.9
15.3
22.6
12.2
30.7
39.7
125.4
7.7
27.6
43.3
22.9
50.0
64.5
216.0
4.8
RE-RANKING ANALYSIS
An important balancing factor is introduced for re-ranking the existing models in Eq. 15) investigate the impact a on final performance, we conduct zero-shot image-text matching experi- ments using MASK (CLIP) in 6 The results show that the best performance on both datasets obtained a = 0.15. When a is relatively small, re-ranking method yields limited improve- ments, matching performance is largely dominated by the pre-trained CLIP model. In contrast, as a increases, the influence of the MASK on matching performance becomes more pronounced. Since the knowledge is not learned from domain-specife paired image-text data, potential distribu- tional discrepancies may arise, which can adversely affect the overall performance. Therefore, an appropriate value of must be carefully chosen to achieve optimal balance between semantic alignment capability and generalization performance.
To
Table 6: Zero-shot image-text matching by MASK (CLIP) using different a on the Flickr30k and MSCOCO datasets.
Hlick330k dataset
MSCOCO
Image Retrieval
Image Annotation
Rs
Image Retrieval
Image Annotation
R@
Rw3
RØ 10
R@
R
RØ10
R@
R@s
RØ 10
R@
R
R 10
0
65.4
87.2
91.7
85.4
97.1
98.7
525.5
35.3
60.0
70.1
55.2
78.7
86.7
386.0
0.1
67.1
$8.7
93.7
87.2
97.3
99.1
533.1
37.1
62.3
73.6
56.3
80.3
88.6
398
0.15
93,9
87.9
97.6
98.9
534.3
37.6
62.7
73,9
56.7
80.8
400.4
0.2
66.9
88.5
93.6
86.4
97.1
99.0
531.5
36.9
62.2
73.4
55.6
80.5
88.3
396.9
5
CONCLUSION
This work has studied a practically important but seldom investigated problem as unpaired image- text matching. To deal with this problem, we propose multimodal aligned semantic knowledge, which leverages word embeddings as bridges to associate words with prototypes, capturing semantic relationships between words, and further utilizing information from OOD words. Additionally, the introduction of prototype consistency contrastive loss effectively mitigates the impact of variance in unpaired matching. Code is available http:///hhuuu.coommmmmmmdddeeeee
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62376138, 92367202), and in part by the Jinan-NTU Green Technology Research Institute (GreenTRI).
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Shi Chen and Qi Zhao. Divide and conquer: Answering questions with object factorization and compositional reasoning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6736-6745, 2023.
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 4125 2019.
Zhongtian Fu, Kefei Song, Luping Zhou, and Yang Yang. Noise-aware image captioning with progressively exploring mismatched words. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 12091-12099, 2024.
Yuxiao Gao, Fuwei Zhang, Zhao Zhang, Xiaoshuang Min, and Fuzhen Zhuang. Mixed-curvature multi-modal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 11699-11707, 2025.
Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 3shnet: Boosting image-sentence retrieval via visual semantic-spatial self-highlighting. Information Pro- cessing & Management, 61(4):103716, 2024.
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10323-10332, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yan Huang, Jingdong Wang, and Liang Wang. Few-shot image and sentence matching via aligned cross-modal memory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6): 2968-2983 2021.
Yan Huang, Yuming Wang, Yunan Zeng, and Liang Wang. Mack: Multimodal aligned concep- tual knowledge for unpaired image-text matching. Advances in Neural Information Processing Systems, 35:7892-7904, 2022.
Yan Huang, Yuming Wang, Yunan Zeng, Junshi Huang, Zhenhua Chai, and Liang Wang. Unpaired image-text matching via multimodal aligned conceptual knowledge. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, pp. 1-17, 2024a. doi: 10. (09/TPAMM.2024.3433
Yan Huang, Yuming Wang, Yunan Zeng, Junshi Huang, Zhenhua Chai, and Liang Wang. Unpaired image-text matching via multimodal aligned conceptual knowledge. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2024b.
José lovino. Stable banach spaces and banach space structures, Fundamentals, In Models, alge- bras, and proofs, pp. 77-95. CRC Press, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
11

---

Published as a conference paper at ICLR 2026
Kuang- -uei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. Proceedings of the European conference on computer vision (ECCV), pp. 201-216, 2018.
Paul Lerner, Olivier Ferret, and Camille Guinaudeau. Cross-modal retrieval for knowledge- -bsse visual question answering. In European Conference on Information Retrieval, pp. 421-438. Springer, 2024.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11336-11344, 2020.
Linyu Li, Zhi Jin, Yichi Zhang, Dongming Jin, Chengfeng Dou, Yuanpeng He, Xuan Zhang, and Haiyan Zhao. Towards structure-awan model for multi-modal knowledge graph completion. arXiv preprint arXiv:2505.21977 2025.
Zhe Li, Lei Zhang, Kun Zhang, Yongdong Zhang, and Zhendong Mao. Improving image-text match- ing with bidirectional consistency of cross-modal alignment. IEEE Transactions on Circuits and Systems for Video Technology. 34(7):6590-6607 2024a.
Zheng Li, Caili Guo, Xin Wang, Hao Zhang, and Yanjun Wang. Integrating listwise ranking into pairwise-based image-text retrieval. Knowledge-Basee Systems, 287:111431 2024b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang, and Yongdong Zhang. Graph structured network for image-text matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10921-10930, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Övgü Özdemir and Erdem Akagündüz. Enhancing visual question answering through question- driven image captions as prompts. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 1562-1571, 2024.
Zhengxin Pan, Fangyu Wu, and Bailing Zhang. Fine-grained image-text matching by cross-modal hard aligning network. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 19275-19284, 2023.
Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-text matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14354-14363, 2024.
Leigang Qu, Meng Liu, Wenjie Wang, Zhedong Zheng, Liqiang Nie, and Tat-Seng Chua. Learnable baslar-based re- re-ranking for image-text retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1252-1261, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
12

---

Published as a conference paper at ICLR 2026
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng.
Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
Liya Wang, Haipeng Chen, Yu Liu, and Yingda Lyu. Regular constrained multimodal fusion for image captioning. IEEE Transactions Circuits and Systems for Video Technology. 2024a.
Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Multimodal IIm enhanced cross-lingual cross cross-modal retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1296-8305 2024b.
Wenzhang Wei, Zhipeng Gui, Changguang Wu, Anqi Zhao, Dehua Peng, and Huayi Wu, Dynamic visual semantic sub-embeddings and fast re-ranking for image-text retrieval. IEEE Transactions on Multimedia, 2025.
Dongging Wu, Huihui Li, Cang Gu. Lei Guo, and Hang Liu. Dual stream relation learning network for image-text retrieval. IEEE Transactions on Multimedia, 2024.
Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang. Learning fragment self-attention embeddings for image-text matching. In Proceedings of the 27th ACM international conference on multimedia, pp. 2088-22 2096, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations, 2023, URL https:
//openreview ew net. http://oorrmmmmdmmmmmmdm
Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z Li. Context-aware attention network for image- text retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recog- nition, pp.
2020.
13

---

Published as a conference paper at ICLR 2026
A
TABLE OF NOTATION
We list the notation used in this paper in Table 7 for the convenience of reference.
Table Notation used in the paper.
Symbol
Description
Word embedding directly from pre-trained word vectors Learnable prototypical region representation (protolype) of k-tt semantic concept Raw region representation from pre-trained object delection model Faster-RCNN
Uk
rj
The mean of the latent space corresponds to the region representation
jj
The variance of the latent space
Random vector sampled from Gaussian distribution
R
{rx)n=i.... (R E R BBXM
P4
Region representation associated with the prototypical region representations Uj
V
{wk) k=ml......... (V E R BxN)
dg
Distance metries, e.g., cosine similarity
U
{eg {ugb==1..... (U E R2x2)
p())
Max-mean pooling
Wout
Word embedding Wout corresponding to OOD word Similarity score
8g
Dout
Prototypical region representation Vout corresponding to word embedding wont Top-k similarity vector from existing multimodal model Top-k similarity vector from MASK
gh
k
B PROOF: EQ. 8 FACILITATES THE CONSTRUCTION OF CORRESPONDING PROTOTYPES FOR OOD WORDS.
Given a paired multimodal knowledge {(w:; v&) k=)).==...... where wy RN and ea RM the word embedding and prototypical region representation of the k-th semantic concept, respec- tively, and K is the total number of semantic concepts. We sample from the multimodal knowledge {(w,, vx) k==.....K to obtain a subset {(wk, vw) word embedding Wout and {(WR, vn) k=))k=,..... to construct Vout such that the following equation is satisfied:
Our objective is to utilize the known
do(f(ww), [(Vow)) (ek, vout), K (k.
(16)
and f'(onut Re Wont-
(17)
We first rewrite Eq. 16 into an equidistant form. Then, there exists 3 >> 0 such that:
df(())), f(Voua) = B dokuks Vout).
(18)
Define the scaled mapping F := f. For any k out:
d. (F(w)), F(Vous) = de (VR, Vout).
(19)
That is, F' precisely preserves the euclidean distance among these prototypes. An isometric map- ping in euclidean space exhibits a well-defined structural property, which is a fundamental result in classical geometry and functional analysis. Specifically, any distance preseeving mapping defined on the euclidean space must be a rigid transformation Iovino (2021). In other words, there exists an orthogonal matrix A that represents a rotation or reflection, and translation vector I, such that the mapping can be expressed as:
F(ex) = Auk + t.
(20)
Therefore, the of can be expressed as:
((xx) BAv, + 31.
(21)
14

---

Published as a conference paper at ICLR 2026
Since A is an orthogonal matrix, its inverse A must exist and is equal to its transpose AT. There- fore, we can derive:
U = A (F(()) - t) - ( f(vx) t).
(22)
For any given wi in the knowledge, according to Eq. 17 we have:
U% = A
wk l)-
(23)
Therefore, only need to determine the values of A. and 3 to obtain the visual prototype vector Vout corresponding to the OOD words Wout- Extending Eq. 20 to the general case where A is no longer a strictly orthogonal matrix, obtain the following equation using the column orthogonality condition:
Au Avout 1 Uouell2 + A A mmnn
(24)
Here, there exists an T such that Eq. 23 holds. We do need know the values of A, t, and priori, as they can be uniquely determined from the paired knowledge {[we, 0&) j==],= K using standard similarity Procrustes decomposition. First, we construct matrix W1 = , ukl and matrix W2 [w,...., WK]. Next, compute the centroids of each set:
K
K
2k E
RM,
1
E R.".
(25)
K
K
We centralize the knowledge using the computed centroids ú and
Wi [i] 6, ....,, UK o) E RM W₂ [wo wú WK w] E NxK From here, separate out and subsequently only need consider the linear transformation and scaling. Next, we construct the covariance matrix C:
(26)
C Wi W. M
(27)
Let M > N. And then, we utilize SVD decomposition to obtain the values of A, I, and 3::
C U-V.
U R MXM E EERMXN V NNN
(28)
VT, AERMXN O(M-NN)K NJXN
IN
(29)
3 trace trace(AWi, (WW )
(30)
Irace (W2W)
== T BAū, EE R.M
(31)
Finally, we substitute the values of A, t, and B into Eq. to obtain the prototype vector Vout corresponding to the word embedding Wont-
C PROOF: THE COSINE SIMILARITY EQ. 9 IS A REASONABLE DISTANCE METRIC dg.
Euclidean space possesses strict linear structure preservation properties. the only metric sat- isfying translation and rotation invariance, and is naturally compatible with linear mappings. This allows the structural alignment problem between visual and linguistic spaces to be transformed into standard orthogonal Procrustes problem. Therefore, can equate Eq. 19 with Eq. 20 in euclidean space.
Cosine similarity and euclidean distance are not inherently equivalent. However, after vector nor- malization, there exists a strict monotonic mapping between cosine similarity and euclidean dis- tance. This implies that, cosine similarity be regarded as form of "Euclidean- distance," thereby satisfying the prerequisites for similarity transformations. Let two vectors wy and w/ their cosine similarity is defined
We w
cos( w/))
(32)
|w 1o
15

---

Published as a conference paper at ICLR 2026
where ||
denotes the L1-norm The euclidean distance is defined as:
(wk w/)) = ||w - w/6l--
(33)
We normalize each vector such that || |||||/1| ||_ yieldiinn
dr(wa, w/)² l| w/ = 1| 1| + | wowk = 2c00( w/)-
(34)
co w/)
d w/))
(35)
Combining Eq. 34 and Eq. 35 it can be concluded that after normalization, one-to-one monotonic functional relationship exists between cosine similarity and euclidean distance.
Lem JE [[1 cos
w W
))
+E [[(oos(
w Nulll2
w/ W 112
cosl
Pr
)))
(36)
1eil22 |
ll
word allggmeett
structure preserving
The loss fem comprises two components: the first item enforces alignment between predicted and pre-trained word embeddings, while the second ensures that region representations capture the struc- tural relationships among words.
0(1 cos
W
(w) "': /oo
ut
(37)
112
||wall2
|
0(1
cos
10
we
w/-
(38)
10
@[(cos(
0
w0
cos(
111
11p5l ))
w
[pella
(39)
Acoss
2(cos(
w/
vo
cos(
H4
Hy
|
1
|wlll22
|,00112
||Ms 11
w'
A(cos(
|o
ml
[10
m/
|T= ))
w/
(w) wz)ww
(40)
ww'lls
||lill
|| //u/ll
|
a[[(cos(
w/ | 10/
cos(
11 1)
|| 11
0
w
(41)
w
14 1y
cos(
cos(
|
2
11
Tuul1
Tugl12
D PROOF: THE OBJECTIVE EXISTENCE OF THE ISOMETRIC HYPOTHESIS.
Proof: For any finite set of region representations {ui there necessarily exists mapping f in an appropriate Euclidean space such that:
(f (14: θ₁), suy; θ₁)) da(pi, 15))
(42)
16

---

Published as a conference paper at ICLR 2026
Given non-zero vectors Mi..... , E R2, normalize them to obtain:
14
A ==
,.... n2
(43)
We aim to construct set of vectors {3/, 1/: in a Euclidean space as f(u)), and demonstrate that cos( )/) cos(Mi, (4j) can be achieved. Next, organize the pairwise similarities into matrix, facilitating the use of spectral decomposition to construct vectors that satisfy the given inner product relationships. We define the n x Gram matrix G as follows:
Gij == cos(at, (3)) P. Pg.
(44)
Here a real symmetric positive semi-definite (PSD) matrix, so for any vector = E R", satisfies:
= = 1>= = 0,
(45)
where the G 1. According to the properties of real symmetric PSD matrix, there exist orthogonal matrix U E R=xn and a diagonal matrix A = di 2 0) such that:
G = =UA
(46)
where rank(G) S n. Take the non-zero eigenvalue part of A as Ap E R and take the first r columns of the corresponding eigenvectors to obtain U, R"xr;
G -=U,A, :: == E R
(47)
(48)
where we write in rewtorctor form, and denote its i-th row as V. (y/s E R"): (YY Da v. yj )U
U,A,, Gij-
(49)
Therefore, we have constructed n vectors y1....., Un E R that satisfy the inner- inner-product relation V. yj Giz:
llv: V1 y1 = 1, v. y
(50)
Gij
cos 81) = |3::|] ||
cos (it, رz))
(51)
1.1
cos(yi, Fj) = cos (n) = cos (ps Mn))
(52)
By setting f(4)) = ys (i =1,,,... I......).. we obtain:
cos(f(us), f(u))) = cos(pi V Vi,j. do(J(m: θ,), Auys θy)) x dolue, 15).
(53)
(54)
E PROOF: Lem (EQ. (9)) ENCOURAGES f TO APPROACH THE ISOMETRY ASSUMPTION (EQ. (8)
Given the pre-trane trained word embeddings wa and the predicted word embeddings w = f(4;; Oy) normalize them to obtain:
we
w |
W. |wsl
(55)
pi Cam consists of Cnvrrd and Similarly, we normalize the region representations to obtain A. Te.
Lstruct (Appendix D. For a given batch size B, there exist constants e > 0 and e, > such that, when Lword cw and Lstruct I c., the following inequality holds:
|cos(ri mi/) cos(ui, M4)1 S 4 4
V M
(56)
where M denotes the number ordered pairs (i + j). and i,j E B. When ew - and e, - O, cos( cos(i/ (0²)) cos(p;, 24) admits upper bound arbitrarily close to 0, meaning that the cosine sim- ilarity between the mapped vector pairs approaches that in the original p-space, thereby achieving d,( (fi;; θy), fly; Q2)) do(pe, (t))
17

---

Published as a conference paper at ICLR 2026
We first convert the point-wise cosine error into the Euclidean difference between vectors. Accord-
ing to Appendix C we have:
|00 wùil2 2(1 - cosocos (1/ i))))
(57)
For each case where i E B, let 6, == - cos(iit, vi:z) 2 0. Then:
1100// -mill2 v /200,
(58)
Σ 6 = Liword S Ews
(59)
iEB
8, S Ex
|| 104--ppp wiil22 S 22
(60)
To quantify how point-wise alignment errors influence the pairwise cosine discrepancies, we char- acterize the variation of individual entries in the cosine similarity matrix using the norm difference. Then:
cos( (0 wi/)) cos( (wi, uty) = =l0ig- w// uss w5/1
(61)
b ri/ wú - dtgl
= (co) - w)) 10/ + 0; (10// - wر¹)l
(62)
< ||00 - will2 1 11 + 100bl22 1000 ررك 1/2
S ||00 will2 + 1 -0yl12
i u0j w; . wرن S New + /2Ew = 2V/2..
(63)
To quantify the influence of the structural loss Lstruct on the reconstruction error, we convert the structural loss into an upper bound on the per-pair error. Let the per-pair error be defined as Cij cos(ii w.) cos cos(uu, Mj). Then we obtain:
Σ cj Lstruct S Exi
(64)
max leyg <
S
Vea
(65)
(
we distribute es uniformly across all pairs, then the absolute error of each pair is bounded above by e./. /M. Then:
(w0), ()}) cos (pts 11))
(w0, (61)) cos (0y) + cos(css úh,) - cos(p:. H4)]
S Icos(10, (1,,, (3)) - cos(u,, 00,) + (cos(úr wZy) - cos(u:,
(66)
S 22 + cossui; w5)) (1) (if)) + cos(1il, w//) - cos(u:,14))
5 2e +
Ea M
Therefore, by enforcing 0 and + 0 during training, we obtain the following expression:
|cos(Wi, w/)) cos(ui, (5))) - 0.
(67)
d θy), f(y; θ₁)) da(pi, 15))-
(68)
18

---

Published as a conference paper at ICLR 2026
F PROOF: THE w' IN EQ. (9) CANNOT BE REPLACED BY W.
The w' in Eq. (9) cannot be replaced by w. Doing so would remove the "structure-preservation
supervision imposed the M model f, the second term in Eq. (9) would no longer
tribute a meaningful loss for training f. The loss function recommended in the article for preserving structure (the second term of (q)) is defined sstructi
C stract
[E[[os((
w/
cos( |usll2 ww3ll2
w/
Mj 1) 1)
(69)
Twilla
||ay|12
The loss function obtained after replacing w' with w in Catract is defined as Catruct
C struct (cos(
We |lwsl22
Wwyll cos(
wy
Mi
)) ))
P4
(70)
Muill
||mil12
Next, compare the constraint capabilities of Catruct and Extract on the MTM model (with parameters 9
F.1 FROM THE PERSPECTIVE OF THE GRADIENT OF THE LOSS FUNCTION
For the loss function C Catruct, we apply the chain rule to 9 as follows:
i cos
dL 2E(cos(
w/ ||o0/122 w|w'll2
w/
14
MJ ))
) cos(
(71)
,9,
||11l12"
||m
ae,
i cos( wi |wiill ||w3 8
Ô cos( w/: |will2" 1/ 11
① cos(
wf ||
mn |wslll
Ow!
Orl
(72)
Ow!
00
Ow'
00
As long as the parameters of influence w (i.e., vanish. Consequently, LA Letruct produces a nonzero gradient that drives the update of O,, forcing
aul * 0), the above equation generally does not de
the mapping f to bring cos( imposes constraints on the MTM
closer to cos(
He Teel 1T Therefore, Litruct directly
M
model f.
Moreover, the cos( struct
Wi wella
in CB
Sarwet is a constant. We apply the chain rule to θ as follows:
11
B
E [cos((
Wj
-cos(
14
Mi )) ||ا,)|2
(73)
| ll2
|15l122
|| Iu
constant
i Strratt
0.
(74)
CB Strrett provides no constraints gradient information on and therefore cannot compel the map- ping f to preserve input similarity in any manner. Although is affected, this influence does not propagate to O,, making impossible to fulfill the objective of structural preservation.
F.2 FROM THE PERSPECTIVE THE JOINT OPTIMIZATION OBJECTIVE
The loss function recommended in the article for word alignment (the first term of Eq) is defindd as Lword:
Wt
w
C Cword = E[(1 - cos(
))
(75)
|wsll2
|
Next, we compared the differences between the two loss functions when jointly optimized with Lwoord The joint loss function for Lword and LA is expressed follows:
LA = Lword + Struct-
(76)
Both loss terms involve θ, (through w\) and therefore jointly constrain the mapping. Specifically, Lword pulls each single-point mapping ii toward its corresponding wi, while L4
strruct enforces that
19

---

Published as a conference paper at ICLR 2026
preserve the pairwise similarity structure among samples, consistent with the semantic relationships in the p-space (prototype). The joint effect of these two losses ensures that the learned mapping f achieves both accurate pointwise alignment and global structural preservation.
The joint loss function for Lword and LB struct is expressed as follows:
L = Loword + LB strrct
(77)
It is worth noting that Lstruct does not impose any constraints on the model f. Consequently, fword is the only term that directly supervises f. and minimizing it merely pulls w toward its corresponding wi. In contrast, L. Lettuct adjusts the μ--pacee (governed by the parameters of the PAE wi; W, but provides no mechanism to model) to make cos( Tp TT approach cos( [w ||
enforce structural preservation in the mapping produced by f.
G PROOF: THE PROTOTYPE CONSISTENCY CONTRASTIVE LEARNING LOSS L ENHANCES THE DISCRIMINABILITY OF REGION REPRESENTATIONS.
The prototype consistency contrastive learning loss enhances the discriminability of region represen- tations by reducing intra-word variance and increasing inter-word separation. Intra-word variance of and inter-word separation DA.K' can be defined as follows:
Je
o
=
Jk
1x
(78)
j=1
D == 110x - 22
(79)
For any Mg:
|ag uxll 2-a12=1110 111gg- - ++ ī - 13 - = 111g -11112 + 11 ux
(11 - -. Uk)
(80)
Jn
1
ot
1 - [112 + || 2k
(81)
Jn
1=1
where ü serves the temporary mean for given batch. The intra- word variance can be decomposed into the within-batch intra-word variance and the deviation between the batch-specific word centers and the prototypes. The loss Lat is updated along the gradient direction:
Pi 15 77115 vx.)
(82)
My (1
+ (1 (1- (1) + vou.
(83)
(84)
As iterations progress, of gradually approaches zero, causing samples within the same word to cluster tightly around their prototypes and therehy enhancing the discriminability of the region rep- resentations. The loss Cat is updated by moving along the gradient direction while simultancously being repelled away from other prototypes:
P3 - n3 +1 Σ (uy v.)-
(85)
1
0/ Jk
Σ
My 1) Uk + 17 Σ (uk -.r)
(86)
j=1
kylk!
DE. (t = 111//- t/y 1|2 = || Up + n
(wk
Ui.)
Σ (we 11
(87)
At convergence, each prototype is positioned as far as possible from all others, resulting in distinct separation between different prototypes and further enhancing the discriminability the region representations.
20

---

Published as a conference paper at ICLR 2026
H FINE-TUNED DOMAIN KNOWLEDGE
To better adapt the multimodal aligned semantic knowledge to specific datasets, we can further fine- tune the prototypical region representations to obtain domain-specific knowledge. This fine-tuning step is optional and depends on the availability of unpaired data in the target dataset. Notably, the multimodal aligned semantic knowledge alone can also be applied directly and achieves strong performance. Given that annotating paired image-fext data is costly, while unpaired images and texts are typically more accessible, we adopt bidirectional region -wod cycle-consistent learning approach in an unpaired learning setting.
In particular, given a batch of unpaired images and texts, we first obtain a of raw region rep- resentations R = {r .n=1......RRR RR using the pre-trained Faster-RCNN above and a set of parsed words through tokenization operation implemented via NLTK. Then, we use the knowl- edge as cross-modal bridge to represent all the words into the corresponding prototypical region representations u {vilijt...... (u E R R)x2). For the set regions R, we extract region repre- sentations j E RIxZ by utilizing the PAF model h.
We utilize a bidirectional region-word cycle-consistent loss to learn parametric transformation matrix E 22x2 This loss incorporates two cross-modal similarity measurement processes: region-to-word (R2W) and word-to-region (W2R). In the R2W process, similarities between each word and all regions are first computed, and these similarities are then used as weights to aggregate all regions into reconstructed word representation. Conversely, in the W2R process, similarities be- tween each region and all words are computed to reconstruct regions from the word representations. The corresponding formulations
S = u. WWWW))
(88)
à = softmar (S) pW, f = softmar wW, where S E RJxT is the similarity matrix between transformed word and region representations. R contains the reconstructed word representations from region representations, and ja E R R!x2 contains the reconstructed region representations from word representations.
(89)
Each original word (or region) representation is then compared with its reconstructed counterpart to determine whether they correspond the same entity. By minimizing the cross-entropy between the predicted and ground-truth labels, we obtain self-supervised loss, Lss. which is used to optimize W:
R-2 noftmaz(uWūi) w2R softmor(pW TT.
(90)
B22 = Σ v log(ū;);
(91)
CW2R Σ
Nn T
log
(92)
12
h R22 W2R
(93)
where R2w RJJJ and WWR RIxI are two matrices including the predicted labels R2W and W2R directions, respectively. In Re2w and W22 the j-th and n-th columns are denoted as ūز and Gn. respectively. y₁ and yn are two groundtruth label vectors, whose the j-th and n-th values are ones and the rest are zeros. After the cycle consistent learning, we can use the learnable W to
transform all prototypical region representations into the fine-tuned domain knowledge, denoted as {Wwe, (x)]k==.....K where 0z = U&W E R2
I EXPERIMENTAL SETTING
I.1 DATASET
The details of experimental datasets and metrics are described follows.
We also stack multiple transformation matrices in a nonlinear way, but find it does not necessarily lead to hetter performance
21

---

Published as a conference paper at ICLR 2026
Flickr30k Young al. (2014) consists of 31,783 images collected from the Flickr website. Each image has human annotated texts. We use the public training, validation and testing splits, which contain 29.783, 1,000 and 1,000 images, respectively.
MSCOCO Lin et al. (2014) consists of 123,287 images, each of which is associated with 5 texts. We use the public training. validation and testing splits, with 113,287, 5,000 and 5,000 images, respectively.
1.2 IMPLEMENTATION DETAILS
In the multimodal aligned semantic knowledge, we collect all words from the VG dataset and filter out some special characters and rare words, resulting in a total of K=12,385 semantic concepts. For each image, we initially employ the pre-trained object detection model Faster-RCNN D to extract raw region representations, setting the number of detected regions to /=36 and the dimensionality of each region representation to M=2048. For each word, we obtain its word embedding using the pre-trained word vectors glove-840B- sove--44B--00d The batch size is 4096 for the first 200 epochs and 2048 for the next 200 epochs. The trade-off factors λι and X2 are set to 3. The sampling size is set to 10. We use the Adam to optimize the loss with a learning rate of le-4.
I.3 PRETRAINED MODELS AND WORD VECTORS
Master-RCNy a widely used deep learning model for object detection, tasked with both identify- ing and localizing objects within image. Building on earlier approaches such as R-CNN and Fast R-CNN, it introduces a Region Proposal Network (RPN) that generates object proposals directly within the model. This integration significantly improves both speed and accuracy, enabling effi- cient, real-time detection of multiple objects with high precision. We use Detectron2 as the backend to support comprehensive functions, including training, testing, and feature extraction. Additionally, we migrate the pre-traine trained Caffe-basee model from the original repository, ensuring that it extracts visual features consistent with the original model, with deviations of less than 0.01.
Glo Ve is an unsupervised learning algorithm designed to generate vector representations of words. trained aggregated global word- word co-occurrence statistics from corpus, producing embed- dings that capture meaningful semantic relationships and exhibit interpretable linear substructures within the word vector space. We obtain word embedding using the pre-trained word vectors glove- 840B-300d. It is set of pre-trained word embeddings derived from the Glo model developed by Stanford University, trained specifically on Common Crawl data. The model is built using approxi- mately 840B tokens, resulting in a vocabulary of 2.2 million words and producing 300-dimensional word vectors.
CLIP6 is cross-modal pre-trained model proposed by OpenAI, designed to learn shared semantic embedding space for images and text. trained on large-scale natural image-text pairs, mapping images and text into the same-dimensional vector space using image encoder and text encoder. Contrastive learning is employed to pull corresponding image-text pairs closer in the embedding space while pushing non-corresponding pairs farther apart.
ALBEF is a langunel pre-trained model proposed by Salesforce, designed to enhance cross-mott codsl semantic representations through "align before fuse" strategy. Its core idea is to first align image and text features in a shared space via image-text contrastive learning, and then fuse them using multimodal encoder capture richer cross-modal interactions. Additionally, AL- BEF incorporates a momentum distillation mechanism, where a continuously updated momentum model generates pscudo-labels to improve training robustness. The model demonstrates strong per- formance on tasks such as image-text retrieval, visual question answering, and natural language visual reasoning, making a key approach in the vision assion-language pretraining field.
pytorch
22

---

Published as a conference paper at ICLR 2026
1.4 MODEL DETAILS
In this section, present the architectures of the models involved in the proposed MASK frame- work, as shown Table The [B M, 2048] indicates that batch contains B images, each image divided into M regions, and each region has feature dimension of 2048. In experiments, set B = 128 and M = 36. It is worth noting that the model architectures are not fixed. In later sections, will discuss how model size impacts the accuracy of image-text matching.
Table Overview the model architectures integrated into the MASK framework.
Model
Layer
Input
Output
Params
FC
[B, M. 2048
[B, M, 512
1.55M
ReLU
B, M, 512
B. M, 512
Self-Attention
B. M12
B, M, 512
1.5MM
PAE
Self-Attention
B. M, 512
[B, M, 512
1.0SM
Self-Attention
B. M152
B. 5122
1.05M
Normalization
B, M, 512]
B, M,,21
1024
FC
B. 512)
[B, M, 512
0.26M
ReLU
B. 512
B. M,512 512
FRM
Self-Attention
B. 512
B.M. M,521 512 [B, M, 2048
1.05M
FC
B, M,512
1.05M
FC
B. 612
B. MM 300
0.15M
ReLU
B. M, 300
B. M. 300
Self-Attention
B. M, 300
B. M. 300
0.37M
Self-Attention
B. M, M,300
B. M. 300
0.37M
M
FC
B, M, 300
B. M. 300
0.09M
ReLU
B, 300 300
B, M, 300
Self-Attention
B, 3,0 300
B, M. 300
0.37M
FC
B. M,300
[1, M, 300]
0.09M
Total
8.1M
ADDITIONAL EXPERIMENTS
J.1 CROSS-DATASET IMAGE-TEXT MATCHING
Our proposed MASK can also enhance the generalization of conventional image-text matching mod- els when applied unseen datasets. Specifically, we try to re-rank conventional image-text matching models for the task of cross- image-text matching. The experimental setup is as follows: (1) two representative image-text matching models (i.e., VSRN Radford al. (202T) and SAEM Wu et al. (2019) are trained on source dataset (e.g., Flickr30k or MSCOCO), (2) these models are then evaluated on different target dataset (c.g., MSCOCO or Flickr30k), and (3) using the pro- posed MASK to re-rank these models the target dataset. It is important to note that neither the re-ranking methods nor the base models are trained on the target dataset.
Table 9; Cross-dataset image-text matching by re-ranking existing models on the Flickr30k and MSCOCO Datasets.
MSCOCO
PlickrADk
Flickrikk
MSCOCO
Method
Image Retneval
Image Annotation
Image Retneval
Annotation
VSRN
42.3
69.3
78.1
$3.1
79.5
37.5
409.9
14.0)
31.7
42.2
2/1.4
40.0
50.0
198.4
M.CK
VSNNN LeaPRRaas
42.9
69.6
79.8
79.3
414.9
16.2
43.8
20.9
51.4
VSRN M MACKNG-M
44.8
71.3
79.5
56.7
81.7
39.0
422.9
16.5
35.8
45.0
23.3
43.3
$2.0
215.9
VSRN FR
73.4
80.1
56.8
83.3
89.1
429.1
16.7
37.1
44.4
24.2
44.5
56.8
223.7
SAEM
21.4
70.2
33.4
5019
80.6
215.3
14.3
15.0
45.0
23.2
45.4
37.4
219.
8AEM ACNos
41.8
70.7
80.0
54.2
81.2
39.9
417.9
15.4
34.9
45.9
23,6
46.0
57.7
223,4
SARM LRR
70.9
23.8
SAEM MACK V0-10 2004
43.4
71.7
80.7
58.8
82.1
427,0
17.4
37.8
47.4
26.6
49.1
58.7
237.0
SAEM FRazzn
43.7
72.4
$1.8
59.6
84.2
90.6
432.3
17.9
39.8
47.1
26.8
51.3
60.2
243.1
SAEM MASK
45.1
72.6
$2.0
86.9
91.9
440.1
19,1
40.5
47.8
27.6
53.9
62.3
251,2
23

---

Published as a conference paper at ICLR 2026
In Table 9 the results of two kinds of cross-dataset image-text matching are both presented, which are explained as follows. MSCOCO - Flickr30k: training existing models on the MSCOCO dataset and testing them the Flickr30k dataset. Flickr30k + MSCOCO: training existing models on the Flickr30k dataset and testing them on the MSCOCO dataset. We observe that applying MASK to re-rank the outputs of VSRN and SAEM consistently enhances their generalization performance on unseen datasets, with substantial relative improvements observed across both sub-tasks. For example, V SRN + M ASK performs much better than VSRN by 5.0% and 4.8% in R I on the MSCOCO + Flickr30k task, and by 3.6% and 6.3% in @@ on the Flickr30k + MSCOCO task. Compared with VSRN FFR, the relative improvements are also large, i.e., 7.2% ~ 8.7% and 7.8% 8. 1 in Rs when re-ranking VSRN and SAEM, respectively. Comparing the results in Table D with those in 2 we observe that the relative performance gains are more substantial for VSRN and SAEM, primarily because CLIP and ALBEF exhibit much higher baseline performance. In other words, re-ranking yields larger improvements when applied to less accurate models.
J.2 VISUALIZATION OF POSITIVE AND NEGATIVE EXAMPLES FOR IMAGE RETRIEVAL AND IMAGE ANNOTATION
To better understand the OOD words, we show some representative examples of retrieved images or texts based on text image queries by CLIP in Table TO These examples are selected according to the following criteria: 1) Positive examples are those in which the ground-truth matched image or text is not ranked top-1 by CLIP but is successfully promoted to higher rank by MASK through re-ranking, and 2) Negative examples refer to situations where the ground truth image or text is initially ranked top-1 by CLIP but is pushed the positive examples, it seems that the CLIP cannot well understand the semantic concepts such as "glasses", "pierced", and "broken". For instance, in the top-1 retrieved image for the first text query, there are no clear clues indicating "pierced" or "glasses", yet its rank is still higher than that of the ground-truth one. Similarly, in the top-1 retrieved text for the second image query, the annotation contains the word "glasses", even though the image itself does not include such information. While our MASK especially focuses on understanding these semantic concepts and can thus increase the corresponding similarity scores between the matched regions and words. However, MASK may also make incorrect decisions. For example, in the images retrieved for the fourth text query, MASK reduces the rank of the ground-truth image. This behavior can be attributed to the presence of adverbs (c.g.. "very", "quite"), adjectives (e.g.. "large", "excited"), and pronouns (c.g., "they", "this") in the text. Our further analysis indicates that the OOD words negatively affecting MASK are typically those that cannot correspond to specific visual regions. Attempting to construct region prototypes for such words introduces substantial semantic noise. Finally, it is important to note that negative examples contain not only non-visual adjectives, adverbs, and pronouns, but also some informative OOD words with tense or plural variations. Consequently, the final image-text matching accuracy is influenced by the combined effects of all these words.
lower position after re-ranking by MASK. In
J.3 DIFFERENT DETECTOR COMPARISON
This work derives region representations using the Bottom-Up and Tow-Down (BUTD) model, i.e., FaNter-RCNN, which consists of a Region Proposal Network (RPN) Ren et al] (2015) and a 101- layer Residual Network (ResNet101) He et al. (2016) pretrained on the VG dataset. Since our constructed multimodal knowledge is also based on the VG dataset, this pretraining step is crucial for learning discriminative region representations and achieving strong performance. To validate this, we experiment with three alternative detectors within the proposed MASK framework. The first is DETR Carion et al. (2020). a recently popular Transformer-bise based detector. The second is DINOZZangg et al. (2023), evaluated in two versions: the original model and variant pretrained on the VG dataset in the same manner as BUTD. The third is an enhanced version of BUTD, referred to as BUTD+, which employs 55s ConvNeXt Liu et al. (2022). and Swin Transformer Liu ctt al. (202I) as the backbone networks to replace ResNet101.
We evaluate these detectors on the task of unpaired image-text matching and compare their perfor- mance on the Flickr30k dataset in Table IT The results show that directly using either DETR or DINO leads to poor performance. This is primarily because they fail to extract region representa- tions as accurately as BUTD when constructing multimodal knowledge. Specifically, BUTD uses Faster R-CNN as its backbone, a two-stage object detector that allows ground-truth bounding boxes
24

---

Published as a conference paper at ICLR 2026
Table 10: Examples of retrieved top-3 images/texts based on text/image queries by CLIP. Groundtruth matched images are marked by black bounding boxes, which can be re-ranked higher by the proposed MASK. Similarly, the groundtruth is annotated using the symbol "(GT)". The underlined words indicate the representative OOD words relative to the multimodal knowledge.
Query troskmage
up- iragrabeais
spp- iregettrac by
The mas with pienced workin glanes
drew
aitiin is benkan sroce
yellow chats
standing
coepti sitting sand with Uhite 15n is the wake:
Nea
teatidd ins vhiee Gi and Mack bek is pinning Gi
performing disamardd lick bett hicking
brokk a hean beld
young
the panson in heiding
Bos
break boord held
Aommard kick
young
bot is kicking pal
the person
hoiding
back bel: piesing
whlte and
man in
(Gr)
(GT)
gloses
pensen has botiiing os rocentaia (CT)
mocustileeer mounin with blue belued ou.
parsss has bobling
girt werning plasen while malk rilihiing
mrountohseer mounin with like behoet on.
liee whill nade climbing
on monurann (GT)
young buy qulte excital in itrras
A Addiid aearing thaz jumping
A buy weaning jeans lxage in
A boy sicry join leage in air
A chiid weatdiwaarigg A yourte bay quiss shin jumping
Nog
eactted the managaamms.....
diere more von The trampolise noz (GT)
beanely
back meralls and not ever
Neg
badk immppline sea
young
This fomely down mas jah. (GT)
mersilt and nut sher halding: chalmase
halding chalmmate.
Table 11: Unpaired image-text matching using different detectors on the Flickr30k dataset.
Image Retrieval
Image Annotation
Detector
Architecture
Pretrained
Boxes
R
R@S
R 10
RØ1
R@3
R 10
DETR
ResNet50+Transforrs
No
36
0.5
3.6
1.5
6.9
19.4
DINO
DesNet5O+Transfomee
No
36
0.8
2.7
4.2
1.9
4.6
7.4
21.6
DINO
Yes
36
3.1
8.3
12.5
6.4
19.2
28.9
78.4
BUTD
RPN + ResNet101
Yes
36
4.8
22.0
12.1
39.0
122.8
BUTD+
RPN 55
Yes
36
5.3
16.7
24.2
11.9
34.6
43.4
136.1
BUTD+
RPN ResNet152
Yes
100
5.2
15.2
24.4
16.6
39.0
47.3
147.7
BUTD+
RPN ConvNeXt
Yes
36
5.7
18.0
26.1
12.9
37.4
46.9
147.0
BUTD+
RPN Swin
Yes
36
5.9
18.7
27.1
13.3
38.7
48.6
152.3
to be directly provided, yielding highly accurate feature representations. In contrast, DETR and DINO require generating hundreds candidate bounding boxes and then selecting the one with the highest Intersection
Union (IoU) for each ground-truth box. This additional step inevitably introduces noise, thereby degrading the quality of the constructed knowledge and resulting in lower performance. In addition inaccuracies in bounding box generation, the superior performance of
25

---

Published as a conference paper at ICLR 2026
BUTD may also be attributed to its more powerful feature extraction network. To investigate this, we replace the ResNet101 backbone with ResNet152, ConvNeXt, and Swin Transformer, result- ing in enhanced version called BUTD+. This modification leads to substantial performance improvement, demonstrating the benefit of stronger feature extraction.
J.4 MODEL. ARCHITECTURE ANALYSIS
This work acquires multimodal aligned knowledge based on three models: PAE, FRM, and MTM. Since the architecture of these models can significantly influence image-text matching accuracy, we use different architectures to perform the experiment of unpaired image-text matching and compare their performance on the Flickr30k dataset in Table 12
We can see that increasing the number of fully connected (FC) layers leads a decline in image-text matching performance, whereas adding more self-attention layers results in performance gains. For example, in the PAE model, increasing the number of fully connected layers from to results in performance drop of approximately 19.4% in Rs on the Flickr30k dataset. When the number of layers is further increased from 3 to 5, accuracy decreases by an additional 11.3%. These findings demonstrate that FC layers have substantial impact model performance, and excessive depth severely degrade accuracy. This can be explained as that FC layers apply fixed nonlinear transfor- mations, where even minor training errors accumulate the network deepens, gradually distorting the spatial distribution of features. In contrast, self-attention layers explicitly model similarity re- lationships among entities, inherently preserving or even reinforeing the semantic structure of the representations.
Table 12: Unpaired image-text matching using different model architectures on the Flickr30k dataset.
Image Retrieval
Image Annotation
Model
Layer
Params
R
R@s
R&T0
R
R@5
R TO
Rs
Self-Attention layer
2.11M
13.4
11.2
26.6
35.4
110.6
I FC Self-Artention layers
4.21M
4.8
14.8
22.0
12.1
30.1
39.0
122.8
PAE
I FC 5 Self-Atteation layers
6.31M
4.6
15.3
26.3
12.3
28.5
41.7
128.7
Self-Attention layers
4.73M
2.6
11.8
10.3
26.1
36.2
103.4
5 FC Self-Attention layers
5.25M
9.3
12.6
9.6
24.8
33.9
92.1
FC + Self-Artention layer
2.1M
4.2
13.9
21.2
11.7
29.6
37.5
118.1
FC Self-Attention layer
2.36M
4.8
14.8
22.0
12.1
30.1
39.0
122.8
FRM
3 FC Self belf-Attention layer
2.62M
3.7
12.6
20.2
10.8
28.7
36.1
112.1
2 FC Self-Attention layers
4.46M
4.9
12.8
24.5
13.3
32.6
38.2
126.3
2 Sclf-Attention layers
6.56M
4.6
24.2
13.9
33.8
38.9
130.7
I FC Self-Arteation layers
1.26M
3,9
12.4
18.8
10.0
26.1
34.5
105.7
Self-Attention layers
1.44M
4.8
14.8
22.0
12.1
30.1
39.0
122.8
MTM
Self-Attention layers
L62M
2.7
11.3
9.2
23.5
32.4
95.2
3 FC + Self-Attention layer
0.7M
4.2
13.6
20.4
11.7
28.2
37.5
115.6
3 FC 5 Self-Attention layers
2.18M
5.1
13.9
22.7
12.8
32.4
41.3
128.2
J.5 DIFFERENT PRETRAINED WORD VECTORS COMPARISON
The semantic geometry of word embeddings directly influences the construction of OOD proto- types. To evaluate the impact of different pretrained word vectors matching accuracy, per- form the experiment of unpaired image-text matching by MASK using different word vectors on the Flickr30k and MSCOCO datasets Table Experimental results show that GloVe consistently outperforms Word2 Vec and FastText across both datasets, demonstrating its superior suitability for constructing OOD prototypes. Specifically, GloVe achieves 122.8% @s on Flickr30k and 209.5% R@s MSCOCO, surpassing Word2Vec by 2.7 ~ 5.2% and FastText by 8.4 ~ 14.4%. These performance differences can attributed to the distinct characteristics the word vectors. GloVe encodes global word-word co-occurrence statistics, enabling it to capture broader contextual relat- edness. Such global semantic structure is crucial cross-moud cross-modal matching, where visual regions and textual words need to align through high-level associative semantics rather than strict synonymy. In contrast, Vord2Vec, which learns from local context windows, excels at modeling fine- fine-graine grained synonymy but is less capable of capturing the broader semantic relations required for cross-modal
26

---

Published as a conference paper at ICLR 2026
alignment. FastTTxx places greater emphasis morphological similarity. However, morphologi- cal similarity does not necessarily imply semantic similarity, which introduces significant noise and ultimately reduces matching accuracy.
Table 13: Unpaired image-text matching by MASK using different pretrained word vectors on the Flickr30k and MSCOCO datasets.
Flickr30k
MSCOCO dalaset
Word vectors
Image Reineval
Image
Anmotation
Image Retrieval
Image Annotation
Rs
RØ
Re3
30.1
22.7
209.5
Word2Vee FastText
4.6
14.0
21.3
11.3
28.6
37.8
117.6
7.4
26.2
41.5
22.2
47.8
61.7
206.8
26.1
J.6 MODEL EFFICIENCY AND SIZE COMPARISON
As last, compare the testing time and model size of our proposed MASK with those of CLIP, AL- BEF, and MACK on the same machine (Intel(R) Xeon(R) Platinum 8468V, 512 GB RAM memory, and NVIDIA L40). based on their publicly available codes in Table 14 To eliminate the influence of other factors, such as the matching framework and feature extraction, we fix the batch size to for models during testing. the table, the testing time measured by seconds (s), and the model size is measured by the millions of model parameters (M).
From the table, we observe that MACK and MASK have significantly smaller model sizes compared to CLIP and ALBEF. The parameters of MACK come from the BUTD module used for extracting region representations. While the knowledge-band no additional parameters. Similarly, for MASK. the majority of the testing time is attributed to the inference processes of the BUTD and PAE models, which considerably faster than the testing time required by CLIP and ALBEF.
image-text matching is lightweight and requires
Table 14: Model Efficiency and Size Comparison.
Detector
Testing Times (S)
Model Size (M)
CLIP 0.08
291.0
ALBEF
0.29
209.5
MACK
0.03
42.5
MASK
0.04
50.6
J.7 HYPERPARAMETER ANALYSIS EXTENSION
To further evaluate the impact of different hyperparameters A and de matching accuracy, different values hyperparameters to perform the experiment of unpaired image-text matching and compare their performance the Flickr30k and MSCOCO datasets in 15 The results indicate that MASK achieves the best performance in unpaired image-text matching when A = A = 3. This further indicates that the cross-modal alignment loss and the prototype consistency contrastive loss are complementary. By jointly optimizing the entire loss function in a balanced manner, the model's generalization capability can be substantially improved.
K LIMITION AND FUTURE WORK
is important to acknowledge certain limitations of the proposed MASK, which will be addressed in future work. First, the raw region representations are extracted using the pre-trained object detection model BUTD. It would be better to pretrain more advanced detectors on the VG dataset to provide more discriminative region presentations. Second, relying solely nouns for unpaired image-text matching is suboptimal. It would be better to take all the other words into consideration for more accurate tmate-text matching.
27

---

Published as a conference paper at ICLR 2026
Table 15: Unpaired image-text matching by MASK using different X and X2 on the Flickr30k and MSCOCO datasets.
Flickr30k datasel
MSCOCO dalusel
Hyperparameter
Image
Image
Image
Image
RG
RB3
R810 10
RE1
RC83
RB 10
R@1
R33
RC 10
RE
R88
RG10
Rs
1.0
4.2
13.0
19.2
11.6
27.9
36.8
112.7
688
22.9
35.5
20.9
43.2
55.8
185.]
3.0
4.8
14.8
22.0
12.1
30.1
39.0
122.8
7.6
26.7
41.8
12.7
48.5
62.2
209.5
3.1
10.2
16.6
11.2
25.5
34.3
100.9
5.2
17.8
28.3
19.8
37.1
48.3
156.5
1.0
3.9
12.2
18.0
11.2
27.5
36.4
109.2
6.3
21.0
32.3
19.8
41.5
53.9
174.8
3.0
4.8
14.8
22.0
12.1
30.1
39.0
122.8
7.6
26.7
41.8
22.7
48.5
62.2
209.5
165.4
L THE USE OF LARGE LANGUAGE MODELS
We use LLMs solely to assist in checking grammatical correctness. After the initial check by the model, we further refine and correct any remaining grammatical issues manually. Therefore, the role of LLMs in this work is limited.
28


Reviews:
[{"review_id": "RlV6mxiFm9", "reviewer": "Reviewer_PJM3", "summary": "The authors propose MASK to address the issue of unpaired ITM by focusing on OOD words. \n(1) They use relationship among word embeddings to construct prototypes of OOD words. \n(2) A new consistency contrastive loss is proposed to make compact prototypes. \n(3) Pretrained word embeddings are used for relation-preserving equivalent mapping. \nAbound experiments are performed to illustrate the effectiveness and efficiency of MASK.", "questions": "[1] In Table 7，it seems that the table title should NOT be “detectors” but “architecture”. Otherwise, it will be the same as the title of Table 9. \n※ p4 ln210: Lcl is defined as follows D: WHAT is D? Appendix D? \n※ p5 ln233: f should satisfy B: WHAT is B? B!=8? You mean Eq.(8)? \n\n[2] It will be better if the authors give an answer about Fig. 2 that \nWHETHER the (1) Pretrained F-RCNN and (2) Pretraind word vectors \nwill BP (which means: requires_grad == True)? \n※ I guess BOTH of them do NOT need grad to BP. \n※ Is there MORE module that also NOT need BP? F-RCNN? \n\n[3] OOD word/vocab is relative, NOT absolute. Because Glove (pretrained word emb) is likely to know so called “OOD word”. So, “OOD” is relative to CLIP/ALBEF (pretrained VLM) or F30k/COCO (dataset). \n※ In addition, in Fig. 2, “cat” seems to be a quite common seen word, NOT OOD. \n“otter[≈≈freshwater carnivorous mammal]/manga[≈≈comic]” may be more suitable. \nSo, a more SOUND example should be given, especially in REAL TEST VISUALIZATION. \nPlease give us some REAL test examples about your MASK with REAL OOD word. \ne.g. in T2I retrieval, a text query with OOD word makes original CLIP model search the GT image on Top 3, but after your MASK rerank, the GT image gets up to Top 1. \nYou can make VISUALIZATION about the Pos as well as Neg on I2T and T2I (F30k). \n\n[4] What about the Neg (Wrong) example visualization? Is there any OOD word? \nHow to EXPLAIN it? WHY it makes wrong? What TYPE of word it is? verb/prep or n./adj.？\n※ We know that, verb(v.), preposition(prep.) are harder than noun(n.), adjective(adj.). \nIs there any method to make v./prep. understanding more precise? \n※ How to make these MISTAKES to be alleviated? MORE effective methods? \n※ We AGGRE WITH your work, but we also wonder HOW it is WORKED indeed ! \n\n[5] Big Models (Large Language Models) are popular, and may have super high score now. \nWe want to know how many scores your MASK method can improve? \n※ Just Eval, DO NOT need any finetune/train! WHY NOT Keep pace with the times! \n※ SigLIP v1 can get 570.84 score on f30k 1K Test. We have already tested! \nsee: https://huggingface.co/google/siglip-so400m-patch14-384/tree/main\n※ Since Big Model can get 570+ score, CAN Big Model+MASK get 600 score (full score)? \n\n[6] More powerful region encoder, may lead to more score! \n※ F-RCNN ResNet 152 is more powerful than F-RCNN ResNet101. \n※ More SOTA image/region encoder for More Ablation studies? Swin Transformers? \n\n[7] What if w’ in the second term of Eq.(9) change into w? \n※ Because we think w is GT, but w’ not. Maybe GT is better. \n\n[8] p5 ln261: we first sample m paired ...\n※ How to sample? I don’t understand. In Appendix? \n※ I guess: using GloVe(=pre-trained word embeddings) to get Top m high score word.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] MASK surpasses MACK[NeurIPS 22] and MACK++[TPAMI 24], which are the 1st work for unpaired ITM. So, MASK is the new SoTA work. \n\n[2] MASK made several smart designs(contributions): (1) Information retention loss: a self-supervised noise-adding objective, in which, \nFRM module will ONLY be used in knowledge construction phase, but NOT in infer phase. (2) External Textual knowledge(=GloVe) is used, which can be considered as auxiliary GT info, especially in Eq.(9) with the help of MTM module. (3) The semantic relationship between(among) words are captured(used/exploited) for enhancing generalization, which is established in Eq.(9) and Eq.(12).", "weaknesses": "[1] All of the weaknesses, please see the following Questions Part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y845fgheif", "reviewer": "Reviewer_VshV", "summary": "The paper proposes MASK (Multimodal Aligned Semantic Knowledge), a framework for unpaired image-text matching that leverages external semantic knowledge to bridge the gap between textual and visual modalities.\nInstead of relying on paired image-text data, the method constructs visual prototypes and uses pretrained word embeddings to generate prototypes for out-of-distribution (OOD) words through a weighted combination of existing ones.\nA prototype-consistency contrastive loss is further introduced to reduce intra-class variance and enhance alignment stability between textual and visual representations.\nExtensive experiments on Flickr30k and MSCOCO demonstrate competitive retrieval performance and validate the benefit of incorporating external knowledge.\nThe approach can also serve as a lightweight re-ranking module to enhance existing vision-language models.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel cross-modal semantic alignment method, MASK, which constructs representative prototypes for OOD words by exploiting the intrinsic relationships amongword embeddings.\n\n2. The paper provides solid comparisons with both model-based and knowledge-based baselines on Flickr30k and MSCOCO, together with ablation studies, sensitivity analysis, and visualization of learned representations.\n3. The proposed method can serve as a lightweight plug-in re-ranking module, enhancing existing vision-language models such as CLIP or ALBEF without retraining them end-to-end.\n\n4. This paper introduce a prototype consistency contrastive learning loss to structurally regularize the feature space, mitigating the adverse impact of distributional variance.", "weaknesses": "1. Over-strong assumption and circular reasoning in the OOD prototype proof\nThe theoretical analysis in Appendix B relies on a strong linear/isometric assumption that equates textual and visual prototype spaces. Moreover, the proof assumes the existence of a relation-preserving mapping \nf,which is exactly what the model is trained to learn—thus creating a circular argument. Additional experiments or a more rigorous justification are needed to support this claim.\n\n2. High dependence on region proposals\nThe method heavily relies on the quality of the region proposals provided by upstream object detectors. The experiments show large performance gaps between different detectors (e.g., BUTD vs. DETR/DINO), indicating that the framework’s robustness and generalization depend strongly on detector performance.\n\n3. Insufficient analysis of pretrained word embeddings\nThe approach uses pretrained word vectors to construct OOD prototypes, but the effect of different embeddings (e.g., GloVe, word2vec, fastText) is not explored.  Since the semantic geometry of word embeddings directly affects OOD prototype generation, additional ablation or sensitivity analysis would strengthen the paper.\n\n4. Missing discussion on the parameter m for OOD prototype construction.\nIn Eq.(8), OOD prototypes are computed by weighting m known prototypes, but the paper does not explain how m is chosen or provide related experiments. The selection of m could significantly affect the semantic quality of constructed prototypes.\nIncluding an analysis or sensitivity study on this parameter would help clarify its impact and strengthen the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eerKa14ehs", "reviewer": "Reviewer_EZrZ", "summary": "This paper addresses an underexplored issue in cross-modal matching tasks: image-text matching without pairing. The authors propose a new method called MASK, whose core is to construct a cross-modal aligned semantic knowledge base. The model uses word embeddings as a bridge to match words with their prototypes, and by introducing prototype consistency contrastive learning loss to regulate the feature space, it overcomes the shortcomings of existing methods in handling out-of-distribution words and variance of visual representation distribution. Experiments on the Flickr30K and MSCOCO datasets show that MASK achieves leading performance in the unmatched matching task and can effectively serve as a re-ranking module to enhance the performance of existing pre-trained multimodal models.", "questions": "1.\tDoes the MASK method have good generalization ability on other types of datasets? For instance, how does the model perform on some datasets that contain more domain-specific words or more complex visual scenarios? Is there a plan to verify it on more diverse datasets?\n2.\tIf certain words are not included in the pre-trained word vectors, or if the semantics of the words have changed in a specific domain, how will the MASK method handle this situation? Has there been any consideration of reducing reliance on the pre-trained word vectors or introducing other methods to obtain the semantic information of the words?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe framework proposed in this paper is intuitive and effective. It utilizes the semantic relationships of word embeddings to construct prototypes for out-of-distribution (OOD) vocabulary and a prototype consistency contrastive learning loss, clearly pointing out the key weaknesses of existing methods and having a clear motivation.\n2.\tThis paper introduces external pre-trained word vectors as an auxiliary supervisory signal to establish an equivariant mapping that preserves the relationship between regional representations and word embeddings. This enables regional representations to effectively capture the semantic relationships between words.\n3.\tThe experimental results on Flickr30K and MSCOCO show that MASK outperforms existing models and knowledge-based methods significantly in the image-text matching task, without paired images, proving its effectiveness.\n4.\tMASK can acquire a kind of logical semantic knowledge based on conceptual prototypes through structured learning. Uses MASK as an reordering enhancement plugin for CLIP and ALBEF models, demonstrating its effectiveness in prototype consistency contrastive learning and semantic relationship alignment.\n5.\tThe appendix provides mathematical proofs for concepts such as peer-to-peer transformation mapping and the rationality of loss functions, enhancing the rigor and depth of the method. This is particularly valuable in applied research papers.", "weaknesses": "1.\tThe explanation of the OOD vocabulary processing mechanism is unclear. This is one of the core innovations of this paper. However, the description in Section 3.4 is overly mathematical and lacks an intuitive, step-by-step concrete example. For instance, when encountering a word that does not exist in the knowledge base, how can we use the Glove word vectors to find similar known words and weight them to synthesize their visual prototypes? The current explanation remains at the formula level, which is less readable and reduces the comprehensibility of the method.\n2.\tTo construct the prototype of the OOD vocabulary, we need to sample m pairs of representations from the knowledge base. The paper does not explain at all how this sampling is carried out. Is it random sampling? Or is it Top-K sampling based on the semantic similarity with the OOD words? Different sampling strategies will greatly affect the quality of the final prototype, which is an important unspecified hyperparameter and must be clarified.\n3.\tThe paper did not analyze the overlap between the vocabulary in the test set and the VG's 12,385 concepts. If the vast majority of nouns in the test set are already present in the knowledge base, then the challenge and performance improvement attributed to OOD words might be overestimated. It is necessary to clearly define the specific definition and proportion of OOD in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)