Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval
Authors: Hani Alomari, Anushka Sivakumar, Andrew Zhang, Chris Thomas
Accepted at: acl25

Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval

Hani Alomari
Virginia Tech
hani@vt.edu

Anushka Sivakumar
Virginia Tech
anushkas01@vt.edu

Andrew Zhang
Virginia Tech
azhang42@vt.edu

Chris Thomas
Virginia Tech
chris@cs.vt.edu

Abstract

Cross-modal image-text retrieval is challenging because of the diverse possible associations between content from different modalities. Traditional methods learn a single-vector embedding to represent semantics of each sample, but struggle to capture nuanced and diverse relationships that can exist across modalities. Set-based approaches, which represent each sample with multiple embeddings, offer a promising alternative, as they can capture richer and more diverse relationships. In this paper, we show that, despite their promise, these set-based representations continue to face issues including sparse supervision and set collapse, which limits their effectiveness. To address these challenges, we propose Maximal Pair Assignment Similarity to optimize one-to-one matching between embedding sets which preserve semantic diversity within the set. We also introduce two loss functions to further enhance the representations: Global Discriminative Loss to enhance distinction among embeddings, and Intra-Set Divergence Loss to prevent collapse within each set. Our method achieves state-of-the-art performance on MS-COCO and Flickr30k without relying on external data.

1 Introduction

Cross-modal retrieval methods aim to align different modalities, such as images and text, by learning shared semantic representations (Frome et al., 2013). The challenge lies in bridging the semantic gap between modalities: each modality contributes unique information, and aligning them requires identifying which features to preserve or discard. Moreover, the relationship between modalities is often one-to-many (e.g., a single image can be described by multiple captions, each emphasizing different aspects). Handling this complexity requires learning robust embeddings that capture the diverse semantic relationships that exist across modalities (Song and Soleymani, 2019; Chun et al., 2021).

![A diagram showing an image with multiple colored regions corresponding to different captions, and a shared embedding space with clusters of visual and textual elements.](page_1016_370_388_312.png)

Figure 1: We illustrate how the same image can be paired with multiple captions. Color-coded regions link visual elements to corresponding textual descriptions. Our method produces multiple embeddings for each sample (image or caption) in a shared space where similar visual and textual elements cluster (indicated by colored groups and dashed boundaries). Icons on embeddings illustrate the model’s capacity to map multiple image regions to relevant text descriptions.

Most existing approaches rely on learning a single shared embedding space for images and text (Faghri et al., 2018; Fu et al., 2023; Pham et al., 2024) performing well on benchmarks like Flickr30k and COCO (Plummer et al., 2015; Lin et al., 2014) but struggling with abstract or nuanced cross-modal relationships. To address this, some methods use cross-attention networks that predict similarity between images and text by attending to both modalities simultaneously (Wei et al., 2020b; Diao et al., 2021; Lee et al., 2018; Miech et al., 2021), yet are computationally intensive, requiring joint processing of image-text pairs for each query, which limits their scalability. A common approach is to use separate visual and textual encoders, which allow for efficient search on precomputed embeddings (Gu et al., 2018; Messina et al., 2021). This method scales well to large datasets but often fails
to capture the diversity of cross-modal relationships because each input is reduced to a single embedding vector limiting the representation’s ability to handle one-to-many relationships between queries and potential matches.

A few recent works have proposed architectures that aim to capture these diverse relationships using separate encoders (Song and Soleymani, 2019; Chun, 2024; Kim et al., 2023). The goal is to reduce ambiguity and improve alignment between modalities by representing multiple facets of each input with a set of embedding vectors. Figure 1 shows how these embedding sets cluster similar features from both modalities in a shared space.

While set-based embedding models offer a promising solution for capturing diverse cross-modal relationships, the loss and similarity functions to train these methods lead to issues like sparse supervision (where some embeddings are unused) and set collapse (where embeddings collapse into a single point). For example, PVSE (Song and Soleymani, 2019) uses a distance function that relies on maximum similarity within the set, leaving many embeddings undertrained. Our analysis shows that Smooth-Chamfer similarity in SetDiv (Kim et al., 2023), proposed as a solution to this issue, leads to low variance across the embeddings within the set, reducing the ability to capture diverse relationships in the data.

In this paper, we address these limitations by introducing MaxMatch, a new Maximal Pair Assignment Similarity mechanism and diversity-promoting losses. Unlike previous work, our approach optimally matches pairs within the embedding sets using a permutation-based similarity, ensuring that every embedding in the set contributes to the final objective. In addition, we design a global discriminative loss that encourages each embedding to diverge from a global reference vector, and an intra-set divergence constraint that pushes embeddings apart within the set. These strategies effectively prevent set collapse and improve alignment across modalities. Specifically, our contributions include:

• We introduce a new similarity measure for measuring the distance between sets of embeddings. MaxMatch uses permutation-based assignments and the Hungarian algorithm to match embeddings between sets and compute their distance. We show that MaxMatch prevents degenerate embedding sets unlike previous methods.
• We propose a novel loss which enforces a margin between residual embeddings and a global reference embedding to encourage semantic diversity. We also introduce intra-set divergence constraints to further discourage set collapse. We show that these functions promote semantic diversity and improve the model’s discriminative capability.
• We conduct a comprehensive experimental analysis of our proposed technique against a number of state-of-the-art methods, including embedding space visualizations and analysis. Our results show that MaxMatch learns richer, more diverse embedding sets while outperforming similar methods.

2 Related Work

Cross-modal Retrieval: Methods fall into two main categories: independent-embedding and interactive-embedding approaches. Independent-embedding approaches use separate encoders for each modality to learn a shared embedding space (Faghri et al., 2018; Yan and Mikolajczyk, 2015; Gu et al., 2018), enabling efficient retrieval through precomputed embeddings. While researchers have explored various improvements to similarity metrics (Wei et al., 2020a; Kim et al., 2023), loss functions (Thomas and Kovashka, 2020; Chun et al., 2021), and architectures (Gu et al., 2018), these methods still struggle with complex cross-modal relationships (Song and Soleymani, 2019). Recent work like CORA (Pham et al., 2024) and 3SHNet (Ge et al., 2024) achieve improvements by incorporating external knowledge (scene graphs, segmentation), but this makes direct comparisons with methods that do not use external data more difficult.

Interactive-embedding approaches instead use cross-attention networks for direct similarity estimation (Wei et al., 2020b; Diao et al., 2021), particularly when they are built on large pretrained architectures like CLIP (Radford et al., 2021). While effective, these methods are computationally intensive as they require joint processing of each query-candidate pair at inference time. In contrast, MaxMatch focuses on improving independent embeddings to maintain efficiency while better capturing complex relationships.

Multi-embedding Representations: Recently, a few papers have emphasized the limitations of representing each sample as a single embedding vector for cross-modal retrieval (Pham et al., 2024; Faghri et al., 2018; Chen et al., 2021; Fu et al., 2023;
Figure 2: Left: An overview of the model architecture based on Kim et al. (2023) consisting of a visual encoder, textual encoder, and set prediction modules for \( f^V \) and \( f^T \). Local and global features are extracted from each modality and input into a set prediction module to generate embedding sets \( S^V \) and \( S^T \). We contribute three key components which enable the model to learn diverse embedding sets: a Maximal Pair Assignment Similarity function, Global Discriminative Loss, and Intra-Set Divergence Loss. Right: Our Global Discriminative Loss pushes embeddings within each set away from the global embedding, preventing set collapse, while the Intra-Set Divergence Loss ensures that individual embeddings within each set are distinct, promoting intra-set diversity.

Thomas and Kovashka, 2022). A single vector often fails to capture the rich and diverse semantics present in the data. One strategy to address this challenge is set-based representation learning, where multiple embeddings are generated for each input, like in SetDiv, PVSE, and PCME (Chun et al., 2021). These approaches aim to reduce ambiguity by capturing the varied semantics of each modality using a set-based embedding representation. SetDiv introduces smooth-Chamfer similarity to overcome sparse supervision issues, but it leads to a new problem of set collapse, where embeddings lose diversity. To address sparse supervision and set collapse, we introduce a Maximal Pair Assignment Similarity function that explicitly preserves diversity. Even though PVSE and SetDiv use diversity loss to encourage distinct representations by penalizing pairwise similarity but lack global context, focusing only on residual embeddings before fusion. To address this, we propose novel pushing losses to enhance intra-set diversity and better capture modality relationships, achieving improved cross-modal retrieval performance compared to prior methods.

3 Method

3.1 Feature Extraction

MaxMatch architecture consists of two encoders, shown in Figure 2, each connected to a set prediction model. For vision, a visual encoder \( f^V \) takes an image as input \( x \) and produces the image embedding vector \( v = f^V(x) \in \mathbb{R} \), and a text encoder \( f^T \) that takes in the text caption \( y \) and produces its embedding \( t = f^T(y) \in \mathbb{R} \). Each encoder has two branches that compute local features and global features from the input sample. The extracted local features are given as input to the set prediction modules, each of which fuses the local and global features to encode them into an embedding set. We follow the settings of previous works (Song and Soleymani, 2019; Kim et al., 2023) for the visual and textual feature extractors.

Visual Feature Extractor: We use one of two types of visual feature extractors following previous works like PVSE and SetDiv: (1) Flattened convolutional feature maps as local features \( \psi^V(x) \in \mathbb{R}^{N \times D} \) and their average pooled features as the global feature \( \phi^V(x) \in \mathbb{R}^D \). (2) ROI features from a pretrained object detector as local features \( \psi^V(x) \in \mathbb{R}^{N \times D} \) and their max-pooled representation as the global feature \( \phi^V(x) \in \mathbb{R}^D \).

Textual Feature Extractor: We use one of two types of text feature extractors: (1) Bi-GRU: An input caption \( y \) of \( L \) words is encoded using GloVe (Pennington et al., 2014), producing 300-dimensional word vectors as local features \( \psi^T(y) \in \mathbb{R}^{L \times 300} \). A Bi-GRU with \( H \) hidden units processes these features, with the final hidden state representing the global features \( \phi^T(y) \in \mathbb{R}^D \). (2) BERT (Devlin et al., 2019): The hidden state outputs serve as local features \( \psi^T(y) \in \mathbb{R}^{L \times D} \), and their max-pooled representation forms the global
Figure 3: t-SNE visualization of learned embedding spaces, where colors indicate different elements within each set, and markers differentiate modalities (dots for image embeddings, crosses for text embeddings). MIL produces scattered embeddings, Smooth Chamfer collapses them, while MaxMatch preserves semantic distinctiveness and strong alignment.

feature \( \phi^T(y) \in \mathbb{R}^D \).

3.2 Embedding Set Prediction Module

We next describe the set prediction module adapted from Kim et al. (2023) that we utilize to generate set embeddings. As shown in Figure 2, slots in the module iteratively compete to attend to input features. Each of the L aggregation blocks in the module contains a cross-attention layer followed by a feed-forward network. K learnable queries ("slots") cross-attend to keys and values from \( \psi^V(x) \), denoted \( E^l = AggBlock(E^{l-1}) = MLP(\tilde{E}^l) + \tilde{E}^l \in \mathbb{R}^{K \times D} \), where \( \tilde{E}^l \) represents the slot states after cross-attention. The final embedding set S is computed as \( S = LN(E^L) + [LN(\phi(y))...^{x^K}] \), where LN is a normalization layer, and \( [LN(\phi(y))...^{x^K}] \in \mathbb{R}^{K \times D} \) denotes K repetitions of global features \( \phi(y) \). Conceptually, each slot represents an offset from the global feature \( \phi^V(x) \). However, embeddings in standard SetDiv have a tendency to cluster closely around the global feature (i.e. set collapse). To encourage diversity, MaxMatch introduces a Maximal Pair Assignment Similarity, Global Discriminative Loss, and Intra-Set Divergence Loss.

3.3 Maximal Pair Assignment Similarity

MaxMatch aims to overcome issues related to set-based embeddings and set-based similarity measures, particularly set collapse and sparse supervision. To better understand these limitations, we first examine existing set-based similarity functions such as Multiple Instance Learning (MIL) similarity, introduced in PVSE, and smooth-Chamfer similarity, introduced by SetDiv.

Given two sets of embeddings \( S1 = \{x_1, x_2, ..., x_K\} \) and \( S2 = \{y_1, y_2, ..., y_K\} \), and a similarity function \( c(x, y) \) between vectors \( x \in S1 \) and \( y \in S2 \), the MIL loss objective is defined as \( SMIL(S1, S2) = \max_{x \in S1, y \in S2} c(x, y) \); while MIL provides a simple mechanism for similarity calculation between embedding sets, it suffers from sparse supervision because the max function passes gradients to a single element in each set. By focusing only on maximum similarity during training, it underutilizes the remaining elements in the sets and leads to embedding sets where only one element contributes during retrieval. Kim et al. (2023) proposed smooth-Chamfer similarity to address the sparse gradient issue. This averages the similarity scores between all pairs from \( S1 \) and \( S2 \), which allows all embeddings to receive gradients:

\[
S_{SC}(S1, S2) = \frac{1}{2\alpha|S1|} \sum_{x \in S1} \log \left( \sum_{y \in S2} e^{\alpha c(x, y)} \right)
+ \frac{1}{2\alpha|S2|} \sum_{y \in S2} \log \left( \sum_{x \in S1} e^{\alpha c(x, y)} \right)
\]

However, despite providing gradient signals for the entire set, we show in Figures 3 and 4 that Smooth-Chamfer similarity leads to set collapse (i.e. all embeddings become the same), while MIL leads to unused embeddings. Even with a high \( \alpha \) value, intended to emphasize stronger similarities, the log function smooths out contributions and leads to uniformly high similarity scores across all elements (i.e. collapse). See our Appendix for additional analysis. In sum, both existing techniques for training set-based representations lead to degenerate conditions. We propose Maximal Pair Assignment Similarity, which learns optimal matching between elements of embedding sets. By no longer pulling all embeddings towards the mean, sets are no longer encouraged to collapse, yet all embeddings remain active in the matching process.

Figure 4: Heatmaps showing average pairwise similarities between image-text embeddings across all test samples for models trained with different similarity functions: MIL shows sparse supervision, leading to isolated high similarities (only one embedding in the set is used), Smooth Chamfer causes set collapse, resulting in uniformly high similarities, and MaxMatch maintains distinct alignment, and fully utilizing embeddings set.
MaxMatch leverages permutation-based assignments, which we efficiently implement using block processing and masking techniques. It finds the optimal pairing of embeddings from \( S_1 \) and \( S_2 \), where the overall matching score of the sets is maximized. We describe our approach in detail below.

Cosine Similarity Calculation: We begin by computing the cosine similarity between the L2-normalized set of image embeddings \( \mathbf{V}_i = \{v_{i1}, v_{i2}, \cdots, v_{iK}\} \in \mathbb{R}^{K \times D} \) and the set of text embeddings \( \mathbf{T}_j = \{t_{j1}, t_{j2}, \cdots, t_{jK}\} \in \mathbb{R}^{K \times D} \), where \( K \) represents the number of embeddings per sample. The similarity score between each pair of sets is \( Sim(V_i, T_j) = S(V_i, T_j) \), where \( S(V_i, T_j) \in \mathbb{R}^{K \times K} \) is the similarity matrix for the sets: \( S_{mn}(V_i, T_j) = \frac{v_{im}^T t_{jn}}{\|v_{im}\|\|t_{jn}\|} \), where \( S_{mn}(V_i, T_j) \) denotes the element in the m-th row and n-th column of \( S(V_i, T_j) \).

Hungarian Algorithm: To obtain the optimal matching, we apply the Hungarian algorithm to blocks of the similarity matrix \( S(V_i, T_j) \) between embedding sets \( V_i \) and \( T_j \) which yields the permutation \( \pi \) that maximizes the total similarity by selecting the most semantically meaningful pairs:

\[
\pi^* = \arg\max_{\pi} \operatorname{tr}(S(V_i, \pi(T_j)))
\]

where \( \pi \) is a permutation of indices representing the optimal matching between the image and text embeddings and \( \operatorname{tr} \) denotes trace. After that, we apply the optimal assignment on \( S_{ij} \) that maximizes the total similarity, where the outputs are a binary mask \( M_{ij} \) indicating the selected matches:

\[
M_{ij} = \begin{cases}
1, & \text{if } (V_i, \pi^*(T_j)) \text{ is an optimal match} \\
0, & \text{otherwise}
\end{cases}
\]

The mask \( M_{ij} \) is applied to the similarity matrix to extract only the optimal matches:

\[
MaxSim(S_i^V, S_j^T) = \sum_{k=1}^K M_{ij} \odot S_{ij}
\]

where \( M_{ij} \) is a binary mask matrix that indicates whether a particular pair \( (V_i, \pi^*(T_j)) \) is part of the optimal one-to-one matching between the sets \( S_i^V \) and \( S_j^T \). Finally, we scale and sum them to calculate the final similarity score \( S_H \) for each image-text pair:

\[
S_H(S_i^V, S_j^T) = \frac{1}{K} \sum_{i=0}^K \sum_{j=0}^K \left( \left( \exp \left( MaxSim(S_i^V, S_j^T) \right) \right) - 1 \right)
\]

where the exponential function \( \exp \) is used to amplify the influence of pairs with higher similarity. This approach aligns with the intuition that the closest pairs, those that best capture the semantic relationship between the image and text, should have more weight in the final similarity score. For example, if a particular image-text pair is closely related, their similarity score after exponential scaling will dominate, ensuring that these critical pairs contribute more significantly to the overall score.

3.4 Training and Inference

We train MaxMatch to minimize standard objectives presented in previous works (Kim et al., 2023; Pham et al., 2024; Song and Soleymani, 2019): triplet loss (TRI), diversity regularizer (Div), Maximum Mean Discrepancy (MMD) regularizer, and Contrastive loss (CON). In addition, we introduce a Global Discriminative Loss (GD) and Intra-Set Divergence Loss (ISD) to prevent set collapse and encourage the model to learn diverse embeddings.

\[
\mathcal{L} = \mathcal{L}_{TRI} + \lambda_{GD} \mathcal{L}_{GD} + \lambda_{ISD} \mathcal{L}_{ISD} 
+ \lambda_{MMD} \mathcal{L}_{MMD} + \lambda_{Div} \mathcal{L}_{Div} + \lambda_{CON} \mathcal{L}_{CON}
\]

Triplet loss with hardest negatives: Following prior work (Kim et al., 2023; Song and Soleymani, 2019), we incorporate the a hinge-based Triplet Loss with Hardest Negatives mining. Given a batch of embeddings \( B = \{(S_i^V, S_i^T)\}_{i=1}^N \), the triplet loss is formulated as follows:

\[
\mathcal{L}_{TRI} = \sum_{i=1}^N \max \left[ \delta_1 + S_H(S_i^V, S_j^T) - S_H(S_i^V, S_i^T) \right]_+
+ \sum_{i=1}^N \max \left[ \delta_1 + S_H(S_i^T, S_j^V) - S_H(S_i^T, S_i^V) \right]_+
\]

where \( \delta_1 > 0 \) denotes the margin. For each positive pair \( (S_i^V, S_i^T) \), the model identifies the hardest negative sample \( S_j^T \) closest to \( S_i^V \), and similarly the hardest negative \( S_j^V \) closest to \( S_i^T \).

Regularization and Stabilization Losses: To ensure robust and stable training, we employ a set of losses following prior work (Kim et al., 2023). The Maximum Mean Discrepancy (MMD) (Gretton et al., 2006) minimizes the MMD between embedding sets of images and text, helping to prevent early divergence between modalities. The diversity regularizer (Song and Soleymani, 2019) penalizes similar element slots. Consistent with previous findings (Chen et al., 2021), we observed that using the hardest triplet loss can lead to instability
during early training. Therefore, following (Pham et al., 2024), we also leverage contrastive loss to align all matching image and text representations.

Global Discriminative Loss: pushes each embedding away from the corresponding global reference embedding (within a margin and with a scaling factor) within each set to ensure the embedding set captures meaningful semantics beyond what is already captured in the global representation. The key idea is to ensure that embeddings within each set are gradually pushed away from their respective global, encouraging distinct representations within the set. The loss is defined as

\[
\mathcal{L}_{GD} = \frac{1}{2N} \sum_{i=1}^{N} \left( \exp \left( s \cdot (\cos(v_i, \phi^V(y_i)) - \delta_2) \right) \right. \\
\left. + \exp \left( s \cdot (\cos(t_i, \phi^T(y_i)) - \delta_2) \right) \right)
\]

where \( N \) is the batch size, \( \delta_2 \) is the separation margin, and \( s \) is the scaling factor that controls gradient smoothness. Here, \( \cos(v_i, \phi^V(y)) \) and \( \cos(t_i, \phi^T(y)) \) represent the similarities between each visual embedding \( v_i \) or textual embedding \( t_i \) and their respective global references, \( \phi^V(y) \) and \( \phi^T(y) \). The exponential function applied to the scaled similarity encourages embeddings to spread apart in the feature space if they exceed the margin \( \delta_2 \), which promotes a well-distributed feature space across both modalities.

Intra-Set Divergence Loss: is designed to encourage diversity among embeddings within each set by penalizing high similarity between embeddings. This loss reduces redundancy within the set to encourage sets to learn unique representations which capture different aspects of the data. The loss is formulated as:

\[
\mathcal{L}_{ISD} = \frac{2}{M(M-1)} \sum_{j=1}^{M-1} \sum_{k=j+1}^{M} \left( \exp \left( s \cdot (\cos(v_{i,j}, v_{i,k}) - \delta_3) \right) \right. \\
\left. + \exp \left( s \cdot (\cos(t_{i,j}, t_{i,k}) - \delta_3) \right) \right)
\]

where \( M \) is the number of embeddings per set, \( \delta_3 \) is the similarity margin, and \( s \) is a scaling factor that smooths the gradients. The cosine similarity terms \( \cos(v_{i,j}, v_{i,k}) \) and \( \cos(t_{i,j}, t_{i,k}) \) measure the alignment between each pair of non-identical embeddings within the visual and textual sets, respectively. By applying the exponential function with scaling, the loss pushes embeddings further apart in the feature space if they exceed the margin \( \delta_3 \), encouraging a diverse distribution. The normalization factor \( \frac{2}{M(M-1)} \) averages the loss across all non-matching pairs.

Inference: During inference, we select the top-\( k \) most similar embeddings from the predicted sets without applying the one-to-one matching process used during training. This approach reduces the computational complexity and significantly speeds up the inference phase, making the model efficient while still leveraging the diverse embeddings.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Flickr30k Test Images</th>
    <th colspan="2">Text-to-Image</th>
    <th rowspan="2">RSUM</th>
  </tr>
  <tr>
    <th>Image-to-Text</th>
    <th>Text-to-Image</th>
    <th>Image-to-Text</th>
    <th>Text-to-Image</th>
  </tr>
  <tr>
    <td>Resnet152 + Bi-GRU</td>
    <td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>VSE++ (Faghri et al., 2018)</td>
    <td>52.9 80.5 87.2</td><td>39.6 70.1 79.5</td><td>409.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>PVRE (Yang and Socher, 2019)</td>
    <td>59.1 84.2 91.0</td><td>43.4 72.1 81.5</td><td>423.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>PCME (Chen et al., 2021)</td>
    <td>58.5 81.4 89.3</td><td>44.3 72.7 81.9</td><td>428.1</td><td></td><td></td>
  </tr>
  <tr>
    <td>Sit Div (Kim et al., 2023)</td>
    <td>61.8 85.5 91.1</td><td>46.1 78.4 83.3</td><td>442.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>68.6 89.6 94.6</td><td>51.5 78.9 86.8</td><td>469.9</td><td></td><td></td>
  </tr>
  <tr>
    <td>Faster R-CNN + Bi-GRU</td>
    <td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SCAN (Lee et al., 2018)</td>
    <td>67.4 90.1 95.8</td><td>48.6 77.7 85.2</td><td>465</td><td></td><td></td>
  </tr>
  <tr>
    <td>VRSN (Li et al., 2019)</td>
    <td>71.3 90.6 96.0</td><td>54.7 81.8 88.2</td><td>482.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>CAAN (Zhang et al., 2020)</td>
    <td>70.1 91.6 97.2</td><td>52.8 79 87.9</td><td>478.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>SGRAF (Duan et al., 2021)</td>
    <td>77.8 94.1 97.4</td><td>58.5 83.0 88.8</td><td>499.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>VSEc (Chen et al., 2021)</td>
    <td>76.5 94.2 97.7</td><td>56.4 83.5 89.9</td><td>498.1</td><td></td><td></td>
  </tr>
  <tr>
    <td>NAAF (Chang et al., 2022)</td>
    <td>79.1 95.3 98.3</td><td>60.2 85.3 90.7</td><td>513.2</td><td></td><td></td>
  </tr>
  <tr>
    <td>CHAN (Park et al., 2023)</td>
    <td>79.7 94.5 97.3</td><td>60.2 85.3 90.7</td><td>507.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>HREM (Fu et al., 2023)</td>
    <td>81.4 96.5 98.5</td><td>60.9 85.6 91.3</td><td>514.2</td><td></td><td></td>
  </tr>
  <tr>
    <td>CORA1 (Pham et al., 2024)</td>
    <td>82.3 96.1 98.0</td><td>63.0 87.4 92.8</td><td>519.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>Sit Div (Kim et al., 2023)</td>
    <td>77.8 94.0 97.5</td><td>57.5 84.0 90.0</td><td>500.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>80.8 95.9 97.4</td><td>59.3 84.7 90.3</td><td>509.1</td><td></td><td></td>
  </tr>
  <tr>
    <td>MaxMatch †</td>
    <td>82.1 95.5 98.3</td><td>61.0 86.3 91.9</td><td>515.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>Faster R-CNN + BERT</td>
    <td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>DSRAN (Wen et al., 2021)</td>
    <td>77.8 95.1 97.6</td><td>59.2 86.0 91.9</td><td>507.6</td><td></td><td></td>
  </tr>
  <tr>
    <td>VSEc (Chen et al., 2021)</td>
    <td>81.7 95.4 97.6</td><td>61.4 85.9 91.5</td><td>513.5</td><td></td><td></td>
  </tr>
  <tr>
    <td>MV-VSE (Duan et al., 2021)</td>
    <td>83.7 96.1 98.3</td><td>63.0 87.4 92.8</td><td>517.5</td><td></td><td></td>
  </tr>
  <tr>
    <td>CHAN (Park et al., 2023)</td>
    <td>80.6 96.1 97.8</td><td>63.9 87.5 92.6</td><td>518.5</td><td></td><td></td>
  </tr>
  <tr>
    <td>CODER (Wang et al., 2022)</td>
    <td>83.2 96.5 98.0</td><td>63.1 87.1 93.0</td><td>520.9</td><td></td><td></td>
  </tr>
  <tr>
    <td>HREM (Fu et al., 2023)</td>
    <td>84.0 96.1 98.6</td><td>64.4 88.0 93.1</td><td>524.2</td><td></td><td></td>
  </tr>
  <tr>
    <td>CORA1 (Pham et al., 2024)</td>
    <td>83.4 95.9 98.6</td><td>64.1 88.1 93.1</td><td>523.3</td><td></td><td></td>
  </tr>
  <tr>
    <td>Sit Div (Kim et al., 2023)</td>
    <td>81.3 95.3 97.7</td><td>62.4 86.5 91.4</td><td>514.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>84.2 96.1 97.9</td><td>65.2 87.3 92.8</td><td>520.8</td><td></td><td></td>
  </tr>
  <tr>
    <td>MaxMatch †</td>
    <td>86.2 96.7 98.4</td><td>68.8 88.8 93.2</td><td>527.1</td><td></td><td></td>
  </tr>
</table>

Table 1: Recall@K (%) and RSUM on the Flickr30k dataset. The best RSUM scores are marked in bold, and the second-best scores are underlined. CA, †, and ‡ indicate models using cross-attention, models that use external data, and ensemble models of two hypotheses.

4 Experiments

4.1 Datasets and Evaluation Metric

We evaluate MaxMatch on Flickr30k and COCO datasets. We follow the standard train, validation, test splits and evaluation practices from prior work. For COCO, we average performance over five folds of 1K test images and on the full set of 5K test images, while for Flickr30k, we use the test set. In both datasets, each image has five matching captions. Performance is measured using Recall@K (\( K \in \{1, 5, 10\} \)) and the RSUM metric, which sums Recall@K scores for image-to-text and text-to-image retrieval (Faghri et al., 2018). Implementation details are provided in the Appendix.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">CA</th>
    <th colspan="6">COCO 1K Test Images</th>
    <th colspan="6">COCO 5K Test Images</th>
  </tr>
  <tr>
    <th colspan="3">Image-to-Text</th>
    <th colspan="3">Text-to-Image</th>
    <th colspan="3">Image-to-Text</th>
    <th colspan="3">Text-to-Image</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>RSUM</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>RSUM</th>
  </tr>
  <tr>
    <th colspan="18"><i>Resnet152 + Bi-GRU</i></th>
  </tr>
  <tr>
    <td>VSE++ (Faghri et al., 2018)</td>
    <td></td>
    <td>64.6</td>
    <td>90.0</td>
    <td>95.7</td>
    <td>52.0</td>
    <td>84.0</td>
    <td>92.0</td>
    <td>478.6</td>
    <td>41.3</td>
    <td>71.1</td>
    <td>81.2</td>
    <td>30.3</td>
    <td>59.4</td>
    <td>72.4</td>
    <td>355.7</td>
  </tr>
  <tr>
    <td>PVSE (Song and Soleymani, 2019)</td>
    <td></td>
    <td>69.2</td>
    <td>91.6</td>
    <td>96.6</td>
    <td>55.2</td>
    <td>86.5</td>
    <td>93.7</td>
    <td>492.8</td>
    <td>45.2</td>
    <td>74.3</td>
    <td>84.5</td>
    <td>32.4</td>
    <td>63.0</td>
    <td>75.0</td>
    <td>374.4</td>
  </tr>
  <tr>
    <td>PCME (Chun et al., 2021)</td>
    <td></td>
    <td>68.8</td>
    <td>-</td>
    <td>-</td>
    <td>54.6</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>44.2</td>
    <td>-</td>
    <td>-</td>
    <td>31.9</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Set Div (Kim et al., 2023)</td>
    <td></td>
    <td>70.3</td>
    <td>91.5</td>
    <td>96.3</td>
    <td>56.0</td>
    <td>85.8</td>
    <td>93.3</td>
    <td>493.2</td>
    <td>47.2</td>
    <td>74.8</td>
    <td>84.1</td>
    <td>33.8</td>
    <td>63.1</td>
    <td>74.7</td>
    <td>377.7</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td></td>
    <td>74.54</td>
    <td>93.98</td>
    <td>97.54</td>
    <td>58.35</td>
    <td>87.63</td>
    <td>94.31</td>
    <td><b>506.75</b></td>
    <td>51.84</td>
    <td>79.86</td>
    <td>87.62</td>
    <td>36.35</td>
    <td>66</td>
    <td>77.28</td>
    <td><b>398.95</b></td>
  </tr>
  <tr>
    <th colspan="18"><i>Faster R-CNN + Bi-GRU</i></th>
  </tr>
  <tr>
    <td>SCAN † (Lee et al., 2018)</td>
    <td></td>
    <td>72.7</td>
    <td>94.8</td>
    <td>98.4</td>
    <td>58.8</td>
    <td>88.4</td>
    <td>94.8</td>
    <td>507.9</td>
    <td>50.4</td>
    <td>82.2</td>
    <td>90</td>
    <td>38.6</td>
    <td>69.3</td>
    <td>80.4</td>
    <td>410.9</td>
  </tr>
  <tr>
    <td>VSRN † (Li et al., 2019)</td>
    <td></td>
    <td>76.2</td>
    <td>94.8</td>
    <td>98.2</td>
    <td>62.8</td>
    <td>89.7</td>
    <td>95.1</td>
    <td>516.8</td>
    <td>53</td>
    <td>81.1</td>
    <td>89.4</td>
    <td>40.5</td>
    <td>70.6</td>
    <td>81.1</td>
    <td>415.7</td>
  </tr>
  <tr>
    <td>CAAN (Zhang et al., 2020)</td>
    <td></td>
    <td>75.5</td>
    <td>95.4</td>
    <td>98.5</td>
    <td>61.3</td>
    <td>89.7</td>
    <td>95.2</td>
    <td>515.6</td>
    <td>52.5</td>
    <td>83.3</td>
    <td>90.9</td>
    <td>41.2</td>
    <td>70.3</td>
    <td>82.9</td>
    <td>421.1</td>
  </tr>
  <tr>
    <td>SGRAF† (Diao et al., 2021)</td>
    <td></td>
    <td>79.6</td>
    <td>96.2</td>
    <td>98.5</td>
    <td>63.2</td>
    <td>90.7</td>
    <td>96.1</td>
    <td>524.3</td>
    <td>57.8</td>
    <td>84.9</td>
    <td>91.6</td>
    <td>41.9</td>
    <td>70.7</td>
    <td>81.3</td>
    <td>428.2</td>
  </tr>
  <tr>
    <td>VSE+ (Chen et al., 2021)</td>
    <td></td>
    <td>78.5</td>
    <td>96</td>
    <td>98.7</td>
    <td>61.7</td>
    <td>90.3</td>
    <td>95.6</td>
    <td>520.8</td>
    <td>56.6</td>
    <td>83.6</td>
    <td>91.4</td>
    <td>39.3</td>
    <td>69.9</td>
    <td>81.1</td>
    <td>421.9</td>
  </tr>
  <tr>
    <td>NAAF† (Zhang et al., 2022)</td>
    <td></td>
    <td>80.5</td>
    <td>96.5</td>
    <td>98.8</td>
    <td>64.1</td>
    <td>90.7</td>
    <td>96.5</td>
    <td>527.2</td>
    <td>58.9</td>
    <td>85.2</td>
    <td>92.0</td>
    <td>42.5</td>
    <td>70.9</td>
    <td>81.4</td>
    <td>430.9</td>
  </tr>
  <tr>
    <td>CHAN (Pan et al., 2023)</td>
    <td></td>
    <td>79.7</td>
    <td>96.7</td>
    <td>98.7</td>
    <td>63.8</td>
    <td>90.4</td>
    <td>95.8</td>
    <td>525.0</td>
    <td>60.2</td>
    <td>85.9</td>
    <td>92.4</td>
    <td>41.7</td>
    <td>71.5</td>
    <td>81.7</td>
    <td>433.4</td>
  </tr>
  <tr>
    <td>HREM† (Fu et al., 2023)</td>
    <td></td>
    <td>80.0</td>
    <td>96.0</td>
    <td>98.7</td>
    <td>62.7</td>
    <td>90.1</td>
    <td>95.4</td>
    <td>522.8</td>
    <td>58.9</td>
    <td>85.3</td>
    <td>92.1</td>
    <td>40.0</td>
    <td>70.6</td>
    <td>81.2</td>
    <td>428.1</td>
  </tr>
  <tr>
    <td>CORA ‡ (Pham et al., 2024)</td>
    <td></td>
    <td>81.7</td>
    <td>96.7</td>
    <td>99.0</td>
    <td>66.0</td>
    <td>92.0</td>
    <td>96.7</td>
    <td><b>532.1</b></td>
    <td>63.0</td>
    <td>86.8</td>
    <td>92.7</td>
    <td>44.2</td>
    <td>73.9</td>
    <td>84.0</td>
    <td><b>444.6</b></td>
  </tr>
  <tr>
    <td>Set Div (Kim et al., 2023)</td>
    <td></td>
    <td>79.8</td>
    <td>96.2</td>
    <td>98.6</td>
    <td>63.6</td>
    <td>90.7</td>
    <td>95.7</td>
    <td>524.6</td>
    <td>58.8</td>
    <td>84.9</td>
    <td>91.5</td>
    <td>41.1</td>
    <td>72.0</td>
    <td>82.4</td>
    <td>430.7</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td></td>
    <td>80.4</td>
    <td>96.4</td>
    <td>98.4</td>
    <td>64.6</td>
    <td>91.0</td>
    <td>96.0</td>
    <td>526.6</td>
    <td>59.5</td>
    <td>85.8</td>
    <td>94.0</td>
    <td>42.3</td>
    <td>72.8</td>
    <td>83.0</td>
    <td>436.1</td>
  </tr>
  <tr>
    <td>MaxMatch †</td>
    <td></td>
    <td>81.7</td>
    <td>96.5</td>
    <td>98.7</td>
    <td>65.4</td>
    <td>91.6</td>
    <td>96.5</td>
    <td><b>530.3</b></td>
    <td>61.9</td>
    <td>86.86</td>
    <td>93.1</td>
    <td>43.1</td>
    <td>73.8</td>
    <td>83.9</td>
    <td><b>443.1</b></td>
  </tr>
  <tr>
    <th colspan="18"><i>Faster R-CNN + BERT</i></th>
  </tr>
  <tr>
    <td>VSE∞ (Chen et al., 2021)</td>
    <td></td>
    <td>79.7</td>
    <td>96.4</td>
    <td>98.9</td>
    <td>64.8</td>
    <td>91.4</td>
    <td>96.3</td>
    <td>527.5</td>
    <td>58.3</td>
    <td>85.3</td>
    <td>92.3</td>
    <td>42.4</td>
    <td>72.7</td>
    <td>83.2</td>
    <td>434.2</td>
  </tr>
  <tr>
    <td>CODER (Wang et al., 2022)</td>
    <td></td>
    <td>82.1</td>
    <td>96.6</td>
    <td>98.8</td>
    <td>65.5</td>
    <td>91.5</td>
    <td>96.2</td>
    <td>530.7</td>
    <td>62.6</td>
    <td>86.6</td>
    <td>93.1</td>
    <td>42.5</td>
    <td>73.1</td>
    <td>83.3</td>
    <td>441.2</td>
  </tr>
  <tr>
    <td>MV-VSE† (Diao et al., 2021)</td>
    <td></td>
    <td>80.4</td>
    <td>96.6</td>
    <td>99.0</td>
    <td>64.9</td>
    <td>91.2</td>
    <td>96.0</td>
    <td>528.1</td>
    <td>59.1</td>
    <td>86.3</td>
    <td>92.5</td>
    <td>42.5</td>
    <td>72.8</td>
    <td>83.1</td>
    <td>436.3</td>
  </tr>
  <tr>
    <td>CHAN (Pan et al., 2023)</td>
    <td></td>
    <td>81.4</td>
    <td>96.9</td>
    <td>98.9</td>
    <td>66.5</td>
    <td>92.1</td>
    <td>96.7</td>
    <td>532.5</td>
    <td>59.8</td>
    <td>87.2</td>
    <td>93.3</td>
    <td>44.9</td>
    <td>74.5</td>
    <td>84.2</td>
    <td>443.9</td>
  </tr>
  <tr>
    <td>HREM† (Fu et al., 2023)</td>
    <td></td>
    <td>82.9</td>
    <td>96.9</td>
    <td>99.0</td>
    <td>67.1</td>
    <td>92.0</td>
    <td>96.6</td>
    <td>534.5</td>
    <td>64.0</td>
    <td>88.5</td>
    <td>93.7</td>
    <td>45.4</td>
    <td>75.1</td>
    <td>84.3</td>
    <td><b>451.0</b></td>
  </tr>
  <tr>
    <td>CORA‡ (Pham et al., 2024)</td>
    <td></td>
    <td>82.8</td>
    <td>97.3</td>
    <td>99.0</td>
    <td>67.3</td>
    <td>92.4</td>
    <td>96.9</td>
    <td>535.6</td>
    <td>64.3</td>
    <td>87.5</td>
    <td>93.6</td>
    <td>45.4</td>
    <td>74.7</td>
    <td>84.6</td>
    <td><b>450.1</b></td>
  </tr>
  <tr>
    <td>Set Div (Kim et al., 2023)</td>
    <td></td>
    <td>82.1</td>
    <td>95.9</td>
    <td>95.9</td>
    <td>65.1</td>
    <td>91.2</td>
    <td>96.2</td>
    <td>529.1</td>
    <td>62.3</td>
    <td>86.2</td>
    <td>92.6</td>
    <td>42.8</td>
    <td>72.8</td>
    <td>82.8</td>
    <td>439.6</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td></td>
    <td>83.0</td>
    <td>96.9</td>
    <td>98.9</td>
    <td>66.4</td>
    <td>91.9</td>
    <td>96.6</td>
    <td>533.8</td>
    <td>63.3</td>
    <td>87.9</td>
    <td>93.2</td>
    <td>44.2</td>
    <td>73.9</td>
    <td>83.9</td>
    <td>446.5</td>
  </tr>
  <tr>
    <td>MaxMatch †</td>
    <td></td>
    <td>83.8</td>
    <td>97.0</td>
    <td>98.9</td>
    <td>67.3</td>
    <td>92.4</td>
    <td>96.9</td>
    <td><b>536.3</b></td>
    <td>65.4</td>
    <td>88.7</td>
    <td>93.8</td>
    <td>45.1</td>
    <td>74.9</td>
    <td>84.8</td>
    <td><b>452.6</b></td>
  </tr>
</table>

Table 2: COCO Recall@K (%) and RSUM results on both the 1K test setting (average of 5-fold test dataset) and 5K test setting are presented. The best RSUM scores are marked in bold, and the second-best scores are underlined. CA, ‡, and † indicate models using cross-attention, models that use external data, and ensemble models, respectively.

4.2 Comparisons with Other Methods

We report results for the Flickr30k and COCO datasets in tables 1 and 2, respectively. We evaluate our method across two visual feature extractors; ImageNet pretrained ResNet152 (He et al., 2016), and Faster R-CNN with pre-extracted region-of-interest (ROI) features (Ren et al., 2015). We use either Bi-GRU or BERT for the text feature extraction. Following PVSE, SetDiv, CORA, etc., we report ensemble results by averaging similarities from two checkpoints with different seeds. Additionally, the results for the Faster R-CNN + BERT configuration were obtained using the official codebase, as this setting was not reported in the original paper (Kim et al., 2023).

We observe that MaxMatch outperforms state-of-the-art methods by an impressive margin except when using Bi-GRU as a semantic concept encoder with the Faster R-CNN visual features. However, MaxMatch achieves the second-best results in this setting. Note that the top-performing method, CORA enhances text representations through a graph generated by an external large language model-based parser (Li et al., 2023), introducing an additional model and text processing step. Using ResNet152 extracted visual features, MaxMatch substantially outperforms all other methods with a jump in RSUM by +27.3 on Flickr30k dataset and an increase in RSUM by +24.55 on COCO 5K. With BERT as the semantic concept encoder and Faster-RCNN as the visual feature extractor, MaxMatch outperforms all state-of-the-art methods by a significant margin. In particular, MaxMatch achieves +2.9 RSUM improvement over HREM on Flickr30k, and +1.6 RSUM on COCO 5K. It is also worth mentioning that MaxMatch outperforms CHAN by +2.3 RSUM on Flickr30k, and +2.6 RSUM on COCO 5K which requires cross-attention during inference time, making CHAN more computationally expensive.

4.3 Ablation Study

Table 3 ablates the different similarity functions (MIL, smooth-Chamfer, and Maximal Pair Assign-
<table>
  <tr>
    <th>Similarity</th>
    <th>Div</th>
    <th>MMD</th>
    <th>GD</th>
    <th>ISD</th>
    <th>RSUM</th>
  </tr>
  <tr>
    <td>MIL</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>438.98</td>
  </tr>
  <tr>
    <td>Smooth-Chamfer</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>439.57</td>
  </tr>
  <tr>
    <td>Smooth-Chamfer</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td>441.76</td>
  </tr>
  <tr>
    <td>Smooth-Chamfer</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>442.43</td>
  </tr>
  <tr>
    <td>Smooth-Chamfer</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td>443.11</td>
  </tr>
  <tr>
    <td>Smooth-Chamfer</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>444.10</td>
  </tr>
  <tr>
    <td>Maximal Pair Assignment</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>441.23</td>
  </tr>
  <tr>
    <td>Maximal Pair Assignment</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td></td>
    <td>442.67</td>
  </tr>
  <tr>
    <td>Maximal Pair Assignment</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>443.49</td>
  </tr>
  <tr>
    <td>Maximal Pair Assignment</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>444.49</td>
  </tr>
  <tr>
    <td><b>Maximal Pair Assignment</b></td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td><b>446.53</b></td>
  </tr>
</table>

Table 3: Ablation study of the proposed similarity function and different similarity measures (MIL and Smooth Chamfer) combined with various loss settings, including Div, MMD, GD, and ISD, on the overall performance, measured by RSUM on COCO.

ment) in combination with various loss functions (Div, MMD, GD, and ISD) to analyze their impact on overall performance, as measured by RSUM on the 5k MS-COCO testing dataset. The Maximal Pair Assignment function achieves the highest RSUM score of 446.53 when combined with all four loss settings (Div, MMD, GD, and ISD), resulting in the best performance among all tested configurations. This highlights Maximal Pair Assignment as the most effective similarity function when combined with the appropriate losses. In contrast, other combinations, such as MIL or smooth-Chamfer with selected loss functions, achieve lower RSUM scores, suggesting reduced effectiveness at capturing the desired relationships. Notably, Maximal Pair Assignment with GD and ISD losses exhibits strong performance, achieving RSUM scores above 441, showing the importance of these loss functions in improving the model’s performance.

4.4 Embedding Set Element Analysis

We analyze the role of individual embedding set elements in both \( S^V \) and \( S^T \) modalities in Table 4. Applying a max operation to pair embeddings from both modalities yields identical results, indicating a robust alignment mechanism. Retaining all embeddings consistently achieves the best performance, with the highest RSUM score of 446.53, highlighting their collective contribution. When using only \( S^V(3) \) or \( S^T(1) \), the RSUM drops to 437.91, emphasizing the critical role of all components. These findings underscore the advantage of leveraging complete embedding sets, unlike prior methods (e.g., PVSE and SetDiv). The bottom section reports circular variance = \( 1 - \left\| \frac{\sum_{s \in S} x}{|S|} \right\|^2 \), where lower log(Var.) values indicate stronger set collapsing, where MaxMatch maintains a higher log(Var.), indicating less set collapse compared to other method, ensuring better disentanglement and a more diverse embedding space.

<table>
  <tr>
    <th>Evaluation</th>
    <th colspan="4">RSUM</th>
  </tr>
  <tr>
    <th>S^V(1)</th>
    <th>S^V(2)</th>
    <th>S^V(3)</th>
    <th>S^V(4)</th>
    <th>MIL</th>
    <th>Smooth Chamfer</th>
    <th>MaxMatch</th>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>438.98</td>
    <td>439.57</td>
    <td>446.53</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>439.08</td>
    <td>421.12</td>
    <td>440.89</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>393.94</td>
    <td>439.93</td>
    <td>447.78</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>400.06</td>
    <td>439.50</td>
    <td>437.91</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>302.14</td>
    <td>438.84</td>
    <td>442.39</td>
  </tr>
  <tr>
    <th>S^T(1)</th>
    <th>S^T(2)</th>
    <th>S^T(3)</th>
    <th>S^T(4)</th>
    <th>MIL</th>
    <th>Smooth Chamfer</th>
    <th>MaxMatch</th>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>438.98</td>
    <td>439.57</td>
    <td>446.53</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>438.84</td>
    <td>439.24</td>
    <td>437.91</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>393.10</td>
    <td>438.12</td>
    <td>440.89</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>293.50</td>
    <td>438.12</td>
    <td>442.39</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>412.32</td>
    <td>439.14</td>
    <td>443.78</td>
  </tr>
  <tr>
    <th colspan="2">Circular variance of embedding set</th>
    <th colspan="2">MIL</th>
    <th colspan="2">Smooth Chamfer</th>
    <th colspan="2">MaxMatch</th>
  </tr>
  <tr>
    <td colspan="2">RSUM</td>
    <td>483.5</td>
    <td>500.8</td>
    <td>509.1</td>
  </tr>
  <tr>
    <td colspan="2">log(Var.)</td>
    <td>-7.35</td>
    <td>-2.13</td>
    <td>-1.68</td>
  </tr>
</table>

Table 4: RSUM on COCO dataset for \( S^V \) (top) and \( S^T \) (middle). This highlights the impact of selectively removing components on overall performance and embedding stability across three methods: MIL, Smooth Chamfer, and MaxMatch . The circular variance of the embedding set (bottom) on Flickr30k, where lower log(Var.) values indicate stronger collapse.

![Visual comparison of image retrieval results between our method and Set Div. For each embedding in the set, we show the top retrieved image. Our method generates embeddings that retrieve diverse images, while Set Div’s embeddings tend to collapse, resulting in retrieval of identical or highly similar images.](page_1012_1342_388_246.png)

Figure 5: Visual comparison of image retrieval results between our method and Set Div. For each embedding in the set, we show the top retrieved image. Our method generates embeddings that retrieve diverse images, while Set Div’s embeddings tend to collapse, resulting in retrieval of identical or highly similar images.

4.5 Qualitative Analysis

To compare the semantic diversity of our learned embeddings with prior work, we perform cross-modal retrieval using each embedding in our embedding set independently. Given a set of embeddings for a query, we retrieve the closest test sample for each query embedding by selecting the best-matched embedding from the test sample’s set. This evaluates how well our embeddings capture
diverse semantics. Qualitative results in Figure 5 show that SetDiv representations collapse, retrieving identical or highly similar samples, whereas our embeddings retrieve diverse yet semantically consistent results. The left side of the figure indicates which query embedding was used for retrieval.

5 Conclusion

We introduced a novel set-based embedding framework for cross-modal retrieval with Maximal Pair Assignment Similarity and a combination of new loss functions. The Maximal Pair Assignment Similarity, utilizing permutation-based assignments, resolves challenges of sparse supervision and set collapsing, improving accurate image-text similarity scoring. Global Discriminative Loss further enhances the model’s ability to differentiate between embeddings, while the Intra-Set Divergence Loss mitigates set collapsing by encouraging diversity within each set. Our approach achieves superior performance on standard benchmarks compared to previous methods, and we plan to extend it to other modalities and tasks in future work.

6 Acknowledgements

We acknowledge Advanced Research Computing at Virginia Tech for providing computational resources and technical support that have contributed to the results reported within this paper. We also thank all reviewers for their comments which helped improve the paper.

7 Limitation

While our proposed MaxMatch demonstrates strong performance for cross-modal retrieval and overcomes many important limitations related to representation set collapse, several important limitations remain unaddressed. While MS-COCO and Flickr30k are standard benchmarks, they represent a relatively constrained subset of real-world cross-modal scenarios. Most image captions in these datasets focus on describing concrete objects and actions, rather than abstract concepts, emotional content, or diverse cultural interpretations that might benefit from multiple distinct representations. Future work should explore more diverse and challenging datasets that better reflect the complexity of human visual and linguistic understanding. In addition , while effective for image-text pairs, the current formulation of MaxMatch is specific to bi-modal retrieval. Extending the framework to handle additional modalities or simultaneous alignment across three or more modalities would require substantial modifications to both the matching mechanism and loss functions. This limits its immediate applicability in broader multimodal scenarios involving audio, video, or other data types. These limitations point to several promising directions for future research, including more efficient matching algorithms, adaptive set size mechanisms, and generalization to broader multimodal contexts.

References

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086.

Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40.

Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. 2021. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15789–15798.

Sanghyuk Chun. 2024. Improved probabilistic image-text representations. In International Conference on Learning Representations (ICLR).

Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. 2021. Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8415–8424.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.

Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. 2021. Similarity reasoning and filtration for image-text matching. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):1218–1226.

Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. Vse++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference (BMVC).
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’ Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. 2023. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 2024. 3shnet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting. Information Processing & Management, 61(4):103716.

Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. 2006. A kernel method for the two-sample-problem. Advances in neural information processing systems, 19.

Jiuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang. 2018. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7181–7189.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Dongwon Kim, Namyup Kim, and Suha Kwak. 2023. Improving cross-modal retrieval with set of diverse embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23422–23431.

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pages 201–216.

Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. 2019. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4654–4662.

Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. 2023. Factual: A benchmark for faithful and consistent textual scene graph parsing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6377–6390.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing.

Ilya Loshchilov and Frank Hutter. 2017. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maillet. 2021. Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. ACM Trans. Multimedia Comput. Commun. Appl., 17(4).

Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826–9836.

Zhengxin Pan, Fangyu Wu, and Bailing Zhang. 2023. Fine-grained image-text matching by cross-modal hard aligning network. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19275–19284.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W Workshop on Automatic Differentiation.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Computational Linguistics.

Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. 2024. Composing object relations and attributes for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14354–14363.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Yale Song and Mohammad Soleymani. 2019. Polyse-mous visual-semantic embedding for cross-modal re-trieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1979–1988.

Christopher Thomas and Adriana Kovashka. 2020. Pre-serving semantic neighborhoods for robust cross-modal retrieval. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 317–335. Springer.

Christopher Thomas and Adriana Kovashka. 2022. Emphasizing complementary samples for non-literal cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4632–4641.

Haoran Wang, Dongliang He, Wenhai Wu, Boyang Xia, Min Yang, Fu Li, Yunlong Yu, Zhong Ji, Errui Ding, and Jingdong Wang. 2022. Coder: Coupled diversity-sensitive momentum contrastive learning for image-text retrieval. In European Conference on Computer Vision, pages 700–716. Springer.

Jiwei Wei, Xing Xu, Yang Yang, Yanli Ji, Zheng Wang, and Heng Tao Shen. 2020a. Universal weighting metric learning for cross-modal matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13005–13014.

Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. 2020b. Multi-modality cross attention network for image and sentence matching. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10938–10947.

Keyu Wen, Xiaodong Gu, and Qingrong Cheng. 2021. Learning dual semantic relations with graph at-tention for image-text matching. IEEE Transactions on Circuits and Systems for Video Technology, 31(7):2866–2879.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transfor-mations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500.

Fei Yan and Krystian Mikolajczyk. 2015. Deep corre-lation for matching images and text. In Proceedings of the IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR).

Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. 2022. Negative-aware attention framework for image-text matching. In 2022 IEEE/CVF Con-ference on Computer Vision and Pattern Recognition (CVPR).

Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z. Li. 2020. Context-aware attention network for image-text retrieval. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3533–3542.

A Appendix

This Appendix provides additional details and results that complement the main paper. Section B outlines our comprehensive implementation for various model configurations. In Section C, we show that smooth Chamfer similarity encourages set collapse. Methodological considerations regarding external knowledge integration are discussed in Section D. Section E evaluates our approach using stronger visual backbones, and Section F examines its performance on the PVSE architecture. Finally, Section H presents pseudocode for our key components.

B Implementation Details

Following prior work (Kim et al., 2023; Song and Soleymani, 2019), we use two types of feature extractors and set the embedding dimension \( D = 1024 \). We obtain convolutional visual features by applying a \( 1 \times 1 \) convolution to the final fea-ture map of a CNN. Then, we use the pre-extracted 2048-dimensional region features from Faster R-CNN (Ren et al., 2015) as BUTD (Anderson et al., 2018). Both feature types were transformed to the embedding space via a two-layer MLP with residual connections. We set \( K = 4 \) and \( D_h = 1024 \) for convolutional features and \( D_h = 2048 \) for region features (Kim et al., 2023). We process textual features using a BiGRU with either GloVe (Pennington et al., 2014) or BERT (Devlin et al., 2019) embeddings.

Our implementation uses PyTorch (Paszke et al., 2017) v2.0.1, leveraging the model architecture from the SetDiv codebase (Kim et al., 2023). We employ automatic mixed precision to improve training speed and efficiency. Each input batch consists of either 128 or 200 images along with their corresponding captions. We train our model using the AdamW optimizer, with both MMD loss (\( \lambda_{MMD} \)) and diversity loss (\( \lambda_{Div} \)) weighted at 0.01 across all configurations. While the main paper describes the key implementation details, specific training parameters vary based on the feature extractors used.

ResNet152 + BiGRU: We train the model for 120 epochs using initial learning rates of 1e-3 for MS-COCO (Lin et al., 2014) and 2e-3 for Flickr30k (Plummer et al., 2015). For the set prediction module, these learning rates are scaled by factors of 0.1 and 0.01 for MS-COCO and Flickr30k, respectively. Following (Song and Soleymani, 2019; Kim
et al., 2023), we apply step-wise learning rate decay with a factor of 0.1 every 20 epochs. During the first 50 epochs, the CNN weights are kept frozen. Training is performed on dual NVIDIA A40 GPUs with a batch size of 200. For the loss configuration, the triplet margin (\( \delta_1 \)) is set to 0.2, while both Global Discriminative Loss (\( \lambda_{GD} \)) and Intra-Set Divergence Loss (\( \lambda_{ISD} \)) use weights of 0.1. The margin parameters (\( \delta_{2,3} \)) and scaling parameter (\( s \)) are set to 0.6 and 0.5, respectively. Contrastive loss is not included in this configuration.

Faster R-CNN + BiGRU: The model is trained for 80 epochs with an initial learning rate of 1e-3, employing cosine annealing (Loshchilov and Hutter, 2017) and weight decay of 1e-4. For the set prediction module, the learning rate is scaled by 0.1 for both datasets. Following (Chen et al., 2021), we apply a dropout of 20% to both ROI features and word embeddings during training. Training is conducted on a single NVIDIA A40 GPU with a batch size of 200. For the loss configuration, the triplet margin (\( \delta_1 \)) is set to 0.3, while both Global Discriminative Loss (\( \lambda_{GD} \)) and Intra-Set Divergence Loss (\( \lambda_{ISD} \)) use weights of 0.1 for MS-COCO and 0.05 for Flickr30K. The margin parameters (\( \delta_{2,3} \)) and scaling parameter (\( s \)) are set to 0.6 and 0.5, respectively. The contrastive loss weight is set to 0.001.

Faster R-CNN + BERT: The model trains for 45 epochs with an initial learning rate of 1e-3, employing cosine annealing and weight decay of 5e-4. For the set prediction module and BERT, the learning rates are scaled by 0.5 and 0.1, respectively. Following previous configurations, we apply dropout of 20% to both ROI features and word embeddings during training. Training is conducted on dual NVIDIA A40 GPUs with batch sizes of 200 for MS-COCO and 128 for Flickr30k. For the loss configuration, the triplet margin (\( \delta_1 \)) is set to 0.15, while both Global Discriminative Loss (\( \lambda_{GD} \)) and Intra-Set Divergence Loss (\( \lambda_{ISD} \)) use weights of 0.1. The margin parameters (\( \delta_{2,3} \)) and scaling parameter (\( s \)) are set to 0.8 and 0.5, respectively. The contrastive loss weight is set to 0.001.

C Smooth Chamfer Leads to Set Collapse

We consider the smooth Chamfer similarity defined as
\[
S_{SC}(S_1, S_2) = \frac{1}{2\alpha|S_1|} \sum_{x \in S_1} \log \left( \sum_{y \in S_2} e^{\alpha c(x, y)} \right)
+ \frac{1}{2\alpha|S_2|} \sum_{y \in S_2} \log \left( \sum_{x \in S_1} e^{\alpha c(x, y)} \right).
\]
In what follows, we show, under suitable assumptions (e.g. when \( c(x, y) \) is affine in \( x \), as in \( c(x, y) = x^\top y \)), that the objective is minimized when all elements in each set are identical. For clarity, we detail the argument for \( S_1 \), and a symmetric argument applies to \( S_2 \).

Let \( S_1 = \{ x_1, x_2, \ldots, x_n \} \). Assume that \( S_2 \) is fixed. Define the function \( L(x) \triangleq \log \left( \sum_{y \in S_2} e^{\alpha c(x, y)} \right) \). Then, the contribution of \( S_1 \) to the overall objective is \( f(S_1) = \frac{1}{2\alpha n} \sum_{i=1}^n L(x_i) \). (We omit the constant prefactor \( 1/(2\alpha) \) in what follows, as it does not affect the location of the minimum.)

Assume that \( c(x, y) \) is affine in \( x \) for every fixed \( y \); for example, if \( c(x, y) = x^\top y \), then for fixed \( y \) the map \( x \mapsto \alpha c(x, y) \) is affine. Since the exponential is convex and increasing, the composition \( x \mapsto e^{\alpha c(x, y)} \) is convex. Moreover, the sum over \( y \) of convex functions is convex, and the logarithm of a sum of exponentials (the log-sum-exp function) is also convex. Thus, the function \( L(x) = \log \left( \sum_{y \in S_2} e^{\alpha c(x, y)} \right) \) is convex in \( x \).

Let the arithmetic mean of the embeddings in \( S_1 \) be \( \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i \). Then by Jensen's inequality we have:
\[
\frac{1}{n} \sum_{i=1}^n L(x_i) \geq L \left( \frac{1}{n} \sum_{i=1}^n x_i \right) = L(\bar{x}) .
\]
Equality holds if and only if all \( x_i \) are equal. That is, for any fixed mean \( \bar{x} \), the minimum of
\[
f(S_1) = \frac{1}{n} \sum_{i=1}^n L(x_i)
\]
is achieved if and only if
\[
x_1 = x_2 = \cdots = x_n = \bar{x}.
\]
Thus, any deviation (i.e., any diversity among the \( x_i \)) increases the objective.
This also becomes obvious when analyzed from the perspective of the gradient. Consider the gradient of \( L(x) \) with respect to \( x \). By the chain rule,

\[
\nabla_x L(x) = \frac{\sum_{y \in S_2} \alpha e^{\alpha c(x, y)} \nabla_x c(x, y)}{\sum_{y \in S_2} e^{\alpha c(x, y)}}. \tag{13}
\]

Thus, each \( x_i \) receives a gradient that is a softmax-weighted average of the vectors \( \alpha \nabla_x c(x, y) \) (for example, when \( c(x, y) = x^\top y \), one has \( \nabla_x c(x, y) = y \)).

Now, consider a small perturbation of the set \( S_1 \) that preserves the overall mean.

\[
x_i = \bar{x} + \delta_i, \quad \text{with} \quad \sum_{i=1}^n \delta_i = 0. \tag{14}
\]

Because \( L \) is convex, its Hessian denoted by

\[
H(x) = \nabla_x^2 L(x),
\] (15)

is positive semidefinite:

\[
H(x) \succeq 0. \tag{16}
\]

A standard result in convex analysis is that the average

\[
\frac{1}{n} \sum_{i=1}^n L(\bar{x} + \delta_i)
\] (17)

is minimized (with respect to the deviations \( \{\delta_i\} \) under the constraint \( \sum_i \delta_i = 0 \)) when all \( \delta_i = 0 \). In other words, any nonzero differences \( \delta_i \) (i.e., any diversity among the embeddings) increases the objective’s value. Hence, the Hessian is strictly positive in the directions that would cause dispersion in \( S_1 \). Thus, the gradients computed above will push any \( x_i \) that deviates from the mean \( \bar{x} \) back toward \( \bar{x} \). Hence, during training the optimal (or at least a stationary) point is one where all embeddings in \( S_1 \) are equal. A symmetric argument applies to \( S_2 \).

D Discussion on CORA and 3SHNet External Knowledge Integration

Recent work by CORA (Pham et al., 2024) achieves impressive results through the integration of an external LLM as a scene graph parser (Li et al., 2023), particularly when using Bi-GRU encoders. While this approach demonstrates strong performance, it introduces important considerations for fair comparison. CORA’s key innovation lies in enhancing text representations by augmenting raw captions with structured scene graphs derived from an LLM parser, effectively incorporating external semantic knowledge into the representation learning process. This external knowledge particularly benefits simpler text encoders like Bi-GRU, where the additional semantic structure helps bridge the gap in language understanding capabilities. CORA achieves state-of-the-art results even with embeddings trained from scratch (as reported in their supplementary material), highlighting how the LLM parser significantly improves textual representations.

In contrast, 3SHNet (Ge et al., 2024) takes a different approach by integrating external visual knowledge through semantic segmentation features. Rather than relying on a language-based external module, 3SHNet incorporates structured spatial and object-level information using UPSNet to enhance object-region representations. This structured semantic-spatial self-highlighting method improves image representation, making retrieval more robust without requiring external textual scene graphs.

These approaches raise important methodological considerations. Unlike CORA and 3SHNet, our method operates directly on raw captions using a standard text encoder (GloVe or BERT) without incorporating external knowledge or additional preprocessing, and does not rely on UPSNet segmentation beyond the basic steps used in prior work (Song and Soleymani, 2019; Kim et al., 2023; Fu et al., 2023; Faghri et al., 2018; Chen et al., 2021). This distinction becomes particularly evident when using stronger language models like BERT, where the benefits of external parsing diminish as the encoder itself becomes more capable of understanding complex semantic relationships. While CORA’s paper acknowledges that BERT excels at encoding longer text sequences, their approach primarily uses it to encode short phrases, which loses the global context of the sentence. Our method, in contrast, leverages BERT’s full capability to process complete captions, achieving competitive performance without requiring additional scene graph parsing or external knowledge integration. This demonstrates the effectiveness of our approach in maintaining semantic richness while keeping the architecture simpler and more efficient.

E Testing with a Larger Backbone

In alignment with prior work (Kim et al., 2023), we select ResNeXT101 (Xie et al., 2017) pretrained
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Image-to-Text</th>
    <th colspan="3">Text-to-Image</th>
    <th rowspan="2">RSUM</th>
  </tr>
  <tr>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
  </tr>
  <tr>
    <td>MS-COCO 1K Test Images</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>VSE∞</td>
    <td>84.5</td>
    <td>98.1</td>
    <td>99.4</td>
    <td>72</td>
    <td>93.9</td>
    <td>97.5</td>
    <td>545.4</td>
  </tr>
  <tr>
    <td>Set Div</td>
    <td>86.3</td>
    <td>97.8</td>
    <td>99.4</td>
    <td>72.4</td>
    <td>94</td>
    <td>97.6</td>
    <td>547.5</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>86.9</td>
    <td>98.12</td>
    <td>99.38</td>
    <td>72.22</td>
    <td>94.12</td>
    <td>97.6</td>
    <td>547.84</td>
  </tr>
  <tr>
    <td>MS-COCO 5K Test Images</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>VSE∞</td>
    <td>66.4</td>
    <td>89.3</td>
    <td>94.6</td>
    <td>51.6</td>
    <td>79.3</td>
    <td>87.6</td>
    <td>468.9</td>
  </tr>
  <tr>
    <td>Set Div</td>
    <td>69.1</td>
    <td>90.7</td>
    <td>95.6</td>
    <td>52.1</td>
    <td>79.6</td>
    <td>87.8</td>
    <td>474.9</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>69.62</td>
    <td>90.76</td>
    <td>95.54</td>
    <td>52.18</td>
    <td>79.46</td>
    <td>87.58</td>
    <td>475.14</td>
  </tr>
  <tr>
    <td>Flickr30k Test Images</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>VSE∞</td>
    <td>88.4</td>
    <td>97.3</td>
    <td>99.5</td>
    <td>74.2</td>
    <td>93.7</td>
    <td>96.8</td>
    <td>550.9</td>
  </tr>
  <tr>
    <td>Set Div</td>
    <td>88.8</td>
    <td>98.5</td>
    <td>99.6</td>
    <td>74.3</td>
    <td>94</td>
    <td>96.7</td>
    <td>551.9</td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>90.5</td>
    <td>99</td>
    <td>99.8</td>
    <td>74.96</td>
    <td>94.2</td>
    <td>96.78</td>
    <td>555.24</td>
  </tr>
</table>

Table 5: Recall@K (%) and RSUM on MS-COCO and Flickr30k dataset with ResNeXT101 and BERT backbones. The best RSUM scores are marked in bold, and the second-best scores are underlined.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Image-to-Text</th>
    <th colspan="3">Text-to-Image</th>
    <th rowspan="2">RSUM</th>
  </tr>
  <tr>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
  </tr>
  <tr>
    <td>VSE++</td>
    <td>52.9</td>
    <td>80.5</td>
    <td>87.2</td>
    <td>39.6</td>
    <td>70.1</td>
    <td>76.5</td>
    <td>409.8</td>
  </tr>
  <tr>
    <td>PVSE</td>
    <td>59.1</td>
    <td>84.5</td>
    <td>91</td>
    <td>43.4</td>
    <td>73.1</td>
    <td>81.5</td>
    <td>432.6</td>
  </tr>
  <tr>
    <td>PCME</td>
    <td>58.5</td>
    <td>81.4</td>
    <td>89.3</td>
    <td>44.3</td>
    <td>72.7</td>
    <td>81.9</td>
    <td>428.1</td>
  </tr>
  <tr>
    <td>Set Div</td>
    <td>61.8</td>
    <td>85.5</td>
    <td>91.1</td>
    <td>46.1</td>
    <td>74.8</td>
    <td>83.3</td>
    <td>442.6</td>
  </tr>
  <tr>
    <td><b>MaxMatch + PVSE</b></td>
    <td><b>65.4</b></td>
    <td><b>88.1</b></td>
    <td><b>93.7</b></td>
    <td><b>45.4</b></td>
    <td><b>74.7</b></td>
    <td><b>83.2</b></td>
    <td><b>450.6</b></td>
  </tr>
  <tr>
    <td>MaxMatch</td>
    <td>68.6</td>
    <td>89.6</td>
    <td>94.6</td>
    <td>51.5</td>
    <td>78.9</td>
    <td>86.8</td>
    <td>469.9</td>
  </tr>
</table>

Table 6: Recall@K (%) and RSUM on Flickr30k dataset on PVSE architecture. The scores for PVSE with our method is in bold.

F Evaluating Method Performance on PVSE

We evaluate the versatility and effectiveness of our proposed method by applying it to the PVSE architecture, demonstrating its ability to enhance various baseline models. While PVSE offers more modest performance compared to recent methods like SDE, it provides an excellent test case for assessing our method’s generalizability. The results in Table 6 show substantial improvements across all metrics when our method is integrated with PVSE. Specifically, our method improves PVSE’s Image-to-Text R@1 performance from 59.1% to 65.4%, a significant gain of 6.3 percentage points, surpassing even the more recent Set Div model (61.8%). In Text-to-Image retrieval, we achieve consistent improvements, with R@1 increasing from 43.4% to 45.4%. The overall RSUM metric improves by 18 points (from 432.6 to 450.6), demonstrating the comprehensive enhancement our method brings to the baseline model.

Notably, our method’s performance with PVSE not only surpasses the original PVSE results but also outperforms several more recent approaches including PCME and approaches the performance of Set Div. This is particularly impressive given PVSE’s relatively simpler architecture. A key distinction of our approach is its ability to utilize the entire embedding set during training and inference, whereas traditional Multiple Instance Learning (MIL) approaches in PVSE only select a single embedding from the set. This comprehensive utilization enables our method to better capture diverse nuanced meanings from the samples, leading to more robust representations. When further combined with Set Div, our method achieves even more substantial gains, reaching state-of-the-art performance with an RSUM of 469.9, highlighting its complementary nature to existing advanced techniques. These results convincingly demonstrate that our method is architecture-agnostic and can
Figure 6: Visual comparison of image retrieval results between our method and Set Div. For each embedding in the set, we show the top retrieved image. Our method generates embeddings that retrieve diverse images, while Set Div’s embeddings tend to collapse, resulting in retrieval of identical or highly similar images.

effectively enhance both simple and sophisticated models, validating its broad applicability in cross-modal retrieval tasks.

G Qualitative Analysis for Image-to-Text Retrieval

Following Section 4.5, we extend our qualitative analysis to image-to-text retrieval. We perform cross-modal retrieval using each embedding in our set independently, selecting the closest test sample for each query embedding. This evaluates the semantic diversity captured by our embeddings compared to prior work. As shown in Figure 6, SetDiv representations tend to collapse, retrieving nearly identical samples, whereas our embeddings yield diverse yet semantically coherent results. The left side of the figure indicates the query embedding used for retrieval.

Failure Cases and Limitations: While our method generally produces more semantically diverse and accurate retrievals compared to SetDiv, we also observe certain failure cases where diversity is reduced or retrievals collapse. As shown in the rightmost column of Figure 7, our embeddings occasionally yield highly similar or even repeated captions across different slots, especially for images with limited semantic variation or dominant foreground objects. This suggests that, in scenarios where visual content is overly focused or lacks contextual richness, the learned embeddings may converge to similar representations. To further characterize these limitations, we include additional qualitative examples in the appendix, including cases where our model retrieves semantically plausible yet not part of the ground truth captions, a common occurrence in image-text datasets.

Figure 7: Failure cases in image-to-text retrieval using our method. While our approach typically retrieves semantically diverse captions, some examples particularly those with visually homogeneous content result in repeated or overly similar captions across the set. This highlights a limitation in capturing fine-grained diversity for certain input images.

H Code Availability

We provide pseudocodes of our key components illustrating the Global Discriminative Loss, Intra-Set Divergence Loss, and Maximal Pair Assignment Similarity functions. These code snippets demonstrate the conceptual implementation of our method in PyTorch notation, highlighting both the efficiency and simplicity of our approach. We will release the codebase on GitHub.
Maximal Pair Assignment Similarity

def mask_max_similarity(sim_matrix, text_indices, img_indices):
    """
    Generate a binary mask to identify maximum similarity pairs in a similarity matrix.

    Args:
        sim_matrix (Tensor): Similarity matrix between text and image embeddings.
        text_indices (Tensor): Indices corresponding to text embeddings.
        img_indices (Tensor): Indices corresponding to image embeddings.

    Returns:
        Tensor: Binary mask with 1s at maximum similarity pairs.
    """
    # Step 1: Extract similarities at specified indices
    selected_similarities = sim_matrix[text_indices, img_indices]

    # Step 2: Find indices of maximum similarity for each pair
    max_indices = selected_similarities.argmax(dim=1)

    # Step 3: Create binary mask with 1s at maximum similarity indices
    mask = torch.zeros_like(sim_matrix)
    mask[text_indices[max_indices], img_indices[max_indices]] = 1

    return mask

Listing 1: Generate a mask for maximum similarity pairs in a similarity matrix.

def create_index_permutations(num_embeddings, row_size, col_size):
    # Generate permutations of indices for matrix operations.
    # Step 1: Generate all permutations of the embeddings
    permutations = torch.tensor(list(itertools.permutations(range(num_embeddings))), dtype=torch.long)
    # Step 2: Repeat permutations to match the required row size
    row_indices = permutations.repeat(row_size // num_embeddings, 1)
    # Step 3: Create column indices to match the required column size
    col_indices = torch.arange(col_size).repeat(row_indices.size(0), 1)
    # tuple: Two tensors representing row and column indices.
    return row_indices, col_indices

Listing 2: Create index permutations for matrix operations.

def maximal_pair_assignment_similarity(img_embs, txt_embs):
    """
    Compute assignment of maximum similarity between image and text embeddings.

    Args:
        img_embs (Tensor): Image embeddings.
        txt_embs (Tensor): Text embeddings.

    Returns:
        Tensor: Maximum similarity tensor.
    """
    # Step 1: Compute cosine similarity between normalized embeddings
    dist = cosine_sim(l2norm(img_embs), l2norm(txt_embs))

    # Step 2: Determine row and column sizes based on batch and set sizes
    row_size = image_batch_size * img_set_size  # Number of image embeddings
    col_size = text_batch_size * txt_set_size  # Number of text embeddings

    # Step 3: Generate all possible index permutations for matching
    text_index_all, image_index_all = create_index_permutations(
        img_set_size, row_size, col_size
    )

    # Step 4: Create a mask for maximum similarity pairs
    mask = mask_max_similarity(dist.detach(), text_index_all, image_index_all)

    # Step 5: Calculate maximum similarity scores
    max_similarity = mask * dist

    # Step 6: Apply exponential scaling and average pooling
    max_similarity = avg_pool(torch.exp(max_similarity.unsqueeze(0)) - 1) * img_set_size

    return max_similarity.squeeze()

Listing 3: Function to compute assignment of maximum similarity between embeddings.
Intra-Set Divergence Loss

def Intra_Set_Divergence_Loss(set_emb, margin, scale):
    # Compute pairwise similarity between all embeddings in the set
    A = torch.bmm(set_emb, set_emb.transpose(1, 2))  # [bs, K, K]

    # Create a mask to exclude self-similarities and include only upper triangular matrix elements
    num_embeddings = set_emb.size(1)
    mask = torch.triu(torch.ones(
        num_embeddings, num_embeddings,
        device=set_emb.device), diagonal =1).bool()
    mask = mask.unsqueeze(0)  # Expand mask to match batch dimension

    # Apply the mask to extract unique pairwise similarities
    A_masked = A.masked_select(mask)

    # Compute the loss using the smooth exponential function with margin and scale
    loss = torch.exp((A_masked - margin) * scale)

    # Normalize the loss over the batch and number of unique pairs
    return loss.sum() / (set_emb.shape [0] * (set_emb.shape[1] * (set_emb.shape[1] - 1) / 2))

Listing 4: Intra-Set Divergence Loss

Global Discriminative Loss

def Global_Discriminative_Loss(Set_emb, global_emb, margin, scale):
    # Add a singleton dimension to global_embedding for batch matrix multiplication
    global_emb= global_emb.unsqueeze(1) #[bs,1,D]

    # Transpose embeddings to match dimensions for batch matrix multiplication
    Set_emb= Set_emb.transpose(1, 2) #[bs,D,K]

    # Compute pairwise similarity between global embedding and the set of embeddings
    A= torch.bmm(global_emb, Set_emb).squeeze() #[bs,K]

    # Compute the loss by applying the margin and scale, then take the mean
    loss= torch.mean(torch.exp(scale * (A - margin)))
    return loss

Listing 5: Global Discriminative Loss

Reviewer paper summaries: This paper aims to the task cross-modal image-text retrieval. To address the issues of sparse supervision and set collapse, this paper proposes a Maximal Pair Assignment Similarity function to optimize one-to-one matching between embeddings sets. This paper also introduce two loss functions to enhance the representations. The experimental results achieves SOTA performance.
Summary of strengths: - This paper is well-written and is easy to follow.
- Both this motivation and its analysis are clear.
- The experimental results is convinced. The proposed method achieves the SOTA performance.
Summary of weaknesses: - It is strange that in Eq. (6), the two terms on either side of the plus sign are the same, as S_H seems be a symmetric operation.
- The visual and textual encoders are outdated. How does the performance change when using more modern encoders?
- The effectiveness of k during inference is not analyzed.
- The impact of the four lambda values in Eq. (5) is not examined.
Comments, suggestions and typos: Could the authors briefly explain the purpose of the five loss functions?
Overall assessment scores: 3.5