Partially Aligned Cross-modal Retrieval via  Optimal Transport-based Prototype Alignment Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:48:16

Abstract: Supervised cross-modal retrieval (CMR) achieves excellent performance thanks to the semantic information provided by its labels, which helps to establish semantic correlations between samples from different modalities.  However, in real-world scenarios, there often exists a large amount of unlabeled and unpaired multimodal training data, rendering existing methods unfeasible. To address this issue, we
propose a novel partially aligned cross-modal retrieval method called Optimal Transport-based Prototype Alignment Learning (OTPAL). Due to the high computational complexity involved in directly establishing matching correlations between unannotated unaligned cross-modal samples, instead, we establish matching correlations between shared prototypes and samples. To be specific, we employ the optimal transport algorithm to establish cross-modal alignment information between samples and prototypes, and then minimize the distance between samples and their corresponding prototypes through a specially designed prototype alignment loss. As an extension of this paper, we also extensively investigate the influence of incomplete multimodal data on cross-modal retrieval performance under the partially aligned setting proposed above. To further address the above more challenging scenario, we raise a scalable prototype-based neighbor feature completion method, which better captures the correlations between incomplete samples and neighbor samples through a cross-modal self-attention mechanism. Experimental results on four benchmark datasets show that our method can obtain satisfactory accuracy and scalability in various real-world scenarios.

Main paper: 
 introduction: 
 Cross-modal retrieval (CMR) is a cross-modal search task: retrieving relevant information from different modalities, such as text, image, audio, etc. With the introduction of deep learning, cross-modal retrieval has made great progress. It has become the core research in many multi-modal applications, such as audio-video retrieval [30] , automatic story generation [38] , visual question answering [16, 23, 41] , and medical image-report retrieval [3, 37] . Thus, CMR has attracted increasing attention in both academia and industry due to its importance in real-world applications.Common cross-modal retrieval methods include the following types: (1) unsupervised cross-modal retrieval (US-CMR) [7, 8, 35, 36, 39] , (2) supervised cross-modal retrieval (S-CMR) [20, 29, 33, 34, 42, 43, 47] and (3) semi-supervised cross-modal retrieval (SS-CMR) The semi-supervised CMR method operates on both partially labeled paired and partially unlabeled paired data. To enhance the practicality of the aforementioned CMR methods in open environments, we introduce two types of partially aligned CMR methods tailored for unlabeled unpaired data (c) and unlabeled incomplete data (d), respectively.[ 12, 17, 18] . S-CMR methods can access semantic labels, which are generally superior to US-CMR performance. However, it usually incurs substantial labeling costs. Thus, SS-CMR methods [12, 17, 18] have been proposed to balance the conflict between the performance and cost of labeling data. Semi-supervised methods [12, 17, 18] exploit the intrinsic structure, pairwise relationships, and semantic information between different modalities to improve retrieval performance on the overall data. However, despite existing semi-supervised cross-modal methods performing well, most of them require pairwise data to establish representation consistency [17] and prediction consistency [18] , which is only applicable to well-paired data. For instance, as depicted in Fig. 1 (𝑏), even though some of the images and texts in the aforementioned semi-supervised methods are unlabeled, they still leverage the pairwise information of the images and texts. Unfortunately, in real-world applications, this assumption is always violated due to the diversity of multimodal data sources and noise issues, the pairwise correspondences are also partially available and inevitably suffer from partial mismatch problems. For example, images collected from web pages are difficult to be semantically related to text descriptions, or they can easily be mistakenly collected as irrelevant image-text pairs (as shown in Fig. 1 (𝑐)). As a result, real-world multi-modal data often consists of a small piece of well-paired multi-modal data and a massive amount of unpaired multi-modal data. Furthermore, in addition to irrelevant image-text pairs resulting from erroneous acquisition, multimodal pairwise data are inevitably limited by complete requirements. Yet this requirement is also equally demanding. For instance, in practical scenarios, multimodal training datasets inevitably suffer damage or loss during data collection, processing, storage, and transmission. Therefore, incomplete modal data will result in some pairwise relationships being unavailable (as shown in Fig. 1 (𝑑)). We unify the above two problems, referring to them collectively as cross-modal retrieval under two types of partially aligned data.To tackle the above challenges of two types of partially aligned data (Fig. 1 (c ) and (d)) in cross-modal retrieval, we propose an Optimal Transport-based Prototype Alignment Learning (OT-PAL) framework, which comprises a dual optimal transport-based prototype alignment module, by leveraging semantic prototypes learned from labeled data to facilitate the alignment of unlabeled unpaired data with prototypes. Specifically, we initially employ multimodal classification and invariance learning to capture modalitydiscriminative and invariant representations on labeled paired data. Furthermore, since the prototype can be characterized as the semantic center of each category in the feature space and can serve as the bridge between data and semantics, we define a prototype for each category and learn and update it through training. Besides, to learn the alignment relationship between unpaired data and prototypes, minimizing intra-class variation, and maximizing interclass distance, our OTPAL tackles the partially aligned problem caused by unlabeled unpaired data (Fig. 1 (c )) from the perspective of an optimal transport strategy, leveraging the inherent correlation among multi-modal data to construct the effective transport cost, and progressively transport unlabeled unpaired samples into the correct prototype at the minimum cost. Finally, to address the partial alignment problem caused by unlabeled incomplete data (Fig. 1 (d) ) and further extend the application of partially aligned cross-modal retrieval, we construct a prototype-guided neighborbased feature completion method, which can perform feature reconstruction based on the correlation between missing features and neighbor features. The major contributions of this paper can be summarized as follows:• We clarify two key challenges in enhancing the robustness of cross-modal retrieval systems in practical applications. Furthermore, we propose a prototype-based optimal transport learning strategy for partially aligned cross-modal retrieval.• We propose a novel framework for partially aligned crossmodal retrieval, which minimizes the distance between samples and their corresponding prototypes, thereby enhancing modality discriminability and invariance. • We design a scalable cross-modal prototype and neighbor completion method to address incomplete partially aligned intractable scenarios. • Experimental results on four benchmark datasets demonstrate the effectiveness of our method, which can handle cross-modal retrieval tasks in various complex scenarios. 
related work: 
 Cross-modal retrieval (CMR) methods can be roughly divided into two groups: traditional multimodal representation learning methods and deep multimodal representation learning methods.Traditional CMR methods primarily learn linear or simple nonlinear mappings through statistical analysis. Canonical Correlation Analysis (CCA) [8] is one of the most typical unsupervised subspace learning methods, which leverages canonical correlation analysis to maximize pairwise correlations between two sets of heterogeneity. Many CCA-based extensions and similar methods have been proposed, such as Kernel Canonical Correlation Analysis (KCCA) [36] and Partial Least Squares (PLS) [28] . To learn the common space with semantic information, a joint representation learning (JRL) [44] is proposed to jointly explore the correlation and semantic information in a unified optimization framework. To exploit the correlations on semi-supervised data, Generalized Semisupervised Structured Subspace Learning (GSS-SL) [45] is proposed to take the label space as a linkage to model the correlations among different modalities.Deep learning CMR methods exploit the power of deep neural networks to capture non-linear relationships. DAVAE [14] proposes dual-aligned variational autoencoders to learn latent common representations in the view of distribution level and semantic level. To learn intra-modality and inter-modality representation correlations, several cross-modal retrieval methods [19, 21, 22] have been proposed. More recent works [29, 34, 42, 43, 47] exploit pair-wise label and class-wise information to learn modality invariance and discriminability. PAN [43] learns a unified prototype for each semantic category and uses them as anchors to learn cross-modal representations. Cross-Modality Cross-Instance Contrastive Learning for Cross-Media Retrieval (C 3 CMR) [34] proposes intra-modal and inter-modal contrastive learning to capture data associations and enhance the discriminative capability of features. Incomplete Cross-Modal Retrieval with Deep Correlation Transfer (ICMR-DCT) [29] proposes to model incomplete multi-modal data and dynamically capture adjacency semantic correlation for cross-modal retrieval.To balance the conflict between the performance and cost of labeling data, some semi-supervised works [12, 17, 18] are proposed. Label Prediction Framework (LPF) [18] predicts the labels of the unlabeled data utilizing complementary information from both modalities for semi-supervised cross-modal retrieval. Semi-Supervised Multi-Modal Learning with Balanced Spectral Decomposition (SMLN) [12] correlates different modalities by capturing the intrinsic structure and discriminative correlations of multimedia data. However, the role of semantic information is not considered. Self-Supervised Correlation Learning (SCL) [17] leverages label prediction and class-aware contrastive learning to learn modality invariance and discrimination. Although the above works perform well on semi-supervised data, it is difficult to extend to partially aligned data.Optimal Transport Strategy (OT) was originally proposed to depict the distance between two probability distributions, which is often used to find correspondences with learnable features or measure distribution distances. It has been applied to many fields, such as domain adaptation [5] , sequence alignment [46] , vision and language [2] . Cuturi [6] first proposed to use the Sinkorn algorithm to compute an approximate transport coupling with an entropic regularization. This method is lightspeed and can handle large-scale problems efficiently. Unbalanced Optimal Transport [1] (UOT) is proposed to relax equality constraint and the corresponding optimization problem can be reformulated as a non-negative 
proposed method: 
 The purpose of this paper is to train a model that can effectively address the two types of partially aligned cross-modal retrieval we introduce. To this end, we propose that the Optimal Transportbased Prototype Alignment Learning (OTPAL) model (Fig. 3 ) consists of three modules: 1) Multimodal Classification and Invariance Learning (MCIA) in Section 3.2 can help the model capture basic discriminant representation and modality invariance. 2) Dual Optimal Transport-based Prototype Alignment (DOTPA) in Section 3.3 can effectively establish the correspondence between unlabeled unpaired images and texts. 3) Prototype-based Neighborhood Feature Completion (PNFC) in Section 3.4 aims to generate incomplete features to maintain the semantic consistency of multimodal data.Notations. Different from traditional cross-modal retrieval, we propose a more practical task: partially aligned cross-modal retrieval, aimed at addressing the challenges of partially unaligned and partially incomplete multimodal data commonly encountered in real-world scenarios. Next, we formulate the partially aligned problem concerning the image and text modalities. We assume that two types of partially aligned data are illustrated in Fig. D 𝑡 = {𝑥 𝑡 𝑖 } 𝑁 𝑡 𝑖=1denotes only text modal data, while the corresponding image is inaccessible. 𝑁 𝑣 and 𝑁 𝑡 represent the number of image-only samples and text-only samples respectively. The goal of OTPAL is to establish a common space for partially aligned data. In this space, the representations of images and texts can be better semantically aligned.We first explore a type of partially aligned cross-modal retrieval method that we propose, shown in Fig. 2 (a ). In the labeled data in Fig. 2 (a), images and texts are paired with semantic labels oneto-one. However, in the unlabeled data, image samples and text samples are not aligned (unpaired) and lack semantic information. Next, we specify our proposed method. Due to the heterogeneous properties of different modalities, cross-modal retrieval first learns a common representation space. Thus, the dual-stream backbones (e.g., CLIP-RN50 [24] and CLIP-Xformer [24] ) 𝑓 𝑣 (.) and 𝑓 𝑡 (.) are used to extract the image 𝑥 𝑣 𝑖 and text 𝑥 𝑡 𝑖 modal features, and then project the features from different modalities into a modality-shared feature space formulated as: EQUATIONEQUATIONwhere 𝐶𝐿𝐼𝑃 𝑣 (.) and 𝐶𝐿𝐼𝑃 𝑡 (.) define visual and textual encoders.To generate a common representation space, two modality-specific multi-layer perception (MLP) 𝑓 𝑣 (.) and 𝑓 𝑡 (.) are utilized, where 𝜃 𝑣 and 𝜃 𝑡 are trainable parameters of modality-specific multi-layer perception. Finally, we obtain the representation features 𝑧 𝑣 𝑖 and 𝑧 𝑡 𝑖 projected on a common subspace. Multimodal Classification and Invariance Learning (MCIA) over the labeled data can help the model learn basic discriminative representation ability. At first, we design two modality-specific classifiers based on the projected features, and then utilize cross-entropy loss and standard triplet loss [33] to optimize the network parameters expressed as follows:EQUATIONEQUATIONEQUATIONwhere ŷ𝑣 𝑖 and ŷ𝑡 𝑖 are predicted values by the two MLP classifier layers. 𝑋 𝑣 denotes the set of triplets by selecting 𝑧 𝑣 𝑖 as the anchor to find the positive text 𝑧 𝑡 + 𝑗 and the negative text 𝑧 𝑡 - 𝑗 . The same applies to 𝑋 𝑡 . |𝑋 𝑣 | and |𝑋 𝑡 | are their cardinalities.The unsupervised classification loss 𝐿 𝑢 𝑐𝑒 is defined as the crossentropy between the pseudo-labels y𝑖 and the model's predictions ŷ𝑣 𝑖 , ŷ𝑡 𝑖 in the image and text modalities, respectively:EQUATIONThe MCIA loss function (Fig. 3 ) combines 𝐿 𝑙 𝑐𝑒 , 𝐿 𝑙 𝑡𝑟𝑖 and 𝐿 𝑢 𝑐𝑒 , which helps the model to learn certain intra-modality discriminability and inter-modal alignment representations and can be represented as: EQUATIONThe aforementioned MCIA does not directly explore the alignment relationship of unlabeled unpaired data from two modalities, thus it cannot address situations where modality data is partially aligned. Next, we focus on the key challenge of the partially aligned CMR task: unlabeled unpaired image and text data association. To address such an issue, we propose a Dual Optimal Transport-based Prototype Alignment (DOTPA) method (Fig. 3 ). It simultaneously matches the unpaired images and texts with the prototypes and obtains a matching matrix. The prototype [40] can be regarded as a set of modality-shared semantic centers, and features with similar semantics have consistent prototype assignments. The shared prototype for images and texts can be expressed as:EQUATIONwhere 𝑗 denotes the index of the category. During the initialization phase, we randomly generate a set of prototypes that are learned and updated through training. We use prototypes as an intermediate bridge to find correspondences between unlabeled unpaired images and texts. Specifically, for the given 𝑁 𝑢 suppliers (unlabeled unpaired images and texts) and 𝐾 demanders (prototypes). The supplier supplies unlabeled unpaired images and texts to the demander, described as a vector a, and the demander receives unlabeled unpaired images and texts from the supplier, described as a vector b. We formulate the cross-modal prototype assignment task as an optimal transport problem. The problem is to find an optimal transportation plan 𝑃 ∈ 𝑅 𝐵×𝐾 to minimize the transport cost 𝐶, which satisfies the following equation:EQUATIONwhere ⟨𝑃, 𝐶⟩ 𝐹 is the Frobenius inner product between the cost matrix 𝐶 and the matching plan 𝑃. 𝐶 is a cost matrix where each element 𝐶 𝑖 𝑗 represents the similarity from 𝑧 𝑣 𝑖 (𝑧 𝑡 𝑖 ) to 𝑚 𝑗 , and the cost matrix for our optimal transport problem can be denoted 𝐶 𝑖 𝑗 = 1cos(𝑧 𝑣 𝑖 , 𝑚 𝑗 ). We add entropy regularization on 𝑃 as 𝐻 (𝑃) = 𝑖 𝑗 𝑃 𝑖 𝑗 log 𝑃 𝑖 𝑗 . This ensures that 𝑃 is not over-concentrated on a few elements. Finally, the solution of Eq. ( 9) can constrain a transportation polytope:EQUATIONwhere 𝑃 𝑖 𝑗 means the transport plan between the 𝑖 -𝑡ℎ unlabeled unpaired sample and the 𝑗 -𝑡ℎ prototype, and 𝑃 contains all nonnegative 𝐵 × 𝐾 elements, with row and column sums equal to a and b, respectively. We preserve the assignment matrix 𝑃 * and the solution of the transportation polytope, solved efficiently using the Sinkhorn-Knopp algorithm [6] , which can be written as follows:EQUATIONwhere 𝑢, 𝑣 are row and column normalized vectors and can be calculated through the iterative Sinkhorn-Knopp algorithm [6] .To minimize intra-class and maximize inter-class variations, and learn discriminative representations, each sample should be close to the prototype to which it belongs. Specially, for each feature 𝑧 𝑟 𝑖 , 𝑟 = {𝑣, 𝑡 }, we can calculate the softmax of its similarity to the corresponding prototype, formulated as follows:EQUATIONwhere 𝑠𝑖𝑚(.) represents the cosine similarity, 𝑇 is the temperature factor, and 𝐾 is the number of shared prototypes. We then definePartially Aligned Cross-modal Retrieval via Optimal Transport-based Prototype Alignment Learning the data-to-proxy alignment loss via the cross-entropy loss as:EQUATION)where 𝐾 is the number of classes and 𝑁 𝑙 denotes the number of labeled data. Analogously, to effectively leverage a large amount of unlabeled data, we generalize Eq. ( 13) to unlabeled unpaired image and text data, establishing their matching associations with shared semantic prototypes, which can be calculated as follows:EQUATIONwhere 𝑃 𝑖 𝑗 is the optimal transport plan obtained by Eq. ( 9). To obtain a more reliable prototype assignment, we use a prototypebased reliability measurement method. The reliability of the 𝑃 𝑖 𝑗 is measured according to the similarity between the sample and its class prototype, and we select the threshold of unpaired image and text similarity 𝑅(𝑃) to be greater than 𝜏 to train the DOTPA module, where 𝑅(𝑃 𝑖 𝑗 ) = 𝑑 (𝑧 𝑟 𝑖 , 𝑚 𝑗 ), 𝑟 = {𝑣, 𝑡 }. The DOTPA method is well-suited for our proposed partially aligned cross-modal retrieval for the following reasons: (1) We align unlabeled unpaired images and texts with the prototypes to which they belong. Compared with directly aligning unlabeled unpaired images with texts, it can effectively reduce storage space and calculation amount.(2) Prototype-based alignment loss can further reduce intra-class variation and increase inter-class distance, which is conducive to learning semantic discriminative representations on both labeled and unlabeled data.As an extension of this paper and equally important, we delve into another partially aligned cross-modal retrieval method, as illustrated in Fig. 2 (b ). The unlabeled incomplete data consists of two types: solely image data where the corresponding text is inaccessible, and solely text samples where the corresponding image samples are inaccessible. To address the aforementioned issue of unlabeled incomplete data, we propose the Prototype-based Neighborhood Feature Completion (PNFC) method (Fig. 3 ) to reconstruct the missing modality features and further improve the robustness of partially aligned cross-modal retrieval. The idea behind PNFC is that instances from similar semantics are expected to be consistent in modality-shared space. By incorporating neighborhood feature information with semantic prototypes, both modality complementary information and prototype semantic information can be preserved in the recovered data. To reconstruct the missing text features 𝑧 𝑡 𝑖 for available image features 𝑧 𝑣 𝑖 and preserve semantic consistency between incomplete modality data, we first design 𝑁 𝑘 (𝑧 𝑣 𝑖 ) as a list of the 𝑘 nearest neighbors in the corresponding image modality, by using 𝑧 𝑣 𝑖 as the query to rank their distances as:EQUATIONwhere 𝑘 is a hyper-parameter. We transfer the nearest neighbors' relation to text modality and can infer missing text feature neighbors 𝑁 𝑘 ( 𝑧 𝑡 𝑖 ) = 𝑧 𝑡 1 , ..., 𝑧 𝑡 𝑘 from 𝑁 𝑘 (𝑧 𝑣 𝑖 ). Since the prototypes are trained with the discrimination loss, they can provide shared semantic information across modalities. Thus, we leverage the prototype and nearest neighbor features to complete the missing text features:EQUATIONwhere 𝜃 𝑊 is the trainable parameters. 𝜑 denotes the cross-modal self-attention encoder that attends to the correlations between the complete modality feature and neighbor samples. The self-attention encoder can be expressed as:EQUATIONEQUATIONEQUATIONwhere [32] . Finally, we perform a fullyconnected decoder to generate missing feature as follows:EQUATIONwhere 𝐷 (.; 𝜃 𝐷 ) denotes the fully-connected decoder. To maintain representation consistency between the completion features and the corresponding available features, we propose the missing modal feature completion loss computed as:EQUATIONOverall Loss Function. The overall objective 𝐿 is combined with three losses, including the discrimination loss 𝐿 𝑀𝐶𝐼𝐴 , prototype alignment loss 𝐿 𝑃 and missing feature completion loss 𝐿 𝑀𝐹𝐶 :EQUATION)where 𝛼 is the hyper-parameter to balance the different losses. The prototype alignment loss 𝐿 𝑃 is the sum of 𝐿 𝑙 𝑝𝑣 , 𝐿 𝑙 𝑝𝑡 , 𝐿 𝑢 𝑝𝑣 and 𝐿 𝑢 𝑝𝑡 . 
experiments 4.1 datasets and new data splitting: 
 We undertake comprehensive experiments on four widely used cross-modal retrieval benchmarks including NUS-WIDE-10K [4] , Wikipedia [26] , XmediaNet [22] , and Pascal-Sentence [25] . Data type-1: In our proposed partially aligned cross-modal retrieval, we randomly select 20%, 40% and 60% instances from the multi-modal training set as labeled aligned data, while the rest are designated as unlabeled unaligned data (Fig. 2 (a) ). In the labeled aligned data, images and texts are aligned and annotated with class information. Meanwhile, in unlabeled unaligned data, images and texts are not aligned. Note that having only 20% labeled aligned and 80% unlabeled unaligned training data poses the greatest challenge in partially aligned cross-modal retrieval. Data type-2: As shown in Fig. 2 (b), in unlabeled unaligned data, the image sample is available but the corresponding text is missing, or the text is accessible but the corresponding image is missing. For example, we denote 10% as labeled aligned data and 90% as unlabeled unaligned data, where 90% of unlabeled unaligned data comprises 45% image-only samples and 45% text-only samples represented as (10%L, 45%I, 45%T). Similarly, we also define various other ratios. Note that (10%L, 45%I, 45%T) is also the most challenging. We follow the image and text encoder networks utilized by CLIP [24] methods. Specifically, the visual encoder utilizes ResNet50 [9] as its base framework, integrating the style of the Transformer architecture [32] . Through a linear projection layer, it produces 1024-dimensional image representations. For the text transformer encoder, we first employ a lower-cased Byte Pair Encoding (BPE) with a vocabulary size of 49,152 words [27] to tokenize the textual descriptions. The textual descriptions are enclosed within [SOS] and [EOS] tokens indicating the beginning and end of the text sequence.The tokenized text is then fed into the transformer's module, and the textual features at the [EOS] position are normalized and processed by using a linear projection layer to output 1024-dimensional textual representations. To learn modality-shared features and obtain classification predictions for image and text modalities, we employ four fully connected layers with GELU activation function [10] for each modality, i.e., 1024 → 2048 → 1024 → 𝐾. The entire network is optimized by the Adam update rule [15] with the learning rate 10 -3 and batch size 128. The training epoch is set to 400 for the XmediaNet dataset and 200 for the other datasets. In addition, we set the dropout ratio to 0.5, and the early stop to 20.For the hyper-parameters in OTPAL, we set 𝛼, 𝛽, 𝜏 to {10, 5, 0.9} on XmediaNet, {15, 1, 0.5} on NUS-WIDE-10K, Wikipedia and Pascal Sentence. For other hyper-parameters, 𝑇 = 0.5, and the number of nearest neighbors 𝑘 is set to 3 and 5 for Pascal-Sentence, and XMediaNet, respectively. The proposed model is trained on one 24GB Nvidia RTX A5000 GPU in PyTorch. We evaluate the retrieval performance with the mean Average Precision (mAP), which is a widely-used performance evaluation criterion in cross-modal retrieval tasks [11, 13, 19] . In our experiments, we take images and texts as queries to calculate the cosine similarity to retrieve the relevant text and image samples, which can be denoted as Image2Text (I2T) and Text2Image (T2I). To evaluate the effectiveness of the OTPAL, we compare the proposed method with nine representative cross-modal retrieval baseline methods: unsupervised cross-modal methods (US-CMR): DCCA [39] and DCCAE [35] , supervised methods (S-CMR): DSCMR [47] , DAVAE [14] PAN [43] , C 3 CMR [34] , and ICMR-DCT [29] , semi-supervised methods (SS-CMR): SMLN [12] , SCL [17] and our method OTPAL. Notice that all methods use the same feature encoder as our method.Given that OTPAL is designed for partially aligned cross-modal retrieval, its performance is evaluated based on data type-1, where a certain proportion of labeled image-text pairs is retained, while the rest is randomly shuffled to construct unlabeled unpaired data. Tables 1-3 present the retrieval mAP scores under varying label ratios on three datasets, respectively. We use "-" for methods that do not provide source codes or results. As shown in these results, we can draw the following conclusions: 1) Training with partially aligned data may significantly harm the performance of cross-modal retrieval. As the proportion of labeled paired data decreases, the mAP scores of these methods will drop rapidly. 2) When labeled paired data is scarce, the retrieval performance of unsupervised methods does not surpass that of supervised methods. This is because, in partially aligned cross-modal retrieval, unsupervised methods are unable to utilize unlabeled aligned data to learn aligned representations between paired data. 3) Although the semi-supervised methods SMLN and SCL can achieve good performance, their improvement is limited. Mainly because they do not sufficiently exploit the discrimination and intrinsic correlations when encountering unlabeled unpaired data. Hence, their performance may be worse than some supervised methods. 4) Our method outperforms all existing SOTA methods on all datasets with varying proportions of labeled paired data, demonstrating the outstanding robustness of OTPAL in handling partially aligned data. Moreover, when the labeled paired data ratio is lower, the improvement of OTPAL is more evident, which can be observed in XmediaNet Table 3 . In conclusion, the effectiveness of our approach can be attributed to the intrinsic connection established between labeled paired data and unlabeled unpaired data, which can further minimize the intraclass distance and improve model discrimination through optimal transport-based prototype alignment.To validate the effectiveness of each module of our method, we conduct ablation studies for these modules, including the Dual Optimal Transport-based Prototype Alignment (DOTPA), and Prototypebased Neighborhood Feature Completion (PNFC). Experiments are conducted on Pascal-Sentence, NUS-WIDE-10K, and XmediaNet. Effectiveness of DOTPA. Specifically, we ablate the contributions of three key components in our OTPAL, i.e., 𝐿 𝑀𝐶𝐼𝐴 , 𝐿 𝑙 𝑝𝑣 + 𝐿 𝑙 𝑝𝑡 , and 𝐿 𝑢 𝑝𝑣 + 𝐿 𝑢 𝑝𝑡 . All compared methods are trained under 20% labeled paired data using the same settings as our OTPAL. The experimental results are reported in Table 5 . From the results, the following observations can be drawn: 1) We can observe that the full OTPAL achieves the best performance, showing that all three components are important to improve the performance towards partially aligned data. Furthermore, in the visualization comparison between total OTPAL and Baseline (𝐿 𝑀𝐶𝐼𝐴 ), we can observe that the former converges to the best performance faster than the baseline in Fig. 4 (PNFC). By comparing OTPAL and OTPAL1, it is shown that exploring incomplete modality data is very important for incomplete part-aligned cross-modal retrieval. By comparing OTPAL with other methods, it shows that our method leverages cross-modal neighbors guiled by shared prototype information to generate completion features, alleviate the impact of performance degradation caused by missing data, and further improve the robustness of our OTPAL.The main hyper-parameters involved are 𝛼, 𝛽 and 𝜏. The objective function contains two parameters 𝛼 and 𝛽, which control the contributions of different loss functions. We evaluate their influences on the XmediaNet dataset, and report the results in Fig. 5 (𝑎) -(𝑐). These experiments are conducted under the 20% labeled paired multi-modal data scenario. Specifically, to accurately reflect the performance variations affected by hyper-parameters, we verify the effect of one hyper-parameter while keeping other hyper-parameters unchanged. For 𝛼, we set the range of the hyper-parameter in Eq. ( 22) as {0.001, 0.01, 0.1, 1, 10, 100, 1000, 10000}, and we can observe that the mAP scores in both directions tend to be stable when 1 < 𝛼 < 10000. This also proves that OTPAL can keep high performance over a wide range of 𝛼. For 𝛽 in Eq. ( 7), the changing trend of mAP score is opposite to 𝛼. The performance is stable in a larger range of {0.001 100}, and it drops sharply when it exceeds 100. For 𝜏 below Eq. ( 14), the mAP first increases with the growth of 𝜏, and then begins to decrease after 𝜏 surpasses a threshold 0.9, which shows that a suitable threshold can effectively select reliable prototypes that perform accurate alignment between instances and prototypes. In conclusion, OTPAL is relatively insensitive to the selection of 𝛼 and 𝛽, but sensitive to the choice of 𝜏.We visualize the learned visual and textual representations produced by the modality-specific layer using t-SNE in Fig. 6 . The original image and text features are from the testing set of NUS-WIDE-10K. We contrast t-SNE 1024-dimensional embeddings of the original features (Fig. 6 (𝑎) -(𝑏)) and the learned visual and textual features (Fig. 6 (𝑐) -(𝑒)). We can clearly see that the original features of different modalities present different spatial distributions, and instances from different categories cannot be separated (Fig. 6 (𝑎) -(𝑏)). Besides, we can observe that both the image and text features of OTPAL exhibit consistent semantic distribution and are aligned closely (Fig. 6 (𝑐) -(𝑒)), which validates the effectiveness of our method in learning modality invariance and semantic discrimination. This can be attributed to the fact that shared prototypes can establish intrinsic correlations between images and texts and better characterize global semantic information (Fig. 6 (𝑐) stars). 
conclusion: 
 To tackle the challenges of partially aligned cross-modal retrieval in real-world scenarios, we propose a novel optimal transport-based prototype alignment learning for robust cross-modal retrieval. In particular, to learn modality discriminative and invariant representations while minimizing intra-class distance, we propose a dual optimal transport-based prototype alignment strategy. Further, to deal with the problem of incomplete partially aligned data, a prototype-based neighbor feature completion is proposed, which can extend our method to more realistic scenarios. Extensive experiment results demonstrate the superiority of our model in partially aligned cross-modal retrieval by comparing it with SOTA methods.  
 
 Appendix:  


Decision: The paper introduces a novel approach, called Optimal Transport-based Prototype Alignment Learning (OTPAL), for partially aligned cross-modal retrieval. The initial ratings were two Borderline Reject and two Borderline Accept. After the rebuttal, two reviewers changed their recommendation ratings, and all reviewers reached a consensus to Borderline Accept. The authors are suggested to include a more comprehensive review on incomplete multi-modal/view learning and unaligned multi-modal/view learning in their final version.

Reviews: review: paper_summary: paper_summary: This study develops a method called Optimal Transport-based Prototype Alignment Learning (OTPAL). It classifies these multi-modal pairs into two categories: unlabeled unpaired data and unlabeled incomplete data. To solve the optimal transport problem across different modalities (such as images and text) and their prototypes, the authors put forward a method named Dual Optimal Transport-based Prototype Alignment (DOTPA), which promotes alignment across these modalities. Addressing the issue of unlabeled incomplete data, the study proposes a Prototype-based Neighborhood Feature Completion (PNFC) approach. This method utilizes prototypes to approximate and restore the semantics of the absent modalities.
strength_weakness: strength_weakness: This document thoroughly describes the condition of partially aligned multi-modal pairs in real-world settings and suggests specific strategies to overcome the obstacles presented by each type of partially aligned data.
limitations: limitations: - The authors may violate the submission policy in the block of the bottom left corner of page 1.
- The authors do not provide experimental comparison of 100% labeled data, which makes it difficult to conclude that the improvement of experimental results is from the contribution of this paper, rather than the backbone.
- The baselines are out of date. The authors should provide the baselines more recently.
paper_summary: paper_summary: The paper introduces a novel approach for partially aligned cross-modal retrieval called Optimal Transport-based Prototype Alignment Learning (OTPAL). This method tackles the challenge of unannotated, unaligned cross-modal samples by establishing matching correlations between shared prototypes and samples using the optimal transport algorithm. Additionally, OTPAL introduces a scalable prototype-based neighbor feature completion method to handle incomplete multimodal data, thereby enhancing the robustness of cross-modal retrieval systems in practical applications. While the effectiveness of OTPAL is demonstrated, the paper lacks sufficient discussion on related works in the area of partially unaligned or incomplete multi-modal learning.
strength_weakness: strength_weakness: 1. The paper presents a novel setting for the cross-modal retrieval task.
2. Extensive experimental results verify the effectiveness of the proposed method.
limitations: limitations: 1. My major concern is the missing connections with the existing works on partially unaligned and incomplete multi-modal learning studies which are highly related to this work. For example, [A] for partially view-aligned learning, [B for incomplete multi-view learning, [C] for the unified solution of partially aligned and incomplete multi-view learning. [A] Partially View-aligned Representation Learning with Noise-robust Contrastive Loss [B] Incomplete Multi-view Clustering via Prototype-based Imputation, IJCAI2023 [C] Robust Multi-view Clustering with Incomplete Information.
2. The optimal transport algorithm may involve high computational complexity. It is recommended to analyze the time complexity or present the time cost for clarity.
I will consider raising my rating if the authors' feedback addresses my concerns.
paper_summary: paper_summary: ● This paper introduces an Optimal Transport-based Prototype Alignment Learning (OTPAL) to encounter the problem of partially alignment multi-modal pairs in real-world scenarios.
● The paper categorizes the partially aligned multi-modal pairs into unlabeled unpaired, and unlabeled incompleted data.
● The authors propose a dual optimal transport-based prototype alignment (DOTPA) to resolve the optimal transport problem between data (image or text) and prototypes, enabling cross-modal alignment.
● For the unlabled incomplete data, this paper introduces a prototype-based neighborhood feature completion (PNFC) method to generate the representation of the missing modality. The prototypes are adopted to provide similar semantics to recover the missing modality.
strength_weakness: strength_weakness: ● This paper explicitly illustrates the situation of the partially aligned multi-modal pairs in the real world and proposes corresponding approaches to resolve the challenges in each type of partially aligned data.
● The methods are clear and easy to follow.
● The experiments are sufficient and the proposed methods maintain a considerable performance.
limitations: limitations: ● In section 2, it lacks of the review of the incomplete multi-modal learning scenarios.
● The compared baselines are out of date. The authors should compare with the recent research on partially aligned cross-modal retrieval.
● The author should provide the experiment results on 100% labeled data to illustrate the necessity of this paper.
● The font size in Fig.5 is too small to read.
paper_summary: paper_summary: This paper investigates the problem of cross-modal retrieval with unlabeled and unpaired data, and proposes a reasonable approach OTPAL for this problem.
strength_weakness: strength_weakness: 1. The motivation is clear and the method is reasonable.
2. The experiments are comprehensive and the results demonstrate improved performance.
limitations: limitations: 1. Lack of latest and sota baseline methods such as RONO, CLF, ALGCN, GNN4CMR, etc.
2. The proposed method is reasonable but not novel. It seems that the overall framework is just a combination of some well-known techniques.