Graph Convolutional Semi-Supervised Cross-Modal Hashing

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 09:22:58

Abstract: Cross-modal hashing encodes different modalities of multi-modal data into a low-dimensional Hamming space for fast cross-modal retrieval. Most existing cross-modal hashing methods heavily rely on label semantics to boost retrieval performance; however, semantics are expensive to collect in real applications. To mitigate the heavy reliance on semantics, this work proposes a new semi-supervised deep cross-modal hashing method, namely, Graph Convolutional Semi-Supervised Cross-Modal Hashing (GCSCH), which is trained with limited label supervision. The proposed GCSCH first generates pseudo-multi-labels of the unlabeled samples using the simple yet effective idea of consistency regularization and pseudo-labeling. GCSCH designs a fusion network that merges the two modalities and employs Graph Convolutional Network (GCN) to capture semantic information among ground-truth-labeled and pseudo-labeled multi-modal data. Using the idea of knowledge distillation, GCSCH employs a teacher-student learning scheme that can successfully transfer knowledge from the fusion module to the image and text hashing networks. Empirical studies on three multi-modal benchmark datasets demonstrate the superiority of the proposed GCSCH over state-of-the-art cross-modal hashing methods with limited label supervision.

Main paper: 
 related work: 
 According to the amount of label semantic information used, crossmodal hashing [46] can be roughly divided into three categories: supervised cross-modal hashing [1, 26, 27, 31] , unsupervised crossmodal hashing [22, 43] , and semi-supervised cross-modal hashing [4, 24, 45] . Supervised cross-modal hashing methods indeed rely on the availability of labeled multi-modal data, which ensures that the learned hash codes are highly discriminative. This approach use the supervisory signals provided by the labels to guide the learning process, ensuring that the hash codes generated for different modalities are semantically consistent and can effectively capture the relevant information for discrimination tasks. Discrete Latent Factor Hashing (DLFH) [12] utilizes negative log-likelihood of crossmodal similarity, and optimizes in a discrete scheme that directly learns hash code without continuous relaxation. DLFH achieves remarkable accuracy and trains much faster than some relaxationbased hashing methods. By leveraging the power of deep learning, deep cross-modal hashing integrates feature learning and hash code learning into a unified framework. For instance, Deep Cross-Modal Hashing (DCMH) [11] is among the first end-to-end learning frameworks with DNNs for cross-modal hashing. DCMH learns deep hash functions for each modality using DNNs and minimizes negative log-likelihood of cross-modal similarity. In contrast to supervised cross-modal hashing, unsupervised cross-modal hashing does not require label information that is often laborious to collect in real-world applications. Unsupervised deep cross-modal hashing (UDCMH) [34] combines deep learning and matrix factorization with binary latent factor models for multi-modal data retrieval. However, the performance of unsupervised cross-modal hashing is limited due to the lack of label supervision.As labeled multimodal data is expensive to obtain in real-world scenarios, more efforts have been recently made towards semisupervised cross-modal hashing, which considers both a small amount of labeled and a large amount of unlabeled multimodal data. For instance, Multi-view Graph Cross-modal Hashing (MGCH) [24] learns hash code in a semi-supervised manner using the outputs of multi-view graphs processed by a graph-reasoning module. Modalityspecific and Cross-modal GCN (MCGCN) [33] employs two modalityspecific channels and one cross-modality channel to learn modalityspecific and shared representations for each modality respectively, and performs semantic information propagation from labeled data to unlabeled data via GCN. Semi-supervised Semi-paired Crossmodal Hashing (SSCH) [45] performs an alignment-free pseudolabeling process that can strengthen semantic preservation to train effectively and efficiently. Semi-supervised Knowledge Distillation for Cross-modal Hashing (SKDCH) [25] utilizes teacher-student optimization to propagate knowledge, and improves triplet ranking loss to better mitigate heterogeneity gap. The existing semisupervised cross-modal hashing methods mainly consider semantics of labeled and structure of unlabeled multi-modal data, however semantics among unlabeled data have not been exploited effectively. In this work, we propose to exploit semantics among unlabeled multi-label data and employ such semantics to train multi-modal hashing networks. 
approach: 
 This section presents the details of the proposed Graph Convolutional Semi-Supervised Cross-Modal Hashing (GCSCH), including problem setup, formulation, and out-of-sample extension.This work focuses on the image-text cross-modal retrieval. Assume that we are given a multi-modal dataset with 𝑛 image and text samples, where 𝑛 𝑙 samples are labeled and 𝑛 𝑢 samples are unlabeled, and we have 𝑛 = 𝑛 𝑙 + 𝑛 𝑢 . Specifically, the labeled images and texts are denoted as X 𝑙 ∈ R 𝑛 𝑙 ×𝑑 𝑥 and Y 𝑙 ∈ R 𝑛 𝑙 ×𝑑 𝑦 respectively, and their labels are denoted as L 𝑙 ∈ {0, 1} 𝑛 𝑙 ×𝑐 , where 𝑐 is the number of classes, 𝐿 𝑙 𝑖 𝑗 = 1 if the 𝑖-th sample belongs to the 𝑗-th class and 𝐿 𝑙 𝑖 𝑗 = 0 otherwise. In addition, the unlabeled images and texts are represented as X 𝑢 ∈ R 𝑛 𝑢 ×𝑑 𝑥 and Y 𝑢 ∈ R 𝑛 𝑢 ×𝑑 𝑦 respectively. The whole image and text modalities are represented as X = X 𝑙 , X 𝑢 ∈ R 𝑛×𝑑 𝑥 and Y = Y 𝑙 , Y 𝑢 ∈ R 𝑛×𝑑 𝑦 respectively. The goal of the proposed GCSCH is to learn the deep cross-modal hashing model and hash code B ∈ {0, 1} 𝑛×𝑑 that supports efficient large-scale image-text cross-modal retrieval, where 𝑑 is hash code length.The proposed GCSCH is illustrated in Figure 1 . As shown in Figure 1 , GCSCH comprises three components including image/text network, consistency regularized pseudo-labeling module, and GCN fusion module.Network. For image modality, the proposed GC-SCH employs Convolutional Neural Network (CNN) as the backbone to extract image features. Specifically, the proposed GCSCH obtains the high-level deep features of labeled and unlabeled images:EQUATIONwhere H 𝑙 𝑋 ∈ R 𝑛 𝑙 ×𝑑 and H 𝑢 𝑋 ∈ R 𝑛 𝑢 ×𝑑 denote labeled and unlabeled image features respectively, 𝑓 𝑋 (•) represents the image network, Θ 𝑋 denotes its network parameter. The feature of the whole image modality is represented by H 𝑋 = H 𝑙 𝑋 ; H 𝑢 𝑋 ∈ R 𝑛×𝑑 . For text modality, GCSCH uses Deep Neural Network as the backbone to extract text features. Specifically, the proposed GCSCH obtains the high-level deep features of labeled and unlabeled texts:EQUATIONwhere H 𝑙 𝑌 ∈ R 𝑛 𝑙 ×𝑑 and H 𝑢 𝑌 ∈ R 𝑛 𝑢 ×𝑑 denote labeled and unlabeled text features respectively, 𝑓 𝑌 (•) represents the text network, Θ 𝑌 denotes its network parameter. The feature of the whole text modality is represented byH 𝑌 = H 𝑙 𝑌 ; H 𝑢 𝑌 ∈ R 𝑛×𝑑 .3.2.2 Consistency Regularized Pseudo-labeling Module. Leveraging unlabeled data to improve performance is key for semi-supervised learning (SSL). Instead of developing complex models, this work employs a simple yet effective module to accurately generate pseudomulti-labels for unlabeled multi-modal data with consistency regularization [41] . In this work, following [36] , we combine pseudolabeling [13] and consistency regularization on two types of augmentations. We perform two types of feature augmentations: strong and weak, denoted by Λ(•) and 𝜆(•) respectively. Specifically, we first concatenate the image and text features, and apply strong and weak augmentations on fused features using dropout with different parameters [20] .The loss in pseudo-labeling module includes two terms, i.e., a supervised loss J 𝐶 𝑙 applied on labeled data, and an unsupervised loss J 𝐶 𝑢 applied on unlabeled data. Specifically, the supervised loss is defined as the following loss on weakly augmented labeled fused feature:EQUATIONwhere 𝑓 𝐶 denotes a neural network for classification, and Θ 𝐶 is its network parameter, L 𝑙 𝑖 is the multi-label vector of the 𝑖-th labeled sample, 𝐶𝐸 denotes the widely-used standard cross-entropy loss.For the unlabeled samples, we apply the idea of pseudo-labeling to first compute the multi-label distribution of strong augmented fused feature of the unlabeled samplesL 𝑢 = 𝑓 𝐶 Λ([H 𝑢 𝑋 , H 𝑢 𝑌 ]), Θ 𝐶 . We then convert such multi-label distributions to their hard pseudomulti-labels L𝑢 , and the 𝑗-th label of the 𝑖-th unlabeled sample L𝑢 𝑖 𝑗 is set to 1 if it is larger than 𝜏, and set to 0 if it is smaller than 1 -𝜏. We only retain pseudo-multi-labels of unlabeled samples whose label distributions all fall into such two regions defined by 𝜏 for training. With the popular idea of consistency regularization, we assume a good model should output similar predictions when fed different augmentations versions of one sample. Therefore, the unsupervised loss is defined as follows:EQUATIONwhere function𝐼 (l, 𝜏) = 𝑐 𝑖=1 1(𝑙 𝑖 ∈ [0, 𝜏] ∪ [1 -𝜏, 1]) is used to select unlabeled samples with high-confidence prediction. We minimize the final loss that is defined as J 𝐶 𝑢 + J 𝐶 𝑙 to train classifier in pseudo-labeling module, and further employ such learned classifier to generate pseudo-multi-labels of all the unlabeled samples L 𝑢 ∈ {0, 1} 𝑛 𝑢 ×𝑐 by setting 𝜏 to 0.5. To this end, we have labeled all the unlabeled samples, and the labels of all the samples are defined asL = [L 𝑙 ; L 𝑢 ].The purpose of the proposed GCSCH is to preserve semantic similarity structure among image and text modalities. To achieve this goal, GCSCH proposes the following negative log-likelihood EQUATIONwhereΩ 𝑋𝑌 𝑖 𝑗 = 1 2 (H 𝑋 ) 𝑖 (H 𝑌 ) ⊤𝑗 , and (H 𝑋 ) 𝑖 and (H 𝑌 ) 𝑗 are the 𝑖-th and 𝑗-th image and text features respectively. Specifically, based on L, we obtain semantic similarity matrix S ∈ {0, 1} 𝑛×𝑛 to characterize similarity structure between all samples, and 𝑆 𝑖 𝑗 = 1 if the 𝑖-th and 𝑗-th samples share at least one common label, and 𝑆 𝑖 𝑗 = 0 otherwise.In addition, to reduce information loss of quantizing continuous outputs of image and text networks, GCSCH minimizes the following quantization loss to enable continuous network outputs and hash codes to be close:EQUATION)where B denotes latent hash code shared by image and text modalities.to exploit complementary of multiple modalities and generate more discriminative hash code. Specifically, the proposed GCSCH concatenates the outputs of image and text networks, and further feeds the concatenation into fusion network to obtain high-quality fused feature. To further improve the quality of the fused features, the fusion module includes Graph Convolutional Network (GCN) to explore semantic information of both ground-truth and pseudolabeled samples. With this adjacency matrix, GCN is then used to fully exploit semantics of labeled samples and structure of unlabeled samples. To alleviate over-smoothing problem that often occurs in GCN, the proposed GCSCH uses the following weighting scheme:EQUATION)where S is the weighted adjacency matrix, and 𝑝 is the weight assigned to the node itself and the other nodes. The proposed GCSCH first concatenates image and text features, and feeds it into a fusion network that consists of two-layer GCN:EQUATIONwhere 𝑓 𝐹 (•) and H 𝐹 ∈ R 𝑛×𝑑 denote fusion network and fused feature respectively, Θ 𝐹 is its network parameter. The layer-wise propagation rule of GCN is defined as follows:EQUATIONwhere H (𝑙 ) and H (𝑙+1) denote the input and output of the 𝑙-th layer in fusion network respectively, Θ (𝑙 ) denotes the parameter of the 𝑖-th layer, and 𝜎 (•) denotes activation function. We define H (1) = [H 𝑋 , H 𝑌 ] and H (3) = H 𝐹 . To further improve discrimination, the proposed GCSCH has the following negative log-likelihood loss on EQUATIONwhere Ω 𝐹 𝑖 𝑗 = 1 2 (H 𝐹 ) 𝑖 (H 𝐹 ) ⊤ 𝑗 , (H 𝐹 ) 𝑖 and (H 𝐹 ) 𝑗 denote the 𝑖-th and 𝑗-th fused features respectively. To improve the discrimination of outputs of image and text networks, under teacher-student learning framework [38] , GCSCH regards fusion network and image/text network as teacher and student modules respectively, and uses fused feature generated by GCN to guide training of image and text networks. To achieve this, GCSCH minimizes the following simply but effective loss: minEQUATION)With the above loss, the outputs of the image and text networks can be close to the fused features.3.2.4 Total Loss. By combining the above four losses, i.e., J 𝑆 , J 𝐹 , J 𝐷 and J 𝑄 , we have the final objective function of the proposed GCSCH: minEQUATION)where the three parameters 𝛼, 𝛽, and 𝛾 are used to balance the importance of different losses.The proposed GCSCH presents an iterative optimization scheme to find a feasible solution. For each iteration, we first optimize Θ 𝐶 to achieve better classification results and generate pseudo-labels, and then use pseudo-labels to optimize each of Θ 𝑋 , Θ 𝑌 , Θ 𝐹 and B, while fixing the other variables. The proposed GCSCH can be trained in an end-to-end manner, and its overall training process is shown in Algorithm 1.Table 1 : mAPs of all the cross-modal hashing methods on two cross-modal retrieval tasks with respect to 30% of labeled samples. The bold and underline indicate the best and the second best respectively. Once the proposed GCSCH is trained, the learned image and text networks can be used to generate hash code of a new query. Specifically, given an image query x 𝑞 or text query y 𝑞 , its hash code can be generated as follows:EQUATIONOnce b 𝑞 is generated, it can be used for cross-modal retrieval by retrieving similar instances from database in another modality. 
experiments: 
 This section evaluates effectiveness of the proposed GCSCH by performing image-text cross-modal retrieval. The experiment is conducted on an Ubuntu Enterprise 64-bit Linux server equipped with an NVIDIA A6000 GPU. The proposed GCSCH is implemented using PyTorch.4.1.1 Datasets. The experiment is conducted on three multi-label image benchmarks, i.e., MIRFLICKR-25K [7] , NUS-WIDE [3] , and MS COCO [15] . The details of three benchmark datasets are as follows:MIRFLICKR-25K 1 [7] consists of 25,000 images collected from Flickr website. Each image is associated with several textual tags. Hence, each point is a image-text pair which is manually annotated with 24 unique labels. The text for each point is represented as a 1,386-dimensional bag-of-words (BoW) vector. The 20,015 points that have at least 1 label and 1 textual tag are selected for experiment. The 2,000 samples are randomly selected as a query set and the 1 http://lear.inrialpes.fr/people/guillaumin/data.php rest 18,015 samples are used as database. The 10,000 samples are randomly selected from database for training.NUS-WIDE2 [3] contains 260,648 web images, and some images are associated with textual tags. Each image is annotated with one or multiple labels from 81 concept labels. We select 195,834 imagetext pairs that belong to the 21 most frequent concepts. The text for each point is represented as a 1,000-dimensional bag-of-words vector. The 2,100 samples are randomly selected as a query set and the rest 193,734 samples are used as database. The 10,500 samples are randomly selected from database for training.MS COCO3 [15] is constituted by two subsets of images: a training set with 82,783 training images and a validation set with 40,504 images. In our experiments, we merge the training images and validation images and remove those samples that have no text data. Finally, 122,218 image-text pairs are left for our experiments. The text for each point is represented by a 2,000-dimensional bag-ofwords vector. The 5,000 samples are randomly selected as a query set and the rest samples are used as database. The 10,000 samples are randomly selected from the database for training.We compare the proposed GCSCH with nine stateof-the-art cross-modal hashing baselines for comparison, including three semi-supervised cross-modal hashing methods, i.e., SSCH [45] , MGCH [24] , TS3H [4] , two unsupervised cross-modal hashing method, i.e., DGCPN [43] , UCCH [22] , and four supervised crossmodal hashing method, i.e., EDMH [1] , LEMON [31] , HMAH [27] , HCCH [26] . All samples are used for training semi-supervised and unsupervised methods, while only labeled samples are used for training supervised methods. Precision-recall curves of all the cross-modal hashing methods on two cross-modal retrieval tasks with respect to 30% of labeled samples.For the proposed GCSCH, the batch size is set to 128, where the numbers of labeled and unlabeled samples are set to 80 and 48 respectively, and the number of iterations is set to 100. The five trade-off parameters, i.e., 𝛼, 𝛽, 𝛾, 𝜏, 𝑝 are set to 0.1, 0.05, 10 -9 , 0.05, and 0.3 respectively. The Adam optimizer is used for optimizing. For image modality, we employ VGG model pretrained on ImageNet as the backbone, where the classification layer is replaced by hash layer whose output dimension is set to 𝑑. For text modality, we employ text network with three fully-connected layers including a 1,024-dimensional hidden layer. The modality fusion network is a GCN network including two 1,024-dimensional graph convolutional layers and one 1,024-dimensional fully-connected layer. For all networks, ReLU is adopted as activation for hidden layers and tanh activation is used to approximate hash code. Weakly augment and strongly augment use different proportions of dropout operations.4.1.4 Evaluation Metrics. Two cross-modal retrieval tasks are used for evaluation, i.e., I→T that retrieves relevant texts in database given any image query, T→I that retrieves relevant images in database given any text query. We consider the widely used metric, i.e., mean Average Precision (mAP) to evaluate retrieval performance, which is calculated using all the samples in databases.Percentage of Labeled Samples. This section compares the proposed method with state-of-the-art hashing baselines with a small percentage of labeled samples. We set the percentage of labeled samples in training set to 30%, and report the mAPs of all the hashing methods with respect to different bits, i.e., 16, 32, 48, 64 bits in Table 1 , where bold and underline indicate the best and second best in each case. The PR curves of all the methods with respect to 32 bits are shown in Figure 2 . From Table 1 and Figure 2 , we can clearly observe that (1) The proposed GCSCH has the highest mAPs among 24 cases, and outperforms the best baselines averagely by 5.19%, 3.62%, 3.65% on MIRFLICKR-25K, NUS-WIDE, MS COCO respectively. In addition, PR curves of GCSCH are generally above those of the baselines. (2) Among the semi-supervised baselines, TS3H outperforms the other two semisupervised baselines. MGCH achieves similar performance, and SSCH underperforms in the most cases. (3) The unsupervised baselines are generally competitive to the supervised baselines in the setting of limited label information, among which HCCH performs best.Percentages of Labeled Samples. This section compares the proposed method with the semi-supervised baselines with varying percentages of the labeled samples. We set code length to 32, and vary the percentages of labeled samples from the range of [10%, 90%]. The mAPs of all the semi-supervised methods with respect to different percentages are reported in Figure 3 . As can be observed, the proposed GCSCH outperforms the semi-supervised baselines in most cases, indicating its superiority of learning from varying percentages of labels. As the percentage of labeled data increases, the mAPs of the proposed GCSCH improves stably, and consistently higher than the most baselines. Some baselines, e.g., MGCH show some performance fluctuations We visualize the learned hash code to qualitatively verify the proposed method, and compare the proposed method with the hashing baselines. We adopt NUS-WIDE for experiment, and 8,000 samples that are annotated with only one label are randomly selected and code length is set to 32. The hash codes learned by the ten hashing methods are visualized into a 2dimensional space with t-SNE [28] , as illustrated in Figure 5 . From Figure 5 , we see that visualization of the proposed GCSCH is better than the baselines, which is generally consistent with previous quantitative empirical results.We present a case study of image-text crossmodal retrieval, and compare the proposed GCSCH and three semisupervised baseline methods, i.e., MGCH, SSCH, TS3H. We adopt NUS-WIDE for experiment, and set code length and the percentage of labeled samples to 32 and 30% respectively. The top 10 retrieved results of a random image and text query pair on two cross-modal retrieval tasks are illustrated in Figure 6 . Given an image query, the corresponding images of the retrieved texts are shown to enable retrieved results to be intuitive in I→T task. The retrieved sample is marked green if it shares at least one common label with the query, and is marked red otherwise. As can be seen from Figure 6 , compared to the semi-supervised cross-modal hashing baselines, the proposed GCSCH obviously retrieves more similar samples on the two retrieval tasks. The above results qualitatively verify the effectiveness of the proposed method for image-text cross-modal retrieval. 
conclusion: 
 This work studies semi-supervised cross-modal hashing with limited semantic supervision for cross-modal retrieval, and proposes Graph Convolutional Semi-Supervised Cross-Modal Hashing (GC-SCH) to mitigate heavy reliance on semantics. Compared to existing semi-supervised cross-modal hashing, this work can generate pseudo-multi-labels of unlabeled samples using the simple yet effective idea of consistency regularization and pseudo-labeling. In addition, this work fuses image and text modalities, employs GCN to capture semantic information among ground-truth-labeled and pseudo-labeled multi-modal data, and guides training of multimodal hashing networks under teacher-student learning framework. Empirical studies on three benchmarks demonstrate the superiority of the proposed method over the state-of-the-arts in image-text retrieval with limited labels. 
: 
 rapid increase of data [21] . Hashing [30] has attracted considerable interest for large-scale retrieval due to its obvious superiority in terms of storage and computation. Hashing [5, 18, 19, 30] learns hash codes that well preserve similarity structure of original data.Hashing has been successfully applied to large-scale cross-modal retrieval by harvesting its benefits. Cross-modal hashing [47] maps multi-modal data into a common Hamming space, where efficient retrieval is performed. The shallow cross-modal hashing methods [2, 5, 6, 10, 16, 29, 42, 47] typically extract the hand-crafted or deep features using pre-trained network and then learn hash codes based on the extracted features of multi-modal data. The deep crossmodal hashing [8, 9, 14, 17, 32, 35, 37, 39, 40] has been developed to jointly perform feature learning and latent hash code learning in an end-to-end manner, and has shown superior to shallow crossmodal hashing. Deep cross-modal hashing [9] directly takes raw multi-modal data as inputs, e.g., raw image, bag-of-word text, and transforms them into hash codes using DNNs while incorporating semantic supervision. Typically, a similarity matrix is constructed based on whether two samples share common labels to indicate pairwise semantics. However, label semantics are expensive to obtain in real-world tasks, limiting the widespread application of deep supervised cross-modal hashing on cross-modal retrieval.Semi-supervised cross-modal hashing [25, 33, 44, 45] mitigates the heavy reliance on labels by considering semantics of labeled multi-modal data and structure information of unlabeled multimodal data. The hash functions are learned through the joint optimization of supervised losses on a small amount of labeled data and unsupervised losses on a vast amount of unlabeled data. However, semantics among unlabeled multi-modal data have not been exploited effectively, and the learned hash codes are not highly discriminative especially when labeled data is limited. Therefore, there remains a research gap in developing deep semi-supervised cross-modal hashing that is expected to yield improved retrieval performance.To mitigate heavy reliance on labels and harness a vast mount unlabeled multi-modal data effectively, this paper proposes a new deep semi-supervised cross-modal hashing method, i.e., Graph Convolutional Semi-Supervised Cross-Modal Hashing (GCSCH) for cross-modal retrieval. The proposed GCSCH first predicts pseudomulti-labels of unlabeled multi-modal data using consistency regularization, and further leverages the superior capability of Graph Convolutional Network (GCN) to effectively exploit semantic structure of the whole multi-modal data that can effectively supervise multi-modal hashing network training and hash code learning. The proposed GCSCH comprises three components including image/text network, consistency regularized pseudo-labeling module, and GCN fusion module, as shown in Figure 1 . The main contributions of this work are as follows:• We propose Graph Convolutional Semi-supervised Crossmodal Hashing (GCSCH) that is trained with limited label supervision for cross-modal retrieval. The proposed GCSCH can generate pseudo-multi-labels for all the unlabeled multimodal data with the simple yet effective idea of consistency regularization and pseudo-labeling, such that the potential semantics of unlabeled data can be effectively exploited.• The proposed GCSCH leverages GCN to effectively exploit semantic structure of ground-truth-labeled and pseudo-labeled multi-modal data, and effectively transfers the knowledge from fused feature to image and text networks using knowledge distillation. • Extensive empirical results on three benchmark datasets demonstrate that the proposed GCSCH outperforms the state-of-the-arts on image-text retrieval with limited label supervision. 
 
 Appendix: http://lms.comp.nus.edu.sg/research/NUS-WIDE.htmhttp://mscoco.org/ 


Decision: In this paper, the authors propose a semi-supervised method, Graph Convolutional Semi-Supervised Cross-Modal Hashing (GCSCH). This method uses pseudo-labeling, a fusion network, and a teacher-student learning scheme to capture semantic information and transfer knowledge, demonstrating superior performance on three benchmark datasets with limited label supervision. After the rebuttal phase, all the reviewers unanimously conclude acceptance. However, a few minor issues still need to be addressed before the camera-ready version. Please refer to the reviewers' limitations and final thoughts.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel semi-supervised deep cross-modal hashing method to capture semantic information among ground-truth-labeled and pseudo-labeled multi-modal data based on graph convolutional network. Experimental results show the effectiveness of the proposed approach. The following concerns should be addressed.
strength_weakness: strength_weakness: 1.The method has achieved significant improvements in experiments. 
2.This paper is well organized and written.
limitations: limitations: Some concerns and suggestions to enhance this work are as follows:
1. The motivation in the abstract is unclear. I strongly suggest that the description need to be enhanced clarity and succinctness
2. I recommend the author to put the notations into figure 1 as well and further improve the clarity and readability of them.
3.The introduction in manuscript lacks effective organization and coherence. I encourage the authors to carefully revise the manuscript with a focus on improving its overall organization and writing structure. Avoid overly general descriptions, such as “ [2, 5, 6, 10, 16, 29, 42, 47]” and “ [8, 9, 14, 17, 32, 35, 37, 39, 40]”.
4. I noticed several formatting errors such as the lack of punctuation in the formulas in Section 3.2.
5. It is recommended that the authors the language expression errors and enhance the manuscript's overall quality. Such as “Using the idea of knowledge distillation,” in the abstract.
paper_summary: paper_summary: This paper proposes GCSCH to solve the cross-modal hashing retrieval problem under limited supervision. The proposed method employs three main approaches, including pseudo-labeling, GCN and knowledge distillation.
strength_weakness: strength_weakness: 1. The paper is easy to follow.
2. Experimental results demonstrate the effectiveness of the proposed method.
limitations: limitations: 1. The novelty of this paper is incremental, as the proposed framework just integrates some well-developed approaches for performance improvement.  

2. The construction of S is based on the available labels. Then, S will be a very sparse matrix, as very limited samples have labels. In this case, I am confused if the loss functions Eq. (5) and Eq. (10) can be useful with a very sparse adjacency matrix. Or does the construction of S use the pseudo-labels of unlabeled samples?

3. The authors only provide two abstract functions for feature augmentations in Line 304. How to implement these two functions in the experiments? No details are provided. 

4. In Fig. 3, in some cases, why does using more labels lead to a performance drop?

5. From Fig. 5, it seems that TS3H is better than the proposed GCSCH. 

6. If the parameter \gamma is set to 10^-9, does it mean the related part can be ignored with regard to the overall loss?
paper_summary: paper_summary: This paper proposes a novel semi-supervised cross-modal hashing method, i.e., Graph Convolutional Semi-Supervised Cross-Modal Hashing (GCSCH). The proposed method can be trained with limited label supervision for cross-modal retrieval. Specifically, a simple yet effective idea of consistency regularization and pseudo-labeling is introduced to generate pseudo-multi-labels for all the unlabeled multi-modal data; and GCN is adopted to effectively exploit semantic structure of ground-truth-labeled and pseudo-labeled multi-modal data. Experimental results demonstrate that the proposed method can outperform SOTA methods.
strength_weakness: strength_weakness: 1. This paper is well written and easy to follow.

2. The proposed method is reasonable and sound. The idea of using consistency regularization and pseudo-labeling is interesting in semi-supervised cross-modal hashing.

3. Experimental results demonstrate that the proposed method can outperform SOTA methods on image-text retrieval with limited label supervision.
limitations: limitations: 1. It is mentioned that there is a strong augmentation and a weak augmentation. However, related description in the paper is only to augment the fused features with different dropout parameters. It is suggested to provide a more detailed description of this feature augmentation operation.

2. The ablation experiment only involves removing the fusion module and the pseudo-label module separately, which is not very sufficient to show the effect of the proposed method. It is recommended to add extra experimental results about removing both modules.

3. It is suggested to check the paper to correct minor typos.
paper_summary: paper_summary: This paper proposes an effective cross-modal hashing method that combines elements from existing approaches to enhance cross-modal hashing performance across multiple datasets, achieving significant advantages when using a smaller proportion of labelled data.
strength_weakness: strength_weakness: The paper is readable, with a well-organized structure, and the experimental setup is generally complete. The innovation of the paper is moderate and incremental.
limitations: limitations: The main innovation and the module that contributes to performance improvement focuses on the Consistency Regularized Pseudo-labeling Module. This module enhances the quality of pseudo-labels through optimising consistency between strong and weak data augmentation expressions, however, which is not a novel concept. Ablation study results suggest that integrating more advanced technologies into the pseudo/real label graph model fusion does not significantly enhance results compared to the Consistency Regularized Pseudo-labeling Module, serving more as a complement to it.

Some issues are:

1. Could you provide more specific definitions for strong/weak augmentation?
2. The authors claim that the GCN Fusion Module makes hash encoding more distinctive. In the experimental section, Figure 5 visualises the hash code distributions of different methods. A more interesting result could be the visualisation of GCSCH compared to GCSCH-F (perhaps also GCSCH-C).
3. The weight \beta of the L_D loss in the knowledge distillation module significantly impacts the results, which is understandable. However, a more detailed analysis is needed to understand what this implies and possible reasons behind it.