FedCAFE: Federated Cross-Modal Hashing with Adaptive Feature Enhancement

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:21:11

Abstract: Deep Cross-Modal Hashing (CMH) has become one of the most popular solutions for cross-modal retrieval. Existing methods need to first collect data and then be trained with these accumulated data. However, in real world, data may be generated and possessed by different owners. Considering the concerns about privacy, data may not be shared or transmitted, leading to the failure of sufficient training of CMH. To solve the problem, we propose a new framework called Federated Cross-modal Hashing with Adaptive Feature Enhancement (FedCAFE). FedCAFE is a federated method which could use distributed data to train existing CMH methods under the privacy protection. To overcome the data heterogeneity challenge of distributed data and improve the generalization ability of global model, FedCAFE is endowed with a novel adaptive feature enhancement module and a new weighted aggregation strategy. Besides, it could fully utilize the rich global information carried in the global model to constrain the model during the local training process. We have conducted extensive experiments on four widely-used datasets in CMH domain with both IID and non-IID settings. The reported results demonstrate that the proposed FedCAFE achieves better performance than several state-of-the-art baselines. As the topic that training deep CMH in federated scenario is in its infancy, we plan to release the code and data to boost the development of the field. However, considering restriction of anonymous submission and size limitation, we could only upload the source code of FedCAFE as supplementary materials for peer review at the present stage.

Main paper: 
 introduction: 
 Existing deep cross-modal hashing methods [3, 15, 19, 43] have achieved efficient approximate nearest neighbor search by mapping high-dimensional samples to hash codes in low-dimensional Hamming space [20, 25, 28, 34] . Such methods have obvious advantages in large-scale retrieval, providing high retrieval speed and satisfactory retrieval accuracy [24, 26, 48] . These methods often require a large amount of data for training, while in real-world applications, data is typically scattered across different devices or institutions. Collecting and centralizing a large amount of distributed data is not only expensive but also impractical. Considering the legal restrictions and growing concerns about data privacy protection, such training schemes may be forbidden as transmitting data may face potential security risks and threats. Federated learning, as a distributed machine learning framework, can ensure that a global model is trained collaboratively by uniting a series of clients without sharing local raw data [12, 18] . The emergence of federated learning solves the problem that local data cannot be shared under privacy and security requirements. Therefore, using federated learning to accomplish distributed training for deep cross-modal hashing methods is a feasible solution, which could not only protect the privacy of each participant but also improve the global model's generalization ability and robustness. These advantages motivate this paper to propose a new federated cross-modal hashing framework.As the clients come from different devices and have their own private data, the data distributions among different clients are likely to be inconsistent (not independent identically distributed, i.e., non-IID), leading to data heterogeneity, which is one of the fundamental challenges in federated learning. Non-IID problem among client data often affects the stability, convergence, and effectiveness of the global model to varying degrees. To address the issue of data heterogeneity, quite a few works have attempted to propose their solutions. Most of them try to limit the updates of clients. For example, FedProx [18] constrains the models of clients to be closer to the global model by adding a proximal term in local updates, and MOON [17] restricts the direction of local updates to reduce client drift. However, most of the existing federated learning algorithms are all designed and validated based on the image classification task. In this paper, we put our efforts into investigating the heavily understudied cross-modal hashing problem under federated learning settings. Compared with single-modal classification tasks, this research problem is more challenging. First, cross-modal retrieval tasks need to deal with multiple modalities of data samples simultaneously, such as image and text modalities. Second, in the retrieval process, it uses one modality sample (e.g. text) to retrieve another modality sample (e.g. image) with semantic similarity, which requires exploring the semantic relationship between different modality samples. Third, data heterogeneity and distribution imbalance problems in federated learning also need to be considered and solved.In literature, existing researches on federated cross-modal retrieval methods are very limited. FedCMR [50] is the first attempt and is based on one deep cross-modal retrieval model. Considering that hashing methods have high retrieval efficiency and accuracy, deep cross-modal hashing has become one of the most popular solutions for cross-modal retrieval. Thus, a federated cross-modal hashing framework PLFedCMH [23] is proposed which dynamically achieves personalized parameter customization for different layers of each client by introducing a hyperparameter network on the server, to improve the performance of local hashing models. PT-FUCH [16] proposes a strategy for federated unsupervised crossmodal hashing task, which uses global prototypes to promote the alignment of client feature spaces. Although they have promoted the developments of federated cross-modal retrieval, there still exist some problems which are not well solved. First, how to address data heterogeneity is crucial for federated learning and is still an open problem in federated cross-modal hashing domain. Second, discriminative multi-modal features play a fundamental role while existing methods fail to fully adapt the feature extraction of current cross-modal methods with distributed data. In deep cross-modal hashing retrieval tasks, it is essential to learn discriminative features that can correctly distinguish the similarity and differences between different modalities of samples.To overcome the limitations, we propose a new framework called Federated Cross-Modal Hashing with Adaptive Feature Enhancement, which is also named FedCAFE for brevity. As a federated framework, FedCAFE is model-agnostic which means it could be used for any existing deep cross-modal hashing methods. The main parts of FedCAFE involve an adaptive feature enhancement module and a new weighted aggregation strategy. The module could learn better multi-modal features with the help of global semantic information, thereby constraining the raw features. With the new strategy, local models could be better aggregated into the global model whose generalization ability could be boosted. The major contributions are as follows:  
ralated work: 
 Cross-Modal Hashing. In recent years, the development of Deep Neural Networks (DNNs) [5, 13, 32, 39] has brought a new wave of research to cross-modal hashing. On the one hand, compared with traditional methods [6, 21, 40] , deep hashing methods can integrate feature learning and hash learning into an end-to-end framework, letting them promote each other and improve the quality of hash learning [9, 27, 35, 42] . On the other hand, due to the high nonlinear modeling capability, DNNs can adequately exploit the data distribution, so as to obtain satisfactory retrieval accuracy [1, 11, 22, 49] . Although existing deep cross-modal hashing methods have achieved great success, they may fail to handle the new requirements which are proposed by nowaday real-world applications. At present, data may be generated and possessed by different owners. Meanwhile, more and more concerns are raised about privacy protection.Thus, the learning scheme of traditional cross-modal hashing methods, which first collecting all data and then using accumulated data to train a centralized model, becomes impractical as transmitting data may be unsafe and even forbidden. How to train the model with distributedly stored data becomes a tricky challenge. Fortunately, federated learning, which could use data in different silos to train a unified model under privacy protection, is proposed and introduced into cross-modal retrieval domain. FedCMR [50] is the first attempt to explore a deep cross-modal retrieval framework for distributed data. It proposes a federated learning framework and takes existing deep cross-modal retrieval method DSCMR [47] as an example to demonstrate the effectiveness. FedCMR proves the feasibility of deep cross-modal retrieval methods in federated learning scenarios. Afterwards, PLFedCMH [23] is proposed to combine federated learning with cross-modal hashing. It uses a server-side hypernetwork to achieve personalized parameter customization for different clients. PT-FUCH [16] explores unsupervised cross-modal hashing in distributed scenarios. Federated Learning. When data distributions among clients in federated training are different, the local training process of each client will let the local model converge to local optimal. Meanwhile, non-IID data distribution may mislead the optimization direction of the global model, which will further affect the generalization ability of global model and even make global model difficult to converge. In order to alleviate such problems, there have been extensive works to contribute their efforts in different directions. For example, FedProx [18] adds a proximal term in local training process to make the updated local models more similar to global model. Similarly, MOON [17] adds regularization items in local training process to prevent local models drift. FedProto [38] normalizes the training of local models by restricting local category prototypes generated by each client to be closer to global prototypes. However, there are some limitations in this kind of methods. It violates the fact that optimal point of local empirical objective is different from global optimum, and reduces the performance and convergence speed of global model [4] . Some methods try to tackle the problems from another way, i.e., improving the aggregation algorithms to get better global model. For instance, FedNova [41] is a normalized method replacing the widely-used simple weighted aggregation to alleviate objective inconsistency. Personalized Federated Learning (PFL) presents a new solution to data heterogeneity among clients [7, 14, 29, 45] . Specifically, these methods generate a personalized model for each client that is suitable for local data distribution. CD 2 -pFed [33] uses a channel decoupling technique guided by cyclic distillation, which promotes the collaboration between local weights and global weights at channel level to realize personalized local models. Although these improved methods achieve good convergence speed and performance, most of them only explore the performance of algorithms on classification tasks. For more challenging scenarios, such as cross-modal retrieval, directly using above methods are not optimal and even infeasible. Thus, in this paper, we focus on investigating the cross-modal retrieval task for distributed data based on federated learning. 
method: 
 To support existing deep cross-modal hashing to be trained with distributed data, we propose a new framework FedCAFE. Please note that FedCAFE is a federated learning framework, not a specific hashing method. It can be integrated with existing hashing methods. Considering that most existing deep hashing methods have their own feature extraction backbone networks, we directly use these networks and build our FedCAFE upon the extracted features. The illustration of FedCAFE for local client training is presented in Figure 1 . Details of each component and the overall algorithm of FedCAFE are described below.Without loss of generality, we use image and text modalities as examples to illustrate our federated deep cross-modal hashing framework.In the federated learning setting, we assume that there are 𝐾 clients, each of which has its own private data. The multi-modal data on the 𝑘-th client can be represented as𝑫 𝑘 = 𝒙 𝑖 , 𝒚 𝑖 , 𝒍 𝑖 𝑛 𝑘 𝑖=1 ,where 𝑛 𝑘 denotes the number of image-text pairs, 𝒙 𝑖 represents the 𝑖-th image, 𝒚 𝑖 is the corresponding text, the label of the 𝑖-th sample is represented by 𝒍 𝑖 ∈ {0, 1} 1×𝐶 , and 𝐶 is the number of categories.Most existing deep cross-modal hashing methods employ two separate backbone networks, one for image modalities and another for text. These networks are designed to extract features from original images and texts, respectively, which are then used for hash learning. Our FedCAFE is a federated learning framework designed to facilitate the training of deep cross-modal hashing in the federated learning scenario, rather than a specific hashing method. Thus, FedCAFE requires a selected deep cross-modal hashing method and directly utilizes its default architectures to extract features. This approach allows FedCAFE to seamlessly integrate with existing hashing methods, leveraging their established mechanisms for feature extraction. The extracted raw features are represented as 𝑶 * 𝑖 ∈ R 1×𝑟 , where 𝑟 denotes the length of hash codes and * indicates different modalities (gray area of Figure 1 ). Then, to better mine the semantic information, a classifier is added after the raw features, and the predicted labels for different modalities are represented as l * 𝑖 ∈ R 1×𝐶 (yellow area of Figure 1 ). FedCAFE tries to mitigate the effects of non-IID data distribution and improve the generalization ability of the global model by utilizing the proposed adaptive feature enhancement module to generate enhanced features and impose constraints on the raw features (green area of Figure 1 ). As both backbone networks and classifiers have parameters to be learned, we use 𝑾 𝑥 and 𝑾 𝑦 to represent the entire network parameters of the image and text modalities, respectively.The objective of our algorithm is to update and optimize the parameters of the global model for each modality by federated learning to achieve optimal performance for distributed deep crossmodal hashing methods on retrieval tasks. The overall objective of the algorithm can be formulated as follows: min EQUATIONEQUATION)As it is not feasible for the server to access local data of clients directly, we attempt to solve Eq. ( 1) in following manner. In the 𝑡-th communication round, the server sends the global model parameters of different modalities to 𝐾 clients. The corresponding local model parameters are overwritten with the global model parameters and used as the initial parameters of local model for the 𝑡-th round. The clients then train and update their local models using their own data, generating updated local model parameters for different modalities. Finally, the new local model parameters are uploaded to server for aggregation, generating global model parameters for next round.Generally, in federated learning scenarios, the global model is trained on data that is distributed across multiple clients, and the update of global model needs to be transmitted to a central server for aggregation. One of the major challenges is that different clients may have significantly different data distributions, making it challenging to optimize a global model that performs well on all clients. To address this problem, a global memory rich in global semantic information is designed.Specifically, in the 𝑡-th round of communication, the global memory is generated by aggregating the local memories of 𝐾 clients. For the 𝑘-th client, we first use the K-means algorithm to obtain local memories of different modalities. These local memories serve as category prototypes of enhanced features. To reduce the domain gap among feature spaces of different modalities as much as possible, FedCAFE combines the local memories of the two modalities using the K-means algorithm to obtain the final local memory 𝑷 𝑡 𝑘 ∈ R C×𝑟 for the 𝑘-th client in round 𝑡. This approach allows information from different modalities to fully integrate, promoting collaboration among modalities.After obtaining the local memory for each client, the server further uses the K-means algorithm to aggregate them to generate the global memory 𝑷 𝑡 𝑔 . The global memory aggregates the local memories of all participating clients and could better represent the feature distribution of the entire dataset. Therefore, we expect it can be used as a guide for individual client training, thereby alleviating the data heterogeneity problem among clients. It is worth noting that we do not violate the privacy policy of federated learning as we do not share raw data of clients.The design of global memory could effectively promote information exchange and collaborative training among clients, thereby improving the generalization performance of local models on clients.In federated learning, the data distributions among clients are generally non-IID, and clients can only learn knowledge from their local data, leading to a drift in the features learned by each client, which hinders the generalization of the global model. For retrieval tasks, the learned representations of the data are crucial. To learn better representations of multi-modal data, we hope the representations are as discriminative as possible. However, existing methods [23, 50] fail to consider how to enhance representations with the help of semantic information, which may make the learned representations not good enough.Therefore, we propose a module that enables sharing of global semantic knowledge among clients. This means that even across different clients, as long as samples have the same semantics, the features generated will be more consistent. In addition to global memory, labels also contain rich semantic information, which can further promote knowledge sharing among clients. Specifically, by merging the predicted labels with the global memory, memory features rich in global semantic knowledge can be generated. Memory features can be expressed as:EQUATIONwhere L * represents the predicted label matrix of local data and * denotes different modalities, 𝑷 𝑔 stands for the global memory. Using the matmul product ⊗ between them can produce memory features 𝑽 * 𝑀 . Based on this, we introduce an adaptive selector A = Tanh (𝑶 * ) that integrates the original feature 𝑶 * with the memory feature 𝑽 * 𝑀 to enhance the original feature. The design of A, i.e., using the Tanh(•), can not only directly obtain the weight of adaptive selector from original feature 𝑶 * , but also avoid the complicated adjustment of parameters. The enhanced feature 𝑽 * 𝐸 is expressed as:EQUATIONwhere 𝑶 * represents the raw features extracted by backbone networks, ⊙ is the Hadamard product. The enhanced features of image modality can be represented as 𝑽 𝑥 𝐸 ∈ R 𝑛 𝑘 ×𝑟 , and the enhanced features of text modality can be expressed as 𝑽 𝑦 𝐸 ∈ R 𝑛 𝑘 ×𝑟 , typically, they are regarded as continuous surrogate of hash codes.To more effectively constrain the raw features of samples, cosine similarity measurement is adopted to utilize enhanced features 𝑽 * 𝐸 to constrain the original features 𝑶 * , thus improving the quality of feature representation. The formula is as follows:EQUATIONwhere cos(•) represents cosine similarity measurement. In cross-modal retrieval tasks, the information contained in multimodal data features is crucial as it can reveal the inherent correlations and complementary properties between different modalities. In order to fully utilize this information, FedCAFE further utilizes the global information carried in the global model to constrain the training of local models for each client. Specifically, when each client conducts local training, the features extracted by the global model are introduced as prior knowledge to provide guidance for the local training process. In this way, it ensures that local models can closely follow the guidance of the global model and better utilize the correlations between features of different modalities. The specific formula is as follows:EQUATIONwhere 𝑽 * -𝑘 𝐸 represents the enhanced features of different modalities on the 𝑘-th client, * represents different modalities, 𝑽 𝑥 𝐸 represents the enhanced features of image modality when image sample passes through the corresponding global model, and 𝑽 𝑦 𝐸 represents the enhanced features of the text modality when the text sample passes through the global model of the corresponding modality. In this way, samples of different modalities can be effectively compared and matched in a unified feature space.The information carried by the global model is obtained through training on multiple clients and can reflect the overall rules and patterns of data distribution. Transmitting this information to local models provides them with a broader and more comprehensive perspective, enabling them to generalize better to unknown data.Since the learned hash codes not only need to maintain the similarity of the data in the feature space but also should have good classification ability, classification loss is added for supervised learning. FedCAFE utilizes KL divergence to measure the similarity between the predicted labels generated by classifiers and the true distribution of samples. For the training process of local models, the classification loss can be represented by the following formula:EQUATIONwhere L * represents the predicted labels of different modalities, and 𝑳 represents the ground truth of samples.The overall loss function can be constituted of hashing loss and federated loss:EQUATIONwhere 𝛼, 𝜂 and 𝛾 are trade-off parameters. As the FedCAFE framework serves as a bridge to apply deep cross-modal hashing to the federated learning scenario, hashing loss L ℎ𝑎𝑠ℎ will not be specifically designed. As mentioned above, FedCAFE is model-agnostic which means it could be used for any existing deep cross-modal hashing methods. Thus, the objective loss of the chosen hashing method could be directly leveraged as L ℎ𝑎𝑠ℎ in Eq. ( 8).Data heterogeneity may lead to a decrease in model performance, as the local models on clients are trained on data that is not representative of the overall data distribution. To alleviate the problem of local model drift caused by data heterogeneity across clients and fully leverage the characteristics of deep cross-modal hashing retrieval tasks, we propose a novel weighted aggregation strategy to generate the global models of different modalities in each communication round on the server. Specifically, we use the degree of difference between global memory and local memory as a measure of weight. To measure the similarity between the local memories of clients and the global memory, FedCAFE uses the negative log-likelihood function. The formula of the similarity between the local memory of 𝑘-th client and the global memory is:EQUATIONwhere𝜃 𝑖 𝑗 = 1 2 (𝑷 𝑡 𝑘 𝑖 • )(𝑷 𝑡 𝑔 𝑗 • ) 𝑇, and the local memory on the 𝑘-th client can be expressed as 𝑷 𝑡 𝑘 ∈ R 𝐶 ×𝑟 . We assume that the global memory and local memory of the same class should be similar, so we use the similarity matrix S to represent the relationship between them. If 𝑷 𝑡 𝑘 𝑖 • and 𝑷 𝑡 𝑔 𝑗 • are similar, S 𝑖 𝑗 = 1, otherwise S 𝑖 𝑗 = 0. When the value of the negative log-likelihood function is larger, it indicates lower similarity between the local memory of 𝑘-th client and the global memory, suggesting that the specific information or knowledge contained on the 𝑘-th client is more unique compared to the global data distribution. Therefore, in order to fully emphasize these unique pieces of information and enhance the generalization ability of the global model, a higher weight is assigned to the model parameters on this client. The formula is as follows:EQUATION)where 𝛼 𝑘 is the model weight of the 𝑘-th client. This weighting strategy helps ensure that the unique knowledge on the 𝑘-th client is adequately emphasized, while also aiding the global model in better learning and adapting to various data distributions. Thus, the global model parameters for different modalities in round (𝑡 + 1) can be expressed as:EQUATIONwhere 𝑾 For better understanding of FedCAFE, we summarize the overall algorithm in Algorithm 1. 
experiments 4.1 datasets: 
 We evaluated the proposed FedCAFE on four benchmark datasets for cross-modal hashing retrieval tasks, namely MIRFlickr-25K [8] , NUS-WIDE [2] , FashionVC [36] , and Ssense [37] . The specific dataset partitioning strategy is shown in Table 1 . [30] 0.6610 0.6734 0.6836 0.6907 0.7040 0.7143 0.5381 0.5581 0.5567 0.5186 0.5526 0.5609 FedProx [18] 0.6597 0.6724 0.6829 0.6899 0.7033 0.7133 0.5352 0.5565 0.5564 0.5176 0.5501 0.5599 FedCMR [50] 0.6729 0.6856 0.6956 0.7129 0.7229 0.7318 0.5572 0.5636 0.5735 0.5569 0.5781 0.5851 MOON [17] 0.6740 0.6862 0.6948 0.7081 0.7219 0.7317 0.5650 0.5840 0.5758 0.5664 0.5753 0.5867 FedProto [38] 0.6741 0.6892 0.6979 0.7138 0.7277 0.7392 0.5662 0.5863 0.6063 0.5695 0.5818 0.6051 PLFedCMH [23] MIRFlickr-25K contains 25,000 images collected from Flickr. It is multi-labeled, in which each sample is annotated with at least one of 24 semantic labels. Following the settings of [10] , we removed the samples in which the number of tags is less than 20 and obtained 20,015 instances. In our experiments, we randomly selected 2,000 instances as the query set, left the remaining instances as the retrieval set, and randomly selected 10,000 samples from the retrieval set for training. This data partitioning strategy ensures that the model encounters diverse data distributions during both training and testing phases, thereby accurately evaluating its performance.NUS-WIDE contains 269,648 images, each of which contains one or more of 81 labels. Following [10] , we obtained 195,834 instances by using the 21 most frequent labels. In our experiments, we randomly picked up 2,100 instances as the query set, left the remaining samples as the retrieval set, and randomly chose 10,500 samples from the retrieval set for training.FashionVC consists of 20,726 image-text pairs collected from the online fashion community Polyvore. Following [44] , we removed labels with less than 25 samples and obtained 19,862 instances. Each sample has a hierarchical structure label, with 8 coarse-grained categories in the first level and 27 fine-grained categories in the second level. In our experiments, we leveraged the fine-grained labels and randomly selected 3,000 instances as query set, the remaining samples are used as retrieval set and training set.Ssense consists of 25,947 image-text pairs collected from the online fashion community sense. Following the settings of [44] , labels with less than 70 samples are removed and 15,696 instances are obtained. Similarly, the labels of this dataset are hierarchical, with 4 coarse-grained categories in the first layer and 28 fine-grained categories in the second layer. In our experiments, we selected fine-grained labels for annotating samples, and 2,000 instances are randomly selected as the query set, the remaining image-text pairs are used as the retrieval set and training set.Backbone Architectures. We adopted two existing deep crossmodal hashing retrieval frameworks as feature learning backbone networks for performance evaluation: 1) DCMH [10] for MIRFlickr-25K and NUS-WIDE; 2) SHDCH [44] for FashionVC and Ssense. We employed the original codes of both of them as those codes have already been made publicly available by their authors.Baselines. We compared our FedCAFE with several state-ofthe-art baselines, including FedAvg [30] , FedProx [18] , MOON [17] , FedProto [38] , FedCMR [50] , and PLFedCMH [23] .Heterogeneity. To investigate the effectiveness of FedCAFE with varying degrees of heterogeneity, we designed two settings: IID and non-IID. Since the MIRFlickr-25K and NUS-WIDE are multilabel datasets, it is difficult to simulate them as non-IID scenarios. Therefore, evaluations on these two datasets belong to the IID setting, where we randomly shuffled the training set and distributed it equally among the clients. For FashionVC and Ssense datasets, we followed previous works [31, 46] and used the Dirichlet distribution Dir(𝛽) to simulate the non-IID scenarios, where a smaller 𝛽 indicates stronger heterogeneity among clients. In our experiments, we set 𝛽 to 0.5 and 0.2 for two situations.Hyperparameters. For consistency, we set the communication rounds to 100 and the number of clients to 10 for all methods. For local training, the batch size is 128, the local training epoch is 20 on MIRFlickr-25K and NUS-WIDE, and 10 on FashionVC and Ssense. Additionally, we set the hyperparameter 𝛼, 𝜂, and 𝛾 of FedCAFE to 0.1, 0.1, and 1 respectively. For specific hyperparameters in baselines, we kept the same settings as their original papers.Evaluation Metrics. We conducted two cross-modal retrieval tasks, which are retrieving texts using an image query (i.e., Image-to-Text) and retrieving images using a text query (i.e., Text-to-Image). We also conducted experiments on different lengths of hash codes. To evaluate the performance of all methods, we used Mean Average Precision (MAP), the most widely-used evaluation metric in cross-modal hashing retrieval methods. A higher MAP value indicates better performance. The baselines we compared have both traditional federated learning and personalized federated learning settings. For traditional federated learning, we evaluated the algorithm performance directly on the global model. For personalized federated learning, we evaluated the algorithm performance based on the average performance of the optimal local model on each client. To simulate practical personalized federated learning settings, we distributed the retrieval set and query set to clients according to their corresponding data heterogeneity and evaluated the learned models on the clients. [30] 0.5852 0.6753 0.7220 0.7368 0.8582 0.8976 0.4681 0.5502 0.6573 0.6321 0.7425 0.8204 FedProx [18] 0.5680 0.6615 0.7089 0.6837 0.8245 0.8623 0.4731 0.5777 0.6592 0.5171 0.6579 0.7580 FedCMR [50] 0.5868 0.6786 0.7228 0.7331 0.8652 0.8995 0.4745 0.5809 0.6587 0.6343 0.7514 0.8213 MOON [17] 0.5820 0.6808 0.7285 0.7024 0.7933 0.8638 0.4694 0.5722 0.6606 0.6399 0.6419 0.7311 FedProto [38] 0.6978 0.7099 0.7231 0.8808 0.9158 0.9030 0.5630 0.5785 0.5823 0.7663 0.8381 0.8109 PLFedCMH [23] [44] 0.9692 0.9697 0.9720 0.9820 0.9881 0.9880 0.9692 0.9697 0.9720 0.9820 0.9881 0.9880Local [44] 0.7161 0.7776 0.7895 0.6763 0.7436 0.7546 0.4888 0.5322 0.5518 0.4388 0.4752 0.4917 FedAvg [30] 0.8936 0.9517 0.9620 0.9386 0.9734 0.9840 0.7653 0.9059 0.9528 0.8028 0.9447 0.9792 FedProx [18] 0.8567 0.9424 0.9590 0.8789 0.9626 0.9773 0.7952 0.8964 0.9390 0.8152 0.9226 0.9715 FedCMR [50] 0.8953 0.9586 0.9631 0.9411 0.9763 0.9855 0.7956 0.9142 0.9537 0.8094 0.9459 0.9760 MOON [17] 0.9446 0.9628 0.9683 0.9549 0.9782 0.9845 0.7871 0.9205 0.9510 0.8028 0.9238 0.9806 FedProto [38] 0.9247 0.9620 0.9676 0.9390 0.9826 0.9845 0.9579 0.9585 0.9613 0.9789 0.9836 0.9825 PLFedCMH [23] 0.9545 0.9595 0.9557 0.9752 0.9827 0.9824 0.8832 0.8983 0.9014 0.8851 0.9366 0.9437 FedCAFE 0.9665 0.9671 0.9658 0.9856 0.9856 0.9860 0.9582 0.9595 0.9570 0.9801 0.9847 0.9832We conducted extensive experiments so as to evaluate the effectiveness of FedCAFE on retrieval accuracy. MAP Results in IID Scenarios. Table 2 reports the retrieval performance of all compared algorithms on MIRFLICKR-25K and NUS-WIDE datasets. The first line reports the performance of centralized learning, the second line reports the performance of clients utilizing private data for local training. As shown in Table 2 , Fed-CAFE achieves the best performance in all settings. Compared to the local training method without federated collaboration, federated learning methods have both shown performance improvements. For instance, on MIRFlickr-25K with the hash code length of 64 bits, FedCAFE outperformed the local training method by 7.5% in Image-to-Text task and 7.26% in Text-to-Image task, demonstrating the crucial role of federated collaboration during training. This aids models in acquiring stronger generalization capabilities, thereby enhancing overall performance. Additionally, compared to MOON, FedProto achieves higher retrieval accuracy, further proving the effectiveness of global prototypes, which enable clients to share global semantic information. However, in contrast, FedCAFE is more efficient and direct in enhancing sample features through the utilization of global memory.To conclude, our proposed method FedCAFE is superior to all baselines in terms of accuracy.MAP Results in Non-IID Scenarios. Table 3 reports the performance of all compared algorithms on FashionVC using different 𝛽 values. Table 4 lists the results of all methods on Ssense. In both tables, the first line reports the performance of centralized learning, the second line reports the performance of local training. From these tables, we can have following findings. (1) The proposed Fed-CAFE achieves the best or comparable performance in most cases on FashionVC and Ssense. (2) As data heterogeneity increases (𝛽 becomes smaller), retrieval accuracy of all algorithms decreases, showing that non-IID distribution affects model learning and retrieval accuracy. (3) The MAP results of FedCAFE do not exhibit a significant decrease with the increase of Non-IID degree, demonstrating its stability and robustness in handling data distribution differences.In a word, our method FedCAFE could also perform well in non-IID scenarios.Validity of Functional Components. To verify the necessity of each component proposed in FedCAFE, Table 5 shows the retrieval performance of FedCAFE after dropping some elements and losses As shown in Table 5 , some observations can be found. (1) Fed-CAFE achieves the best performance when all components are employed. (2) Removing any component will reduce the performance.(3) FedCAFE will degrade into the conventional FedAvg when all components are removed. (4) By introducing the Adaptive Feature Enhancement module to enhance the raw features and using them to constrain the raw features, not only the drift of local models of each client be effectively alleviated, but also the performance of the global model can be significantly improved. (5) Deleting either L 𝐸 or L 𝐶 results in a decrease in retrieval performance, further highlighting the critical role of these two components. They can effectively unify the local features of clients into the global semantic space. To conclude, all components are necessary and effective.Hyperparameter Sensitivity Analysis. To consider the impact of hyperparameter values, we further conducted experiments on the trade-off parameter 𝛼, 𝜂, and 𝛾 in 32-bit hash code length on FashionVC with the 𝛽 = 0.5 data heterogeneity. We evaluated and analyzed the performance of FedCAFE by taking values for 𝛼, 𝜂, 𝛾 ∈ {0.001, 0.01, 0.1, 1.0, 10, 100}. Figure 3 (a) shows the impact of changing 𝛼 on model performance when 𝜂 and 𝛾 are fixed at 0.1 and 1.0, respectively. Similarly, Figure 3 (b) shows the impact of changing 𝜂 on model performance when 𝛼 and 𝛾 are fixed at 0.1 and 1.0, respectively. Figure 3 (c) shows the impact of changing 𝛾 on model performance when 𝛼 and 𝜂 are fixed at 0.1 and 0.1, respectively. It can be observed that when 𝛼, 𝜂, 𝛾 ∈ {0.001, 0.01, 0.1, 1.0}, even under different combinations of hyperparameters, the performance of FedCAFE remains relatively satisfactory. In our experiments, we set 𝛼, 𝜂, and 𝛾 as 0.1, 0.1, and 1.0, respectively.Communication Rounds Analysis. In order to verify the impact of the number of communication rounds on FedCAFE, experiments were conducted separately on MIRFlickr-25K, NUS-WIDE, FashionVC, and Ssense with 32-bit hash code length. On FashionVC and Ssense, the Dirichlet parameter was set to 𝛽 = 0.5. The experimental results are shown in Figure 2 . It can be observed that as the number of communication rounds increases, the performance of the global model gradually improves and tends to stabilize. By increasing the number of communication rounds, each client can share and exchange knowledge thoroughly, thereby promoting the convergence of the model.  
conclusion: 
 In  
 
 Appendix: √ √ √ √ 0.7603 0.9169 


Decision: This paper introduces a cross-modal hashing and adaptive feature enhancement framework called FedCAFE, which enable training  cross-modal hashing methods in distributed data environments. Utilizing the federated learning framework, FedCAFE is able to collaboratively train global models across multiple clients without sharing raw data. By introducing an adaptive feature enhancement module and a new weighted aggregation strategy, the stability and convergence of the model when dealing with non-independently identically distributed data are improved, achieving better results than existing benchmarks on four widely used datasets. Search accuracy.

The paper is well-written and conducted sufficient experiments to demonstrate the efficiency of the proposed method. The proposed method also has sufficient novelty on the integration of the feature and parameters across the client and servers in distributed environment.

The rebuttal materials has addressed the most problem proposed by reviewers. I checked the all the opinions and the answers from authors. The authors conducted experiments to prove the generalization ability of the framework that doubt by the reviewer.

Considering all the materials, I suggest present this paper which is well-written, clear, novel, comprehensive experimented in the Conference.

Reviews: review: paper_summary: paper_summary: This paper introduces a federated cross-modal hashing and adaptive feature enhancement framework called FedCAFE, which aims to solve the problem that existing cross-modal hashing methods are difficult to train in distributed data environments. Utilizing the federated learning framework, FedCAFE is able to collaboratively train global models across multiple clients without sharing raw data. By introducing an adaptive feature enhancement module and a new weighted aggregation strategy, the stability and convergence of the model when dealing with non-independently identically distributed data are improved, achieving better results than existing benchmarks on four widely used datasets. Search accuracy.
strength_weakness: strength_weakness: Innovative Federated Learning Framework: FedCAFE introduces a novel approach to federated learning by integrating cross-modal hashing with adaptive feature enhancement, which is a significant innovation in the field. Unlike traditional federated learning approaches that focus solely on a single data modality, FedCAFE effectively handles multiple modalities, ensuring that insights can be drawn from diverse data sources without the need to centralize the data. This ability to process and analyze data from different modalities federatively is particularly valuable for applications dealing with complex datasets like images and text where data privacy is a concern.

Robust Experimental Validation: The framework has been thoroughly tested across four widely-used datasets, demonstrating superior performance over existing benchmarks. This comprehensive validation not only shows FedCAFE’s effectiveness in enhancing retrieval accuracy but also proves its robustness in handling non-IID (independently and identically distributed) data across various client distributions. The experiments detailed in the paper include a range of scenarios, showcasing the framework's adaptability and scalability in real-world applications.

Practicality and Adaptability: FedCAFE’s design incorporates an adaptive feature enhancement mechanism that adjusts to the specific characteristics of the data it processes. This adaptability makes it highly practical for real-world applications where data characteristics can vary widely between different nodes in a federated network. Moreover, the inclusion of a new weighted aggregation strategy optimizes the learning process by prioritizing more informative updates from clients, which enhances the overall efficiency and effectiveness of the learning model.

Privacy-Preserving Capabilities: One of the significant advantages of FedCAFE is its ability to maintain high standards of data privacy. By design, the federated learning model ensures that sensitive data remains on local devices, and only essential information is shared across the network. This aspect is crucial for compliance with stringent data protection regulations and is highly relevant in sectors like healthcare and finance, where privacy concerns are paramount
limitations: limitations: Content of the article: First of all, I would like to know what the problem is with cross-modal hashing. Can you give an example? The second point is, in line 293, what is "+" and how to calculate it, whether to sum two vectors or to concatenate them. Hope you can answer me, thank you.

High Complexity of Implementation: One of the significant drawbacks of the FedCAFE framework is its complexity. The integration of cross-modal hashing with adaptive feature enhancements requires sophisticated algorithmic structuring and tuning, which may not only necessitate extensive computational resources but also a high level of expertise to implement effectively. This complexity could hinder the adoption of FedCAFE in environments where technical resources or expertise are limited, thus restricting its practicality in smaller organizations or institutions with constrained budgets.

Insufficient Generalization Validation: While the framework shows impressive results on the datasets used, the paper lacks a comprehensive validation across a broader array of scenarios that might encounter more varied data distributions or real-world conditions. The generalization capabilities of FedCAFE are crucial for its application in diverse environments, and without extensive testing in these contexts, there is a risk that the framework might not perform as expected when deployed under different or more challenging conditions than those presented in the study.

Dependency on High-Quality Input Data: The effectiveness of FedCAFE heavily relies on the quality and consistency of the input data. In federated environments, where data can originate from diverse sources with varying levels of quality and integrity, the performance of the framework could be significantly compromised. This reliance on high-quality input is a considerable limitation, as it might not always be feasible to ensure uniform data quality across all nodes in a federated learning setup.

Potential for Overfitting: The adaptive feature enhancement component, while beneficial for tailoring the learning process to specific data characteristics, also poses a risk of overfitting to the idiosyncrasies of the data available at individual nodes. This could lead to models that perform well on local or seen data but fail to generalize effectively to new or unseen data types or distributions. The paper could benefit from a more detailed discussion on strategies to mitigate this risk, ensuring the model remains robust and applicable to a wide range of data scenarios.
paper_summary: paper_summary: This paper proposes a new framework called Federated Cross-modal Hashing with Adaptive Feature Enhancement (FedCAFE). FedCAFE is a federated method which could use distributed data to train existing CMH methods under the privacy protection. To overcome the data heterogeneity challenge of distributed data and improve the generalization ability of global model, FedCAFE is endowed with a novel adaptive feature enhancement module and a new weighted aggregation strategy. Besides, it could fully utilize the rich global information carried in the global model to constrain the model during the local training process.
strength_weakness: strength_weakness: 1. This paper proposes a novel federated framework FedCAFE for training existing deep cross-modal hashing methods with dis-
tributed data. 
2. The authors propose an adaptive feature enhancement module, which fully embeds global semantic information into feature learn-
ing. This module could ensure discriminative multi-modal features and lead to better hash learning. A new weighted aggregation strategy is also proposed. 
3. The proposed FedCAFE achieves better retrieval accuracy than all chosen baselines.
limitations: limitations: I have some concerns regarding the experimental section.
1.	To enhance the credibility of the results, the author should consider involving more users instead of just 10.
2.	The authors should evaluate their method on more Non-i.i.d settings..
3.	How much computational overhead does the proposed method require? To demonstrate its practicality, a proper comparison of computational costs with existing methods should be provided.
paper_summary: paper_summary: This paper introduces FedCAFE, a federated framework designed to train deep cross-modal hashing methods using distributed data without sharing private client data. FedCAFE employs default backbone networks for feature extraction, followed by an adaptive feature enhancement module that utilizes global semantic knowledge to enhance and constrain raw features. It also utilizes a weighted aggregation strategy during model aggregation on the server to enhance the generalization capability of the global model.
strength_weakness: strength_weakness: 1. The paper is well-structured and generally well-written.

2. The experimental results demonstrate significant performance improvements compared to the baseline methods.

3. The paper introduces an innovative concept, fully embedding global semantic information into feature learning, which is novel.
limitations: limitations: 1.It's difficult to discern the relevance to federated learning in the flowchart depicted in Figure 1.

2 The manuscript still employs the methods of PLFedCMH and PT-FUCH, using feature prototypes to convey cross-modal semantics in the federated learning process, which seems lacking in innovation. The integration of federated learning with cross-modal retrieval raises the crucial issue of transferring cross-modal semantics without compromising user privacy, yet the paper appears to lack innovation in this regard. 

3 The innovative aspect of the paper primarily lies in enhancing local models with global information. When global information is sufficiently learned, enhancing local models seems like a natural progression. However, the paper does not delve deeply into how to learn global information.

4 The paper mentions the transmission of local memories for two modalities, yet when local memories are updated, there is a risk of eavesdroppers capturing different local memories to obtain client-side private data, particularly concerning image data, which seems detrimental to federated learning.
paper_summary: paper_summary: The paper proposes a Federated Cross-Modal Hashing method, FedCAFE, which includes an adaptive feature enhancement module and a new model aggregation strategy. Each client preserves local memory using the k-means method and uploads it to the server for further aggregation using k-means to obtain global memory. Ultimately, local memory is used to compute model similarity during server-side model aggregation, while global memory is employed to introduce global knowledge during client-side training. The authors validate the effectiveness of FedCAFE on four benchmark datasets.
strength_weakness: strength_weakness: 1. This paper further validates the effectiveness of federated learning in the task of deep cross-modal hashing.
2. The article is written fluently, and the method's introduction is clear and easy to understand.
limitations: limitations: 1. When deep cross-modal hashing is combined with federated learning, are there any new challenges that arise? The paper does not seem to address this. Intuitively, it seems that all federated learning methods could be applicable to this task.
2. Despite the authors emphasizing that FedCAFE can obtain more discriminative representations, they have not explained or validated why they are more discriminative.
3. The paper employs a parameter aggregation method based on similarity measurement, which is common in federated learning. The contribution of this paper lies in utilizing local memory for similarity measurement rather than proposing this aggregation strategy.
4. Does sharing memory between the server and clients raise privacy concerns? The paper needs further discussion on this issue.
5. In the ablation experiments in Table 5, when only the Adaptive Feature Enhancement module is added, the model performance hardly changes. The authors need more experiments to prove the effectiveness of AFE.

Other suggestions:
1. In the introduction, the description of contributions, the first and second points do not have a substantial difference, so there is no need to artificially separate them.
2. In the related work section, it is unnecessary to elaborate extensively on federated learning. Some space should be dedicated specifically to introducing "Federated Learning for Cross-Modal Hashing."