FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction
Authors: Ziwei Li
Xiaoqi Wang
Hong-You Chen
Han Wei Shen
Wei-Lun Chao
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Dimensionality Reduction
Unsupervised Learning
Representation Learning
Primary area: other

Abstract: Federated learning (FL) has rapidly evolved as a promising paradigm that enables collaborative model training across distributed participants without exchanging their local data. Despite its broad applications in fields such as computer vision, graph learning, and natural language processing, the development of a data projection model that can be effectively used to visualize data in the context of FL is crucial yet remains heavily under-explored. Neighbor embedding (NE) is an essential technique for visualizing complex high-dimensional data, but collaboratively learning a joint NE model is difficult. The key challenge lies in the objective function, as effective visualization algorithms like NE require computing loss functions among pairs of data. 
In this paper, we introduce \textsc{FedNE}, a novel approach that integrates the \textsc{FedAvg} framework with the contrastive NE technique, without any requirements of shareable data. To address the lack of inter-client repulsion which is crucial for the alignment in the global embedding space, we develop a surrogate loss function that each client learns and shares with each other. Additionally, we propose a data-mixing strategy to augment the local data, aiming to relax the problems of invisible neighbors and false neighbors constructed by the local $k$NN graphs. We conduct comprehensive experiments on both synthetic and real-world datasets. The results demonstrate that our \textsc{FedNE} can effectively preserve the neighborhood data structures and enhance the alignment in the global embedding space compared to several baseline methods.

FEDNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction
Ziwei Li, Xiaoqi Wang, Hong-You You Chen, Han-Wei Shen, Wei-Lun Chao
The Ohio State University (1i.5326, wang. 5502, M3en,9301, shen.94, chao. 209) @osu. edu
Abstract
Federated learning (FL) has rapidly evolved promising paradigm that enables collaborative model training across distributed participants without exchanging their local data. Despite its broad applications in fields such as computer vision, graph learning, and natural language processing, the development of a data pro- jection model that can be effectively used to visualize data in the context of FL is crucial yet remains heavily under-explored. Neighbor embedding (NE) an essential technique for visualizing complex high-dimensional data, but collab- oratively learning joint NE model is difficult. The key challenge lies in the objective function, effective visualization algorithms like NE require computing loss functions among pairs data. this paper. we introduce FEDNE, a novel approach that integrates the FEDAVG framework with the contrastive NE technique, without any requirements of shareable data. To address the lack of inter-client repulsion which is crucial for the alignment in the global embedding space, we develop surrogate loss function that each client learns and shares with each other. Additionally, propose data-mixing strategy augment the local data, aiming to relax the problems of invisible neighbors and false neighbors constructed by the local ANN graphs. We conduct comprehensive experiments and real-world datasets. The results demonstrate that our FEDNE can effectively preserve the neighborhood data structures and enhance the alignment in the global embedding space compared to several baseline methods.
both synthetic
Introduction
Federated Learning (FL) has emerged as highly effective decentralized learning framework in which multiple participants collaborate learn shared model without sharing the data. In recent years, FL has been extensively studied and applied across various domains, including image and text classifications [10
31 23 2), computer vision tasks 3 41 50 and graph learning problems 42 5
16]. Despite the growing interest in FL., the area of dimensionality reduction within this framework has received limited investigation. However, visualizing and interpreting data from distributed sources is important, as real-world applications often generate large volumes of complex datasets that are stored locally by each participant. For example, different hospitals collect nigh-dimensionall electronic health records (EHRs) [ and securely store this patient data within their local systems. As each hospital might only collect limited data focus on particular diseases, conducting data visualization combined dataset can substantially improve disease diagnosis and provide deeper insights. However, sharing sensitive patient information is restricted due to privacy protection. Thus, developing shared dimensionality reduction model in the FL setting is crucial for facilitating collaborative analysis while maintaining data local sites.
Dimensionality reduction (DR) refers to constructing a low-dimensional representation from the input data while preserving the essential data structures and pattern.. Neighbor embedding (NE) 6 49]. family of DR techniques, widely utilized to visualize complex high-dimensional data due to its ability preserve neighborhood structures and handle non-linear relationships effectively. Essentially,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

NE methods (e.g., t-SNE 6 balancing attractive forces that pull similar data points together and repulsive forces that push dissimilar points apart. Defining the objective function requires access to any pairs of data points.
44 43
and UMAP 33
operate on an attraction attraction-reuulson repulsion spectrum
Such a need to access pairwise data for determining the attraction and repulsion terms, however, poses critical challenges to the FL framework. As data are distributed across different clients, computing their pairwise distances becomes non-trivial, making it difficult to recover the centralized objective in an FL setting. Specifically, the absence of inter-client repulsive forces complicates the separation of dissimilar data points. Moreover, within a client, due to the unavailability of others' data, defining a faithful attraction term based on the lop neighbors is challenging, often resulting in the inaccurate grouping of distant data points. This contrasts with traditional FL. tasks, such as image classification, where learning objectives can be decomposed over individual training instances, allowing each client to calculate the loss independently and optimize the model based solely on its local dataset.
To the best of our knowledge, only a few existing works address the problem of decentralized data visualization. Both dSNE [37] and F-dSNE (38] methods require publicly shared data that serves as reference point for aligning the embeddings from different clients. This setup introduces additional assumptions that may not be feasible in real-world applications, and the quantity and
representativeness of the reference data can significantly impact the resulting embeddings.
To this end, we proposed novel Federated neighbor embedding technique, called FEDNE, which follows the widely used FEDAVG pipeline. It trains a shared NE model that is aware of the global data distribution, without requiring any shareable data across participants. To compensate for the lack of much-needed inter-client repulsive force, besides training a local copy of the NE model, each client additionally learns surrogate model designed summarize its local repulsive loss function. During global aggregation, this surrogate model will be sent back to the server along with the local NE model, which other clients can use in the next round of local training. In detail, for client m, its local surrogate model is designed to approximate the repulsion loss from an arbitrary point to its local data points. By sending the surrogate model other clients, another client m' can incorporate it into its local loss function for training the NE model. Additionally, to handle the difficulty of estimating the neighborhood, we introduce an intra-client data mixing strategy to simulate the presence of potential neighbors residing on the other clients. This approach augments the local data to enhance the training of the NE model.
To showcase the effectiveness of FEDNE, we conduct comprehensive experiments using both syn- thetic and real-world benchmark datasets used in the field of dimensionality reduction under various FL settings. Both qualitative and quantitative evaluation results have demonstrated that our method outperforms the baseline approaches in preserving the neighborhood data structures and facilitating the embedding alignment in the global space.
Remark. It is worth discussing that we understand address in the FL framework. However, we want to reiterate the main focus of this paper is identifying the unique challenges in the federated neighbor embedding problem and proposing effective solutions rather than resolving all the FL challenges at once. We discuss the privacy considerations and our further work in section 6
is an important aspect to
Related Work
Federated learning. FL aims to train a shared model among multiple participants while ensuring the privacy of each local dataset. FEDAvG [34] is the foundational algorithm that established the general framework for FL. Subsequent algorithms have been proposed to further improve FEDAvG in terms of efficiency and accuracy. Some of the work focuses on developing advanced aggregation
techniques from various perspectives such as distillation 46 39 model ensemble [30 40]. and
to better incorporate knowledge learned by local models. Moreover, to minimize the deviation of local models from the global model, many works focus on enhancing the local training procedures (21 2 51 29]. FEDXL (15) was proposed a novel FL problem framework for optimizing family of risk optimization problems via an active-passive decomposition strategy. Even though FEDXL deals with the loss decomposition for pairwise relations, our main focus and application are very different.
weight matching 45 52
Neighbor embedding. Neighbor embedding (NE) a family of non-linear dimensionality reduction techniques that rely on k-nearest neighbor (kNN) graphs to construct the neighboring relationships

---

within the dataset [6]. The key of NE methods lies in leveraging the interplay between attractive forces which bring neighboring data points closer and repulsive forces which push uniformly sampled non-neighboring data pairs further apart. t-SNE 44 is well-known NE algorithm. It first converts the data similarities to joint probabilities and then minimizes the Kullback-Leibler divergence between the joint probabilities of data pairs in the high-dimensional space and low-dimensional embedding space, Compared to I-SNE, UMAP [33] is better at preserving global data structure and more efficient in handling large datasets. A later study has analyzed the effective Ioss of UMAP[13] and demonstrated that the negative sampling strategy indeed largely reduces the repulsion shown in the original UMAP paper, which explains the reasons for the success of UMAP. Our federated NE work is built upon a recent work that has theoretically connected NE methods with contrastive loss [22 [9]. Their proposed framework unifies t-SNE and UMAP as a spectrum of contrastive neighbor embedding methods.
Decentralized dimensionality reduction. As nowadays datasets are often distributively stored, jointly analyzing the data from multiple sources has become increasingly important especially when the data contains sensitive information. SMAP 47 is a secure multi-party t-SNE. However, this framework requires data encryption, decryption, and calculations on the encrypted data, SMAP is very time-consuming and thereby can be impractical to run in real-world applications. dSNE was proposed for visualizing the distributed neuroimaging data [37]. It assumes that a public neuroimaging dataset is available to share with all participants. The shareable data points act as anchors for aligning the local embeddings. To improve the privacy and efficiency of dSNE, Faster AdaCliP dSNE (F- dSNE) 3X was proposed with differential privacy to provide formal guarantees. While their goal is not to collaboratively learn global predictive DR model and thus does not follow the formal FL defined the literature. Both methods require a publicly available dataset to serve reference gradients communicating across central and local sites. However, since public dataset may not be available in most real-world scenarios, our FEDNE is designed without any requirements for the shareable data.
protocols [2
FL Framework for Neighbor Embedding
In this section, we first provide background information on neighbor embedding techniques. We then formulate the problem within the context of FL and outline the unique challenges.
3.1 Contrastive Neighbor Embedding
The goal of general NE techniques is to construct the low-dimensional embedding vectors Z1..., NN E Rd for input data points II...., EN E R² that preserve pairwise affinities of data points in the high-dimensional space. The neighborhood relationships are defined via building sparse k-nearest-neighbor (kNN) graphs over the entire dataset with fixed & value. Contrastive NE [1]] is unified framework that establishes clear mathematical relationship among a range of NE techniques including t-SNE (44, UMAP 13], and NCVis 4.. via contrastive loss. For parametric NE, an encoder network fe is trained to map an input data point æ to a low-dimensional representation z, i.c., = fe(x).
In general. the contrastive NE algorithms first build ANN graphs to determine the set of neighbors p; for each data point x; in the high- dighediiensional space. A numerically stable Cauchy kernel is used for converting a pairwise low-dimensional Euclidean distance to a similarity measure: Φ(z;, x,) = 1 1+ 1+1 Then, the contrastive NE
| loss is optimized via the negative sampling strategy:
(())
E (og(6(fo(2s), So (o();))) bE 1ooglol1(g/60000
Attractive force
Repulsive force
where p: denotes the set of neighboring data points of æ;-
3.2 Problem Formulation
In general federated learning with one central server and M clients, each client holds its own training data D., = {a: 1Pr [P.rl and we denote the collective global data as Dptah The clients' datasets are disjoint which cannot be shared across different local sites, i.e., D., n D,n- = 0 for V m, m' E [M], and m a m'. Our goal is to learn a single neighbor embedding model such that the high-dimensional affinities of the global data can be retained in a global low-dimensional embedding (2D) space.
3

---

It is natural to consider employing the FEDAVG [3]] framework since the clients can collaborate by communicating their parametric NE models. Then, the learning objective can be formulated with T; Rj Dm as following:
M Σ
Dm
0" = arg min ((
Lm (0),
(2)
D
m=1
where (0)
log log()(f((2.)) fo
bE log(1 ((fo(x;), fo
(3)
Attractive force
Repulsive Force
As the client data cannot leave its own device, Eqqatiin cannot be solved directly. The vanilla FEDAVG relaxes Equation thrruugh 7 iterations of local training and global model aggregations. The fundamental procedures defined below,
Local: (
arg min L. (0), initialized with ((--1);
0
P
Global:
0
During local training, each participating client m updates its model parameter 6,, for only a few epochs based on the aggregated model ((t-1) received from the server.
3.3 Challenges of Federated Neighbor Embedding
However, besides the challenges posed by the non-IID data, simply decomposing the problem into Equation inndedd overlooks the pairwise data relationships existing across different clients. The major difference between the existing FL studies, e.g.. image classification, and Federated neighbor embedding is that the objective function of the former problems instance- saance-based, where their empirical risk is the sum of the risk from each data point:
1 [Dml
L. (0)
Σ
(x; (;; 0).
1Dmll
As result, the FL objective Equation 2
i.e., ΣM M
Pml
Lm (0), is exactly the one as all the [D]
clients' data were gathered at the server.
In the context of Federated neighbor embedding. Equation 3 only considers æ, to come from the same client as H,, Thus, simply adopting the vanilla FEDAVG framework can result in losing all the attractive and repulsive terms that should be computed between different clients.
Therefore, we redefine the Fl. objective of the contrastive neighbor embedding problem to be
M
(( B. (10
E(..) D [1(x; ((
(6)
172
X72
pf
Intra-client terms
Inter-client terms
where the pairwise empirical risk E(v,, ary; can be further defined as f(x,, xz; = [ E p.]
blg((1 - (oJ))
Attractive force
Repulsive
cannot be solved directly when æ; and æ, come from different clients. Specifically, this decentralized setting brings two technical challenges: (1) Biased repulsion loss: Negative sampling requires selecting neighor pairs uniformly across the entire data space. Under the FL setting, it is difficult from outside of its local dataset. (2) Incorrect attraction loss: Each client only has access to its local data points. This partitioning can result in incomplete INN graph, leading to incorrect Pis as some true neighbors of a data point might reside other clients.
Nonetheless, since the inter-client pairwise distances are unknown, Equation 7
a client to sample

---

Training
Update Surrogate Model
Server
Distribute NE model
Client M
Upload Surrogate Model
Client
+ Distribute Clients'
Client 1
Surrogate Models
S Local Training
Upload Model
fm
An illustration of one round of FEDNE. Besides the general steps in FEDAVG: @ 3 -> @, our local surrogate model training (@) can be smoothly integrated into the whole pipeline. Then, each client conducts its local training (5) using the augmented data and the surrogate models received from all the other clients (3)
Figure 1:
Federated Neighbor Embedding: FEDNE
To address the aforementioned challenges in applying the FL framework to the neighbor embedding problem, we develop learnable surrogate loss function" tunaiion"trainedd data augmentation technique to tackle the problems in repulsion and attraction terms separately. The two components be smoothly integrated into the traditional FEDAVG pipeline shown in Figure
trained by each client and intra-client
4.1 Surrogate Loss Function
The repulsive force plays an important role in ensuring separation among dissimilar data points, contributing the global data arrangement in the embedding space. In the centralized scenario, each data point can uniformly sample its non non-neigle noi-heighbors across the entire dataset. In contrast, in the federated setting, as each client can only access its local data, the dissimilar points residing in other clients are invisible, and all the repulsion will be estimated within its own data space. In particular, under severe non-Ill) situations, where the data distributions across different clients vary significantly 54 the non-nein nonnhhegg samples selected to repel are very likely to come from the same clusters in high-dimensional space. As showcased in FFiurre 2 (a), without explicitly taking care of the repulsion between dissimilar data pairs across clients, those points may still overlap with each other in the embedding space.
R for each client such that fup b loog(1 - <xfe(x;) (x;))) to estimate the repulsion, where z; and Iy come from different clients. This surrogate model, once shared, enables other clients input their local data points and obtain pre-estimated repulsive loss to data points from the originating client.
At high level, seek learn function f
Surrogate model training. We learn forar via supervised learning at each round of FEDAVG. To do so, generate set of owwddimensional query points inputs and pre-compute their corresponding repulsive loss to client m m's data points based on the current projection model. We choose to sample set of points Zg within 2D space for the following are uniformly selected across the data space, query points are not required to maintain any specific affinity with D... Second, because the high-dimensional space is often much sparser than 2D space, generating sufficient high-dimensional samples to comprehensively represent the data distributions of all other clients is impractical. Therefore, each client employs grid sampling strategy every round, using a predefined step size and extensive ranges along the two dimensions. This procedure is informed by client m's current embedding positions, ensuring more manageable and representative sampling process within the embedding space.
reasons. Firstly, non-neighboring points
In sum, given the sampled inputs 'n {z ,......, Sax, computing the repulsive loss between each by and b random data points in Dm, i.e., pup
we prepare the training targets by
log(1- 6(zg; 2(1)). Then, the dataset for supervised training the surrogate repulsion function Ma is constructed as D, = {(2g, tep pepp 1P
We use surrogate loss function and surrogate model interchangeably.
5

---

Implementation details. After building the training dataset, each client trains its surrogate model Jm using an MLP with one hidden layer to learn the mapping between the input embedding vectors and their corresponding repulsive loss measured within the client data by minimizing the mean squared error (MSE). The training objective is formulated as follows:
Del
w = arg min
pcp 41
mmw (2g.)22.)) (2g.))
(8)
D4
4.2 Neighboring Data Augmentation
To mitigate the limitations of biased local ANN graphs and ensure better neighborhood representation, we propose an intra-client data mixing strategy. This approach generates additional neighboring data points within each client, thereby enhancing the local data diversity. To be specific, locally constructed kNN graphs can be biased by the client's local data distribution. As the associated data pairs for computing the attractive loss are distributed across multiple clients, the local neighborhood within each client can be very sparse. Consequently, data points within a client may miss some of their true neighbors (i.e., invisible neighbors) considered in the global space. Moreover, when the local data is extremely imbalanced compared to the global view, constructing the kNN graph with fixed value may result in incorrect neighbor connections between very distant data points (i.e.. false neighbors). As demonstrated in Figure 2 (b), since data is partitioned across different clients, with fixed value, each local k-N graph can be even more sparse and erroneously connect very distinct data points.
Intra-client data mixing, To address these problems, we employ straightforward yet effective strategy, intra-client data mixing, locally generate some data within client by interpolating between data points and their neighbors. Our method shares a similar spirit to the mixup augmentation [53] (3]. In detail, given a data point æ and the set of its is nearest neighbor NN₂ (x:), a new data point generated by linearly interpolating æ; and random sample in (æ; denoted as Z.;:
Xm + X)any
where À is the weight sampled from the Beta distribution i.e., À ~ Beta(o).
4.3 Overall Framework
Once each client has received the surrogate loss functions of all the other participants (i.e., step 4 in Figure I). proceeds to its local training. By combining the original NE loss with the surrogate loss function the augmented local training data, the new objective for client can be formulated as:
Lm (Dm: = Σ (og( fo(())
[Dml
Σ ij
log(1 - ((fo(z""))))))))) fo(o"))
D
Original attractive loss
Original repulsive loss
iDm''
Smer (fo(z)"))
(10)
Surrogate repulsion from client mi
where ai", az) E D', i.e., the augmented training set. For simplicity, we use p to represent the is the surrogate model received from another client m'.
neighbor set constructed within Dmn fmep sw
Computation. At first glance, FEDNE may seem to introduce heavy computational overhead compared the original FEDAVG framework, as it requires additional surrogate model training at every round. Moreover, client needs to use multiple received surrogate models to do inference using its own local data. Nevertheless, we want to emphasize that the surrogate model contains only one hidden layer and takes 2D data points as inputs. Therefore, training and using them is manageable. We conducted experiments using MNIST with 20 clients on a server with NVIDIA GeForce RTX 2080 Ti GPUs Compared to FEDAVG, our FEDNE takes 35% more GPU time to complete one round of training. For future speed-up, we may consider applying strategies such as clustered FL and we leave this for future work.

---

Global View ANN Graph (k=
Global View of the KNN Graph (K=2)
Global
Data Partition
Aggregation
Projection Inter- Repulsion
Local ANN Client
Local KNN of Client
(a)
(b)
Figure 2: Toy examples for illustrating the major challenges in solving Federated NE. Color denotes the client identity, and different shapes represent the true categories of the data points (just for demonstration purposes). (a) Without repelling the dissimilar data from other clients, the projected data points from different clients may overlap with each other in the global embedding space. (b) Biased local kNN graphs may incorrectly connect distant data pairs as neighbors.
5 Experiments
5.1 Experimental Settings
Datasets. We conduct experimental studies on four benchmark datasets that have been widely used in the field of dimensionality reduction [35 555: MNIST (26), Pashion-MNIST 48), mouse retina
single-cell transcriptomes R32, and CIFAR-10 10 25 Their statistical information and general settings
are summarized in Table I| For CIFAR-10, 10, since the Fuclidean distances in the pixel space of natural image dataset are not meaningful to preserve 77, we adopted ImageNet- pererrrn--e ResNet-34 [17 to extract set of 512D feature vectors as input data. The resulting vectors still retain category-relevant structures that can be suitable for the Euclidean metric.
Non-IID data partition. We consider two partitioning strategies to simulate the heterogeneous client distributions: (1) Dirichler: For class
sample 9. from Dir(a) and assign data samples of that class e to a client m proportionally to ge [m]... The hyperparameter a controls the imbalance level of non-III condition [27 [8] (2) Shards:
the data partition where smaller indicates more each client holds data from C' classes, and all samples from the same class are randomly and equally
divided among all clients (27).
Evaluation metrics. We assess the quality of data embeddings by analyzing the input high- dimensional data points and their corresponding 2D positions [14] First, to evaluate the preservation of neighborhood structures, we compute trustworthiness and continuity scores. Trustworthiness quantifies the quality of a low-dimensional embedding by checking whether neighbors in the high- dimensional space remain the same as the ones in the embedded ow-dimensional space. Conversely, continuity verifies whether the neighbors the embedded space correspond neighbors in the original input space. We use ANN classification accuracy to measure the effectiveness in preserving the nearest neighbors in the embedded space, where higher scores indicate better class discrimination.
We fix A 7 for all the neighborhood metrics. Additionally, we employ steadiness and cohesiveness metrics to evaluate the reliability the global inter-cluster structures (20). Steadiness assesses the presence of false groups, while cohesiveness checks for the existence of any missing groups.
Implementation and training details. We employ a fully connected neural network with three hidden layers for MNIST, Fashion-MNIST, and CIFAR- 10 datasets and network with two hidden layers for the RNA-Seq dataset. In all experiments, we use Adam optimizer [24] with learning rate annealing and batch size of 512 where the batch size refers to the number of edges in the constructed kNN graphs. Furthermore, we assume full participation during the federated learning and each client performs one epoch of local training (E 1). In addition, we set a 0.2 to perform the intra-client data augmentation in our study.
Baselines. We consider four approaches compare with our FEDNE, (1) LocalNE: each client trains the NE model using only its local data without any communication. Two baseline methods:
7

---

FedAvg+NE and (3) FedProx+NE are implemented by applying the widely used FL frameworks 34 28 to NE model training. (4) GlobalNE: the NE model trained using aggregated global data, serving as the upper bound for performance comparison. Moreover, we want to emphasize that we do not include dSNE and F-dSNE for comparison. Although, at first glance, their titles might imply that they tackled a similar problem, their method is built upon the non-paaaeetric t-SNE and heavily relies on the shareable reference dataset. Thus, they are not comparable with our framework.
Table 1: Dataset statistics and learning setups.
Dataset MNIST
#Class
#Training
#lest
#Clients(MM)
#Dimension
10
60K
10K
20/100
784
Fashion-MNin
10
60K
10K
20/100
784
scRNA-Seq
12
30K
4.4K
20/50
50
CIFAR-10
10
50K
10K
20/100
512
5.2 Results
We conduct comprehensive experiments under various non-lID conditions and then evaluate on the global test data of each four datasets. For the highly imbalanced scRNA-Seq dataset, we only consider the Dirichlet partitions. The results of partitions under Dirichlet distributions are summarized in Table Overall, our FEDNE outperforms the LocalNE by 2.62%, 6. 1.%, 14.31% 12.69%, and 7.31% on average under the five metrics (i.e., conti., trust., KNN acc., stead., and cohes.) respectively. In addition, the results of the Shards setting can be found in the appendix, i.c., Table 9
Improved preservation of true neighbors. Both FEDNE and the baseline approaches achieved relatively high continuity scores, indicating that the models can easily learn how pull the data points that are similar in the high-dimensional space closer in the 2D space. However, the lower trustworthiness scores observed with the two baselines, Ne and NedProx+NE, imply that without properly addressing incorrect neighborhood connections and separation of data points across different clients, the resulting embeddings may contain false neighbors. Consequently, points that are positioned closely in the 2D space might not be neighbors in the original high- fech-dimensionnll space.
Enhanced class discrimination in the embedding space. Our method has significantly improved the ANN classification accuracy compared to the baseline results. This improvement highlights the limitations of locally constructed k-NN graphs, which may incorrectly pull distant data pairs closer in the embedding space. In particular, if two data points from different classes are mistakenly treated as neighbors, class separation will be largely reduced even when inter-client repulsion is considered. Our intra-client data mixing method is specifically designed to relax this problem, and when combined with our surrogate loss function, it ensures an enhanced class separation. For instance, the embedding visualization of FedAvg+NE in Figure Judder the Dir(0.1) setup shows significant overlap between points from different labels. In contrast, FEDNE effectively separates the top groups of features in the visualization.
Better preservation of global inter-cluster structures. Furthermore, observe large improve- ments in preserving the clustering structures according to measures of steadiness and cohesiveness. Specifically, higher steadiness achieved by FEDNE indicates that the clusters identified in the pro- jected space better align with the true clusters in the original high-dimensional space. The higher cohesiveness scores imply that the clusters in the nigh-dimensional space in general can be retained in the projected space, i.e., not splitting into multiple parts. Overall, even though FEDNE is not explicitly designed to improve feature clustering, it can produce relatively reliable inter-cluster structures.
5.3
Ablation Study
To verify the effect of our design choices, we conduct ablation studies on removing one of the proposed technical components from the FEDNE pipeline in Figure I] First, we remove the data augmentation by intra-client data mixing technique and only keep the surrogate repulsion model. We then remove the surrogate model and only augment the local data using the intra-client data mixing approach. The comparison results under the setup of Dir(0.1) with 20 clients are shown in Table 3 With any of the components removed, our FEDNE can still outperform the baseline FedAvg+NE. However, we cannot simply conclude which component impacts the most on the final embedding results since the data characteristics and client partitions are very different across different setups. Further studies on our design choices are included in the appendix.
8

---

Table 2: Quality of the global test 2D embedding under the non-lID Dirichlet distribution (Dir(0.1) and (ir().5)) on four datasets. FEDNE achieves top performance on preserving both neighborhood structures (i.e,, continuity, trustworthiness, and kNN classification accuracy) and global inter-cluster structures (i.e., steadiness and cohesiveness).
MNIST
RNA-Seq
Metric
Method
M
20
M
100
M
20
M
100
Ad
20
J
50
J1 20
Jd 100
0.5
0.1
0.5
0.1
0.5
LocalNE
0,91
0.95
0.93
0.95
0.96
0.98
0.97
0.98
0.95
0.97
0.96
0.97
0.87
0.92
0.87
0.91
FedAvg+NE
0.97
0.98
0.96
0.97
0.98
11.99
0.99
0.99
0.97
0.98
0.97
0.98
0.993
0.94
0.93
0.94
Comi.
FedlProxNN:
0.96
0.97
0.99
0.97
0.98
0.93
FEDNE
0.97
0.97
0.96
0.97
0.99
0.99
0.99
0.97
0.98
0.97
0.98
0.93
0.94
0.93
0.94
GlohaINE
0.97
0.99
0.98
0.95
LocalNts
0.93
0.86
0.74
0.81
FedAvg+NE:
0.78
0.88
0.96
0.85
0.89
0.82
0.86
Trust.
FodPhox-NB
0.78
0.91
0.75
0.58
0.95
0.97
0.89
0.96
0.84
0.90
0.83
0.89
0.81
0.86
0.80
0.85
PEDNE
0.85
0.93
0.82
0.90
0.95
0.97
0.95
0.96
0.87
0.91
0.85
0.91
0.83
0.86
0.80
0.85
LocalNE
0.44
0.66
0.43
0.58
0.53
0.64
0.53
0.60
0.81
0.89
0.80
0.89
0.44
0.58
0.43
0.55
TedAvg+NE:
0.48
0.76
0.43
0.67
0.60
0.70
0.55
0.66
0.85
0.94
0.84
0.93
0.55
0.72
0.48
0.68
FedProxeN::
0.68
0.54
0.66
0.83
0.94
0.93
0.55
0.71
0.69
PEDNE
0.72
0.89
0.65
0.78
0.66
0.70
0.66
0.67
0.90
0.96
0.88
0.95
0.63
0.77
0.54
0.73
GlobalNi
0.93
0.73
0.97
0.78
LocalNE
0.57
FedAvg+NE:
0.54
0.73
0.43
0.69
0.79
0.84
0.62
0.81
0.64
0.79
01.88
0.78
0.67
0.71
0.63
0.69
Stead.
FedPhox+NE
0.51
0.72
0.43
0.68
0.79
0.84
0.62
0.81
0.59
0.79
0.66
0.78
0.66
0.71
0.63
0,71
0.73
0.67
GlohaINE:
0.76
0.83
0.81
0.73
LocalNE
0.70
0.81
0.70
0.77
0.62
0.69
0.62
0.68
0.61
0.66
0.60
0.66
0.65
0.72
0.64
0.70
Ne
0.68
0.72
Cobes.
FedProx-NE
0.77
0.89
0.76
0.86
0.71
0.73
0.69
0.73
0.68
0.68
0.65
0.69
0.71
0.76
0.70
0.76
PEDNE
0,82
0.90
0.82
0.87
0.72
0.74
0.70
0.74
0.70
0.68
0.68
0.72
0.73
0.76
0.70
0.75
GlobalNi:
Shards(2)
Shards(3)
Dir(0.1)
Dir(0.5)
GlobaINE
Oroa
Cal
FedNE
NedAvg+NE
Figure 3: Visualization of the resulting global test 2D embeddings on the Fashion-MNIS: dataset under four FL setups of 20 clients (M 20). Table Ablation study on removing one of the components from FEDNE pipeline with MNIST and scRNA-Seq datasets
Fe
w/o Data Mixing
w/o Surrogate
FEDNE
Metric
MNIST
RNA-Seq
MNIST
RNA-Seq
MNIST
RNA-Seq
MNIST
RNA-Seq
Cont.
0.97
0.97
0.96
0.97
0.96
0.97
0.97
0.97
Trust.
0.78
0.85
0.85
0.86
0.80
0.86
0.85
0.87
ANN
0.48
0.65
0.54
0.88
0.72
0.90
Stead.
0.54
0.64
0.63
0.71
0.54
0.65
0.63
0.73
Cohes.
0.77
0.68
0.84
0.71
0.81
0.69
0.82
0.70

---

6 Discussion
Privacy Preserving. As introduced in (ecciinn 4] FEDNE incorporates the proposed surrogate models into the traditional FEDAVG framework where the surrogate models only take the low-dimensional randomly sampled data as inputs. After training, each surrogate model contains information about the corresponding client. Thus, we consider the privacy concerns to be alleviated as one cannot directly reconstruct the original high-dimensional client data. To further enhance privacy protection, our framework can be integrated with various stages. For example, Gaussian mechanisms (GM) can be applied to the parameters of the surrogate model before it is sent to the server.
techniques different
Scalability and Computational Complexity. To our knowledge, the field of dimensionality reduction (DR) focuses on relatively smaller-scale datasets, compared to the studies of classification problems. This is because computational complexity is never a trivial problem even for many outstanding DR techniques, particularly for non-linear methods such as Isomap and t-SNE which have non-convex cost functions [4]. In our experiments, we have included the most widely used benchmarks in the DR literature. Moreover, we have considered more participants and larger scales of data compared to prior work 38). While. unlike the other FL studies focused on classification, have not yet included much larger datasets or with increased numbers of clients, we expect our approach to be applicable in real-world settings, for example, cross-silo settings with manageable amounts of clients. In terms of computation, as discussed in section 4 our approach requires only 35% additional GPU time compared to FEDAVG, and we expect such overhead to remain similar when going to larger datasets. When the client number increases, we may optionally drop a portion of surrogate models in local training.
experiments
7 Conclusion
In this paper, we develop a federated neighbor embedding technique built upon the FEDAVG frame- work, which allows collaboratively training data projection model without any data sharing. To tackle the unique challenges introduced by the pairwise training objective in the NE problem, we propose learn surrogate model within each client to compensate for the missing repulsive forces. Moreover, we conduct local data augmentation via an intra-client data mixing technique to address the incorrect neighborhood connection within a client. We compare FEDNE four baseline methods and the experiments have demonstrated its effectiveness in preserving the neighborhood data structures and clustering structures.
ACKNOWLEDGMENT
We would like to thank the reviewers and Xinyu Zhou for their valuable feedback. This work is supported in part by the NSF-ffundd AI institute Grant OAC-21 12606 and Cisco Research.
References
[1] Sheikh S Ahdullah, Neda Rostamzadeh, Kamran Sedig, Amit X Garg, and Eric McArthur. Visual analytics for dimension reduction and cluster analysis of high dimensional electronic health records. In Informatics, volume 7, page 17. MDPI, 2020.
[2] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough. and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.042633 2021.
[3] Mohammed Adnan, Shivam Kalra, Jesse C. Cresswell, Graham W Taylor, and Hamid R Tizhonsh. Federated learning and differential privacy for medical image analysis. Scientific reports, 12(1):1953, 2022.
[4] Aleksandr Artemenkov and Maxim Panov, Novis: noise contrastive approach for scalable visualization. In Proceedings of The Web Conference 2020, pages
2020.
[5] Jinheon Baek, Wonyong Jeong. Jiongdao Jin, Jachong Yoon, and Sung Ju Hwang. Personalized subgraph federated learning. In International Conference on Machine Learning, pages 1396-1415. PMLR, 2023.
[6] Jan Niklas Böhm, Philipp Berens, and Dmitry Kohak. Attraction-repulsion spectrum in neighbor embed- dings. The Journal of Machine Learning Research, 23(1):118-44 4148,,444,,4444 2022.
[7] Jan Niklas Böhm, Philipp Berens. and Dmitry Kobak. Unsupervised visualization of image datasets using contrastive learning, arXiv preprint 2777992210009877 2022.
10

---

[8] Keith Bonawitz. Hubert Eichner. Wolfgang Grieskamp. Dzmitry Huba, Alex Ingerman, Vladimir Ivanov. Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi. Brendan McMahan, al. Towards federated learning at scale: System design. Proceedings of machine learning and systems, 1:374-388, 2019.
[9] Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, and Pankaj Gupta. Federated continual learning for text classification via selective inter-client transfer. arXiv preprint (..iv:22100060000 2022.
[10] Hong- You Chen and Wei-Lun Chao, On bridging generie and personalized federated lcarning for image classification, arXiv preprint arXiv:2107.0077: 2021.
[11] Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. Contrastive learning unifies I-sne and umap. arXiv preprint arXiv:2206.00 00666 2022.
[12] Sebastian Damrich, Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. From f-sne to umap with contrastive learning. In The Eleventh International Conference on Learning Representations, 2022.
[13] Sebastian Damrich and Fred A Hamprecht. On umap's 's true loss function. Advances in Neural Information Processing Systems,
2021.
[14] Antonio Gracia, Santiago González, Victor Robles, and Ernestina Menasalvas. A methodology to compare dimensionality reduction algorithms in terms of loss quality. Information Sciences, 270:1-27, 2014.
[15] Zhishuai Guo, Rong Jin, Jiebo Luo, and Tianbao Yang. Fedxl: Provable federated learning for doep x-risk optimization. 2023.
[16] Chaoyang He. Keshav Balasubramanian, Emir Ceyani, Carl Yang. Han Xie. Lichao Sun, Lifang He. Liangwei Yang, Philip S Yu, Yu Rong, et al. Fedgraphnn: A federated learning system and benchmark for graph neural networks. arXiv preprint arXiv:2104.07144 2021.
[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. IEEE, 2016.
[18] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribu- tion for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
[19] Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. arXiv preprint Xrx::2205. 14814, 2022.
[20] Hyeon Jeon, Hlyung-Kwon Ko, Jaemin Jo, Youngtaek Kim, and Jinwook Seo. Measuring and explaining the inter-cluster reliability of multidimensional projections. IEEE Transactions on Visualization and Computer Graphics, 28(1):551-56) 2021.
[21] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national conference on machine learning, pages 5132-5143. PMLR, 2020.
[22] Mashal Khan, Frank G Glavin, and Matthias Nickles. Federated learning a privacy solution an overview. Procedia Computer Science, 217:316 -265, 2023.
[23] Fahad Ahmed KhoKhar, Jamal Hussain Shah, Muhammad Attique Khan, Muhammad Sharif, Usman Tariq, and Seifedine Kadry. A review on federated learning towards image processing. Computers and Electrical Engineering, 99:107818, 2022.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[25] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, Canada, 2009.
[26]
Yann LeCun. The mnist database handwritten digits. http://yann. lecun, com/exdh/nnist//, 1998.
[27] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965-978. IEFF, 2022.
[28] Tian Li. Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi. Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Muchine learning and systems, 2:429-450 2020.
[29] Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance reduced local sgd with lower communication complexity. arXiv preprint Mrxiv:1992. 12844, 2019.
[30] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems, 33:2351-234 2020.
11

---

[31] Ming Liu, Stella Ho. Menggi Wang, Longxiang Gao, Yuan Jin, and He Zhang. Federated learning meets natural language processing: A survey. arXiv preprint arXiv:2107.12600 2021.
[32] Evan Macosko, Anindita Basu, Rahul Sauja. James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison R Bialas, Nolan Kamitaki, Emily M Marterstock, et al. Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, (01(5):122--2221 2015.
[33] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018
[34] Brendan McMahan, Eider Moore. Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficci learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1282-122.. PMLR, 2017.
[35] Michael Moor, Max Hom, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Interna- rional conference on machine learning, pages 7045-7054. PMLR, 2020.
[36] Deepak Ravikumar, Sangamesh Kodge, Isha Garg, and Kaushik Roy. Intra-class mixup for out-of- distribution detection. IEEE Access, 0725968-22988 2023.
[37] Debbrata K Saha, Vince D Calhoun, Sandeep R Panta, and Sergey M Plis. See without looking: joint visualization of sensitive multi-site datasets. In IJCAI, pages 2672-2678, 2017.
[38] Debbrata Kumar Saha, Vince Calhoun, Soo Min Kwon, Anand Sarwate, Rekha Saha, and Sergey Plis. Federated, fast, and private visualization of decentralized data. In Federated Learning and Analytics in Pructice: Algorithms, Systems, Applications, and Opportunities, 2023.
[39] Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-L.un Kim. 16 federated knowledge distillation. Machine Learning and Wireless Communications, page 457, 2022.
[40] Naichen Shi, Fan Lai, Raed Al Kontar, and Mosharaf Chowdhury. Fed-ensemble: Improving generalization through model ensembling in federated learning. arXiv preprint 1076622007.0063,, 2021.
[41] Debaditya Shome and Tejaswini Kar. Fedaffect: Few-shot federated learning for facial expression recognition. In Proceedings the IEEE/CVF international conference on compuder vision, pages 4168-
4175, 2021.
[42] Toyotaro Suzumura, Yi Zhou, Natahalie Baracaldo, Guangnan Ye, Keith Houck, Ryo Kawahara, Ali Anwar, Lucia Larise Stavarache, Yuji Watanabe, Pablo Loyola, et al. Towards federated graph learning for collaborative financial crimes detection. arXiv preprint arXiv: /909. 19909./2946 /2946, 2019.
[43] Laurens Van Der Maaten, Accelcrating t-sne using tree-based algorithms. The journal of machine learning research, ((1))3221-32455 2014.
[44] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9)1),, 2008.
[45] Hongyi Wang. Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos. and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.00r 064400 2020.
[46] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-eflinn federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[47] Jiazhi Xia, Tianxiang Chen, Iei Zhang, Wei Chen, Yang Chen, Xiaolong Zhang, Cong Xie, and Tobias Schreck. Smap: A joint dimensionality reduction scheme for secure multi-party visualization. In 2020 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 107-118. IEEE, 2020.
[48] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnin a novel image dataset for benchmarking machine learning algorithms. arXiv preprint (1Vv:17080007477, 2017.
[49] Zhirong Yang. Jaakko Peltonen, and Samuel Kaski. Scalable optimization of neighbor embedding for visualization. International conference on machine learning, pages 127-135. PMLR, 2013.
[50] Peihua Yu and Yunfeng Liu. Federated object detection: Optimizing object detection model with federated learning. In Proceddings of the 3rd international conference on vision, image and signal processing, pages 1-6, 2019.
[51] Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems. 14:5532--3444 2020.
[52] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pages 7252-7261. PMLR, 2019.
12

---

[53] Hongyi Zhang, Moustapha Cisse. Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint urXiv:1710.09412.22 2017.
[54] Hangyu Zhu, Jinjin Xu, Shiging Liu, and Yaochu Jin. Federated learning non-iid data: survey. Neurocomputing. 465:371-390, 2021.
[55] Xinrui Zu and Qian Tao. Spacemap: Visualizing high-dimensional data by space expansion. In ICML, pages 27707-27722 2022.
13

---

Appendix
A Experimental Details
A.1 Local training
In the experiments of MNIST and Fashion-MNIST datasets, we use Adam optimizer with learning rate of 0.001 and batch size of 512 (i.e., the number of edges in the kNN graphs not the number of data instances). The learning rate was decreased by 0.1 at 30% and 60% of the total rounds. For the experiments with the single-cell RNA-Seq and CIFAR-10 dataset, the learning rate was initially set to x 10 10-4.0..0000 For negative sampling, we fix the number non-neighboring data points sampled per edge to be 5 (b 5).
Moreover, the surrogate loss function is introduced into the local training at 30% of the total rounds, primarily due to the following concerns: during each round of local training, the surrogate loss function in use was constructed using the global NE model from the previous round. Thus, to avoid dramatic deviations between the surrogate function in use and the NE model newly updated by local training, the surrogate function is integrated after the model has already gained a foundational understanding of the data structures and thereby the optimization process tends to be more stable.
A.2 Surrogate loss function training
The surrogate loss function of each client is fine-tuneds at every round from the previous model but the training set (i.e., D, = [Z (rr)) !PPp in the main paper) needs to be rebuilt according to the current global NE model. The surrogate function is optimized by minimizing the mean squared error (MSE) using the Adam optimizer with learning rate of 0.001.
B Design Choices and Hyperparameter Selections
B.1 Choice of k in building local KNN graphs
In the main experiments, a fixed number of neighbors (k=7) is used for building clients' &NN graphs. We conduct further experiments using different & values under the setting of Dirichlet(0.1) on the MNIST dataset with 20 clients. The results are shown in Table We found that within a certain range (i.e., 7 to 30), the performance of FedNE is relatively stable. When k is too large (e-g., k=50), the performance drops but our FedNE still outperforms the baseline methods, FedAvg+NE, Overall, this trend aligns with the general understanding of dimensionality reduction methods.
Table 4: Experiments different numbers of neighbors i in building the local &NN graphs. The experiments are conducted under the setting of Dirichlet(0.1) on the MNIST dataset with 20 clients. We found that within a certain range (i.e., to 30), the performance of FedNE is relatively stable.
Metric
Method
k: = 7
h: = 15
k = 30
k = 50
FedAvg+NE
0.97
0.96
0.96
0.96
Conti.
FEDNE
0.97
0.97
0.96
0.96
FedAvg+NE
0.78
0.77
0.77
0.77
Trust.
FEDNE
0.85
0.85
0.84
0.79
FedAvg+NE
0.48
0.47
0.45
0.45
INN
FEDNE
0.72
0.69
0.67
0.54
FodAvg+NE
0.54
0.50
0.51
0,51
Stead.
FEDNE
0.63
0.64
0.62
0.55
FodAvg+NE
0.77
0.75
0.77
0.77
Cohes.
FEDNE
0.82
0.84
0.82
0.77
B.2 Weights in intra-client data mixing strategy
We fixed the to be 0.2 in the main experiments to perform intra-client data augmentation. Here, we alter the weight used in the intra-client data mixing strategy. We adjust the mixing weight by changing the value, where À ~ Beta(or). The experiments are conducted under the setting of Dirichlet(0.1) on the MNIST dataset with 20 clients. In the ablation study shown in Section 5.3 Table 3, we demonstrated the effectiveness of adding intra-client data mixing strategy. These
14

---

Figure 4: Experimental results different step sizes in grid sampling for training the surrogate models. The experiments are conducted under the setting of Dirichlet(0.1) on the MNIST dataset with 20 clients. In the main paper, the default step size is set 0.3. The results demonstrate that the performance of FedNE is stable when the step size is below 1.0.
0.9
0.8
5c00
0.7
Cont.
Trust.
KNN Acc.
Stead.
0.5
Cohes.
FedAvg
0.4
0.0
0.4
0.8
1.2
1.6
2.0
Step size
additional results shown in Table demonstrate that FedNE is very stable across different mixing weights.
Table 5: Experimental results on different weights used in intra-client data mixing strategy. We adjust the mixing weight À by changing the a value. where À ~ (eta(a). The experiments are conducted under the setting of (irichlet(0.1) on the MNIST dataset with 20 clients. These additional results demonstrate that FedNE is very stable across different mixing weights.
Metric
Method
a 0.1
0.2
a = 0.3
a 0.4
FedAvg+NE
0.97
Conti.
FEDNE
0.97
0.97
0.97
0.97
FedAvg+NE FEDNE
0.78
Trust.
0.85
0.85
0.85
0.85
FedAvg+NE FEDNE
0.48
KNN
0.72
0.72
0.72
0.72
FedAvg+NE FEDNE
0.54
Stead.
0.63
0.63
0.62
0.63
FedAvg+NE
0.77
Cohes.
FEDNE
0.83
0.82
0.83
0.84
B.3 Choice of step size for grid sampling
The step size is used to control the resolution of grid sampling for training the surrogate models. In the main paper, the default step size sizes. The experiments are conducted under the setting of Dirichlet(O.11 on the MNIST dataset with 20 clients. The results in Figure 4 demoostrate that the performance of FedNE is stable when the step size is below 1.0. However, when we increase the step size beyond 1.0, we observe gradual decrease performance, especially terms of kNN classification accuracy and steadiness metrics. Despite this, FedNE still maintains better performance than FedAvg+NE.
0.3, and here, we experiment with using different step
B.4 Data source for training surrogate models
To construct the training set of the surrogate loss function in a more comprehensive and manageable way, each client employs grid-sampling strategy the 2D space. Here, we conduct experiments on MNIST and Fashion-MNIS' datasets to compare the performance between using our grid-sampling strategy and using only local 2D embeddings as the training data. (Table and Table 7 show the comparison results for MNIST and Fashion-MNIST test data, respectively. We highlight the better
15

---

results in both tables. The -ampling method outperforms the baseline approach (i.e., only using local embedding in surrogate function training), while the baseline still achieves better performance compared to FedAvg+NE. Overall, the results validate the effectiveness of employing the surrogate loss function during local training and also support our proposed grid-sampling strategy.
Table 6: Quantitative comparison between the performance of using our grid-sampling strategy and using only local 2D embeddings surrogate training data. The following results are experimented with the FL setting of 20 clients and two different Shards partitions on the MNIST test data.
Shards(2)
Shards(3) (3)
Metric
Method
Local
Grid
Local
Grid
GlobalNE
0.97
0.97
Cont.
FEDNE
0.96
0.96
0.96
0.97
NedAvg+NE
0.96
0.97
GlobalNE
0.94
0.94
Trust.
FEDNE
0.86
0.87
0.90
0.91
FedAvg+NE
0.83
0.89
GlobalNE
0.93
0.93
ANN
FEDNE
0.69
0.71
0.84
0.87
FedAvg+NE
0.54
0.73
Table Quantitative comparison between the performance of using our grid-sampling strategy and using only local 2D embeddings surrogate training data. The following results are experimented with the FL setting 20 clients and two different Shards partitions on the FasHion-MNIST test data.
Shards(2)
Shards(3)
Metric
Method
Local
Grid
Local
Grid
GlobalNE
0.99
0.99
Cont.
FEDNE
0.98
0.99
0.98
0.99
FedAvg+NE
0.99
0.99
GlobalNE
0.97
0.97
Trust.
FEDNE
0.95
0.96
0.93
0.96
NedAvg+NE
0.93
0.96
GlobalNE
0.74
0.74
ANN
FEDNE
0.65
0.67
0.64
0.69
FedAvg+NE
0.59
0.66
B,5 Frequency of surrogate function update
In all experiments, the surrogate loss functions are retrained at every round. While frequent retraining introduces computational burdens for each client, using outdated surrogate loss functions bias the optimization process of the repulsive loss. Thus, we conduct experiments on the MNIST dataset to showcase the impacts of the frequency of surrogate function updates. We conducted experiments with the other four setups, i.e., retraining the surrogate function every 5, 10, 15, or 20 more rounds. The default one is updating the surrogate function at every round. According to the line chart in Figure 5) the performance decreased dramatically only when the surrogate loss function was updated more than every 10 rounds.
B.6 Time to integrate the surrogate loss function
In the main experiments, the surrogate loss function is integrated into local training after 30% of the total rounds have been finished. Here, we conduct further experiments on introducing the surrogate function at different time periods to confirm our decision, and the results are demonstrated in Figure 6 First, the continuity not affected too much and retains high scores under various setups. However, when the surrogate loss function is introduced too early, the trustworthiness and kNN accuracy drops, which may indicate that the inter-client repulsion better be involved after the intra-client forces have become relatively stable. Moreover, the performance of 55% also drops, particularly on ANN accuracy. This could be because the training process of FEDAVG has converged, making it too late to integrate additional constraints into the training procedure.
16

---

0.7
Cont.
0.6
Trust.
KNN
10
Rounds
Figure 5: The quantitative evaluation on testing the surrogate function under five different retraining frequencies. The line chart shows the results of the MNIST test data with the FL setting of 20 clients and the Shards partition with two classes of data per client.
0.95
0.85
Cont.
Trust.
KNN Acc.
0.75
0.70
10%
Percentage rounds
Figure 6: The quantitative evaluation on four different periods to introduce the surrogate function. The line chart shows the comparison results the MNIST test data with the FL setting of 20 clients and the Shards partition with two classes of data per client.
C Partial Client Participation
We simulate the straggler problem by randomly sampling 10% of the clients involved in each round of communication under the setting of (irlchlet(O.1) the MNIST dataset with 100 clients. While the performance under partial client participation is worse than under full client participation, the results in Table 8
show that FEDNE still performs notably better than the baseline method, FedAvg+NE.
D Evaluation Results on the Shards Setting
addition to the table the main paper, we also report the results the Shards setting on MNIST, Fashion-MNIST, and CIFAR-10 10 datasets in Table 9
E Visualization Results by FEDNE
We demonstrate the comprehensive visualization results from the centralized setting i.e., GlobaINE, our FEDNE and Fe,Avg+NE, and FedProx+NE on MNIST, Fashion-MNISTT RNA-Seq, and CIFAR- 10 global test data. The five metrics are introduced in the main paper.

---

Table 8: We experiment with partial client participation by randomly sampling 10% of the clients involved in each communication round under the setting of (irichlet(0.1) on the MNIST dataset with 100 clients. While the performance under partial client participation worse than under full participation, the results of FEDNE are still notably better than the baseline method.
Metric
Method
100% Participation 10% Participation
FedAvg+NE
0.96
0.95
Conti.
FEDNE
0.96
0.96
NedAvg+NE FEDNE
0.74
0.73
Trust.
0.82
0.79
FedAvg+NE
0.43
0.41
kNN
FEDNE
0.65
0.58
FedAvg+NE FEDNE
0.43
0.41
Stead.
0.57
0.50
FedAvg+NE
0.74
0.72
Cohes.
FEDNE
0.82
0.80
Table 9: Quality of the resulting 2D embedding under the Shards setting (C = 2 and C 3) on global test data of the tree datasets. FEDNE achieves top performance on preserving both neighborhood structures (i.e., continuity, trustworthiness, and ANN classification accuracy) and global inter-cluster structures (i.c., stcadiness and cohesiveness).
MNIST
Fashion-MNIST
CIFAR-10
Metric
Method
M =
M 100
M
20
M 100
M 20
M 100
C=
C'
C C=2
C= 2
C 3
C C=
C
C C=
C
C ==2 2
C
LocalNE
0.97
FodAvg4NE
0.96
0.97
0.97
0.97
0.99
0.99
0.99
0.99
0.93
0.94
0.93
0.94
Conti.
FedProx+NE.
FedNE
0.96
0.97
0.97
0.97
0.97
0.99
0.99
0.99
0.99
0.99
0,93
0,93
0.94
0.93
0.94
0.94
GlobaINE
LocalNE
0.74
0.77
0.75
0.78
0.88
0,91
0.90
0.92
0.72
0.77
0.72
0.76
FedAvg+NE
0.83
0.89
0.84
0.89
0.93
0.96
0.94
0.96
0.80
0.85
0.80
0.83
FedProk+NE
0.82
0.88
0.84
0.89
0.94
0,96
0.93
0.96
0.80
0.85
0.80
0,83
0.91
0.87
0.96
0.96
0.81
0.84
GlobaINE
0.94
0.97
0.87
LocalNE
0.41
0.47
0.41
0.50
0.53
0.56
0.52
0.57
0.40
0.47
0.40
0.48
FedAvg+NE
0.54
0.73
0.54
0.73
0.59
0.66
0.60
0.65
0.51
0.64
0.533
0.65
ANN
FedProxtNE.
0.55
0.73
0.55
0.73
0.59
0.66
0.60
0.65
0.52
0.65
0.524
0.65
FedNE
0.71
0.87
0.73
0.80
0.67
0.69
0.66
0.67
0.56
0.70
0.582
0.72
GlobaINE
0.93
0.73
0.78
LocalNE
0.47
0.46
0.61
0.68
0.70
0.54
0.55
0.59
FodAvgeNE
0.53
0.66
0.52
0.68
0.75
0.83
0.70
0.82
0.63
0.69
0.66
0.69
Stead.
Ne
0.52
0.60
0.55
0.75
0.81
0.63
0.66
0.69
FedNE
0.64
0.72
0.64
0.71
0.81
0.82
0.80
0.82
0.66
0.71
0.67
0.70
GlobaINE
0.76
0.83
0.73
LocalNE
0.69
0.74
0.70
0.74
0.61
0.65
0.63
0.66
0.63
0.67
0.63
0.67
BedAvg+NE
0.72
0.74
Cobes.
FedProk+NE
0.81
0.86
0.83
0.88
0.69
0,73
0.69
0.72
0.71
0.76
0.71
0.75
FedNE
0.75
GlobaINE
0.89
0.74
0.78
18

---

Shards(2)
Shards131
Dir0.1)
Dirto.9
Sharda/2)
Sharda
Der(B.9)
Dir(o.5)
GlobaINE
PedAvg*Re
Figure 7: Visualization results from centralized setting, FEDNE and FEDAVG on MNIST test dataset under eight different FL settings (i.e., Shards with two classes or three classes per client; and Dirichlet with 0.1 0.5).
20 Clients
Clients
Sharda(2)
Shards(3)
Dir0.1)
Dir(.5)
Sharda(3)
Dir(R.1)
Dir(o.5)
GlobaINE
Year
FedNl
FedAvg+NE
Figure 8: Visualization results from centralized setting, FEDNE and FEDAVG on Fashion-MNIST test dataset under eight different FL settings (i.e., Shards with two classes or three classes per client; and Dirichlet with 0.1 or 0.5).
20 Clients
50 Clients
Dir(0.1)
Dir(0.5)
Dir(o.1)
Dir(0.5)
GlobaINE
FedAug+NE
Figure 9: Visualization results from centralized setting, FEDNE and FEDAVG on ScRNA-Seq test dataset under four different FL settings (i.e., Dirichlet with 0.1 or = 0.5).
19

---

20 Clients
100 Clients
Sharda(2)
Shards131
Dir00.1)
Dir10.5)
Shards|21
Sharde
Dir10.11
Div(o.5)
GlobaINE
FedNE
FedAvg+NE
Figure 10: Visualization results from centralized setting, FEDNE and FEDAVG on scRNA-Seq test dataset under four different FL. settings (i.e., Dirichlet with a = 0.1 0.5).
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: Our claims made in the abstract and introduction sections accurately reflect our paper's scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our work in terms of computational efficiency and future work for improving current method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
21

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have provided detailed formulations for our proposed approach and analysis made in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the implementation details and experimental setups in our experiments and results sections, and also in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
22

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The main implementations will be available soon
under
http://gginbb.om//www0002///
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have included the training and experimental details in our experiments section and also in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We did not report the error bars or any statistical significance tests, but we did include the necessary statistical information our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we have included the information in the main paper and also the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
24

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: Yes, our work was conducted with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not aim to address the societal impacts, and discussing societal impacts is not applicable to our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not have any risks related to this.
Guidelines:
25

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: Our paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo/caaaoooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our paper does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
26

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not need to involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "Zq4gzMiJys", "reviewer": "Reviewer_BJv7", "summary": "This paper addresses the challenge of distributed neural embedding (NE) with a focus on privacy protection. To achieve this, the authors extend the concept of federated learning (FL) to NE. However, NE tends to diverge because FL prevents clients from accessing each other's data, leading to inconsistent feature spaces across clients. To mitigate this issue, the authors employ surrogate loss models trained locally, which are then broadcast to all other clients to serve as an anchor. The experiments show promising performance compared to existing baselines.", "questions": "see weakness.", "limitations": "The authors have discussed some limitations. However, the authors are encouraged to discuss the use cases of the proposed method, such as cross-silo settings. Moreover, they are encouraged to discuss the additional privacy risks potentially introduced by their method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated and well-written.\n2. The problem is practical and useful for many real-life applications, though scalability may be the main constraint.\n3. The idea is straightforward, and the experiments seem to verify its effectiveness.", "weaknesses": "1. **Communication complexity**: If I understand correctly, every client in the proposed method must broadcast the surrogate models to all other clients. Although the surrogate models consist of only one hidden layer, this design results in a communication complexity of $\\mathcal{O}(N^2)$. As the number of clients in the system increases, the additional communication costs will rise dramatically. This might be manageable in some cross-silo settings, where only a few clients participate.\n\n2. **Straggler effect**: Following point (1), the proposed method requires communication among clients. However, clients may drop out during training. It would be insightful if the authors could analyze how missing surrogate loss models would affect overall performance.\n\n3. **Additional privacy concerns**: Sharing surrogate models introduces additional privacy risks, e.g., enabling reconstruction attacks or membership inference. While some recent work empirically shows that such private information is less leaked after distillation (e.g., [1] and [2]), the proposed method might be more vulnerable to privacy attacks without differential privacy.\n\n[1] Dong, Tian, Bo Zhao, and Lingjuan Lyu. \"Privacy for free: How does dataset condensation help privacy?.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Wang, Hui-Po, et al. \"Fedlap-dp: Federated learning by sharing differentially private loss approximations,\" Proceedings on Privacy Enhancing Technologies, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Communication complexity ... this design results in a communication complexity of $O(N^2)$ … This might be manageable in some cross-silo settings, where only a few clients participate.\n\nThanks for the thoughtful comment. Since each client will receive the surrogate models of all other clients from the server, we acknowledge that the size of the communication will be $O(N^2)$. We totally agree that our framework is more manageable in some cross-silo settings. However, as the size of the surrogate model is very small containing only one hidden layer (e.g., around 20K bytes), and nowadays data transfer has become much more efficient and affordable (e.g., 1.2 GB/s in modern wireless services), in our humble opinion, communication cost should not be a big problem.\n\n> Straggler effect … However, clients may drop out during training. It would be insightful if the authors could analyze how missing surrogate loss models would affect overall performance.\n\t\t\nThank you for the nice practical question. We follow your suggestion to experiment with the idea that only a random 10% of the clients are involved in each communication round. Specifically, these clients will only receive the surrogate models from the other 10% of the clients. We summarize **our results in the attached PDF**, under the setting of Dirichlet(0.1) on the MNIST dataset with 100 clients. We can see that while the performance under 10% client participation is worse than under full client participation, **the results of FedNE are still notably better than the baseline methods, FedAvg+NE**, demonstrating the effectiveness and applicability of our FedNE framework.\n\n> Additional privacy concerns: Sharing surrogate models introduces additional privacy risks, … the proposed method might be more vulnerable to privacy attacks without differential privacy.\n\nThank you so much for the comment. First, we want to apologize for the confusion caused by our title. The main focus of our paper is on a Federated setting, aiming to bring up the unique FL challenges and propose effective solutions for the pairwise training objective in the problem of Federated Neighbor Embedding. We appreciate your suggestions and will extend our discussion by adding a section in the camera-ready version to incorporate the points that you mentioned.\n\nBesides the privacy-preserving properties that a Federated setting has introduced, we do not specifically develop any privacy-preserving mechanisms. Therefore, we will surely refine our title and any writing to clarify this.\n\nWith that being said, privacy-preserving is an important topic and we will explore incorporating differential privacy (or other techniques) in our future work. We will also investigate some characteristics of our algorithm to enhance privacy. For example, our surrogate models take the “low-dimensional” data as inputs. We will investigate whether this design improves privacy preservation since one cannot directly reconstruct the high-dimensional data. Moreover, since it is the server that collects surrogate models from all the clients and distributes them to each client, certain anonymization techniques may be applied so that each client does not know the owner of the surrogate function it receives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vFgp7SQMqq", "reviewer": "Reviewer_4ZYS", "summary": "The paper presents a new federated learning approach named FEDNE for dimension reduction using contrastive neighbor embedding (NE). The key idea is the introduction of a surrogate loss function that each client learns and shares, which compensates for the lack of inter-client repulsion essential for global alignment in the embedding space. Additionally, the paper proposes a data-mixing strategy to augment local data, addressing issues of invisible and false neighbors in local kNN graphs. Comprehensive experiments demonstrate that FEDNE effectively preserves neighborhood data structures and enhances alignment in the global embedding space compared to several baseline methods.", "questions": "1. How would the parameter k affect the performance of FEDNE? How to set k for different settings?\n\n2. What are the major differences between the metrics? Why the improvement of FEDNE differ a lot across different metrics?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The studied problem is important. There could be many downstream tasks after applying federated neighbor embedding.\n\n2. Many metrics are included in the experiments to evaluate the quality of the resulting embeddings", "weaknesses": "1. The paper lacks investigation on the effect of choice of hyperparameter k.\n\n2. The improvement of FEDNE is significant on some metrics (e.g., kNN) but is very limited in other metrics (e.g., conti.). The paper lacks a detailed exploration of why FEDNE produces different behavior for different metrics.\n\n3. I suggest to highlight the best results in Table 2. Currently the results of FEDNE are highlighted although it may not achieve the best performance in some cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How would the parameter k affect the performance of FEDNE? How to set k for different settings? \n\nThank you for the valuable comment. First, we want to reiterate that the value k is used for building local kNN graphs to capture the neighboring data structures. In general, as k increases, we may lose the neighboring information. When k is too small, it may result in many isolated clusters in the embedding space. \n\nHowever, there is no clear consensus on how to select k in the dimensionality reduction (DR) community. As many recent DR papers use fixed k values [11], we followed them as well. We experimented with different k values under the setting of Dirichlet(0.1) on the MNIST dataset with 20 clients. **Please find the results in the attached PDF**.  We found that **within a certain range (i.e., 7 to 30), the performance of FedNE is relatively stable**. When k is too large (e.g., k=50), the performance drops but **our FedNE still outperforms the baseline methods**, FedAvg+NE. This trend aligns with the general understanding of DR methods. Once there is a best practice to select k for different settings, it can be applied to our problem.\n\t\n[11] “From t-SNE to UMAP with contrastive learning”. In ICLR, 2023.\n\n\n> What are the major differences between the metrics? Why the improvement of FEDNE differ a lot across different metrics?\n\nThanks for your thoughtful comment. We had some discussions in Sections 5.1 and 5.2, but we should have made this more clear. \n\nContinuity, trustworthiness, and kNN classification accuracy are mainly used to evaluate the preservation of neighborhood data structures in the global reduced-dimensional space. Steadiness and cohesiveness aim to measure the preservation of inter-cluster structures since clusters can be distorted when projecting to the global low-dimensional space. Among the five metrics, trustworthiness, kNN classification accuracy, and steadiness have much more noticeable increases compared to continuity and cohesiveness.\n\nThe improvement differs a lot among the metrics mainly because in a simple FedAvg framework without any special treatment for the data pairs across different clients, the model cannot penalize those data pairs to be apart, which results in introducing false neighbors in the embedding space (Line 215 and 292). These false neighbors can further cause data points from different classes to overlap in the embedding space. This explains why the baseline methods achieve much lower trustworthiness scores and kNN classification accuracy, as trustworthiness measures whether neighbors in the embedding space are also neighbors in the high-dimensional space.\n\nDue to similar reasons, without mitigating the incorrect neighbor connections and addressing the missing repulsion terms, the overlap in the embedding space will also mistakenly introduce false data clusters. Thereby, steadiness is also low for the baseline methods as it measures how well the model can avoid false clusters. However, both our surrogate models and intra-client data mixing strategy aim to prevent false neighbors in the embedding space. When true neighbors are preserved by FedNE, the trustworthiness and steadiness scores have increased and the points that are close in the embedding space are more likely to belong to the same class. Thus, the kNN classification accuracy is also notably increased (see Table 2 in the original manuscript).\n\nContinuity measures how well the neighborhood of a data point in the high-dimensional space is preserved in the embedding space. Achieving higher continuity is often easier even under the context of FL because the original attraction term in the client’s local objective function already pulls the neighbors closer in the embedding space. Cohesiveness is for measuring how well the projection model can avoid missing clusters. We still can observe relatively larger improvements in cohesiveness compared to continuity. Improved cohesiveness demonstrates that FedNE will not mistakenly break the true clusters in the embedding space. \n\n\n> I suggest to highlight the best results in Table 2. Currently the results of FEDNE are highlighted although it may not achieve the best performance in some cases. \n\nThank you for the suggestion. We will change to highlight the best performance in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3iE9Fc2nID", "reviewer": "Reviewer_mP1Q", "summary": "The paper \"FEDNE: Surrogate-Assisted Federated Neighbor Embedding for Privacy-Preserving Dimensionality Reduction\" presents a method for visualizing high-dimensional data while maintaining privacy without requiring any shareable reference data. \n\nFederated Neighbor Embedding (FEDNE): A framework combining federated averaging (FEDAVG) with contrastive neighbor embedding (NE) to create a joint NE model across multiple clients without compromising data privacy.\n\nSurrogate Loss Function: An innovative loss function to enhance inter-client repulsion in the global embedding space, ensuring better separation of data points from different clients while preserving local data structures.\n\nData-Mixing Strategy: A technique to counter issues like invisible and false neighbors in local k-nearest neighbor (kNN) graphs by mixing data from various clients during training, thus improving the quality of the learned embeddings.", "questions": "Privacy Guarantees: The paper lacks a thorough discussion on the privacy guarantees of the proposed method, especially against adversarial attackers.The experimental evaluation focuses solely on utility results, with no evaluation of data privacy. How is privacy preservation quantified and ensured? What is the acceptable level of privacy preservation? The paper should include theoretical arguments and experiments demonstrating actual privacy management. From a privacy perspective, it would be helpful to provide guidance on the limitations of this method, particularly regarding transparency and explainability (e.g., OECD AI principle 1.3). What measures are in place to address these concerns?\n\nExperiments: No experiments are conducted on downstream tasks related to the problem, aside from analyzing structural properties? Moreover,  no experiments conducted on large datasets? Lastly, how do the authors plan to tackle the problem of heavy computation? The method appears computationally intensive, which could hinder its practicality.\n\nReconstruction from Gradients: According to Zhu and Han’s study [1], model gradients can be used in some scenarios to partially reconstruct client data. How does the proposed method address this issue?\n\nThe paper claims to operate without relying on shareable reference data, yet it utilizes additional 2D data points from grid sampling to estimate the training targets via repulsion loss and additional augmented data points via interpolation for the attractive loss. Given this, how does the strategy address the significant computational burden it introduces, and is it feasible for real-world applications where computational efficiency is critical?\n\n[1] Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep leakage from gradients. Advances in neural information processing systems 32 (2019)", "limitations": "No societal impacts.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Well-Presented: The paper is clearly and coherently written, making it easy to follow.\nNovel Approach: The study addresses an important problem with a novel approach, combining federated learning with neighbor embedding techniques.", "weaknesses": "Privacy Concerns: While the approach is innovative, the paper does not sufficiently address privacy concerns. It lacks experiments and guarantees demonstrating the privacy preservation of the FedNE approach.\n\nComputational Inefficiency: The method appears to be computationally inefficient. There are no experiments conducted on large datasets, such as those in real-world medical or other privacy-critical domains, where computational complexity could be a significant issue.\n\nInadequate Analysis of Related Work: The related works section is not thoroughly analyzed or discussed, missing critical comparisons and context necessary for a comprehensive understanding of the state of the art.\n\nThe study's applicability could be strengthened by extending beyond benchmark datasets to encompass real-world, privacy-sensitive datasets found in domains such as healthcare or finance. This expansion would provide a more robust demonstration of the method's practical relevance and effectiveness. \n\nAdditionally, addressing pairwise issues associated with attraction terms is essential for improving the preservation of neighborhood structures and enhancing clustering quality. \n\nFurthermore, it is crucial to conduct thorough analyses aimed at optimizing the computational efficiency and scalability of the algorithms, ensuring their capability to handle large-scale datasets effectively. Moreover, the method currently lacks explicit consideration of privacy guarantees. And on elucidating how privacy concerns are addressed within the framework and formalizing privacy guarantees to assure users and stakeholders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Questions related to privacy concerns\n\nWe found that major concerns mentioned in the weakness and questions are related to “privacy-preserving”, and these concerns may arise from the “privacy-preserving” term in our paper title. First, we want to apologize for the confusion caused by our title. The main focus of our paper is on a FL setting, aiming to bring up the unique FL challenges and propose effective solutions for the pairwise training objective in the problem of Federated Neighbor Embedding. Besides the privacy-preserving properties that a Federated setting has introduced, we do not specifically develop any privacy-preserving mechanisms and we certainly do not claim that our approach introduces additional privacy guarantees. **We will surely refine our title and any writing to clarify this**.\n\nIntroducing formal privacy guarantees is not a trivial task. Among the only two existing methods, dSNE [37] proposes a decentralized neighbor embedding framework and later, they extend dSNE to be privacy-guaranteed by incorporating differential privacy (DP) as F-dSNE [38]. We have to admit that, developing privacy-guaranteed Federated NE methods while addressing the lack of inter-client data relationships is an important but challenging task. Therefore, **we will consider the privacy of FedNE as an orthogonal problem and will work toward incorporating DP or other techniques in our future work**. We will add discussions on how to enhance FedNE with privacy-preserving techniques.\n\n> Inadequate Analysis of Related Work\n\nThanks for the valuable comment. As our work intersects with two research areas: FL and DR, we have structured the related work section into three subtopics in Sect. 2. In fact, we have compared FedNE with two closely related works, dSNE and F-dSNE in Sect. 2. However, due to the page limitations, we might have overlooked some relevant literature and we are happy to include additional work on FL and DR. However, we kindly believe that our literature review in the field of decentralized DR is thorough. \n\n> The study's applicability could be strengthened … to encompass real-world, privacy-sensitive datasets …\n\nThanks for the constructive suggestions. As clarified in our above response, our main focus is on FL for DR instead of particularly improving the privacy-preserving properties. Thus, we focus more on benchmark datasets in the field of DR. Specifically, we have included the standard datasets in the DR community but with more practical settings (i.e., more clients and larger datasets) compared to prior work (please see the general response). \n\nWith that being said, your comments are well received, and we will consider extending our study to some real-world privacy-sensitive datasets. However, we do want to point out some considerations. Datasets from different domains have unique properties. For example, the prior work [37] mentioned that, in the neuroimaging domain, not all data is private/unshareable, and many public MRI datasets are accessible. The only existing works [37,38] related to our problem focusing on neuroimaging data have made this unreasonable assumption which makes their method hard to generalize. In contrast, to prevent loss of generalizability, our FedNE was developed without assuming any domain-specific properties. Furthermore, we believe that adapting FedNE to the domain-specific datasets should not be difficult as we did not make any prior assumptions about the data properties. \n\n> Questions regarding scalability, complexity, and computational efficiency\n\nPlease see the general response.\n\n> The paper claims to operate without relying on shareable reference data, yet it utilizes additional 2D data points from grid sampling to estimate the training targets … \n\nThanks for your comment. We want to reiterate that our FedNE does not use any shareable reference data, as the publicly available dataset is often inaccessible in many real-world applications (Line 112-114). Moreover, the quality of the reference data can significantly affect the performance [37]. Given these more practical considerations, we propose surrogate models and intra-client data mixing to address the constraints and challenges brought by the Federated setting. We think **the computational cost is still manageable**, and more importantly, **our work is valuable as it was developed within a more practical and general FL setting**, even with some extra burden from data sampling.\n\n> No experiments are conducted on downstream tasks …\n\nThanks for your suggestion. We want to emphasize that the main focus of this work is to propose a DR framework within the FL setting. Therefore, most of our evaluation metrics are designed for analyzing the structural properties of the embeddings. This is because metrics like preservation of neighborhood relationships and preservation of data clusters can provide a clear indication of how well our DR model performs [57,58] regardless of whether it is under the federated or centralized setting. We totally agree that evaluating on other downstream tasks might be valuable to provide a different understanding of our model performance. However, the metric to evaluate other downstream tasks might not be able to accurately assess our capability of preserving the neighborhood relationship as a DR framework. Thus, we respectfully think conducting experiments on downstream tasks e.g., classification might be biased to assess our effectiveness in DR.\n\n[37] \"See without looking: joint visualization of sensitive multi-site datasets.\" IJCAI. 2017.\n\n[38] \"Federated, Fast, and Private Visualization of Decentralized Data.\" Workshop of Federated Learning and Analytics in Practice (2023).\n\n[57] \"Toward a quantitative survey of dimension reduction techniques.\" IEEE TVCG 27.3 (2019): 2153-2173.\n\n[58] \"Dimensionality reduction: A comparative review.\" JMLR 10.66-71 (2009): 13.\n\n[59] \"Feature dimensionality reduction: a review.\" Complex & Intelligent Systems 8.3 (2022): 2663-2693.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7S1j7GnnvY", "reviewer": "Reviewer_y2ZN", "summary": "The paper introduces a novel approach to address the challenge of collaboratively visualizing high-dimensional data in a federated learning (FL) environment. The proposed method, FEDNE, integrates the FEDAVG framework with contrastive neighbor embedding (NE) techniques, aiming to preserve data privacy while ensuring effective data visualization. By employing a surrogate loss function and an intra-client data mixing strategy, FEDNE seeks to enhance the alignment and preservation of neighborhood structures in the global embedding space. The paper includes comprehensive experiments on both synthetic and real-world datasets, demonstrating the effectiveness of FEDNE in outperforming several baseline methods in terms of neighborhood data structure preservation and clustering.", "questions": "1.\tCould you provide more details on the process of training the surrogate models? Specifically, how do you ensure that these models effectively capture the repulsive forces between dissimilar data points across different clients?\n2.\tNon-IID data is a common challenge in federated learning. How does FEDNE handle extreme cases of non-IID data distribution? Have you considered any additional mechanisms to ensure robustness in such scenarios?\n3.\tHow sensitive is FEDNE to the choice of hyperparameters, such as the step size for grid sampling, the number of neighbors in kNN, and the weight in intra-client data mixing? Have you performed any sensitivity analysis?", "limitations": "The authors have addressed the works limitations and social impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. FEDNE introduces a novel integration of FEDAVG with contrastive NE techniques, addressing the unique challenges of pairwise data relationships in federated learning environments without requiring data sharing.\n2. The intra-client data mixing strategy effectively enhances local data diversity, mitigating the limitations of biased local kNN graphs and ensuring better neighborhood representation.\n3. The paper provides a thorough evaluation of FEDNE using various datasets and metrics, showcasing its superior performance compared to baseline methods in preserving neighborhood structures and clustering.", "weaknesses": "1.\tWhile the authors mention that FEDNE introduces only 35% more GPU time compared to FEDAVG, the overall complexity and scalability in a more extensive, real-world setting are not fully addressed. The authors should further investigate how FEDNE scales with a significantly larger number of clients and more complex datasets or models.\n2.\tThe paper proposes intra-client data mixing as a solution to the bias in local kNN graphs. However, this approach might not entirely mitigate the issue of incorrect neighbor connections, especially in highly imbalanced datasets. More detailed comparisons with alternative methods or further enhancements could provide a more robust solution.\n3.\tThe focus is primarily on dimensionality reduction. The validation results are performed only on the vision classification tasks. Extending the discussions and analyses to include applications in other domains could be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Questions regarding scalability and complexity\n\nPlease see the general response.\n\n> The paper proposes intra-client data mixing … However, this approach might not entirely mitigate the issue … More detailed comparisons with alternative methods …\n\nThank you for the insightful comment. As we pointed out in the paper, the bias in the local kNN graphs is one key blocker to the problem of Federated NE. Dealing with attraction term is more challenging than repulsion term as it requires knowing true data affinity across different clients (as discussed in Sect. 4.2). Such a problem has never been pointed out in prior work, but as emphasized in our paper (Line 209-217).\n\nWith this in mind, we made one of the first attempts to mitigate it. We acknowledge that our approach may not fully address the problem, but it demonstrates that reducing its effect can notably improve the overall performance (see Sect.5.3), setting up the foundation for future work to inspire more research in this new field including but not limited to addressing the attractive loss within the settings of FL.\n\nWe appreciate your comments on “alternative methods.” To our knowledge, none of the prior work in FL for dimensionality reduction aimed to address this issue. We would appreciate if you could provide references to the alternative methods you mentioned, and we will be happy to provide more discussions in the reviewer/author discussion phase.\n\n\n> … The validation results are performed only on the vision classification tasks … other domains could be beneficial.\n\nThanks for your valuable comment. We totally agree that the evaluation of our method is mainly on the vision datasets. The main reason we designed the experimental study in this way is mainly to follow a similar paradigm as other DR studies [33,44,55], with the purpose of fair comparisons. However, we admit that only evaluating vision data might be not enough. Therefore, in addition to the commonly used benchmark vision data, we also evaluated our method on **a biological dataset, scRNA-Seq** (see Tables 1 and 2 in the original manuscript). The scRNA-Seq dataset is a real-world dataset containing a collection of gene expression profiles obtained from individual cells of the mouse retina. We hope this additional biological dataset can give you a more comprehensive understanding of our effectiveness in different domains.\n\n[33] “UMAP: Uniform manifold approximation and projection for dimension reduction.” arXiv preprint arXiv:1802.03426, 2018.\n\n[44] \"Visualizing data using t-SNE.\" JMLR 9.11 (2008).\n\n[55] \"SpaceMAP: Visualizing High-Dimensional Data by Space Expansion.\" ICML. 2022.\n\n> Could you provide more details on the process of training the surrogate models? Specifically, how do you ensure that these models effectively capture the repulsive forces between dissimilar data points across different clients? \n\nThanks for the comment. We have discussed in Sect. 4.1 and A.2 in the appendix, but we should have made this more clear. To train the surrogate repulsion model for a client $m$, we generate a set of low-dimensional (low-D) query points via grid sampling to serve as potential embedding positions of other clients’ data points. Then, for each newly sampled low-D query data $z_q$, we pre-compute the repulsive loss between $z_q$ and $b$ data points sampled within client $m$, which serve as the training targets. \n\nThe surrogate model is expected to learn a mapping from these newly sampled embedding positions to their corresponding repulsion loss, as measured within client $m$. According to the repulsive loss term introduced in Section 3.1, a larger low-D Euclidean distance will result in a smaller repulsive loss value [6]. Thus, the repulsion loss decreases as the distance between the data pairs increases. If two embedding points are close within a threshold, our surrogate model is able to approximate the repulsion force; if the points are far apart, the model estimates the repulsion force between them to be close to zero.\n\n[6] \"Attraction-repulsion spectrum in neighbor embeddings.\" JMLR 23.95 (2022): 1-32.\n\n> How does FEDNE handle extreme cases of non-IID data distribution? Have you considered any additional mechanisms to ensure robustness in such scenarios? \n\nWe agree that the ability to handle non-IID data is very important. Therefore, we have conducted experiments on the settings of Dirichlet(0.1) and Shards with C=2 to demonstrate the effectiveness of FedNE in handling extreme non-IID cases, as shown in Tables 2 and 6 in the original manuscript. \n\nFedNE **addresses non-IID data distribution *exactly* via the surrogate models**. For example, under an *extreme* non-IID condition, each client may have data exclusively for certain clusters. In this scenario, the issue with the attraction term might be *relaxed* because the true neighboring points are likely to reside within the same client. However, in the global context, the dissimilar data pairs contributing to the repulsive loss term are located across different clients. As discussed in Sect. 3.3 and 4.1, our surrogate model is specifically designed for this condition by filling in the missing repulsion terms.\n\n> How sensitive is FEDNE to the choice of hyperparameters\n\nThanks for the valuable suggestions. We have included experiments in Appendix B to analyze the sensitivity of some important hyperparameters in our method such as the frequency of surrogate function updates and the time to integrate the surrogate loss function into the local training.\n\nFurthermore, we want to provide additional results on analyzing the sensitivity for the following hyperparameters: (1) Number of neighbors in the kNN graphs, (2) Step sizes in the grid sampling, and (3) The weight in intra-client data mixing. Due to the space limit, we included the detailed analysis in the **figure captions**. **Please find the results and analysis in the attached PDF**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for all the valuable comments and constructive suggestions. We are glad that the reviewers found that our paper is “well-motivated” and “well-presented” (Reviewer BJv7, 4ZYS, mP1Q), and our approach is “novel” (Reviewer mP1Q, y2ZN).

In the following, we want to first reiterate our contributions. We then respond to each individual reviewer’s comments separately. To address some of the questions, we have included additional experimental results in the attached PDF.  We will also incorporate all the feedback in the camera-ready version.

### **Overall contributions and setting**

We want to emphasize that we study a less-explored problem in Federated Learning (FL), FL for Neighbor Embedding (NE), where NE is a family of non-linear Dimensionality Reduction (DR) techniques. As discussed in Section 3.3, in comparison to classification tasks, FL for NE has a unique challenge associated with the pairwise objective function in the NE problem.

One of our key contributions is to point out these challenges systematically and provide effective solutions. We have used the benchmark datasets commonly studied in existing DR literature, and our experiments are larger in scale compared to prior work. We appreciate some reviewers’ feedback regarding complexity and scalability, and we have tried our best to respond to them.

Overall, there are many properties/challenges that one would expect a sophisticated FL algorithm to achieve and overcome. The area of FL has been significantly advanced in the past five years, thanks to hundreds, if not thousands, of papers addressing various aspects of FL. In this paper, we follow this trend by identifying a new challenge specific to FL for NE and dedicating ourselves to mitigating it. Thus, while our algorithm may not be as computationally efficient or scalable as some existing FL methods, we respectfully believe that our contributions are significant and valuable for future study in the field of FL for DR.


### **Regarding scalability, complexity, and computational efficiency**

Our paper focuses on a less-explored combination of FL and DR. Please allow us to address the questions regarding scalability, complexity, and computational efficiency in three aspects.

First, to our knowledge, DR has focused on relatively smaller-scale datasets, compared to classification. This is because computational complexity is never a trivial problem even for many outstanding DR techniques, particularly for non-linear methods such as Isomap and t-SNE which have non-convex cost functions [44]. Indeed, our experiments have included the most widely-used benchmarks used in the DR literature. 

Second, in terms of FL, we work on a less-explored problem, compared to other learning problems such as classification. Compared to the latter, in which the instance-based loss can closely approximate the centralized loss with frequent communication (Line150-154), the Federated Neighbor Embedding problem has a unique challenge associated with its pairwise objective function. Thus, we focus our study mostly on addressing this challenge using standard DR datasets. With that being said, compared to prior work on decentralized data projection, we already considered more clients and larger datasets. Specifically, d-SNE [37] and F-dSNE [38] conducted experiments with *only 3 to 10 local sites* on the *subsampled* MNIST and biomedical datasets which contain *only hundreds to thousands* of data samples. However, our FedNE is evaluated on the *full benchmark datasets* with the settings of *100 local sites*.

Third, while we have not studied larger datasets and more clients like other FL works in classification, we expect that our approach is applicable in real-world settings, for example, cross-silo settings with a manageable amount of clients. As pointed out in Sect. 4.3 (Line 235-237), our approach only requires 35% additional GPU time compared to FedAvg, and we expect such overhead to remain similar when going to larger datasets with a similar number of clients. When the number of clients increases, we may optionally drop a portion of surrogate models in local training. As shown in our response to Reviewer BJv7, such a setting does not lead to a significant performance drop but can maintain scalability. 


[37] "See without looking: joint visualization of sensitive multi-site datasets." IJCAI. 2017. 

[38] "Federated, Fast, and Private Visualization of Decentralized Data." Workshop of Federated Learning and Analytics in Practice (2023). 

[44] "Visualizing data using t-SNE." JMLR 9.11 (2008).

Meta-review: The paper presents a new federated learning approach for dimension reduction using contrastive neighbor embedding (NE). The key idea is the introduction of a surrogate loss function that each client learns and shares, which compensates for the lack of inter-client repulsion essential for global alignment in the embedding space. Additionally, the paper proposes a data-mixing strategy to augment local data, addressing issues of invisible and false neighbors in local kNN graphs. Comprehensive experiments demonstrate that FEDNE effectively preserves neighborhood data structures and enhances alignment in the global embedding space compared to several baseline methods.

1.	FEDNE introduces a novel integration of FEDAVG with contrastive NE techniques, addressing the unique challenges of pairwise data relationships in federated learning environments without requiring data sharing.

2.	The intra-client data mixing strategy effectively enhances local data diversity, mitigating the limitations of biased local kNN graphs and ensuring better neighborhood representation.

3.	The paper provides a thorough evaluation of FEDNE using various datasets and metrics, showcasing its superior performance compared to baseline methods in preserving neighborhood structures and clustering.

Weaknesses:
The title should not include the “privacy-preserving” as it doesn’t show additional privacy guarantee besides federated learning itself.

1.	Communication complexity: The method appears to be computationally inefficient. There are no experiments conducted on large datasets, such as those in real-world medical or other privacy-critical domains, where computational complexity could be a significant issue.

2.	Additional privacy concerns: Sharing surrogate models introduces additional privacy risks, e.g., enabling reconstruction attacks or membership inference. While some recent work empirically shows that such private information is less leaked after the proposed method might be more vulnerable to privacy attacks without differential privacy. It lacks experiments and guarantees demonstrating the privacy preservation of the FedNE approach. The improved performance is built upon the inherent compromise of privacy. Such compromise also makes future comparisons unfair and challenging if the threat model is not stated clearly, i.e., what is assumed to be safe.
Decision: Accept (poster)