Federated Black-Box Adaptation for Semantic Segmentation
Authors: Jay Nitin Paranjape
Shameema Sikder
S. Swaroop Vedula
Vishal M. Patel
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Blackbox Learning
Split Networks
Segmentation
Primary area: machine_vision
TL;DR: Federated semantic segmentation without transfer of gradient or model weights, enabling more privacy preserving networks

Abstract: Federated Learning (FL) is a form of distributed learning that allows multiple institutions or clients to collaboratively learn a global model to solve a task. This allows the model to utilize the information from every institute while preserving data privacy. However, recent studies show that the promise of protecting the privacy of data is not upheld by existing methods and that it is possible to recreate the training data from the different institutions. This is done by utilizing gradients transferred between the clients and the global server during training or by knowing the model architecture at the client end. In this paper, we propose a federated learning framework for semantic segmentation without knowing the model architecture nor transferring gradients between the client and the server, thus enabling better privacy preservation. We propose \textit{BlackFed} - a black-box adaptation of neural networks that utilizes zero order optimization (ZOO) to update the client model weights and first order optimization (FOO) to update the server weights. We evaluate our approach on several computer vision and medical imaging datasets to demonstrate its effectiveness. To the best of our knowledge, this work is one of the first works in employing federated learning for segmentation, devoid of gradients or model information exchange. Code: https://github.com/JayParanjape/blackfed/tree/master

Federated Black-Box Adaptation for Semantic Segmentation
Jay Paranjape
Shameema Sikder Wilmer Eye Institute The Johns Hopkins University Baltimore, USA
Dept. of Electrical and Computer Engineering
The Johns Hopkins University
Baltimore, USA jparanj j@. edu
S. Swaroop Vedula
Vishal M. Patel
Malone Center for Engineering in Healthcare The Johns Hopkins University Baltimore, USA
Dept. of Electrical and Computer Engineering The Johns Hopkins University Baltimore, USA
Abstract
Federated Learning (FL. is form of distributed learning that allows multiple institutions or clients to collaboratively learn global model to solve task. This allows the model utilize the information from every institute while preserv- ing data privacy. However, recent studies show that the promise of protecting the privacy of data is not upheld by existing methods and that it is possible recreate the training data from the different institutions. This is done by utilizing gradients transferred between the clients and the global server during training or propose
by knowing the model architecture at the client end. In this paper, a federated learning framework for semantic segmentation without knowing the model architecture nor transferring gradients between the client and the server, thus enabling better privacy preservation. We propose BlackFed a black-box adaptation of neural networks that utilizes zero order optimization (ZOO) update the client model weights and first order optimization (FOO) to update approach several computer vision and
the server weights. We evaluate medical imaging datasets to demonstrate its effectiveness. To the best of our knowledge, this work is one of the first works in employing federated learning for segmentation, devoid of gradients or model information exchange. Code: tattps://githug cww/Jayaaaaaaaaaaaaaaaaaaaaaaaaa wawp///aappaaaeaeeeeeeeeaa ed/ tree//reeeeeeeeee// eameemmmttrrrr
Introduction
With data-driven methods becoming immensely popular in Artificial Intelligence (Al) research and applications, there has been surge data collection and curation across the world. This has, in turn, led to the development of AI models that require substantial amounts of data to train. Federated Learning (FL) (34) was developed as viable approach towards training such models by effectively harnessing the data collected at different centers across the world. Through FL, it becomes possible for multiple institutions to collaborate and build a joint model that learns from all of them, while reducing the burden of collecting more data individually. However, collaborations between different institutions present a non- con-trivial challenge due to disparity in data distributions as well as the imperative of safeguarding data privacy. Consequently, FL aims to jointly learn shared model that performs well on data from all participating institutions without exchanging raw data. Various FL approaches have Among
been proposed in the literature 10
25
55
30
29
31
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Legend
Server
Server
Server
cliend weights
I- oagrogated ghts
Lp
9 gradients
Ep formard pasor
Cllrr
Ciient
Cienn
Client
Crien
Clent
Traditional
with Vanilla Spiit-nn
BlackFed (Ours)
eg. FedAvg
(Whitebox)
(Blackbox)
Weight Sharing
Gradient Sharing
No Weight Sharing
No Gradient Sharing
Figure Comparison of our method against traditional FL methods. Existing FL methods are primarily wwhite-box" they involve transfer of model weights [34], gradients (2)] In contrast, our method only utilizes forward passes to update the client and does not require sharing weights or gradients, making it "black-box" model.
R4 was one the pioneering FL methods, which proposes training local models at every center using local data and periodically averaging the local model weights to craft global server model. Various subsequent works improve FedAvg by improving the local model updates
these, FedAvg
30 31 or global updates [24 29].
While FL was primarily proposed for classification tasks, it is also suited for other computer vision tasks such segmentation. Generating annotations in segmentation entails creating pixel-Ievel annotations per image, that are more tedious to label than classification tasks. Consequently, is not always possible for single institution to collect a large amount of data, underscoring the importance of collaborative efforts. A few approaches have been proposed in the literature for (33 [4] However, all these methods for segmentation and classification, while not involving raw data transfer, employ techniques like model information
FL-based segmentation (36 5 18
43
transfer [34 30 [6] or gradient transfer [24 (20), as shown in Figure I However, recent research has revealed that these techniques are vulnerable to attacks that can recreate training data from the participating centers, thus undermining the privacy preserving characteristic of FL [T] 17 57]
27 26 58 23]. These attacks employ methods like gradient inversion [54] 17 57 or adapting
approach, named BlackFed, for segmentation using FL. that does not involve gradient transfer between the server and the client and at the same time, passes no knowledge about the client model architecture to the server, thereby avoiding the necessary conditions for these attacks, as shown in Figure || This is done by formulating the FL learning problem distributed learning problem using split neural networks (split-nn) 20) and combining first order and zero order optimization techniques for training. Our contributions are as follows:
model architecture and weights [1]. In this work, propose
1. We introduce BlackFed black-box -bx algorithm that facilitates distributed learning for semantic segmentation without transferring model information or gradients between the client and the server. For this, we formulate the FL problem using split-nn and use first and zero order optimization for training the server and the clients, respectively.
2. We suggest method to reduce the effect of catastrophic forgetting in BlackFed by retaining client-wise checkpoints in the server.
3. We evaluate the proposed approach on four segmentation datasets and show its effectiveness distributed learning method by showing improvements over individual training.
2 Related Work
FL for the segmentation tasks was motivated by its immense application in the medical domain. Consequently, various methods were introduced in this
Federated Learning for Segmentation:
field 33
6
13
19
28
32
42
49
50
52
38
For instance, [32] attempts to learn a model for brain

---

tumor segmentation by utilizing data from multiple institutions. FedSM [38 attempts to mitigate the effect of non-iid -id nature of the data from different centers on the global server model. However, most of the approaches for medical segmentation focus on the problem of segmenting out the foreground from the background (one class problem). There are relatively fewer works in the literature for FedSeg [36] deals with the class label inconsistency problem that may be present at the local clients. In other words, it builds a robust system that works well even when the clients have annotations for only subset of the classes. FedDrive [14,, on the other hand, sets up various benchmarks for FL algorithms on multi-class datasets like Cityscapes [9]. However, all the existing algorithms require the global server to know the model architecture used in
multi-class segmentation [14 36 18
45 53
the clients and thus, these methods are vulnerable to recent attacks [.]] In our work, we develop an algorithm to perform multi-class segmentation which does not require gradients or client model sharing.
were introduced as an alternative to FedAvg- techniques which require model sharing. They offer an approach to perform collaborative learning by splitting larger network into two segments. The latter segment of the network is shared across all centers and placed at the global server, while the former part is distributed such that each center possesses its own sub-network. During training, the clients perform forward pass using their data and send the encoded features to the server, which further passes the features to the subsequent layers in the network. The server and client models are trained using backpropagation, where the gradients from the first layer of the server model are sent back to each of the clients. Split networks have mainly been used in the literature for the task of classification, mostly in the medical domain 20 40
Split Neural Networks: Split networks 20
Our work marks one of the initial explorations of split learning in the context of semantic segmentation. Furthermore, split networks are shown to be more robust reconstruction attacks which arise from sharing model information [48]. Nonetheless, they remain susceptible to gradient leakage attacks since there is gradient transfer between the server and the client. In this work, we formulate the problem of distributed semantic segmentation using split network and introduce a training algorithm that does not require gradient-sharing.
3 Black-box Adaptation
In the following section, we define the problem of semantic segmentation under the federated setting. Then, describe our proposed formulation for the black-box setting and the algorithm.
3.1 Preliminaries
The setting of FL consists of N clients. denoted by C.,, where E 1,2, N and global server G. Each of the clients has a local dataset, consisting of images X; = and their corresponding ground truth segmentation maps Y, Y= {u/ E BRHWWNN E (1,,,..,, Here, zi and y) represent the jth image and its cor- responding segmentation mask, while nt' represents the number of data points in client i and Ne represents the number of classes in the output. Note that the distributions of the input images vary among the clients. Each client uses its dataset to learn function f I Hxxx BHX xN. which is parameterized by O', that minimizes its local loss function as follows:
{st E Hxxxxx (1,2....
1 n² j=1
arg min L,,
where denotes the loss per data point. The global server, on the other hand, tries to aggregate the knowledge of all the clients by fusing the parameters of the individual clients to optimize joint loss function as follows:
N
arg min L @
1 L
Σ
N
commonly used algorithm, called FedAvg 34], proposes a linear combination of client parameters to optimize the loss function, defined in Eq. follows:
0
(3)
n
i=1
3

---

However, this formulation requires the server to be aware of the exact model architecture or function of the client, which can results in data leakage [T].
3.2 Proposed Algorithm
Proposed Problem Formulation: In this work, we model the problem of distributed learning using a split neural network, which takes away the requirement of the server being aware of the client architecture. In this case, each client learns a function f' R xXXx + R xW'xC' xC', which is parameterized by @'. Similarly, the global server learns a function which is parameterized by hb Thus, the forward pass for given center is given follows:
"R""""" x + R
g ((f e"); T),
where 9j denotes the predicted segmentation map. Hence, the objective function of a given client is as follows:
1 72 Σ of. Φ).
arg {1,2, ..., M) 06
(5)
As in the existing FL literature, the goal of our approach is that after training, all clients should benefit from each other. Hence, during evaluation, given any client, we aim to perform well on data from other clients as well as its data, thus showing good generalization. More specifically, want to optimize any combination of data and client as follows:
Li i
(1,2, N1 = n2 Σ
(g)) 8/3 0*, ②).
(6)
One way optimize this equation is by attending to every client in round-robin fashion. This involves selecting a client, performing the forward pass, as defined in Eq. and then performing backpropagation to update the server and client using the client loss function defined in Eq. B Performing this operation several times in a round-robin fashion enables the server to learn from all client sources, and hence, improves the overall performance. We term this method as "White-box Bound-Robin FL.", and is similar to the existing methods in the literature [24] 20). However, recent works have shown that such method which involves transfer of gradients between the server and client can be utilized to regenerate the training data, thus undermining the privacy preservation
principle of FL [54] (23] Hence, we add one more constraint i.e. no gradient flow back from the
server to the client in Eq. 5
BlackFed Algorithm: To optimize the clients without using gradients, we utilize ZOO method called Simultancous Perturbation Stochastic Approximation with Gradient Correction (SPSA-GE -C)) 39] which involves perturbing the weights of the client model slightly and approximating two-sided gradient based on the change in the loss function due to the perturbations. However, this method was developed to perform black-box adaptation of pretrained foundation models and hence, expects the server model to be initialized with good weights, which is not the case in our formulation, making non-trivial. To overcome this, propose iteratively use alternating optimization technique, which factorizes the optimization problem in Eq. 5 into two optimization problems as follows:
n²
arg min L LiE (1,2,...,n) = n'
100 O
1=1
arg min L', E (1,2, n. = 100 w): 110)
During training, we first select client using the round-robin policy. Next, keeping the server weights fixed, we train the client weights using SPSA-GC for a few iterations. Next, we fix the client weights and use first order optimizer (i.e. Adam-W) to optimize the server for a few iterations. This process is repeated multiple times. During inference for client, we simply run the forward pass as described Eq. 4 using the final weights of the client and server. The training process described in Algorithm We refer to this approach as BlackFed vl.

---

Algorithm 1 Proposed Algorithm for BlackFed vl
Input: (i) N, number of clients
(ii) c_e, number of epochs to train one client
(iii) s_e, number of epochs to train server
(iv) runs, number of complete round-robin runs n 2 0
Output: (i) L, server model weights
(ii) O array of client model weights
BeackFed (N, c_ e,e, $__, e, runs) Begin: initialize 2, θ for r = 1,2.....Iunss do for i 1,2 do 1. do
for [ < SPSA-GC(OTi)
//Zero order optimization for client
end for
for
== S_e do
81. ② - ④- - 1)s 74
//First order optimization for server, where 7/ is the learning rate
end for
end for end for Return: 4, O End
Reducing the Effect of Catastrophic Forgetting: Since the model at the server is shared among all the clients and is updated in a round-robin fashion, it may happen that training with the data from given client may cause it to unlearn patterns for the previous client. This phenomenon often called catastrophic forgetting. This effect is observed in BlackFed v1 especially when the number of clients increases or if there is a significant change the data distribution among clients. This causes the algorithm to perform well on certain clients and poorly the rest of the clients. To reduce the effect of catastrophic forgetting, we propose simple additional step during training with Algorithm [T]
After updating the server weights for given client during training, store the updated weights of the server model a hashmap indexed by the index the client. During inference for a given client, use the latest weights of the client model and the indexed weights of the server model to perform the forward pass. Note that the server state is loaded from the hashmap only during inference. During training, the server still benefits from the data from all clients and updates its weights well as the hashmap. This approach is visualized in Figure 2 We refer to this method as BlackFed v2.
Experiments and Results
Datasets: For evaluating our method, we consider four publicly available datasets, namely (i)
Cityscapes 9 (ii) CAMVID 4. (iii) ISIC 22 45 and (iv) Polypgen (2). Cityscapes and
CAMVID are two road-view semantic segmentation datasets with 19 and 32 classes of interest respectively, collected from multiple cities. In the FL setup. we consider each the cities as separate clients. While CAMVID has predefined train, test and validation splits, for Cityscapes, we divide the data from each client into training, validation and testing splits for that client in 60:20:20 ratio. In this manner, we generate 18 clients for Cityscapes and 4 clients for CAMVID, Further details about the dataset size in each center is provided in the supplementary document. The ISIC dataset corresponds to skin lesion segmentation challenge. We generate three clients for this dataset, which utilize the datasets from ISIC 2016 21 ISIC 2017 [8] and ISIC 2018 (45), respectively. The data every year is collected from different centers and hence, has different distribution amongst centers. Finally, PolypGen is a colon polyp segmentation dataset which was collected from six different centers. The training, validation and testing splits for each of these were done in 60:20:20 ratio for Polypgen whereas they were already provided for ISIC. We visualize the pixel-intensity histograms of the CAMVID and ISIC datasets to verify different data distributions amongst clients in Figure
Experimental Setup: We use a two-layer convolutional network for modeling / in the client and DeepLabv3 77) for modeling g in the server. The number of output channels from the client is kept
5

---

12g
Ng
mom
Legend
eiight
Save pat netwerk
2
1 Ad
Ng
Ng
Clesst part natwoek
Server Hashmap
$2
INe
First ordir opaniaationn using badighop
Ia Forward Propagation in client
1a
2
No
tb Forward Propagation
Zelo order opt optiritation using SPSA (C pro gradient transter invelved)
Optimizsion Client
1dl- Order Optimization of Servor
1a Store Model in Heshmap with client index key
Ciient
Client
Clent
Clients selected In round rabin fashion
Traning
Inforence
The BlackFed v2 Algorithm. During training, the client is selected in round-robin fashion. Then (a) client performs forward pass using its part of the network (b) Server performs forward pass using its part of the network (c) With server weights fixed, client weights updated using ZOO (d) Keeping client weights fixed, server weights updated using FOO (e) The best server weights are stored in the hashmap corresponding to client index. During inference, the client performs a forward pass and calls the server with the output. Server queries the hashmap using the client index and gets its set of weights, using which the prediction is obtained. Note that there is no gradient transfer, thus making this black-box setup.
Figure 2:
Cetto
6:35
Canter
1:30
0.908
0.004
1.75
0.064
1.50
115
1200
Poot
100
200
Figure 3: Pixel Density distribution of (L) the CAMVID Dataset and (R) the ISIC Dataset. Since majority of ISIC pixels are either O or 255 for all centers, these have been omitted for better visualization. Since each of the clients has different distribution, data from one client can be considered as out-of-Distributio (OOD) for other clients.
as 64. Consequently, we start the DeepLabv3 network in the server from the second layer, which expects 64-channel input. During training, we use c_e = 10 and s_e 10. The server is optimized using an Adam optimizer and the client is optimized using SPSA-GC. The learning rates of both the client and the server are set to 10-4, based on validation set performance. The batch size for all experiments is 8, and all images undergo random brightness perturbation with brightness parameter set as 2. The images for Cityscapes and CAMVID are resized to 256 x 512 to maintain their aspect ratio, whereas the images for ISIC and Polypgen are resized to 256 x 256. All experiments are done using single Nvidia RTX A5000 GPU per client and single Nvidia RTX A5000 GPU at the server.
Results: Given the trained client models and the trained server model, we assess a given client's performance with test datasets from its own local data repository and present the mloU in the "Local" column. This metric represents the local performance of the model on its own dataset. In addition, we assess each client on test data from other centers and note down the average mloU scores in the Out-o--Distributioon ("OOD") column. This metric serves as an estimate of the general performance of given client post-training. We consider the latter more important as it can be considered to be the direct outcome of the Fl. paradigm. These results are presented in Table for CAMVID and Cityscapes, and in Table 2 for ISIC and Polypgen. For both tables, in the first row, each client is

---

Locel
00D
TYSCAPES
BIC
POLYPGEN
BlooFFedi2
Indistud
BlazsFed ve
Figure 4: Comparison our method against individual training. The third and fourth columns denote testing with the local test data, while the fifth and sixth columns denote OOD testing. Our method improves OOD performance clients without harming their local performance.
trained on its own dataset, independently of others, indicating no collaboration. For this case, while the client performance on its own test set is high, its general performance suffers. The next two and v2). Notably, BlackFed v2 generally exhibits better performance than vl since it addresses the catastrophic forgetting that occurs during training. The following three rows represent the expected upper bounds to our performance. "Combined Training" represents the scenario where raw data can be freely shared and a single model is trained using the combined data from all clients. "White-box training" represents the case where gradients can flow freely between the server and the client. Thus, instead of the ZOO optimization, to optimize the client part of the model. Finally, the last three rows represent the performance of traditional FL using FedAvg, and recent methods like FedPer and FedSeg, where model sharing is allowed. Here, all the clients follow the same model architecture as the server (DeepLabv3) and the server aggregate weights from the client models. All three methods represent relaxation the constraints imposed in our approach, thus acting as an upper bound to the black-box performance. During training, the client and server models are trained for the This produces a more uniform distribution of results across centers, in contrast with traditional FL methods like FedAvg. where the aggregation of weights is weighed by the client dataset size. We observe that for the Polypgen dataset, BlackFed v1 and v2 perform slightly better or on par with individual client performance for OOD case. For this case, there is little difference between the performance of v1 and v2. This behavior may be related to the data distribution of Polypgen and suggests that BlackFedv2 is not able to correctly avoid the catastrophic forgetting for centers C5 and C6. However, for rest of the scenarios, we see that v2 significantly outperforms vl as well as the individual training cases on OOD mloU metric. At the same time, the performance on local data does not suffer significantly as compared to the individual training. Moreover, BlackFed v2 performs on par with "Combined" and "White-box" Training. All results BlackFed have a p-value less than 0.001, showing the statistical significance of our black-box approach. The visual comparisons for our method with the individual training is given in Fig. As can be seen all four rows, individual training can lead to overfitting, which harms the general OOD performance. Using our method, we are able to improve the OOD results across all datasets without significant decrease in the local results.
rows represent our method (BlackFed
use FOO
number of epochs per client.
Additional Model Architectures While we evaluate DeepLabv3-based server in our main exper- iments, we show the effectiveness of our method on additional segmentation architectures. More specifically, for the CAMVID dataset, we choose UNext (46) and SegFormer [51] as server models and present average mloU results across the test datasets from each client in Table As can
be seen from the table, using our approach can improve the performance over individual aedividuaatrrinniin training for all three models. As the model complexity increases from UNext to DeepLab to Segformer,
7

---

Table 1: mIoU scores for BlackFed vI and v2 in comparison with individual and FL-based training strategies for natural datasets. "Local" represents test data from the center. "OOD" represents mean mloU on test data from rest of the centers. For FedAvg and Combined Training, just one model is trained. Hence, its performance noted only each of the local test datasets. For Cityscapes, we only present the average local and OOD performance across centers for brevity. The supplementary contains an expanded version for Cityscapes.
CAMVID
Cityscapes Average across Centers
C1
C2
C3
C4
Method
Local
0OD
Local
OOD
Local
OOD
Local
OOD
Local
OOD
Individual
0.63
0,46
0.83
0,48
0.85
0.65
0.79
0.64
0,50
0,44
BlackFed v1
0.55
0.67
0.79
0.65
0.78
0.68
0.66
0.66
0.71
0.71
BlackFeed v2
0.70
0.72
0.78
0.66
0.81
0.70
0.75
0.72
0.75
0.74
Combined Training
0.74
0.81
0.84
0.77
0.77
White-box Training
0.73
0.72
0.68
0.70
0.75
FedAvg 34
0.64
0.76
0.84
0.76
0.79
FodSeg
56
0.71
0.79
0.83
0.77
0.81
FedPer
0.57
0.77
0.68
0.82
0.71
0.76
0.66
0.78
0.72
Table mIoU scores for BlackFed v1 and v2 comparison with individual and FL-based training strategies for medical datasets, "Local" represents test data from the center. "OOD" represents mean mloU on test data from rest of the centers. For FedAvg and Combined Training, just one model is trained. Hence, its performance is noted only in each of the local test datasets.
Pobyges
Methed
Local
0DD
Locol
000
Local
000
Local
0OD
Local
00D
Local
000
Locol
D00
Local
000
Local
OOD
Indicntial
0.41
0.75
0.37
0L52
0.86
0.79
0.76
0.75
0.55
0.6
0.28
Blokhhd
11.66
0.64
0.00
0.65
0.1
0.47
0.49
0.36
0.86
0180
(U0)
0.79
Combrall Traming
0.71
0.79
0.81
0.67
0.57
0.00
0.67
0.59
0.76
White box Ttatsing
64
0.77
0.59
0.72
0.58
0.6
0.57
0.08
062
0.50
0.62
0.56
0.00
0.73
0.66
0.77
0.68
Fed/fg
0.78
observe a decrease in individual training performance. This trend is reversed for the combined case where there is more training data. This observation indicates overfitting in the individual case due to less individual training data. Using our method, able to correctly match the performance and trend of combined training.
5 Ablation Studies
Analysis of the Order of Optimization: In the alternating optimization proposed in Eq. 7 we choose to first update the client followed by the server. This order is important since it allows to store the correct server weights the hashmap. If the server is trained before the client, we found that SPSA-GC often gives unstable results and reduces the metric on the validation set after an epoch. This corrected only when its the turn the same client again. Conversely, in the case where the client is updated first, the server adapts in stable manner to the client weights since backpropagation is allowed in the server. Consequently, in the same epoch, we get a higher and more stable performance
Table 3: mloU for CAMVID dataset with varying model architectures of the global server.
Lhv3 C2
Segformer C2
UNext
Method
C1
C3
C4
C1
C3
C4
C1
C2
C3
C4
Individual
0.50
0.57
0.69
0.67
0.27
0.36
0.50
0.34
0.36
0.49
0.60
0.51
BlackFed v2
0.72
0.69
0.73
0.72
0.71
0.69
0.73
0.72
0.61
0.53
0.43
0.56
Combined Training
0.74
0.81
0.84
0.77
0.66
0.77
0.77
0.72
0.59
0.67
0.73
0.67
White-hox Training
0.72
0.76
0.84
0.76
0.54
0.55
0.58
0.57
0.36
0.49
0.60
0.51
8

---

Table 4: Comparison of client and server-side GFLOPS for different algorithms.
D GFLOPS
Segformer GELOPS
UNext GFLOPS
Method
Clicat
Server
Client
Server
Clicat
Server
Centerwris
353.6
475.39
FodAvg [4]
353.6
353.6
7.5
7,5
475.39
475.39
Ours 26.6
326.98
5.12
2.38
53.13
422.26
Table 5: Average MIoU scores for different training strategies of BlackFed. Optimizing the client followed by the server improves performance, which is further improved by maintaining the server- side hashmap.
CAMVID
City CityScapes
PolypGen
Method
Aveg across 3 centers
Average across 4 conters
Avg across 18 conters
Avg across 6 centers
Optimize Server, then client Optimize Client, then Server (BlackFed vl) Optimize Clicat, then Server (BlackFed V2)
0.67
0.66
0.53
0.43
0.67
0.76
0.53
0.50
0,72
0.83
0.74
0,50
for each client, which can be saved the hashmap for usage during inference. We demonstrate this empirically by comparing the performance of the two training strategies in Table
Analysis of Computational Cost: For each of the three model architectures, namely DeepLab V3, UNext and Segformer, we calculate the floating point operations required in a single forward pass at the institution end. These are shown in Table 4 If the clients were to train individual models with their local data, they would require running the entire forward pass on their local systems. This is also the case in existing FL. approaches like FedAvg. In the proposed approach, as can be seen from Row 3 in the table, the client has significantly reduced load, with the majority of computation being offloaded to the server. The server uses the respective architectures starting from the second layer, while all the clients use a two-layer convolution network, with the second convolutional layer being similar to the first layer of the respective architecture.
Effect of Client and Server rounds: The BlackFed algorithm takes in two hyperparameters, namely c_e and s_e, that determine the number of epochs for training the client and server, respectively. We conduct experiments for observing the effect of changing these parameters on the model performance. We use the CAMVID dataset with server architecture being DeepLabv3 and note down the average mIoU score Table for different values of c_e and s_e in {10, 20, 30, 40, 50). Interestingly, we observe that there was no significant difference in performance by
Table 6: Average mloU scores for BlackFed v1 and v2 in comparison with individual and Fl--based training strate- gies.
Number of Clien: Epuchs
Number of Server Epochs
10
20
40
0.71
11:69
0.68
0.69
0.68
0.70
0.69
0.69
0.69
0.69
increasing the number of server epochs client epochs. While increasing the server epochs can improve training, it specific given client, thus
20
0.65
0.68
0.70
0.70
0.56
0.71
12.00
0.68
0.70
0.70
0.6
0.89
0.70
0.66
0.69
also makes the server reducing average performance. On the other hand, more -aseb iterations marginally improve the model. In this context, ZOO primarily serves guide
the server towards a more robust minima, resilient to perturbations client features. Consequently, a greater number of client epochs does not notably impact model performance.
6 Conclusion
In this work, we introduce BlackFed, an FL algorithm that enables distributed lcarning without transfer of gradients or model weights. This characteristic distinguishes our approach as a black-box model compared to existing FL. methods, which can be considered as white-box approaches. Recent research on attacking FL methods require the knowledge of either the gradients or the model information, thus rendering BlackFed more resistant to such attacks since gradients or weights are not shared. BlackFed consists a global server and multiple clients, each possessing their own data. The server is optimized using first order optimization while the client weights are updated using zero order optimization in round-robin fashion. This introduces the effect of catastrophic forgetting in the network, for which propose simple hashmap-base based approach. During training, store the client weights per server

---

so that they can be utilized during inference. With these modifications, our approach demonstrates superior results compared to non-collaborative training and matches the performance of white-box methods, despite being black-box method itself. Extensive experimentation on the natural and medical domain datasets highlights the effectiveness of BlackFed. Through this endeavor, we aim to propel research towards the development of better privacy preserving federated learning systems. Potential directions for future research can include analysis other policies for selecting client order during training, and its relation with the disparity in data distribution. One more interesting direction for future research would be the effect of adversarial attacks using generative models on this method. Since masks are shared between the client and server, it would be interesting check if existing methods are able to recreate client data using them.
Acknowledgments and Disclosure of Funding
This research was supported by grant from the National Institutes of Health, USA; RO1EY033065. The content solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors have competing interests in the paper.
References
[1] Acar, D.A.E., Zhao, Y., Matas, R., Mattina, M., Whatmough, P., Saligrama, V.: Federated learn- ing based on dynamic regularization. In: International Conference on Learning Representations (2021), /ttttps:/oppnreeiier wttpeewwwwwieeeettenreeeeeeeeeee 2it
[2] Ali, S., Jha, D., Ghatwary, N., Realdon, S., Cannizzaro, R., Salem, O.E., Lamarque, D., Daul, C., Riegler. M.A., Anonsen, K.V., Petlund, A., Halvorsen, P., Rittscher, J., de Lange, T., East, J.E.: A multi-centre polyp detection and segmentation dataset for generalisability assessment. Scientific Data 10),), 75 (Feb 2023). https://doi.or///1 http:://ddb.rrr///...00/4/111111 https://doi. http:///oo.org//..1/ 1tt///10..10/////1 000010/////11577023-100090
1033444177777702301998
[3] Arivazhagan, M.G., Aggarwal, V. Singh, A.K., Choudhary, S.: Federated learning with person
alization layers (2019), battps://aree http://awxxvvv.v.aabba wrwg.//////.../////// 00818
[4] Brostow, G.J., Fauqueur, J., Cipolla, R.: Semantic object classes in video: A
high-definition ground truth database. Pattern Recognition Letters 30(2), 88-9 97 (2009). https::/doiiorr/httpsss http:////oooooo/////// 1016/ 2008. 0.008..4.005,500. ftts........ttts////aee //www. sciencedirect video-based Object and Event Analysis
[5] Caldarola, D., Caputo, B., Ciccone, M.: Improving generalization in federated learning by seeking flat minima. In: European Conference Computer Vision. pp. 654-672. Springer (2022)
[6] Chang, Q. Qu, H., Zhang, Y., Sabuncu, M., Chen, C., Zhang, T., Metaxas, D.N.: Syn- thetic learning: Learn from distributed asynchronized discriminator gan without shar- ing medical image data. In: 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 13853-13863. IEEE Computer Society, Los Alami- tos, CA, USA (jun 2020). https://doi.org//// 10. 109/CVPR42000 WWW.C//VPRR220000200033333/7 https://doi
leescomputersociett.y org/10 1109/ 111.:01111/////22000000 2020 01387
[7] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation (2017)
[8] Codella, N.C.F., Gutman, D., Celebi, M.E., Helba, B., Marchetti. M.A., Dusza, S.W., Kalloo, A., Liopyris. K., Mishra, N., Kittler, H. Halpern, A.: Skin lesion analysis to- ward melanoma detection: A challenge at the 2017 international symposium on biomedi- cal imaging (isbi), hosted by the international skin imaging collaboration (isic). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). pp. 168-172 (2018). hanps://doi.rg//10
[9] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3213-3223 (2016), https://aan semanticscholar ot wetg/CorpsID:::2220
10

---

[10] Dayan, I., Roth, H.R., Zhong, A., Harouni, A., Gentili. A., Abidin, A.Z., Liu, A., Costa, A.B.,
Wood, B.J., Tsai, C.S., Wang. C.H., Hsu, C.N., Lee, C.K., Ruan, P., Xu, D., Wu, D., Huang. E.. Kitamura, F.C., Lacey, G., de Antônio Corradi, G.C., Nino, G., Shin, H.H., Obinata, H., Ren, H., Crane, J.C., Tetreault, J., Guan, J., Garrett, J.W., Kaggie, J.D., Park, J.G., Dreyer, K., Juluru, K., Kersten, K. Rockenbach, M.A.B.C,, Linguraru, M.G., Haider, M.A., MaselMaseeh, M., Rieke, N., Damasceno, P.F. e Silva, P.M.C., Wang. P., Xu, S., Kawano, S., Sriswasdi, S., Park. S.Y., Grist, T.M., Buch, V., Jantarabenjakul, W., Wang, W. Tak, W.Y., Li, X., Lin, X., Kwon, Y..., Quraini, A., Feng. A., Priest, A.N., Turkbey, B., Glicksberg, B., Bizzo, B., Kim, B.S., Tor-Díez, C., Lee, C.C., Hsu, C.J., Lin, C., Lai. C.L.,, Hess, C.P., Compas, C.,, Bhatia, D., Oermann, E.K., Leibovitz, E., Sasaki, H., Mori, H., Yang. I., Sohn, J.H., Murthy, K.N.K., Fu, L.C., de Mendonça, M.R.F., Fralick, M., Kang. M.K., Adil, M., Gangai. N.. Vateckul, P., Elnajjar, P., Hickman, S., Majumdar, S., McL.od, S.I., Reed, S., Gräf, S., Harmon, S., Kodama, T., Puthanakit, T., Mazzulli, T., de Lavor, V.I.,, vongthaai,, Y., Lee, Y.R., Wen, Y., Gilbert, F.J., Flores, M.G., Li, Q.: Federated learning for predicting clinical outcomes in patients with covid-19 19. Nature Medicine 27(1)), 1735 1743 (Oct 2021). http:://oi.org// https:/do..rrg/0..008//444141// 10000///4111--02--15550000 https://doi http://///oooo////////// 1038/ 0111:///11911--1--111 021-00066-3
[11] Dimitrov, D.I., Balunovic, M., Konstantinov, N., Vechev, M.: Data leakage in federated
averaging. In: Transactions on Machine Learning Research (2022), /attps://openrev:
[12] Dong, J., Wang, L.,, Fang, Z., Sun, G., Xu, S., Wang. X., Zhu, Q.: Federated dlassinccrmmenal learning. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10154-10163 (2022). https://doi.org/II 109/CV/R526/
[13] Dong, N., Voiculescu, I.: Federated contrastive learning for decentralized unlabeled medical images (2021)
[14] Fantauzzo, L., Fani, E., Caldarola, D., Tavera, A., Cermelli, F., Ciccone, M., Caputo, B.: Feddrive: Generalizing federated learning to semantic segmentation in autonomous driving. In: Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (2022)
[15] Fowl, L., Geiping, J., Czaja, W., Goldblum, M., Goldstein, T.: Robbing the fed: Directly obtaining private data in federated learning with modified models (2022)
[16] Gao, L., Fu, H., Li, L.,, Chen, Y., Xu, M., Xu, C.Z.: Feddc: Federated learning with non-iid data via local drift decoupling and correction. In: IEEE Conference on Computer Vision and Pattern Recognition (2022)
[17) Geiping, J., Bauermeister, H., Dröge, H., Moeller, M.: Inverting gradients how easy is it to break privacy in federated learning? In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS '20 Curran Associates Inc., Red Hook, NY, USA (2020)
[18] Gong, X., Sharma, A., Karanam, S., Wu, Z., Chen, T., Doermann, D., Innanje, A.: Ensemble attention distillation for privacy-rreserving federated learning. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15056-15066 (2021). htts://doi.org// 1 w09/CCCC8992...01111.... 001480
[19] Guo, P., Yang, D., Hatamizadeh, A., Xu, A., Xu, Z., Li, W., Zhao, C., Xu, D., Harmon, S., Turkbey, E., Turkbey, B., Wood, B., Patella, F., Stellato, E.,, Carrafiello, G., Patel, V.M., Roth, H.R.: Auto-fedrl: Federated hyperparameter optimization for multi-instiuutionall medical image segmentation (2022)
[20] Gupta, O., Raskar, R.: Distributed learning of deep neural network over multi- ple agents. Journal of Network and Computer Applications 116, 1-8 (08 2018). http:://doi.or//10. i0aa.a.........a..1s......
[21] Gutman, D., Codella, N.C., Celebi, E., Helba, B., Marchetti, M., Mishra, N., Halpern, A.: Skin lesion analysis toward melanoma detection: A challenge the international symposium on biomedical imaging (isbi) 2016. hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: wee::610503390 (2016)
[22] Han, S., Park, S., Wu, F., Kim, S., Wu, C., Xie, X., Cha, M.: FedX: Unsupervised Federated Learning with Cross Knowledge Distillation. In: ECCV (2022)

---

[23] Hitaj. B., Ateniese, G., Perez-Cruz, F.: Deep models under the gan: Information leakage from collaborative deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. p. 603-618. CCS '17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.orr//000 004411111111111133333331011,11 https
//doi.org/10, 2t5//00.11444 013353633356... 3134012
[24] Hu, M., Zhou, P.. Yue, Z., Ling, Z., Huang, Y., Liu, Y., Chen, M.: Fedcross: Towards
accurate federated learning via molti- cross aggregation. ArXiv abs/2210.08285 (2022),
https://api.... semant cichantaclcclarr org/ Brgg/orpusIDD.. 252917723
(25) Huang, W. Ye, M., Du, B.: Learn from others and be yourself in heterogeneous federated
learning. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
pp. 10133-10143 (2022). https//coooor//1..0019Cvvv..222 10910/9R5222268 010.88.222.009999
[26] Jeon, J., Kim, J., Lee, K., Oh. S., Ok, J.: Gradient inversion with generative image prior. In:
Beygelzimer, A., Dauphin. Y.. Liang, P., Vaughan, J.W. (eds.) Advances Neural Information
Processing Systems (2021), https: /ttr:e/s///perrveee...o
[27] Jin, X., Chen, P.Y., Hsu, C.Y., Yu, C.M., Chen, Cafe: Catastrophic data leakage vertical federated learning. ArXiv abs/2110.15122 (2021), (t02)))tttts://///./
https://api..... http:/ppp..smmaticcccccccooooo// org/
[28] Kaissis, G.A., Makowski, M.R., Rückert, D., Braren, R.F.: Secure, crivacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence 2(6), 305- 311 (Jun 2020). https://doi.org// 1038//22256-00000000 642256-020-0186-1
https://doi. org.org//// 1tgg/10.10 1038/
[29] Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: SCAFFOLD: Stochastic controlled averaging for federated learning. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, 5132-5143, PMLR (13-18 Jul 2020), https::/proceeddingg mlr
html
[30] Li, Q. He, B., Song, D.: Model-contrastive federated learning. In: Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition (2021)
[31] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V:: Federated optimization in heterogeneous networks (2020)
[32] Li, W., Milletari, F., Xu, D., Rieke, N., Hancox. J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M.J., Feng, A.: srivacy-preserving federated brain tumour segmentation. In: Suk, H.I.,
Liu, M., Yan, P., Lian, C. (eds.) Machine Learning in Medical Imaging. pp. 131-141. Springer International Publishing, Cham (2019)
[33] Liu, Q., Chen, C., Qin, Dou, Q., Heng, P.A.: Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[34] McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.y.:
Coamuuiiaatio-Effccceen Learning of Deep Networks from Decentralized Data. In: Singh. A., Zhu, J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 1273-1282. PMLR (20-22 Apr 2017), httpss
7/ //proceedings. mlr wwwpppeessvaaeeee http//////mmmaaaaaee 17a. ht..hammmmm...... html
[35] Mendieta, M., Yang, T., Wang. p.. Lee, M., Ding, Z., Chen, C.: Local learning mat- ters: Rethinking data heterogeneity in federated learning. pp. 8387-8396 (06 2022). https//ddo..or/110 1tt.:/or///O/.11110011 119.1111199/V..........
[36] Miao, J., Yang, Z., Fan, L., Yang. Y.: Fedseg: Class-heterogeneous federated learning for
semantic segmentation. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8042-8052 (2023). https://doi.org// 1099CVVRS2229.2023..
[37] Michieli, U., Ozay. M.: Prototype guided federated learning of visual feature represen- tations. ArXiv abs/2105.08982 (2021), https::/api. semanticscholarr httpiicccoooaaaorrrooropssssssss Corpuus 234777819
[38] Naeem, A., Anees, T., Naqvi, R.A., Loh, W.K.: comprehensive analysis of recent deep and -eeernind-lerrinng--e methodologies for brain tumor diagnosis. Journal of Personal- ized Medicine 12(2) 2075-4426/12/2/77
httpps: htpps://wwwt //www/... mdpi com/
12

---

[39] Oh, C.. Hwang, H., Lee, H.y., Lim, Y., Jung, G., Jung, J., Choi, H., Song, K.: Blackvip: Black-box visual prompting for robust transfer learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24224-24235 (June 2023)
[40] Poirot, M.G., Vepakomma, P., Chang, K., Selppathy-Cramce J., Gupta, R., Raskar, R.: Split learning for collaborative deep learning in healthcare (2019)
[41] Reddi, S., Charles, Z.B., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., McMahan.
B. (eds.): Adaptive Federated Optimization (2021), https:// i LKFG31B13U5
[42] Sheller, M.J., Reina, G.A., Edwards, B., Martin, J., Bakas, S.: Multi-institutional deep learn- ing modeling without sharing patient data: A feasibility study on brain tumor segmentation. In: Crimi. A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) Brainle- sion: Glioma, Multiple Sclerosis. Stroke and Traumatic Brain Injuries. pp. 92-104. Springer International Publishing, Cham (2019)
[43] Shenaj, D., Fani, E., Toldo, M., Caldarola, D., Tavera, A., Michicli, U., Ciccone, M., Zanuttigh, P., Caputo, B.: Learning across domains and devices: Style-driven source-frre domain adaptation in clustered federated learning. In: 2023 IEEE/CVF Winter Confer- ence on Applications of Computer Vision (WACV). pp. 444-454. IEEE Computer Soci- ety, Los Alamitos, CA. USA (jan 2023). https://doi.org//00 http://..1111/////// 1019//C/V66688 2023.00052, org/10..1100. 1109 1109/WACVV6668.. 202.::66666622202...22222
[44] Tan, Y., Long, G., Liu, L.., Zhou, T., Lu, Q., Jiang, J., Zhang. C.: Fedproto: Federated prototype
learning across heterogeneous clients. In: AAAI Conference Artificial Intelligence (2022)
[45] Tschandl, P., Rosendahl, C., Kittler, H.: The ham 10000 dataset, a large collection of multi-source
dermatoscopic images of common pigmented skin lesions. Scientific Data 5(1), 180161 (Aug 2018). https://doi.org// 10. 1038//datt.2011555.. 161
https:///oss org /1w/g/////. 1038/ sdata. 2018
[46] Valanarasu, J.M.J., Patel, V.M.: Unext: Mlp-based rapid medical image segmentation network. arXiv preprint arXiv:2203. 224.:22233009667 (2022)
[47] Vepakomma, P., Gupta, O., Swedish, T., Raskar, R.: Split learning for health: Distributed deep
learning without sharing raw patient data (2018)
[48] Vepakomma, P., Singh. A., Gupta, O.,, Raskar, R.: Nopeek: Information leakage reduction to share activations in distributed deep learning (2020)
[49] Wang, J., Jin, Y., Wang, L.: Personalizing federated medical image segmentation via local
calibration. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (cds.) Computer Vision - ECCV 2022. pp. 456-472. Springer Nature Switzerland, Cham (2022)
[50] Wu, Y., Zeng, D., Wang, Z., Shi, Y., Hu, J.: Federated contrastive learning for volumetric medical image segmentation. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) Medical Image Computing and Computer Assisted Intervention MICCAI 2021. pp. 367-377. Springer International Publishing, Cham (2021)
[51] Xie, E.. Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems (NeurIPS) (2021)
[52] Xu, A. Li, W. Guo, P., Yang. D., Roth, H., Hatamizadeh, A., Zhao C., Xu, D., Huang, H., Xu, Z.: Closing the generalization gap of cross-silo federated med- ical image segmentation. In: 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 20834-20843. IEEE Computer Society, Los Alami- tos, CA. USA (jun 2022). https://doi.coo otg/ 109.00/221000 2022.02020. hntps:///aii org/10. 11199//PR52689 20220020202
ieeecomputersociety
[53] Yao, C., Gong, B., Qi, H., Cui, Y., Zhu. Y., Yang, M.: Federated multi- target domain adaptation. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 1081-1090. IEEE Computer Society, Los Alamitos, CA, USA (jan 2022). https://doi.org/100 10tp://0001110//A11151554 011144558.22.00115. lhttps://dda ieeecomputersociety org/10.
13

---

[54] Zhang, H., Hong, J., Deng, Y., Mahdavi, M., Zhou, J.: Understanding deep gradient leakage via inversion influence functions. In: Oh, A., Neumann, T., Globerson, A., Saenko, K.. Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 3921- files// 3944. Curran Associates, Inc. (2023), http:g/g/rrroceeinnn..err....a.p Conferrr--o pdf
[55] Zhang. L., Shen, L., Ding, L., Tao, D., Duan, L.Y.: Fine-tuning global model via data-free knowl- edge distillation for non-iid federated learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10174-10183 (2022)
[56] Zhang, M., Sapra, K., Fidler, S., Yeung, S., Alvarez, J.M.: Personalized federated learning with first order model optimization. arXiv preprint wrXiv:2012.08565 (2020)
[57] Zhao, B., Mopuri, K.R., Bilen, H.: idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001. 1006100 (2020)
[58] Zhu, Blaschko, M.: R-gap: Recursive gradient attack on privacy (2021)
14

---

Table 7: Data counts for CAMVID, ISIC and Polypgen datasets
CAMVID
ISIC
Polypgen
Split
C1
C2
C3
C4
C1
C2
C3
C1
C2
C3
C4
C5
C6
Train
24
61
181
103
6009
720
2000
153
180
274
136
124
52
Val
8
16
50
26
2003
180
150
51
60
91
45
42
18
Test
92
24
74
42
2003
379
600
52
61
92
46
42
18
Table 8: Data counts for Cityscapes dataset
Cityscapes
Split
C1
C2
C3
C4
C5
C6
C7
C8
C9
C10
C11
C12
C13
C14
C15
C16
C17
C18
Train
121
67
221
107
59
154
76
173
137
83
69
65
255
137
100
66
99
85
Val
38
15
15
55
30
22
15
22
19
Test
26
14
47
23
33
16
37
29
18
15
14
55
29
22
14
21
18
A
Centerwise Dataset Information
In this section, we describe the number of data points in the training, validation and testing splits of each center of each dataset. The center represents the client in FL, where each center is in possession of data that cannot be shared directly with the other centers to the public. CAMVID centers, which are described in T Cityscapes [9] has 18 centers, which are described in Table g These centers represent the different cities from which the data is collected for these datasets. ISIC I [1 45] has centers and Polypgen h2] 4 centers, which represent the different hospitals from which the data is collected. These results are present in Table
has
B Centerwise Performance for Cityscapes Dataset
While the average mloU is presented in the main paper, also present the centerwise performance of DecpLaby3-beld b-ased server for the Cityscapes dataset Tables and [0] It can be seen from the tables that individual training performs poorly, the reason being that each of the centers has limited amount of data. In contrast, our method makes use of data from all clients to improve performance. Here, we also that BlackFed v2 performs better than v1 in all cases, thus showing the effectiveness of the server hashmap in countering catastrophic forgetting. With our approach, the performance of the model reaches close the white-box methods, but without sharing any gradients model information.
Table 9: mIoU scores for BlackFed vl and v2 in comparison with individual and FL-based training strategies for Cityscapes. "Local" represents data from the center. "OOD" represents mean mloU on test data from rest of the centers. For FedAvg and Combined Training, just one model is trained. Hence, its performance is noted only in each of the local test datasets.
C3
C4
C5
C5
C7
C4
CV
006
Laxal
Local
000
[uscll
Taxal
Blackted v]
1.76
0.72
0.67
0.71
0.77
0.71
0.75
0.71
0.66
0.30
1.74
0.71
0.71
0.71
0.66
0.2
0.70
0.70
StadkFal v2
0.76
0.74
6.11
0.74
0.11
0.74
0.77
6.74
0.70
0.73
0.56
0.74
6.75
0.73
0.66
635
0.1
6.72
Tranisg
035
0.34
FezlAng
0.5
0.35
0.54
0.90
0.74
0.00
0.78
0.30
0.36
15

---

Table 10: mloU scores for BlackFed v1 and v2 in comparison with individual and FL-based training strategies for Cityscapes. "Local" represents test data from the center. "OOD" represents mean mloU on test data from rest of the centers. For FedAvg and Combined Training, just one model is trained. Hence, its performance is noted only in each of the local test datasets.
C10
CII
C12
C13
C14
Cls
Cló
CI7
C18
Methed
Local
000
Local
000
Local
000
Local
0000
Local
000
Local
000
Local
DO0
Local
000
Local
0000
0.37
SladiFed v]
0.90
0.71
0.74
0.70
0.68
0.71
0.65
0.71
0.74
0.69
0.53
0.65
0.65
0.65
0.69
0.70
0.72
0.70
Rbadhal v.
I.72
0.73
0.77
1.73
0.71
0.73
0.72
0.76
0.78
0.75
1.788
0.74
0.74
06.33
0.80
1378
0.76
6.74
Contonal Traming
0.78
PedlAg
018
0.81
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper introduces black-box based approach for Federated Learning (FL) using split-nn formulation that does not require gradient and model info sharing. Section defines the approach and Section 4 shows experiments using the approach.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention further directions of improvement and future work the conclu- sion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
16

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We have provided the complete mathematical formulation and the algorithm, but theoretical claims are not made.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We will be releasing the code, data splits and pretrained models after review. The algorithm for the approach is included in the paper. The experimental settings also provided in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
17

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and data splits will be released on Github post review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc/ppl cwtppbliccguudeeeeeeeeceeeceeeee henool/cc/uiddes/ddeeeuuuiiiisii /et////ooeSemeeeee commiissionll ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: While the code will be released along with pretrained models, we have mentioned the experimental settings in the paper. We have also mentioned the data splits in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
18

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results of BlackFed have a p-value less than 0.001, showing statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in the experiment setup Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
all guidelines followed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
19

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Introduction discusses about our method's positive impacts. There are no apparent negative impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk posed by the work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
20

---

Justification:
We use public datasets that have been cited in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Code will be released post review and the novel algorithm is described the paper.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer:
[NA]
Justification:
The study does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "eGFbGT01EF", "reviewer": "Reviewer_zJW7", "summary": "A common issue now with federated learning systems is that they are not completely private owing to gradients transferred between the clients and the global server during training or by knowing the model architecture at the client end. \n\nThe paper proposed a workaround to this by removing the need for the passage of the gradients from client to server or knowing the model at the end. For the gradients, the authors propose zero-order optimization (ZOO) to update the client model weights and first-order optimization (FOO) to update the server weights. \n\nWhile achieving the above, they also perform reasonably well on most datasets.", "questions": "1. Why was the white-box training so subpar in ISIC? Further, while I am not well-versed in the literature, I imagine there are ways to tackle catastrophic forgetting in the white-box method as well. Would it be possible to do an ablation with that? Comparing V1 and white box, apart from ISIC, it beats V1 in all cases.\n\n2. Given that CAMVID has a Significantly lesser number of classes and centers, why is the performance better/on-par (and more stable) in the Cityscape dataset for BlackFed but worse in the case of \"individual\"? Even considering the majority classes.", "limitations": "1. \"After updating the server weights for a given client during training, we store the updated weights of the server model in a hashmap\".  - This step may increase significantly with scale, especially with multiple client updates. Lower floating point operations are indeed handy, but this is coming at the cost of high memory requirements.\n\n2. Issues with performance in the most important setting.\n\nSuggestion -\n\n3. I think in Table 1 and 2, Bolding the highest value and underlining the highest value without gradients will be a more proper representation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. An important direction of research where the federated framework does not require gradient transfer from client to server and an approach to tackle catastrophic forgetting in that framework.\n2. Comparable results to gradient accessed methods across all datasets.\n3. Significant reduction in training costs as compared to previous methods.", "weaknesses": "1. The case of polypgen (given the dataset size, medical background, and criteria of data collection) is where there is the most requirement/use of such a setup. And it underperforms over there. \"This behavior may be related to the data distribution of Polypgen and suggests that BlackFedv2 is not able to correctly avoid the catastrophic forgetting for centers C5 and C6. \". --- I think this has less to do with catastrophic forgetting and more to do with the model just not able learn. The data points are very few across all the centers. Given how the latest weights are stored, the weights used are just poor as they are not able to learn appropriately given the tough case of low data centers. It may also be that it is overparameterised by the other centres. If it was the case of it being affected by not being able to solve catastrophic forgetting, this would have been reflected everywhere. Especially in C1 C2 of CAMVID.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments.\n1) Low Performance on Polypgen: This case indeed serves as an important application area. Here, for 4 out of 6 centers (C1, C2, C4 and C5), it is beneficial to use the FL method. But for C3 and C6, the OOD performance decreases. This could be due to other centers dominating the training, thus causing catastrophic forgetting for particular centers like C6. But as pointed out by the reviewer, this should have been a problem elsewhere as well. In such cases, maybe the learnt weights are not good due to a more adverse distribution shift between centers in Polypgen as compared to CAMVID. In such cases, it would be important to control the training of the FL algorithm to make it more robust, which is an interesting and important direction for future research in this area.\n\n2) Counter-Intuitive Performance of Whitebox on ISIC: As pointed out by the reviewer, for ISIC, the whitebox method strangely performs subpar. We verified this through various runs and changes to the hyperparameters. This anomalous behaviour may be due to the model getting stuck in some local minima during training because of overparametrization.\n\n3) Comparing Individual Performance for CAMVID and Cityscapes: In the Blackfed case, for both datasets, the server model benefits from the entire dataset, hence the simliar performance. However, in the individual case, the model performs subpar in some of the clients. For CAMVID, individual performance of C1 is much lower than others. But for Cityscapes, there are some clients with a good performance and some clients with subpar performance. Hence, since the number of clients is more, on an average, the indivual performance is lower. This also depends on the choice of model architecture and data distribution since the number of data points is so less.\n\n4) Additional Memory Overhead: As the reviewer pointed out, the proposed approach comes with an added memory overhead for storing the hashmap, which scales with the number of clients. However, the motivation for the method was based on the assumption that while the client has limited memory, the common server can have a large memory. Based on the model architecture, each entry in the hashmap would be of the order of 100 MB or less. If there are 100 clients, this would translate to 10 GB, which is a realistic number. For larger models, we believe that the memory overhead cost would not be a bottleneck in comparison to the cost of training such models. One interesting direction in such cases would be using Parameter Efficient Finetuning Methods (PEFT) like adapters or LoRA and only storing them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mPwYHGirGU", "reviewer": "Reviewer_Ftdr", "summary": "In this work, the authors introduce BlackFed, an FL algorithm that enables distributed learning without transfer of gradients or model weights. This characteristic distinguishes the approach as a black-box model compared to existing FL methods, which can be considered as white-box approaches. Recent research on attacking FL methods require the knowledge of either the gradients or the model information, thus rendering BlackFed more resistant to such attacks since gradients or weights are not shared. BlackFed consists of a global server and multiple clients, each possessing their own data. The server is using first order optimization while the client weights are updated using zero order optimization in a round-robin fashion. This introduces the effect of catastrophic forgetting in the network, for which the authors propose a simple hashmap-based approach. The white-box methods, despite being a black-box method itself. Extensive experimentation on the natural and medical domain datasets highlights the effectiveness of BlackFed.", "questions": "No. Please see Weakness.", "limitations": "No. Please see Weakness.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In this work, a new approach, named BlackFed, is proposed. For segmentation, it uses FL that does not involve gradient transfer between the server and the client and at the same time, it passes no knowledge about the client model architecture to the server, thereby avoiding the necessary conditions for these attacks.\nThe strengths is as follows:\n1. BlackFed - a black-box algorithm that facilitates distributed learning for semantic segmentation without transferring model information or gradients between the client and the server is proposed. The authors formulate the FL problem using split-nn and use first and zero order optimization for training the server and the clients, respectively.\n2. To reduce the effect of catastrophic forgetting, the authors propose a simple additional step during training. After updating the server weights for a given client during training, the updated weights of the server model are stored in a hashmap indexed by the index of the client. During inference for a given client, the latest weights of the client model and the indexed weights of the server model to perform the forward pass are used.\n3. The proposed approach is evaluated on four segmentation datasets and it shows it’s effective as a distributed learning method by showing improvements over individual training.", "weaknesses": "The weakness is as follows\n1. For Table 1, it's indeed the proposed methods work well. But there is not experimental result for deeplabv3 on a single machine. Deeplab v3 achieves 80.0 on Cityscape val dataset on https://paperswithcode.com/lib/detectron2/deeplabv3-1. The paper generates 18 clients for Cityscapes. What's the relationship between the Table 1 evaluation result and benchmark result on Cityscape? This may make the paper not convincing enough.\n2. It's unclear why \"As the model complexity increases from UNext to DeepLab to Segformer, we observe a decrease in individual training performance. \". \n3. The image size for Cityscape is  256 × 512. why using so small resolution? it needs to do more experiments on higher resolution.\n4. What's the details for \" Consequently, we start the DeepLabv3 network in the server from the second layer, which expects a 64-channel input. \" ? Does it mean the deeplabv3 trained on server only unfreezed from the second layer? so how about the client?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments.\n1) Comparison with the baseline from papers with code: The dataset of Cityscapes has data from 18 different centers. In the original splits of Cityscapes, the training data contains images from some of these centers, while the data from rest of the centers is in the validation and testing sets. In our case, we want each center to have its own training, validation, and testing data. The goal of the Federated Learning model would be to do well on test data from other centers in addition to doing well on the in-house test set, indicated by the “OOD” and “Local” columns in the tables. Hence, the data splits are done differently. Training DeepLab v3 on single machine is the same as “Combined training” in Table 1, where all the training data is merged to train a single model, that achieves a DSC of 0.77, which is close to the number shown on PapersWithCode. This represents an upper bound to what BlackFed can achieve since training on a single machine means that it has access to data from all centers.\n\n2) Clarification on Trend of Model Complexity vs Performance: The model complexity includes the number of trainable parameters which increase from UNext to DeepLab to Segformer. But in table 3, row 1, we see that the individual performance increased for DeepLab but decreased drastically for Segformer, even though it has the most number of parameters. Since the data is limited in this case, we see that increasing the number of trainable parameters beyond a certain point causes the validation performance to suffer. However, for the case of combined training, the data quantity is higher, so val performance increases from UNext to DeepLab v3 to Segformer as expected. This is the desired trend in the FL algorithm as well since it also has access to more data than the limited data case of individual training. This trend is seen in BlackFed as the performance increases from Unext to DeepLab and does not drop when the model complexity increases from DeepLab to Segformer.\n\n3) Reason for Lower Resolution: We wanted to do the training of a given client on a single GPU, since institutes like medical centers do not have a high compute power. Hence, we downscale the image without affecting its aspect ratio. Training on a larger resolution would require much higher computation. Since we are emulating all client and server computations in the same compute center, this would require increased compute, especially in the case of 18 centers like Cityscapes. However, we plan to release a model zoo on Github with pretrained checkpoints at different resolutions since training with larger resolutions can improve performance.\n\n4) Clarification on Server Architecture: All the parameters in the approach are trainable and there are no frozen layers. In our experimental setup, we use a two-layer Convolution Net in the client to reduce the load on each client. The output of this layer is a feature map of the shape H X W X 64. This is transmitted to the server for further processing. Hence, the first layer of the server architecture should expect an input with this shape. We are using DeepLab v3 architecture for the server. However, instead of starting from the Conv1 layer in DeepLab, we are starting from Conv2 (and deleting Conv 1 from the network), which also expects an input with 64 channels. Hence, in order to continue the training in the server from the client's output, we designed the server architecture in such a way. In other words, this can be considered as one network, the first two layers of which are in the client while the rest of it is in the server, hence the term “split network”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVqNvMn8m4", "reviewer": "Reviewer_ykrj", "summary": "This manuscript introduces a federated learning framework for semantic segmentation that neither requires knowledge of the model architecture nor involves transferring gradients, thereby preventing privacy leakage. BlackFed incorporates split neural networks and first/zero order optimization for training the server and clients, respectively.", "questions": "(1) More comparison of Federated learning methods for segmentation should be performed;\n\n(2) All experiments were conducted using DeepLabV3 as the server segmentation network. It is recommended to include experimental results on different segmentation architectures.", "limitations": "The motivation behind BlackFed is privacy protection. However, the feature maps extracted by clients and segmentation masks are also transmitted to the server, making them vulnerable to attacks that could reconstruct the images from different clients. Therefore, this method does not achieve the goal of privacy protection.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The writing of the paper is clear.\n\n(2) The concept presented in this manuscript is interesting.", "weaknesses": "(1)\tIn addressing privacy leakage caused by sharing weights and gradients in current federated learning methods, this manuscript proposes BlackFed, which iteratively updates client and server networks. However, this approach involves transmitting extracted features and labels between the client and server, leading to privacy leakage and security risks. For instance, some algorithms like stable diffusion [1] can reconstruct client images from these features and segmentation masks. Therefore, I believe BlackFed does not effectively protect privacy and fails to show superiority over methods based on network weights and gradients.\n\n(2)\tThere are too few comparison methods. In the Related Work section, many federated learning methods for segmentation tasks are introduced; FedSeg and FedSM could be used for comparison.\n\n(3)\tBlackFed preprocesses client data using a two-layer convolutional neural network on clients, allowing different clients to extract global features. These preprocessed features are then sent to the server for segmentation. The client-side training follows the method proposed in [2], which limits the approach's innovation.\n\n(4)\tIn the inference phase, the features extracted by the client network need to be uploaded to the server each time. Does this significantly increase testing time consumption?\n\n(5)\tFedPer [3] is another federated learning method using split neural networks. It is recommended to introduce it in the Related Work section and add comparison results in the experimental section.\n\n[1] High-Resolution Image Synthesis with Latent Diffusion Models\n\n[2] Blackvip: Black-box visual prompting for robust transfer learning.\n\n[3] Federated Learning with Personalization Layers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments.\n\n1) Reproduction of Training data: As the reviewer suggested, the intermediate representations and segmentation masks can be used along with diffusion models. However, to train these models, one would still require the raw data to act as ground truth labels during training. If one were to use public datasets for training such diffusion models, the diffusion model would still learn to generate images from a similar distribution to the public datasets. This is not the same as regenerating raw data since raw data can have personally identifiable information (PII) that won't be learnt by diffusion models which are trained on public datasets. In such scenarios, in fact, the output from Stable Diffusion can be considered as a good source for synthetic data, which looks similar to raw data but would arise from the public data distribution. In fact, some recent works also highlight the property of stable diffusion and other image generation methods to copy the data they were trained on [1-4]. In such a scenario, in the absence of raw data from the client, the best a diffusion model would be able to do is to generate images from public datasets similar to the segmentation mask, which we think cannot be considered as reproduction of client data. We believe that not requiring model information transfer and gradient transfer is an important step in the direction of better privacy-preserving FL over existing methods.\n\n[1] Frame by Familiar Frame: Understanding Replication in Video Diffusion Models\n\n[2] Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models\n\n[3] Analyzing bias in diffusion-based face generation models\n\n[4] Replication in Visual Diffusion Models: A Survey and Outlook\n\n2) Adding more Comparisons: FedSeg is a method that is built upon FedAvg and works by sharing model weights across server and client. Hence, this would act as an upper bound for our method, just like FedAvg. We verify this for CAMVID and Cityscapes and add it to Table 1 presented in the extra table page for rebuttal. We see that for both the datasets, it generally performs better than FedAvg. Please note that this cannot be directly compared with BlackFed since our approach operates in a more restricted setting where model sharing is not allowed. We also add other methods like FedPer in this table, as suggested.\n\n3) Novelty for ZOO: Please note that we do not claim novelty for the ZOO method used. In fact, we cited the BlackVIP paper as being the work that introduced SPSA-GC as a ZOO method (ref. No. 38 in the paper). The major contributions of our work include formulation of the FL problem in the blackbox setting using split networks to allow gradient-free updates. The proposed approach formulates the FL problem using a lightweight client and a parameter-heavy server, along with a round-robin algorithm that can allow SPSA-GC to work well, since it was originally meant to fine-tune pre-initialized foundation models. The proposed approach shows that by alternating between clients, and updating them with ZOO and FOO, one can achieve good performance. In addition, we identify the catastrophic forgetting problem and come up with a solution using hashmaps.\n\n4) Inference: During inference, uploading the features to server would increase time as pointed out. Some ways to reduce this would be to upload batches of features at the same time that can reduce the amortized time cost. An alternative is to get a copy of the server initialized with the hashmap weights for the client after training is complete. Then, client can perform the inference locally. This assumes that the client has sufficient compute power. Note that the entire hashmap need not be shared, but only the entry corresponding to the client.\n\n5) Additional Method FedPer: We thank the reviewer for pointing out this method. We will add it to the related work and comparisons in the revised paper. In addition, we added results with FedPer in Table 1 in the extra rebuttal page. However, FedPer does not use split networks. Instead, it is similar to FedAvg, the difference being that not all weights are shared with the server. Here, the majority of the weights are shared with the server and aggregated. The remaining weights are \"personal\" to the model and allow for better local performance. These are not shared with the server. In contrast, there are no weights shared between the client and server in Blackfed. The output from the client is given to the server, where it is further processed, making this a split network. Furthermore, The FedPer paper only performs the task of classification. In order to adapt this for segmentation, we considered the weights of the classifier head of DeepLab v3 as personal weights and the rest of the backbone weights were averaged in the server. We find that in most cases, this improved local performance over FedAvg, but reduced OOD performance. \n\n6) Ablations with more architectures: Table 3 in the paper compares performance with three different architectures for the server. This includes Unext and Segformer in addition to DeepLab. Segformer is a transformer-based method while the others are CNN-based.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMX5htaBhm", "reviewer": "Reviewer_XV1b", "summary": "The authors proposed a black-box tuning method to address the problem of Semantic Segmentation in FL. Specifically, they propose to split the network into two parts and store them in server and clients separately. The server modules are optimized via first-order optimization while the client modules are optimized via black-box optimization. The proposed method achieves promising results on different benchmarks.", "questions": "Please refer to the questions in the Weaknesses section.", "limitations": "The authors did not address the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writting is clear and the presentation of the method is easy to follow.\n2. The proposed method of combining split learning and black-box optimization is interesting.\n3. The experiments are thorough and the results of the method are promising.", "weaknesses": "1. From my understanding, the privacy problems still exist. Since the authors claimed that they are using a two-layer CNN at the client, which is relatively light-weight and compact compared to the network used at server, the attackers at server may still be possible to reproduce the training data using the intermediate output (the uploaded values from each client). The authors may need to add further discussion about this issue.\n2. In the process 1b (Figure 2), how is the first-order gradient computed at the server? Are the ground-truth semantic maps uploaded to the central server? Or the network predictions will be downloaded to different clients and then evaluated? For the first solution, there will be privacy threatens. For the second one, the communication costs will be very large. The authors should clarify this in more details.\n3. The authors introduce a new two-layer CNN at each client, could the network DeepLabv3 (the network at central server) directly be split in the clients and server?\n4. It would be interesting to compare with other FL algorithms which also applies Black-box Tuning such as [1].\n\n[1] Sun J, Xu Z, Yin H, et al. Fedbpt: Efficient federated black-box prompt tuning for large language models[J]. arXiv preprint arXiv:2310.01467, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments.\n1) Reproduction of training data:\n\n    a) The clients can arbitrarily choose their networks. In the current setting, there is no dependence between the clients or the client and the server other than the output size of the client's last layer. The current work proposed a two-layer CNN as an example network.\n    \n    b) Even for a two-layer network, lets say an attacker knows the intermediate representations. Then the task for the attacker would be to find the input x and the function f given y where f is a small network and y is the leaked intermediate output. Now, to find out x, one could solve the equations y = f(x) and y’ = f’(x), where f’ denotes the derivative with respect to x. However, this method requires gradients to be known as well, which is not the case with out method.  \n\n    c) Furthermore, one method could be to learn a new network to produce the same outputs as the client models, given some input. In this case, the attacker would initialize a new two layer CNN and would have to query the original client model with learnable inputs. This way, they could train the new model with the client outputs as ground truths. However, for this case as well, the attacker would require access to the actual client models.\n\n2) Sending Masks to the Sever: In the proposed work, masks are sent to the server for FOO. Even with this information, the attacker would require raw data to generate a new model that can emulate the client without gradients. For example, consider stable diffusion or similar methods that can recreate images given their masks. These would be based on public datasets and hence, would not be able to generate PII information that will be present in the raw data. In fact, given the mask, they can still generate synthetic data, but that cannot be considered as replicating raw data since it would still use a distribution of pixels similar to public datasets and not the private client data. We believe that not requiring model information transfer and gradient transfer is an important step in the direction of better privacy-preserving FL over existing methods.\n\n3) Directly Splitting the Server Model:  The network at server could also be split into two parts as mentioned by the reviewer. While this is a valid strategy, this would limit all the clients to use the same network. At the same time, the server would potentially have the complete model architecture information, which is not required in the current setting. The motivation behind proposing a new lightweight client model was that this would allow different clients to have their own design choices for the model with the condition that the output should have a particular dimension. Since the client is updated using ZOO, having a lightweight model might also help in getting a better performance.\n\n4) FedBPT as added black-box method: We thank the reviewer for pointing out the FedBPT method. We will add it in the related work section. Please note that FedBPT was proposed to work with foundation models and as a prompting mechanism. Hence, it uses CMA-ES, a ZOO method to learn a prompt for the pretrained foundation model at each client and aggregates these prompts at the server. Since we want to learn the network parameters themselves and not the prompts, we cannot directly compare with this method. Instead, for the experiments, we tried replacing SPSA-GC with the CMA-ES method for updating the client in our algorithm. However, we see that it kills the program due to the large number of parameters in the network as compared to prompts. CMA-ES is an evolutionary algorithm that maintains a big group of candidate values for each trainable parameter, that is perturbed towards a better loss value in each iteration. Since the number of parameters is very large, we were not able to run CMA-ES as the ZOO algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We tried to address all concerns in the respective rebuttal sections. Here, we would like to write the global rebuttal for two common concerns raised by the reviewers:

1) More comparison experiments: We added two new results in Table 1 of the paper (shown in the attached pdf file) for the suggested methods FedPer [1] and FedSeg [2]. Both of these methods involve model weights transfer similar to FedAvg and so can be considered as upper bounds for our method. 

    (a) In the case of FedPer, the majority of the weights are shared with the server and a small head is retained at the client to allow for more personalization of client models. We see that this improves local performance but is not able to perform as well on OOD data. While this is defined for classification, for comparison, we adapt it for segmentation by keeping the final classifier head of our network 'personal' to the model and don't share its weights with the server. We find that our method comes on par with this method on OOD distribution, despite being a black box method not involving model weight transfer. 

    (b) In the case of FedSeg, all the weights are shared with the server similar to FedAvg. During the training of the client, an additional step is taken to align the behaviour of the model for OOD and local distributions better. This method was introduced as an improvement to FedSeg. Our method performs on par with FedSeg without sharing model weights. 

    (c) One of the reviewers suggested a comparison with FedBPT [3], which uses the ZOO method called CMA-ES for learning a prompt to foundation models. This prompt is shared between the client and the server while the same frozen foundation model is used at all clients and the server. CMA-ES is an evolutionary method and hence, requires lots of candidates for each of the learnable parameters. Over several iterations, these candidates converge to an optimal value for the parameter. In FedBPT, the learnable prompt is very tiny compared to the client part of the network in our case in terms of learnable parameters. Hence, when we used CMA-ES instead of SPSA-GC as the zero-order optimization method, it always crashed the program. Hence, we believe that CMA-ES would not be a feasible solution for optimizing a larger number of parameters.

[1] Federated Learning with Personalization Layers

[2] FedSeg: Class-Heterogeneous Federated Learning for Semantic Segmentation

[3] Fedbpt: Efficient federated black-box prompt tuning for large language models

2) Discussion on regeneration of training data: The reviewers had concerns about whether attackers could target the method with existing methods like Diffusion Models or other mechanisms. We present our views on this topic below:

    (a) For attacking methods that generate new models to approximate raw data with the lowest error, an optimization problem is solved. Here, the goal would be to find input x that minimizes the error between the predicted value of the original model and the predicted value of the new model. However, these methods would either require the gradients or the ability to query the original client model multiple times, both of which are not available in our proposed method.

    (b) Diffusion models present a more interesting challenge since methods like Stable Diffusion can generate synthetic images based on the shared mask and labels. However, various works have found that these Image Generation methods mimic the data distribution that they are trained on [1-4]. Hence, even if such methods are used to attack BlackFed, they would generate data with pixel distribution from existing public datasets and lack the Personally Identifiable Information (PII) in the raw client data, which is private and not accessible to the diffusion model for training. In such a scenario, while the generated synthetic images would be valid data, they should not be considered as replicating the raw input data.

[1] Frame by Familiar Frame: Understanding Replication in Video Diffusion Models

[2] Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

[3] Analyzing bias in diffusion-based face generation models

[4] Replication in Visual Diffusion Models: A Survey and Outlook

Meta-review: Authors present a gradient-free federated learning approach in which knowledge of the client architecture nor the weights are not required by the server. The client and server networks are optimized differently. The main claim for the article is that this approach is more privacy preserving because the gradients and weights of clients remain at the client. However, representations and masks need to be transmitted to the server which may still cause some privacy issues. 

The main concerns for the rebuttal were on experimental analysis and the privacy concerns. In the experimental analysis, more comparisons and clarifications were required. For the latter, transmitting representations and masks seem to violate the privacy concerns. In other words, the proposed technique also cannot guarantee full privacy of the data. 

During the rebuttal authors successfully answered questions on experimental analysis. Questions on the privacy concerns remained open to the best of my understanding. Among four reviewers, three of them suggested to accept the article, two with BA and one with A. Only one reviewer suggested BR. However, I believe concerns this reviewer raised were addressed to some extent during the rebuttal. Unfortunately, the reviewer did not revise their score. 

Given the positive reviews, acknowledged technical novelty, successful rebuttal and the final ratings, I believe this article would be a good contribution to the conference.
Decision: Accept (poster)