Title: A Kernel Perspective on Distillation-based Collaborative Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Kernel Regression

Distillation-based Collaborative Learning

Generalization Analysis

Kernel Distillation
Abstract: Over the past decade, there is a growing interest in collaborative learning that can enhance AI models of multiple parties.
However, it is still challenging to enhance performance them without sharing private data and models from individual parties.
One recent promising approach is to develop distillation-based algorithms that exploit unlabeled public data but the results are still unsatisfactory in both theory and practice.
To tackle this problem, we rigorously analyze a representative distillation-based algorithm in the view of kernel regression.
This work provides the first theoretical results to prove the (nearly) minimax optimality of the nonparametric collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments.
Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture.
Our algorithm successfully bridges the gap between our theoretical assumptions and practical settings with neural networks through feature kernel matching.
We simulate various regression tasks to verify our theory and demonstrate the practical feasibility of our proposed algorithm.

A Kernel Perspective on Distillation-based Collaborative Learning

Abstract

Over the past decade, there is a growing interest in collaborative learning that can enhance AI models of multiple parties. However, it is still challenging to enhance performance them without sharing private data and models from individual parties. One recent promising approach is to develop distillation-based algorithms that exploit unlabeled public data but the results are still unsatisfactory in both theory and practice. To tackle this problem, we rigorously analyze a representative distillationbased algorithm in the view of kernel regression. This work provides the first theoretical results to prove the (nearly) minimax optimality of the nonparametric collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture. Our algorithm successfully bridges the gap between our theoretical assumptions and practical settings with neural networks through feature kernel matching. We simulate various regression tasks to verify our theory and demonstrate the practical feasibility of our proposed algorithm.

1 Introduction

Collaborative learning of AI models in decentralized settings is an important problem covered in various fields of machine learning such as distributed learning [10, 70], Federated Learning (FL) [23], peer-to-peer learning [3], and miscellaneous collaborative learning [47]. In particular, this theme has been most actively discussed in the context of FL [24, 31, 45, 61]. In this context, each local party is typically viewed as a subordinate entity within the collective learning system. For example, most FL algorithms mandate the exchange of local AI model information among participating local parties. Under this scheme, local AI models are usually subjected to restrictions in their architecture. However, from the perspective of collaboration, each local party may have to be regarded as an independent learning agent, meaning they are not obligated to fully share their model information. In short, the model (or parameter) exchange in FL algorithms can emerge as a critical issue in collaborative learning.

Fundamentally, addressing this issue necessitates an alternative medium for sharing learning information distinct from model exchange. Indeed, Distillation-based Collaborative Learning (DCL) [14, 30, 44] provides a good answer. In these algorithms, local training information is shared via the outcomes of AI models on additional unlabeled public data. The collected information is then utilized for knowledge distillation [21] to each local AI model. As mentioned in [14, 48], this procedure is agnostic to model heterogeneity and avoids the direct sharing of local AI model information. This is a key advantage that distinguishes DCL from traditional FL.

Despite its pioneering nature and potential utility, DCL has not been sufficiently explored. A significant reason for this is the lack of theoretical understanding regarding knowledge distillation and its effectiveness in massively distributed statistically heterogeneous environments. Our work stems from the fundamental question of whether DCL algorithms can be theoretically effective in these settings. Inspired by [48, 58], we analyze FedMD [30, 41], the most standard DCL algorithm from a nonparametric perspective. Specifically, we adopt an operator-theoretic approach [5, 15, 37, 53, 65] to obtain an upper rate of convergence for the nonparametric version of FedMD (named DCL-KR ) in the expected sense. Remarkably, our analysis reveals that DCL-KR achieves a nearly minimax optimal convergence rate, where the prefactor is independent of the number of participating local parties. It is worth noting that DCL-KR is the first nearly minimax optimal collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. The novelty of our theoretical results and their comparison to prior works are provided in Section 2 and 3.

Nevertheless, our theoretical analysis does not fully demonstrate the efficacy of DCL algorithms based on neural network architectures. Instead, our theoretical results serve as inspiration for designing a novel DCL algorithm for regression that refines existing approaches. Consequently, we propose a Distillation-based Collaborative Learning algorithm over heterogeneous Neural Networks (named DCL-NN ) for regression tasks. DCL-NN leverages kernel matching to align the feature kernels from the last hidden layer of each local AI model with an ensemble kernel. This procedure brings heterogeneous neural networks into the regime of DCL-KR.

Finally, we conduct experiments on DCL-KR and DCL-NN. To illustrate the superiority of our algorithms, we compare them with several baselines on various regression tasks. Experimental results show that DCL-KR achieves the same performance as the centralized model, even beyond the theoretical results. We also observe that DCL-NN significantly outperforms previous DCL frameworks in most settings.

In summary, our contributions are as follows:

In Section 3, we theoretically prove that a nonparametric version of the most standard distillationbased collaborative learning algorithm (named DCL-KR) is nearly minimax optimal in massively distributed statistically heterogeneous environments.

Inspired by the results provided in Section 3, we propose a distillation-based collaborative learning algorithm with heterogeneous neural networks (named DCL-NN) in Section 4.

In Section 5, we conduct experiments to empirically confirm our theoretical results and show the practical feasibility of our proposed algorithms.

2 Related Work

Federated Learning Most FL algorithms [45] communicate model parameters for collaboration. This approach has been extensively studied under various constraints, including data privacy [1], statistical heterogeneity [24, 31], communication efficiency [51], personalization [13, 59], and robustness [25]. While it has been successful both theoretically and experimentally, this type of FL is limited in terms of the privacy and flexibility of local AI models, as the algorithms directly access the structures and parameters of the local models. Our study focuses on distillation-based collaborative learning, where the privacy and flexibility of local AI models are fully guaranteed.

Distillation-based Collaborative (or Federated) Learning The type of algorithms we investigate operates by communicating the functional information of local AI models. These algorithms typically assume the availability of additional public data points. In this case, the outcomes of local models on the public dataset are used for collaboration. For instance, Li and Wang [30], Lin et al. [41], Park et al. [48] iteratively collect predictions of local models on the public dataset and then aggregate them into a naive ensemble (with or without a fixed linear transformation) to distribute. On the other hand, Cho et al. [7], Zhang et al. [69], Fan et al. [14] apply personalized ensemble strategies by additionally learning the mutual trust between models. Makhija et al. [44] propose FedHeNN, which distills training information in the form of matching feature kernels instead of the predictions of local AI models on the public data. Both FedHeNN and DCL-NN utilize centered kernel alignment [8] to match feature kernels of local models, but DCL-NN uses the ensemble distillation for predictions as well. Thus, DCL-NN enables parties to learn from the entire input space.

Decentralized Learning with Kernel Regression A number of studies have investigated the minimax optimal rate of regularized kernel regression algorithms such as kernel ridge regression and gradient descent-based kernel regression with early stopping [5, 15, 37, 65]. In particular, over the past decade, the growing interest in decentralized learning has led to active research in the generalization analysis of decentralized kernel regression. While divide-and-conquer algorithms [34, 38, 66, 70] play a significant role in this research flow, most of them fail to account for statistical heterogeneity and massively distributed cases, along with privacy preservation, which has received a lot of attention recently. On the other hand, decentralized kernel regression algorithms with multiple communication rounds [40, 43, 48, 58, 67] achieve superior theoretical results compared to the divide-and-conquer algorithms. However, the discussions of these algorithms primarily focus on the efficiency of resource costs [40, 43, 67], while research on relaxing environmental constraints has been scarce. For example, most of these works assume a limited number of parties to prove the optimality in a minimax sense.

To the best of our knowledge, [48, 58] stand as the only investigations that consider general decentralized environments. Similar to our work, Park et al. [48] study the convergence rate of distillation-based collaborative learning with kernel regression. However, their results demonstrate a weaker version of minimax optimality and do not cover statistically heterogeneous environments. In this regard, Su et al. [58] offer a promising methodology. They analyze nonparametric versions of FedAvg [45] and FedProx [31], representative FL algorithms involving model exchange, in general decentralized environments such as statistically heterogeneous and massively distributed scenarios. In this work, we extend their methodology to analyze FedMD [30, 41] from a nonparametric perspective in massively distributed statistically heterogeneous environments. We summarize the comparison between our work and prior studies in Table 1. Note that algorithms that do not employ Nyström scheme (including nonparametric FedAvg [58]) fail to preserve local data privacy due to the inherent characteristics of kernel regression. On the other hand, DC-NY [66] and DKRR-NY-CM [67] can achieve the local data privacy preservation by utilizing the public data as Nyström centers.

3 DCL-KR: A Nonparametric View of FedMD

In this section, we establish the theory of a nonparametric version of FedMD [30, 41], the most standard distillation-based collaborative learning algorithm.

3.1 Preliminaries

Let ρ x ,y = ρ x · ρ y | x be a Borel probability measure on X × R where X is a compact subset of R d and we assume the support of ρ x is X . The goal of the regression problem is to find a minimizer of the population risk, i.e.,

Then, the function f ∗ 0 : X → R defined by x 0 ↦→ E y ∼ ρ y | x ( ·| x 0 ) [ y ] , x 0 ∈ X is a target function.

Let k : X × X → R be a Mercer kernel [9] where κ := (sup x ∈X k ( x , x )) 1 / 2 < ∞ and H k be a reproducing kernel Hilbert space associated to k . We set k x := k ( · , x ) and the covariance operator T k,ν : H k → H k with respect to any Borel probability measure ν on X defined as

Then we can see that T k,ν = ι glyph[latticetop] ν ι ν where ι ν : H k → L 2 ν is a natural embedding, L 2 ν = L 2 ( X , ν ) denotes the L 2 space, and a superscript glyph[latticetop] denotes the adjoint operator of a given operator. We also define the sampling operator S D : H k → R n by h ↦→ [ h ( x 1 ) , · · · , h ( x n )] glyph[latticetop] and T k,X := S glyph[latticetop] D S D when D = { ( x 1 , y 1 ) , · · · , ( x n , y n ) } with X = { x 1 , · · · , x n } is given. Since S D depends only on data inputs X , we can define the sampling operator for unlabeled datasets in the same way. See Appendix A.1 for further details.

3.1.1 Kernel Gradient Descent with Early Stopping

Given a dataset D = { ( x 1 , y 1 ) , · · · , ( x n , y n ) } generated from ρ x ,y , consider the empirical risk ˜ E D : H k → R given by

where y = [ y 1 , · · · , y n ] glyph[latticetop] . Here, ‖ · ‖ 2 denotes a scaled Euclidean norm ‖ v ‖ 2 = ( 1 n ∑ n i =1 v 2 i ) 1 / 2 . From the functional derivative ∇ ˜ E D ( h ) = S glyph[latticetop] D ( S D h -y ) , the gradient descent scheme becomes

where { η t } t ∈ N is a set of learning rates. In this work, we set η t = η , t ∈ N for a fixed η ∈ (0 , 1 /κ 2 ) . Then, a simple calculation gives ν t → S glyph[latticetop] D ( S D S glyph[latticetop] D ) -1 y as t →∞ provided that the operator S D S glyph[latticetop] D is invertible. The limit is known as the minimum norm interpolation [49] of D . Since the interpolation regressor generalizes poorly unless there is no noise [32, 39], early stopping strategies are usually applied to avoid the overfitting issue. With adequate stopping rules, gradient descent-based kernel regression has an optimal rate in a minimax sense [36, 37, 65].

3.2 DCL-KR Algorithm

From now on, we consider the setting that there are m parties and the i th party has a private local data D i = { ( x j i , y j i ) : j = 1 , · · · , n i } for i = 1 , · · · , m . Assume that all data D = ⋃ m i =1 D i are i.i.d. with the distribution ρ x ,y but each local dataset does not need to have the same distribution. Let Z = { z 1 , · · · , z n 0 } ⊂ X be the additional public inputs. The goal of all parties is to have their models that perform well on the distribution ρ x . In other words, each party expects to be able to make good predictions not only for its local data distribution but also for unseen data distribution through collaborative learning.

Similar to [58], we construct a nonparametric version of FedMD (called DCL-KR ), which is presented in Algorithm 1. In Algorithm 1, G i is a one-step local gradient descent update on ˜ E D i , i.e., G i h = h -ηS glyph[latticetop] D i ( S D i h -y i ) where y i = [ y 1 i , · · · , y n i i ] glyph[latticetop] . Similarly, ˜ G t is a one-step gradient descent update on ˜ E ( Z, y p,t ) , i.e., ˜ G t h = h -ηS glyph[latticetop] Z ( S Z h -y p,t ) .

3.3 Theoretical Results

In this subsection, we show the nearly minimax optimality of DCL-KR. To derive theoretical results, we assume the following conditions regarding regularity of noise, the kernel k , and the target function f ∗ 0 as below.

Assumption 3.1. We assume E y ∼ ρ y y 2 < ∞ and

where M and γ are positive constants.

Algorithm 1 DCL-KR Algorithm

1: Hyperparameters: T : total communication round, E : the number of local iterations at each communication round, η : learning rate

2: Initialize local models f i, 0 = 0 for i = 1 , · · · , m .

3: for t = 0 , · · · , T -1 do

4: for party i = 1 , · · · , m do

5: Update the local model E times by gradient descent on the empirical risk ˜ E D i

6: Upload the local predictions on Z to the server

7: end for

8: The server aggregates the local predictions to compute the consensus prediction

and then distributes y p,t to all local parties.

9: For party i ( i = 1 , · · · , m ), update the local model by infinitely many iterations of gradient descent on the empirical risk ˜ E ( Z, y p,t )

with an initialization g chosen from a subspace spanned by k 1 , · · · , k n 0 .

i,t z z 10: end for

Assumption 3.2. Let λ 1 ≥ λ 2 ≥ · · · > 0 be eigenvalues of T k,ρ x . There are fixed positive constants C s and c s such that

for some s ∈ (0 , 1) .

Assumption 3.3. The target function f ∗ 0 satisfies

for some r ∈ [ 1 2 , 1] where T r -1 / 2 k,ρ x is the ( r -1 / 2) power of operator T k,ρ x and R > 0 is a fixed constant. In particular, f ∗ 0 ∈ H k .

The above assumptions determine the minimax lower rate [5] and are standard assumptions in many prior works [5, 15, 33, 35]. In detail,

Assumption 3.1 implies that the noise is not excessively large. This assumption is a general noise condition that encompasses a wide range of cases. For instance, noise with Bernstein condition such as sub-Gaussian noise satisfies Assumption 3.1.

Assumption 3.2 is about the eigenvalue decay of T k,ρ x . From this assumption, one can derive bounds on the effective dimension that is related to covering and entropy number conditions [15].

Assumption 3.3 is related to the regularity of the target function, specifically how well the RKHS induced by the kernel k represents the target function.

Under these assumptions, we can theoretically show the performance guarantee of DCL-KR. The proof is provided in Appendix A.2. Note that E ( h ) -E ( f ∗ 0 ) = 1 2 ‖ ι ρ x ( h -f ∗ 0 ) ‖ 2 L 2 ρ x is the excess risk of a regressor h and so the quantity ‖ ι ρ x ( h -f ∗ 0 ) ‖ L 2 ρ x indicates the generalization ability of h .

Theorem 3.4. Under Assumption 3.1, 3.2, and 3.3, with n 0 ≥ n 1 2 r + s (log n ) 3 public inputs independently generated from ˜ ρ x such that the Radon-Nikodym derivative dρ x d ˜ ρ x satisfies

DCL-KR gives the performance guarantee

for all i = 1 , · · · , m where η ∈ (0 , 1 /κ 2 ) is a fixed learning rate, T is an adequate stopping rule, and the prefactor C does not depend on B , m , and n .

Since the convergence rate n -r 2 r + s is the minimax lower rate under Assumption 3.1, 3.2, and 3.3, Theorem 3.4 implies that DCL-KR has an almost same convergence rate as the minimax optimal central training when there are sufficiently many public inputs. To the best of our knowledge, this is the first work to prove the (nearly) minimax optimality of a collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. For example, divide-and-conquer algorithms work for limited m . Specifically, DCNY [66] assumes m ≤ O ( n 2 r -1 2 r + s ) and DKRR-NY-CM [67] assumes m ≤ O ( n 2 r + s -1 2 r + s ) . However, Theorem 3.4 does not require any condition on m . Moreover, Theorem 3.4 deals with a more general setting than the theory in [48, 58]. For example, Su et al. [58] only cover r = 1 2 of Assumption 3.3. On the other hand, Park et al. [48] do not consider Assumption 3.2 which gives a finer result. Compared with [48], we also reduce the required size of public inputs and drop the statistical homogeneity condition.

The convergence rate in Theorem 3.4 has an additional factor log n compared with a minimax lower rate [5, 15], but this logarithm term grows slower than any polynomial. Note that an additional logarithm term commonly appears in the context of gradient descent-based kernel regression with Nyström scheme [35, 36].

Theorem 3.4 allows that the public input distribution ˜ ρ x can be different from the local input distribution ρ x . It is natural that the condition (2) is required since ˜ ρ x should cover ρ x for fully distilling training information. We can see that the discrepancy between ρ x and ˜ ρ x affects the upper bound in Theorem 3.4 as the multiplication of B r . We can remove B r in the upper bound by increasing public inputs. See Appendix A.3 for details.

3.3.1 Proof Sketch of Theorem 3.4 and Comments

In the proof of Theorem 3.4, we decompose the term ι ρ x ( f i,T -f ∗ 0 ) into four parts, say (I), (II), (III), and (IV) (see Eq. (7)). The proof is to bound the norms of these terms. Note that DCL-KR can also be understood as a Nyström version of nonparametric FedAvg [58] from the recurrence relation (6).

(I) and (II) appear similarly in [58], except that (I) and (II) incorporate projections. To handle these terms, we reinterpret the proof presented in [58] in operator form instead of matrix form and extend it to our setting. We obtain a norm bound of (II) containing a quantity linked to the local Rademacher complexity. (Appendix A.2.2 and A.2.3)

Comparing with [58], (III) and (IV) are additional terms induced by the procedure that distills functional information from the local regressors. We apply techniques used in [35, 48, 53] to bound (III) and (IV). (Appendix A.2.4)

Note that previous works applying local Rademacher complexity-based stopping rule [50, 58] deal with the case of r = 1 2 only. In this work, we set a new stopping rule T which is an extension of previous works [50, 58] and prove an extended version (Lemma A.6) of a well-known property [60]. As a result, our theory covers r ∈ [ 1 2 , 1] which affects the minimax lower rate. (Appendix A.2.5)

4 DCL-NN Algorithm

In this section, we retain the problem setting from Section 3 but employ heterogeneous neural networks as the local models. Based on the theoretical results in Section 3, we propose a novel

distillation-based collaborative learning algorithm DCL-NN across heterogeneous neural networks in a decentralized setting.

A key factor contributing to the successful theoretical guarantee of DCL-KR lies not only in the linearity of kernel regression but also in the equality of kernels across local models. In fact, the public data predictions can vary in different directions, even if the same training data points are used when kernels differ (See Appendix B). Therefore, we match the kernels of local AI models. Specifically, we use linear feature kernels [18, 64] induced by the features from the last hidden layers of local AI models for kernel matching. For example, for a neural network f : X → R where f ( · ) = w glyph[latticetop] g ( · )+ b , g : X → R c , w ∈ R c , and b ∈ R we use

as the feature kernel of f . Through this idea, we can bring the setting closer to the regime of DCL-KR. Note that our theoretical results suggest that the target kernel should be a good kernel. Indeed, we observe that the naive ensemble

has a significantly better performance than individual feature kernels k f 1 , · · · , k f m (See Section 5 and Appendix B). Here, f i is the local model of the i th party with its local feature kernel k f i obtained by (3) ( i = 1 , · · · , m ). Therefore, we align local feature kernels k f 1 , · · · , k f m in a kernel distillation manner with the ensemble kernel k obtained by (4).

For this purpose, we introduce Centered Kernel Alignment (CKA) [8] as a kernel similarity measure. CKA is a typical measure associated with the similarity of two representations of neural networks [27] and is often used for kernel matching in neural networks [44]. To compute empirical CKA between two kernels k 1 and k 2 on inputs { c 1 , · · · , c p } , we first calculate the Gram matrices K 1 = [ k 1 ( c j 1 , c j 2 )] 1 ≤ j 1 ,j 2 ≤ p and K 2 = [ k 2 ( c j 1 , c j 2 )] 1 ≤ j 1 ,j 2 ≤ p . We then compute the empirical CKA via

Here, ̂ HSIC is an estimator of the Hilbert-Schmidt Independence Criterion (HSIC) defined as

where H := I p -1 p 11 glyph[latticetop] is the centering matrix. In the kernel distillation procedure, the i th local party maximizes ̂ CKA ( k f i , k ) on public inputs Z ( i = 1 , · · · , m ). Notably, this procedure requires only a single communication round for exchanging pairwise feature kernel values on public inputs, ensuring that our algorithm operates exclusively within the function space.

After the kernel distillation procedure, all local AI models have similar feature kernels up to constants. So we can follow an analogous process as in DCL-KR. Note that we perform learning rate scaling described in Appendix B to compensate the kernel scale difference. It makes the impact of local iterations consistent. We also provide the complete algorithm (Algorithm 2) and further details for Section 4 in Appendix B.

5 Experiments

In this section, we evaluate the performance of DCL-KR and DCL-NN. We compare them with baselines on various regression tasks.

Datasets We use the following six regression datasets to evaluate the performance. Target variables are one-dimensional in all datasets. (1) Toy-1D [33] and (2) Toy-3D [6] are synthetic datasets with one-dimensional and three-dimensional inputs, respectively. (3) Energy is a tabular dataset from the UCI database [12] to predict appliances energy use with 28 features. (4) RotatedMNIST is an image dataset where it aims to predict the rotation angles for given rotated images of the MNIST [11] images. (5) UTKFace [71] and (6) IMDB-WIKI [42, 52] are image datasets for age estimation.

20

40

60

80

100

The number of parties

2

4

6

8

Average of RMSEs

1e

2

centralKRR

centralKRGD

IED

DC-NY

DKRR-NY-CM

DCL-KR

(a) Toy-1D Dataset

20

40

60

80

100

The number of parties

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Average of RMSEs

1e

1

centralKRR

centralKRGD

IED

DC-NY

DKRR-NY-CM

DCL-KR

(b) Toy-3D Dataset

20

40

60

80

100

The number of parties

1.0

1.5

2.0

2.5

Average of RMSEs

1e

1

=0.1

=0.3

=0.5

=1

=2

(a) DCL-KR

20

40

60

80

100

The number of parties

1.0

1.5

2.0

2.5

Average of RMSEs

1e

1

=0.1

=0.3

=0.5

=1

=2

(b) IED

We compare kernel machine-based collaborative learning algorithms on two datasets Toy-1D and Toy-3D. On the other hand, we compare neural network-based collaborative learning algorithms on five datasets Toy-3D, Energy, RotatedMNIST, UTKFace, and IMDB-WIKI.

Baselines We compare DCL-KR with two central kernel regression models to verify our theoretical results. These two central models have the minimax optimal convergence rate. We also utilize existing decentralized kernel regression algorithms that does not directly share local data and models (DC-NY [66], DKRR-NY-CM [67], IED [48]) as baselines for DCL-KR. On the other hand, we adopt FedMD with unlabeled public inputs [30, 41], FedHeNN [44], and KT-pFL [69] as baselines for DCL-NN.

Setup The number of parties ranges from 10 to 100 for kernel machine-based algorithms and is 50 for neural network-based algorithms. We construct statistically heterogeneous decentralized environments with Algorithm 3. For neural network-based algorithms, we use 4 different neural network architectures for local models in all settings. For instance, we use ResNet-18, ResNet-34, ResNet-50 [20], and MobileNetv2 [54] for large-scale image datasets. We utilize the average of Root Mean Squared Errors (RMSEs) of the local AI models on a test dataset as a performance metric. The test data points have the same distribution as the whole local data distribution. We apply FedMD with a few communication rounds for pretraining of DCL-NN. See Appendix C for detailed experimental configurations.

5.1 Results on Kernel Machine-based Algorithms

The performance of DCL-KR and its baselines is presented in Figure 1. We set the number of parties m = 10 , 20 , · · · , 100 , the number of private data points n = 50 m , and the number of public inputs n 0 = n 1 2 r + s (log 10 n ) 3 . We first set ρ x = ˜ ρ x , i.e., the public data distribution is the same as the entire local input distribution. As shown in Figure 1, DCL-KR outperforms the baselines in all

2.5

3.0

3.5

4.0

4.5

log

m

2.8

2.6

2.4

2.2

2.0

1.8

Logarithm of Average of RMSEs

=0.5

=1

=4,

=0.5

(a) DCL-KR (log-scale)

2.5

3.0

3.5

4.0

4.5

log

m

2.6

2.4

2.2

2.0

1.8

1.6

Logarithm of Average of RMSEs

=0.5

=1

=4,

=0.5

(b) IED (log-scale)

glyph[negationslash]

experimental settings and achieves comparable performance to the central models. This result implies that DCL-KR has not only the nearly optimal convergence rate but also the same performance as central kernel regression models. In contrast, DC-NY and DKRR-NY-CM exhibit significantly lower performance compared with DCL-KR in massively distributed environments where their theory does not cover. IED does not show a significant performance drop in massively distributed environments even though its theory is built on the statistical homogeneity condition of local data distributions.

To further compare the performance of DCL-KR and IED, which show similar results to central models, we analyze the effect of n 0 and ˜ ρ x on their performance (Figure 2 and 3). Figure 2 illustrates that, as expected from the theoretical results, IED requires more public inputs than DCL-KR to achieve good performance. Moreover, when there is a public distribution shift, DCL-KR maintains its convergence rate, whereas the convergence rate of IED deteriorates. (See Appendix C.3.3 for experimental details.) Overall, our experiments validate the theoretical results of DCL-KR and demonstsrate its superiority over previous results. For additional experimental results and analyses, please refer to Appendix C.3.

5.2 Results on Neural Network-based Algorithms

Table 2 shows the performance of DCL-NN and baselines on five regression tasks. We also present the performance of standalone models and centralized models to assess the performance of the collaborative algorithms. For some cases exhibiting training instability, we report the best test error (marked with asterisks) observed across all communication rounds, while relying on a fixed number of communication rounds for the other cases.

As can be seen in Table 2, DCL-NN outperforms the baselines on all regression tasks. Note that FedHeNN employs kernel matching similar to DCL-NN, but it lacks supervision of label prediction through collaboration, resulting in insufficient performance improvement compared to standalone models. Given the superior performance of DCL-NN, it is evident that incorporating supervised learning for label prediction alongside kernel matching is desirable. On the other hand, while FedMD performs significantly better than standalone models, the performance of DCL-NN is consistently better. Considering that we utilize FedMD for pretraining of DCL-NN, we can see that it performs better than FedMD-only collaborative learning by first training local models with FedMD and then using DCL-NN. In conclusion, the experimental results support the practical effectiveness and superiority of DCL-NN over baselines.

Kernel Distillation Procedure To verify the necessity of kernel distillation, we examine the changes in the performance of local feature kernels and the CKA between them during the kernel distillation procedure. We conduct this experiment on UTKFace. We utilize the RMSE of a kernel linear regression model trained on all local data as a kernel performance measure. The results are presented in Figure 4. As shown in Figure 4, both kernel performance and CKA undergo a temporary degradation due to the change of the objective function at the initial stages. However, as training progresses, both metrics recover and kernel performance surpasses its initial level. Since kernel distillation aims to ensure that all local feature kernels are similar with high performance, the experimental results verify the effectiveness of kernel distillation.

0

50

100

150

200

Epochs

0.146

0.147

0.148

0.149

0.150

Local Feature Kernels

Target Kernel

(a) Kernel Performance

0

50

100

150

200

Epochs

0.80

0.82

0.84

0.86

0.88

0.90

Average of CKAs

For additional experimental results, please refer to Appendix C.4.

6 Conclusions

In this work, we analyze distillation-based collaborative learning from a nonparametric perspective and propose DCL-NN, a practical algorithm as an extension. We demonstrate that DCL-KR, a nonparametric version of FedMD, has a nearly minimax optimal convergence rate in massively distributed statistically heterogeneous environments. Inspired by DCL-KR, we propose DCL-NN, a novel distillation-based collaborative learning algorithm for heterogeneous neural networks. Our experiments confirm the theoretical results of DCL-KR and demonstrate the practical effectiveness of DCL-NN. For a discussion of the limitations of our work, please refer to Appendix D.

Broader Impact Our work explores the methodologies of collaborative learning under data and model privacy preservation. In this regard, our research holds the potential to positively impact the facilitation of collaboration among AI models without raising concerns about information disclosure. On the other hand, our work does not pose any particularly noteworthy negative consequences, given its aim to contribute to the advancement of the general field of machine learning.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (Grant No. RS-2019-NR040050).

References

E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2938-2948. PMLR, 2020.

P. L. Bartlett, O. Bousquet, and S. Mendelson. Local Rademacher complexities. The Annals of Statistics , 33(4):1497-1537, 2005.

A. Bellet, R. Guerraoui, M. Taziki, and M. Tommasi. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics , pages 473-481. PMLR, 2018.

O. Bousquet. Concentration inequalities for sub-additive functions using the entropy method. In Stochastic Inequalities and Applications , pages 213-247. Springer, 2003.

A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7:331-368, 2007.

X. Chang, S.-B. Lin, and D.-X. Zhou. Distributed semi-supervised learning with kernel ridge regression. The Journal of Machine Learning Research , 18(1):1493-1514, 2017.

Y. J. Cho, J. Wang, T. Chirvolu, and G. Joshi. Communication-efficient and modelheterogeneous personalized federated learning via clustered knowledge transfer. IEEE Journal of Selected Topics in Signal Processing , 17(1):234-247, 2023.

C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research , 13(1):795-828, 2012.

F. Cucker and D. X. Zhou. Learning theory: an approximation theory viewpoint , volume 24. Cambridge University Press, 2007.

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems , 25, 2012.

L. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

D. Dheeru and E. Karra Taniskidou. UCI machine learning repository, 2017. URL http: //archive.ics.uci.edu/ml .

A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems , volume 33, pages 3557-3568, 2020.

D. Fan, C. Mendler-Dünner, and M. Jaggi. Collaborative learning via prediction consensus. In Advances in Neural Information Processing Systems , 2023.

S. Fischer and I. Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. The Journal of Machine Learning Research , 21(1):8464-8501, 2020.

J. Fujii, M. Fujii, T. Furuta, and R. Nakamoto. Norm inequalities equivalent to heinz inequality. Proceedings of the American Mathematical Society , 118(3):827-830, 1993.

Z.-C. Guo, S.-B. Lin, and D.-X. Zhou. Learning theory of distributed spectral algorithms. Inverse Problems , 33(7):074009, 2017.

B. He and M. Ozay. Feature kernel distillation. In International Conference on Learning Representations , 2021.

C. He, M. Annavaram, and S. Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems , 33:14068-14080, 2020.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , pages 448-456. PMLR, 2015.

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. Foundations and Trends R © in Machine Learning , 14(1-2):1-210, 2021.

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

S. P. Karimireddy, L. He, and M. Jaggi. Byzantine-robust learning on heterogeneous datasets via bucketing. In International Conference on Learning Representations , 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , pages 3519-3529. PMLR, 2019.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

M. Ledoux and M. Talagrand. Probability in Banach Spaces: isoperimetry and processes . Springer Science & Business Media, 2013.

D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 , 2019.

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Y. Li, H. Zhang, and Q. Lin. Kernel interpolation generalizes poorly. Biometrika , 2023.

Y. Li, H. Zhang, and Q. Lin. On the saturation effect of kernel ridge regression. In International Conference on Learning Representations , 2023.

J. Lin and V. Cevher. Optimal convergence for distributed learning with stochastic gradient methods and spectral algorithms. Journal of Machine Learning Research , 21(147):1-63, 2020.

J. Lin and L. Rosasco. Optimal rates for learning with Nyström stochastic gradient methods. arXiv preprint arXiv:1710.07797 , 2017.

J. Lin and L. Rosasco. Optimal rates for multi-pass stochastic gradient methods. The Journal of Machine Learning Research , 18(1):3375-3421, 2017.

J. Lin, A. Rudi, L. Rosasco, and V. Cevher. Optimal rates for spectral algorithms with leastsquares regression over hilbert spaces. Applied and Computational Harmonic Analysis , 48(3): 868-890, 2020.

S.-B. Lin, X. Guo, and D.-X. Zhou. Distributed learning with regularized least squares. The Journal of Machine Learning Research , 18(1):3202-3232, 2017.

S.-B. Lin, X. Chang, and X. Sun. Kernel interpolation of high dimensional scattered data. arXiv preprint arXiv:2009.01514 , 2020.

S.-B. Lin, D. Wang, and D.-X. Zhou. Distributed kernel ridge regression with communications. Journal of Machine Learning Research , 21(93):1-38, 2020.

T. Lin, L. Kong, S. U. Stich, and M. Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems , 33:2351-2363, 2020.

Y. Lin, J. Shen, Y. Wang, and M. Pantic. Fp-age: Leveraging face parsing attention for facial age estimation in the wild. arXiv , 2021.

Y. Liu, J. Liu, and S. Wang. Effective distributed learning with random features: Improved bounds and algorithms. In International Conference on Learning Representations , 2020.

D. Makhija, X. Han, N. Ho, and J. Ghosh. Architecture agnostic federated learning for neural networks. In International Conference on Machine Learning , pages 14860-14870. PMLR, 2022.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics , pages 1273-1282. PMLR, 2017.

S. Mendelson. Geometric parameters of kernel machines. In International Conference on Computational Learning Theory , pages 29-43. Springer, 2002.

C. Mendler-Dünner, W. Guo, S. Bates, and M. Jordan. Test-time collective prediction. Advances in Neural Information Processing Systems , 34:13719-13731, 2021.

S. Park, K. Hong, and G. Hwang. Towards understanding ensemble distillation in federated learning. In International Conference on Machine Learning , pages 27132-27187. PMLR, 2023.

V. I. Paulsen and M. Raghupathi. An introduction to the theory of reproducing kernel Hilbert spaces , volume 152. Cambridge University Press, 2016.

G. Raskutti, M. J. Wainwright, and B. Yu. Early stopping and non-parametric regression: an optimal data-dependent stopping rule. The Journal of Machine Learning Research , 15(1): 335-366, 2014.

D. Rothchild, A. Panda, E. Ullah, N. Ivkin, I. Stoica, V . Braverman, J. Gonzalez, and R. Arora. FetchSGD: Communication-efficient federated learning with sketching. In International Conference on Machine Learning , pages 8253-8265. PMLR, 2020.

R. Rothe, R. Timofte, and L. V. Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision , 126(2-4):144-157, 2018.

A. Rudi, R. Camoriano, and L. Rosasco. Less is more: Nyström computational regularization. Advances in Neural Information Processing Systems , 28, 2015.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4510-4520, 2018.

R. Schaback and H. Wendland. Kernel techniques: from machine learning to meshless methods. Acta Numerica , 15:543-639, 2006.

B. Sen. A gentle introduction to empirical process theory and applications. Lecture Notes, Columbia University , 11:28-29, 2018.

I. Steinwart and A. Christmann. Support vector machines . Springer Science & Business Media, 2008.

L. Su, J. Xu, and P. Yang. A non-parametric view of fedavg and fedprox: Beyond stationary points. The Journal of Machine Learning Research , 24(203):1-48, 2023.

C. T Dinh, N. Tran, and J. Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems , 33:21394-21405, 2020.

M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

H. Wang, M. Yurochkin, Y. Sun, D. S. Papailiopoulos, and Y. Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations , 2020.

C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics , pages 370-378. PMLR, 2016.

G. Yang and E. J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning , pages 11727-11737. PMLR, 2021.

Y. Yao, L. Rosasco, and A. Caponnetto. On early stopping in gradient descent learning. Constructive Approximation , 26:289-315, 2007.

R. Yin, Y. Liu, L. Lu, W. Wang, and D. Meng. Divide-and-conquer learning with nyström: Optimal rate and algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6696-6703, 2020.

R. Yin, W. Wang, and D. Meng. Distributed nyström kernel learning with communications. In International Conference on Machine Learning , pages 12019-12028. PMLR, 2021.

H. Zhang, Y. Li, and Q. Lin. On the optimality of misspecified spectral algorithms. arXiv preprint arXiv:2303.14942 , 2023.

J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu. Parameterized knowledge transfer for personalized federated learning. Advances in Neural Information Processing Systems , 34: 10092-10104, 2021.

Y. Zhang, J. Duchi, and M. Wainwright. Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates. The Journal of Machine Learning Research , 16(1):3299-3340, 2015.

Z. Zhang, Y. Song, and H. Qi. Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5810-5818, 2017.

Z. Zhu, J. Hong, and J. Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning , pages 12878-12889. PMLR, 2021.

A Details on Section 3

Before we start the proof of Theorem 3.4, we present basic notions.

A.1 Basic Notions

In Subsection 3.1, the reproducing kernel Hilbert space H k is a subset of C ( X ) , i.e., all elements in H k are continuous [57]. Since

we have T k,ρ x = ι glyph[latticetop] ρ x ι ρ x . The compactness of ι glyph[latticetop] ρ x [57] gives the fact that T k,ρ x is compact, self-adjoint, and positive. Furthermore, Mercer's theorem [62] gives a Mercer representation

glyph[negationslash]

The fact that H k ⊂ C ( X ) and T k,ρ x = ι glyph[latticetop] ρ x ι ρ x implies the injectivity of T k,ρ x and so λ i = 0 for all i ∈ N . We define

for any probability measure ν . For convenience, N ( λ ) := N ρ x ( λ ) . From [5, 15], we have N ( λ ) ≤ C ′ s λ -s where C ′ s := C s s / (1 -s ) and N ν ( λ ) ≤ κ 2 λ -1 . Given a dataset D = { ( x i , y i ) } n i =1 , a similar argument as above gives

and T k,X : h ↦→ 1 n ∑ n i =1 h ( x i ) k x i .

Note that Assumption 3.1 implies that

We have

and so the excess risk becomes

Therefore, ‖ ι ρ x ( h -f ∗ 0 ) ‖ 2 L 2 ρ x indicates the generalization ability of h .

Table 3 presents meaning of some notations.

A.2 Proof of Theorem 3.4

Without loss of generality, we assume n ∧ n 0 ≥ κ 2 e .

A.2.1 Recurrence Relation of DCL-KR

Consider a subspace W of H k spanned by { k z 1 , · · · , k z n 0 } . We first show that for a fixed h ∗ ∈ H k and a gradient update G u = u -ηS glyph[latticetop] Z ( S Z u -S Z h ∗ ) we have G t u 1 → P Z h ∗ as t → ∞ for any u 1 ∈ W where P Z is an orthogonal projection onto the subspace W . Set u t +1 = G u t for t ≥ 1 . Then

Since S Z h ∗ = S Z P Z h ∗ , we have

Note that there exists { ˜ z 1 , · · · , ˜ z ˜ n 0 } ⊂ Z such that { k ˜ z 1 , · · · , k ˜ z ˜ n 0 } is a basis of W . Define a matrix

such that k z i = ∑ ˜ n 0 j =1 b ij k ˜ z j . Then K Z ˜ Z = BK ˜ Z ˜ Z where

and

Set P Z h ∗ = ∑ ˜ n 0 j =1 a j k ˜ z j . Then we can see that

where [ · ] r is the r th component of the given vector and a = [ a 1 , · · · , a ˜ n 0 ] glyph[latticetop] . Note that K ˜ Z ˜ Z is invertible since v glyph[latticetop] K ˜ Z ˜ Z v = 0 implies v = 0 . We can also see that K ZZ = BK ˜ Z ˜ Z B glyph[latticetop] where

So

Thus, 0 < η n K 1 / 2 ˜ Z ˜ Z B glyph[latticetop] BK 1 / 2 ˜ Z ˜ Z < I and as t →∞ . Similarly, we get ( I -ηS glyph[latticetop] Z S Z ) t u 1 → 0 as t →∞ . Therefore, we attain G t u 1 → P Z h ∗ as t →∞ for any u 1 ∈ W .

From this fact, DCL-KR has the recurrence relation

where f t = f i,t for any i = 1 , · · · , m and T k,X i := I -ηT k,X i for i = 1 , · · · , m . Then we obtain a closed form

We first compute

From we have

where the last equality follows from P Z ( I -P Z ) = 0 . Thus, we obtain the equality

∗

A.2.2 Norm Bound of First Term in (7)

We first bound the norm of the first term in (7) as

where λ > 0 . The first term in (8) is bounded as

Note that

by Lemma A.8. Set A i = T E k,X i ⇔ T k,X i = 1 η ( I -A 1 /E i ) . We observe that

where the equality follows from 0 ≤ A i ≤ I ⇒ A 1 /E i ≥ A i and I ≥ P Z . Since sup x ∈ [0 , 1] x t/r (1 -x ) = r · ( t ) t/r , we attain the inequality

and

∥ ∥ ∥ T 1 / 2 k,ρ x ( T k,X + λI ) -1 / 2 ∥ ∥ ∥ ≤ ∥ ∥ ∥ ( T k,ρ x + λI ) 1 / 2 ( T k,X + λI ) -1 / 2 ∥ ∥ ∥ ≤ ∥ ∥ ( T k,ρ x + λI )( T k,X + λI ) -1 ∥ ∥ 1 / 2 .

holds with confidence at least 1 -δ where δ ∈ (0 , 1) . Combining (9) and (10) and applying r t + r ≤ 1 t and ( t t + r ) t/r ≤ 1 2 yield

with confidence at least -1

∞

Here, we apply the fact that E A = ∫ 0 P ( A ≥ t ) dt for A ≥ 0 . We next turn to bound the second term in (8). Note that

Set A i = T E k,X i . Using a similar argument as before gives

Since Z and X are independent, we have

We already see that

holds with confidence at least 1 -δ where δ ∈ (0 , 1) and so

by putting λ = n -1 2 r + s as before.

The remaining part is to bound E ‖ ( I -P Z ) f ∗ 0 ‖ H k . Applying Lemma A.9 yields ‖ ( I -P Z ) f ∗ 0 ‖ H k ≤ λ 1 / 2 0 ‖ ( T k,Z + λ 0 I ) -1 / 2 T r -1 / 2 k,ρ x ‖‖ g ∗ 0 ‖ H k where λ 0 > 0 . Then Lemma A.8 gives

From dρ x d ˜ ρ x ≤ B , we obtain

Set λ 0 = 128( κ 2 +1) 2 (log n 0 ) 3 /n 0 where we assume n is sufficiently large such that λ 0 ≤ 1 and N ˜ ρ x ( λ 0 ) ≥ 1 for n 0 ≥ n 1 2 r + s (log n ) 3 . By Lemma A.8 and Lemma A.12,

holds with confidence at least 1 -δ where δ ∈ [4 exp( -1 / 4( κ 2 +1) B 0 ) , 1) and

Since we have

From n 0 ≥ κ 2 e , we get

and so

where the last inequality comes from n 0 ≥ n 1 2 r + s (log n ) 3 .

A.2.3 Norm Bound of Second Term in (7)

Set

Note that

Then the second term in (7) becomes

We can see that

We first bound the expectation of the first term in the above. By the Cauchy-Schwartz inequality, we have

Observe that and

Using E ( y -S D f ∗ 0 )( y -S D f ∗ 0 ) glyph[latticetop] ≤ γ 2 I , we have

where the last inequality follows from the fact that ‖ AB ‖ HS ≤ ‖ A ‖‖ B ‖ HS , ‖ AB ‖ HS ≤ ‖ A ‖ HS ‖ B ‖ , and

which follows from (11), we can see that

where λ i ( · ) is the i th largest eigenvalue of a given operator. Therefore,

Using (12) and 1 ∧ u 2 ≤ 1 ∧ u for u ≥ 0 lead to

where ˆ λ 1 ≥ · · · ≥ ˆ λ n are eigenvalues of S D S glyph[latticetop] D , the first inequality comes from the Bernoulli inequality, and the last inequality follows from the fact that ‖ P ‖ ≤ E . We define

Then

Similarly as in Appendix A.2.2, putting λ = n -1 2 r + s gives

Since

Therefore, the Cauchy-Schwartz inequality gives a bound as

We now bound the expectation of

By the Cauchy-Schwartz inequality and the same argument as before, we have

Also, the same argument as before yields

where

To bound E [ tr ( A )] , note that

Let B = ηP 1 / 2 S D P Z S glyph[latticetop] D P 1 / 2 . Then 0 ≤ B ≤ I and

where the first inequality follows from 1 -x t ≤ 1 ∧ t (1 -x ) and the second inequality follows from 1 /x ∧ t 2 x ≤ t ∧ t 2 x for all t ≥ 0 and x ∈ [0 , 1] . From the fact that

we have

Therefore, we obtain

In conclusion, we have an upper bound of the norm of the second term in (7) as

by taking λ = n -1 2 r + s . We will bound E R ( 1 √ ηtE ) 2 in Appendix A.2.5.

A.2.4 Norm Bound of Third and Last Term in (7)

Note that

where 0 < λ ≤ 1 . From (11) and 0 ≤ P ≤ EI , we have 0 ≤ I -∑ m i =1 n i n T E k,X i = ηS glyph[latticetop] D PS D ≤ ηES glyph[latticetop] D S D = ηET k,X . Using this fact, we find that

To bound this, we first observe that

where the last inequality follows by a similar calculation as before. On the other hand, we have

by the same argument. Using a simple calculation, we get

and

Note that the norm of the third term in (7) is bounded as

Therefore, the norm of the sum of the third and last terms in (7) is bounded by

To bound ∥ ∥ ( T k,X + λI ) 1 / 2 ( I -P Z ) f ∗ 0 ∥ ∥ H k , observe that

‖ ( T k,X + λI ) 1 / 2 ( I -P Z ) f ∗ 0 ‖ H k ≤ ‖ ( T k,X + λI ) 1 / 2 ( T k,ρ x + λI ) -1 / 2 ‖‖ ( T k,ρ x + λI ) 1 / 2 ( I -P Z ) f ∗ 0 ‖ H k and

Under Assumption 3.3, we have

Since

which follows from Lemma A.8 and

we have ‖ ( T k, ˜ ρ x + λI ) -( r -1 / 2) T r -1 / 2 k,ρ x ‖ ≤ B r -1 / 2 . On the other hand,

Since X and Z are independent, we have

holds with confidence at least 1 -δ where δ ∈ (0 , 1) . Also, by Lemma A.8 and Lemma A.11

holds with confidence at least 1 -δ where δ ∈ (0 , 1) . Thus,

with confidence at least 1 -δ where δ ∈ (0 , 1) . Set λ = 128( κ 2 +1) 2 n -1 2 r + s where n is sufficiently large such that λ ≤ 1 and N ˜ ρ x ( λ ) ≥ 1 . Then

We now bound E ‖ ( T k, ˜ ρ x + λI ) 1 / 2 ( I -P Z ) ‖ 2 r . By Lemma A.9, we have

By Lemma A.12, ‖ ( T k, ˜ ρ x + λI ) 1 / 2 ( T k,Z + λI ) -1 / 2 ‖ ≤ √ 2 with confidence at least 1 -4 exp( -1 / 4( κ 2 +1) B 0 ) where

Also, ‖ ( T k, ˜ ρ x + λI ) 1 / 2 ( I -P Z ) ‖ ≤ ( κ 2 +1) 1 / 2 almost surely. Thus,

Note that

and so ( κ 2 +1) r · 4 exp ( -1 4( κ 2 +1) B 0 ) ≤ 4( κ 2 +1) r · 1 n . Therefore,

E ‖ ( T k, ˜ ρ x + λI ) 1 / 2 ( I -P Z ) ‖ 2 r ≤ 2 r 128 r ( κ 2 +1) 2 r · n -r 2 r + s +4( κ 2 +1) r · n -1 glyph[lessorsimilar] n -r 2 r + s . We can conclude that

A.2.5 Stopping Rule and Rademacher Complexity Bound

For convenience, we abuse the notation D = { ( x 1 , y 1 ) , · · · , ( x n , y n ) } and X = { x 1 , · · · , x n } . Define the local empirical Rademacher complexity

and the local population Rademacher complexity

where w 1 , · · · , w n are independent Rademacher random variables and ρ x ,n = 1 n ∑ n i =1 δ x i . We also define

where λ 1 ≥ λ 2 ≥ · · · ≥ 0 are eigenvalues of T k,ρ x . We recall the following well-known property. Lemma A.1 ([46], [60]) . We have √

for glyph[epsilon1] > 0 .

We can prove the following lemma using a similar argument as in [46].

Lemma A.2. There is an absolute constant c > 0 which satisfies that for every glyph[epsilon1] > 0 ,

Proof of Lemma A.2. We divide the proof into three parts.

Part 1. Since T k,X = S glyph[latticetop] D S D , ˆ λ 1 ≥ ˆ λ 2 ≥ · · · ≥ ˆ λ n ≥ 0 are eigenvalues of T k,X . For convenience, set ˆ λ i = 0 for i > n and define ˆ n ≤ n such that ˆ λ ˆ n > 0 and ˆ λ ˆ n +1 = 0 . Choose an orthonormal basis { ˆ ψ i } ∞ i =1 of H k such that ˆ ψ i is an eigenvector of T k,X corresponding to ˆ λ i . Then 〈 ˆ ψ i , ˆ ψ j 〉 L 2 ρ x ,n = 〈 S D ˆ ψ i , S D ˆ ψ j 〉 2 = 〈 T k,X ˆ ψ i , ˆ ψ j 〉 H k = δ { i = j } ˆ λ i . We will show that

where x ∈ { x 1 , · · · , x n } . Let W 1 be the subspace of H k spanned by { ˆ ψ i : i = 1 , · · · , ˆ n } and W 2 be the subspace of H k spanned by { k x i : i = 1 , · · · , n } . Observe that W ⊥ 1 = ker T k,X and W ⊥ 2 ⊂ ker T k,X by the reproducing property. Thus, W 1 ⊂ W 2 . Conversely, choose a basis { k ˜ x i : i = 1 , · · · , ˜ n ′ } ⊂ { k x i : i = 1 , · · · , n } of W 2 . Then, using a similar argument as in Appendix A.2.1 implies that there exists a matrix

such that k x i = ∑ ˜ n ′ j =1 b ij k ˜ x j . Then K X ˜ X = BK ˜ X ˜ X where

Since K ˜ X, ˜ X and B glyph[latticetop] B are invertible,

for any b = [ b 1 , · · · , b ˜ n ′ ] glyph[latticetop] ∈ R ˜ n ′ where [ · ] r is the r th component of the given vector. Therefore, W 2 ⊂ ran T k,X = (ker T k,X ) ⊥ = W 1 and so W 1 = W 2 . From this fact, we can see that k x i = ∑ ˆ n r =1 a r ˆ ψ r for some a 1 , · · · , a ˆ n ∈ R . Then a r = 〈 k x i , ˆ ψ r 〉 H k = ˆ ψ r ( x i ) for all r = 1 , · · · , ˆ n and so we are done. Note that k x = ∑ ∞ i =1 ˆ ψ i ( x ) ˆ ψ i where x ∈ { x 1 , · · · , x n } since ˆ ψ i ( x ) = 0 for x ∈ { x 1 , · · · , x n } and i > ˆ n .

Part 2. Define

and and

where 0 0 = 1 . Then E ⊂ F since

for h = ∑ ∞ i =1 h i ˆ ψ i ∈ E . Thus,

where glyph[epsilon1] 1 , · · · , glyph[epsilon1] n are i.i.d. Rademacher variables. By the reproducing property,

where h = ∑ ∞ i =1 h i ˆ ψ i . Thus,

Since

we have

Therefore,

Part 3. By Khintchine's inequality,

By Remark A.14, g is convex and sup h ∈F ( ∑ n i =1 h ( x i ) 2 ) 1 / 2 = √ nglyph[epsilon1] -Lipschitz on [ -1 , 1] n . By Lemma A.13, we have

From we have

where c = ( 1 + ∑ ∞ m =0 ( m +2) 2 exp ( -m 2 32 )) -1 / 2 is an absolute constant. Therefore,

We set the population radius as

We also define

By Lemma A.1, we have glyph[epsilon1] n ≤ ˜ glyph[epsilon1] n . We can easily see that R , R , Q n , and Q n are increasing functions. The following lemma can be shown by a similar argument as in [2].

Lemma A.3. If g : [0 , ∞ ) → [0 , ∞ ) is a function such that g is non-decreasing and r ↦→ g ( r ) /r is non-increasing, then g is continuous on (0 , ∞ ) .

Since

glyph[epsilon1] ↦→R ( glyph[epsilon1] ) /glyph[epsilon1] and glyph[epsilon1] ↦→R ( glyph[epsilon1] ) /glyph[epsilon1] are non-increasing and so R and R are continuous.

Lemma A.4. glyph[epsilon1] ↦→ Q n ( glyph[epsilon1] ) /glyph[epsilon1] and glyph[epsilon1] ↦→ Q n ( glyph[epsilon1] ) /glyph[epsilon1] are non-increasing. In particular, Q n is continuous and glyph[epsilon1] n < ∞ .

Proof. From the fact that

we can easily see that glyph[epsilon1] ↦→ Q n ( glyph[epsilon1] ) /glyph[epsilon1] is non-increasing. Similarly, we can show that glyph[epsilon1] ↦→ Q n ( glyph[epsilon1] ) /glyph[epsilon1] is non-increasing. Note that

Since glyph[epsilon1] ↦→ glyph[epsilon1] 2 r / 16 κ is increasing, goes 0 as glyph[epsilon1] → 0 + , and goes ∞ as glyph[epsilon1] →∞ , we can conclude that glyph[epsilon1] n < ∞ .

Similarly, we have ˜ glyph[epsilon1] n < ∞ . In fact, we can find the lower and the upper bound of ˜ glyph[epsilon1] n under Assumption 3.2.

Lemma A.5. We have

Proof. Since c s i -1 /s ≤ λ i ≤ C s i -1 /s , we have

We first consider the lower bound of ˜ glyph[epsilon1] n . We first observe that

Set

Note that

Also, we can observe that

hold. The first formula is trivial. To show the second formula, we observe that the function u ( t ) = ( s 1 -s ) s · t (1+ t ) 1 -s is an increasing function. Thus, for t ≥ 2 /s

Here, we apply an elementary inequality: a s b 1 -s ≤ sa +(1 -s ) b ∀ a, b > 0 . Since

putting t = ⌊( c s glyph[epsilon1] 2 ) s ⌋ gives

and so the second formula holds. Therefore,

holds and so we have

where the last inequality follows from glyph[epsilon1] ≤ 2 9 / (4 r +2 s ) κ 1 / (2 r + s ) c s/ (4 r +2 s ) s n -1 4 r +2 s . We can conclude that

by Lemma A.4. We now derive the upper bound of ˜ glyph[epsilon1] n . Note that

Since and

we have

Hence, we have

Set

which is equivalent to √ 1 n ( C s s + C s s 1 -s ) glyph[epsilon1] 2 -2 s = glyph[epsilon1] 1+2 r 16 √ 2 κ . Therefore, we attain the upper bound of ˜ glyph[epsilon1] n :

Without loss of generality, we assume n is sufficiently large such that

Then glyph[epsilon1] n ≤ ˜ glyph[epsilon1] n ≤ 1 . We now prove the following lemma. It is an extended version of Theorem 14.1 in [60].

Lemma A.6. We have

for any t ∈ [ glyph[epsilon1] n , 1] where c 1 is a constant independent of t and n .

Proof of Lemma A.6. We use a similar argument as in the proof of Theorem 14.1 in [60]. Define

where t ∈ (0 , 1] . Let

We first show that E ⊂ A . On the event E , there exists h ∈ H k such that ‖ h ‖ H k ≤ 1 and

If ‖ h ‖ L 2 ρ x ≤ t , we have ∣ ∣ ∣ ‖ h ‖ 2 L 2 ρ x ,n -‖ h ‖ 2 L 2 ρ x ∣ ∣ ∣ > 1 2 ‖ h ‖ 2 L 2 ρ x + 1 2 t 2 ≥ 1 2 t 2 . Otherwise, set ˜ h = t ‖ h ‖ L 2 ρ x h then ‖ ˜ h ‖ L 2 ρ x = t and ∣ ∣ ∣ ‖ ˜ h ‖ 2 L 2 ρ x ,n -‖ ˜ h ‖ 2 L 2 ρ x ∣ ∣ ∣ > t 2 ‖ h ‖ 2 L 2 ρ x · ( 1 2 ‖ h ‖ 2 L 2 ρ x + 1 2 t 2 ) ≥ 1 2 t 2 . Therefore, E ⊂ A . Since t 2 2 ≥ t 1+2 r 2 for t ∈ (0 , 1] , we have E ⊂ A ⊂ ˜ A . To find an upper bound of E Z n ( t ) , we use the symmetrization argument as follows:

where w 1 , · · · , w n are i.i.d. Rademacher variables. By Lemma A.15,

For t ≥ glyph[epsilon1] n , we have 4 κ · Q n ( t ) ≤ t 1+2 r 4 since

Since

where c 1 = 1 96 κ 2 . Putting u = t gives

Let us return to our problem. Consider the event

By Lemma A.6, we have P ( E c ) ≥ 1 -exp( -c 1 n ˜ glyph[epsilon1] 4 r n ) where c 1 is a constant that does not depend on n . Define

By the definition, we can easily obtain the upper bound of T as T < 1 + 1 ηE ˜ glyph[epsilon1] 2 n . Since R ( · ) is non-decreasing,

for some absolute constant c l where the second inequality follows from Lemma A.2. Note that

for all h such that ‖ h ‖ H k ≤ 1 on the event E c . Thus,

on the event E c . Set F = { g : X → R : ‖ g ‖ H k ≤ 1 , ‖ g ‖ L 2 ρ x ≤ 2˜ glyph[epsilon1] n } . Then the ranges of functions in F are contained in [ -κ, κ ] and F = -F . By Lemma A.17 we have

with probability at least 1 -exp( -c 1 n ˜ glyph[epsilon1] 1+2 r n ) ≥ 1 -exp( -c 1 n ˜ glyph[epsilon1] 4 r n ) . Hence,

holds with probability at least 1 -2 exp( -c 1 n ˜ glyph[epsilon1] 4 r n ) . Here, the second inequality follows from (13). Therefore,

since R ( 1 √ ηET ) ≤ κ . From the fact that exp( -c 1 n ˜ glyph[epsilon1] 4 r n ) ˜ glyph[epsilon1] 2+4 r n glyph[lessorsimilar] n 2 r +1 2 r + s exp( -c ′ 1 n s 2 r + s ) glyph[lessorsimilar] 1 , we have

A.2.6 Conclusion

Note that

Therefore, we bound the expected risk as

A.3 Corollary of Theorem 3.4

As mentioned in Section 3.3, one can remove B r in the upper bound in Theorem 3.4 by using more public inputs. The precise statement is as follows:

Corollary A.7. Under Assumption 3.1, 3.2, and 3.3, with n 0 ≥ B 1+ glyph[epsilon1] n 1 2 r + s (log( Bn )) 3 public inputs independently generated from ˜ ρ x satisfying (2) DCL-KR gives the performance guarantee

for all j = 1 , · · · , m where glyph[epsilon1] > 0 is a fixed constant, η ∈ (0 , 1 /κ 2 ) is a fixed learning rate, T is an adequate stopping rule, and the prefactor C does not depend on B , m , and n .

Proof. In the proof of Theorem 3.4, there are two terms in the upper bound affected by B . One is

in the norm bound of the first term in (7). The other is

in the norm bound of the third and fourth terms in (7). For the first part, n 0 ≥ B 1+ glyph[epsilon1] n 1 2 r + s (log( Bn )) 3 implies

For the latter part, set λ = 128( κ 2 +1) 2 n -1 2 r + s /B . Then

and hence

Since it eliminates B r in the upper bound, we are done.

A.4 Useful Lemmas

Recall Cordes' inequality [16].

Lemma A.8 (Cordes' Inequality) . Let A,B be two bounded positive linear operators on a seperable Hilbert space. Then for any s ∈ [0 , 1]

holds.

We also recall a property of projection operators.

Lemma A.9 ([53]) . Let Z be a bounded linear operator and P be a projection operator such that ran P = ran Z glyph[latticetop] . Then for any bounded operator X and λ > 0 we have

There are some useful lemmas for PAC bounds.

Lemma A.10 ([17]) . Let X = { x 1 , · · · , x n } be a dataset where data points are independently generated from ν . Then

holds with confidence at least 1 -δ where δ ∈ (0 , 1) .

Lemma A.11 ([38]) . Let X = { x 1 , · · · , x n } be a dataset where data points are independently generated from ν . Then

holds with confidence at least 1 -δ where δ ∈ (0 , 1) .

Lemma A.12 ([48]) . Let X = { x 1 , · · · , x n } be a dataset where data points are independently generated from ν . For λ ∈ (0 , 1] such that N ν ( λ ) ≥ 1 ,

holds with confidence at least 1 -δ where

To prove Lemma A.2 in Appendix A.2.5, we introduce a concentration inequality for Lipschitz functions.

Lemma A.13 ([60]) . Let X 1 , · · · , X n be independent random variables whose supports are contained in [ a, b ] and f : R n → R be convex and L -Lipschitz with respect to the Euclidean norm. Then we have

where X = [ X 1 , · · · , X n ] and t > 0 .

Precisely, we use the following fact in Appendix A.2.5

Remark A.14 ([60]) . Let A ⊂ R n be a bounded set and

where x = [ x 1 , · · · , x n ] ∈ [ -1 , 1] n and a = [ a 1 , · · · , a n ] . Since

f is a sup a ∈ A ‖ a ‖ R n -Lipshitz function where ‖ · ‖ R n is the Euclidean norm on R n . We can observe that f is convex since f is a supremum of convex functions defined on a convex compact set.

To prove Lemma A.6 in Appendix A.2.5, we recall the Ledoux-Talagrand contraction inequality [29, 56] and Talagrand's inequality [4, 56].

Lemma A.15 (Ledoux-Talagrand Contraction Inequality) . If φ : R → R is a L -Lipshitz function, then

Lemma A.16 (Talagrand's Inequality) . Let X 1 , · · · , X n be independent X -valued random variables. Let F be a countably family of measurable real-valued functions on X such that ‖ f ‖ ∞ ≤ U < ∞ and E f ( X i ) = 0 for all f ∈ F . Let

Then

for all t ≥ 0 .

Lastly, we recall the following well-known property used in Appendix A.2.5.

Lemma A.17 ([2]) . Let F be a class of functions with ranges in [ a, b ] and w 1 , · · · , w n be i.i.d. Rademacher variables. Then

holds with probability at least 1 -δ . Also,

holds with probability at least 1 -δ .

B Details on DCL-NN Algorithm

As we mentioned before, DCL-NN considers the same problem as in Section 3 but local models are heterogeneous neural networks. That is, there are m parties and D i = { ( x j i , y j i ) : j = 1 , · · · , n i } is the private dataset of the i th party ( i = 1 , · · · , m ) where D = ⋃ m i =1 D i are i.i.d. whose distribution is ρ x ,y . One remark is that the local data distributions of parties are not the same in general. To communicate training information, we introduce an unlabeled public input dataset Z = { z 1 , · · · , z n 0 } ⊂ X . The goal of parties is to find a minimizer of the population risk E defined in Section 3.

To extend DCL-KR to heterogeneous neural network settings, it is necessary to ensure that the assumptions of DCL-KR are satisfied as much as possible. Specifically, one important assumption in DCL-KR is the equality of kernels across local models. Indeed, public data predictions can vary in conflicting directions after the local training procedure, even when using the same local datasets, if the kernels differ.

For further explanation of this claim, we consider the simple case of E = 1 in DCL-KR where E is the number of local iterations. After the consensus prediction u is distributed to local parties, the server then receives the updated local prediction on Z :

from the i th local party. (The notation is consistent with Appendix A) Suppose two parties have exactly the same dataset. If the same kernel is used in these two parties, the updated local predictions

Algorithm 2 DCL-NN Algorithm

1: Hyperparameters: D i : local dataset of party i ( i = 1 , · · · , m ), Z = { z 1 , · · · , z n 0 } : public inputs, E : the number of local iterations at each communication round, T : total communication rounds, T k : epochs of kernel distillation

2: Pretrain: Pretrain local AI models f i ( · ) = w glyph[latticetop] i g i ( · ) + b i ( i = 1 , · · · , m ).

3: # Feature Kernel Distillation Procedure

4: For party i ( i = 1 , · · · , m ), compute the feature kernel values on public inputs { k f i ( z j 1 , z j 2 ) : 1 ≤ j 1 , j 2 ≤ n 0 } via (3) and upload them to the server.

5: The server aggregates the local feature kernel values to the target kernel values { k ( z j 1 , z j 2 ) : 1 ≤ j 1 , j 2 ≤ n 0 } with (4) and distributes them to all parties.

6: for party i = 1 , · · · , m do

7: for t k = 0 , · · · , T k -1 do

8: for mini-batch Z 0 ⊂ Z do

9: Update parameters of its model f i by maximizing ̂ CKA ( k, k f i ) on Z 0 defined as (16) via gradient descent where k is fixed.

10: end for

11: end for

12: end for

13: # Collaborative Learning Procedure

14: Initialize the consensus prediction y p, 0 = 0 .

15: for t = 0 , · · · , T -1 do

16: for party i = 1 , · · · , m do

17: Update w i and b i by maximizing MSE (Mean Squared Error) on the public inputs Z with consensus prediction y p,t via gradient descent with sufficiently many iterations.

18: for e = 1 , · · · , E do

19: Update w i and b i by maximizing MSE on D i via gradient descent.

20: end for

21: Upload the local prediction y i p,t +1 on Z to the server.

22: end for

23: The server aggregates the local predictions to compute the consensus prediction y p,t +1 = ∑ m i =1 n i n y i p,t +1 and distributes y p,t +1 to all parties.

24: end for

will be identical. However, if the kernels are different, this will not be the case. For kernels like the Gaussian kernel, which have high correlation between close inputs, the updated local predictions will be strongly influenced by data points close to each input. On the other hand, for kernels like the linear kernel, which have high correlation between distant inputs, the updated local prediction on Z will be influenced more by data points farther from each input. We can observe this fact from the above formula (14). This observation implies that aggregating local learning information becomes very challenging when the kernels differ. In short, using the same kernel ensures that the shift mechanisms of predictions on Z at the edges are identical, making it possible for the aggregation through simple weighted averaging to work well. This is a key element of the strong theoretical results of DCL-KR and explains why kernel matching between neural networks is necessary in DCL-NN.

Let f i be a local model of the i th party such that f i ( · ) = w glyph[latticetop] i g i ( · ) + b i , g i : X → R c i , w i ∈ R c i , c i ∈ N , and b i ∈ R for i = 1 , · · · , m . Since most modern neural network architectures have a linear layer as the last layer, this setting is general enough. As (3), we set the feature kernel of f i ( i = 1 , · · · , m ) to be

To bring the setting to the DCL-KR scheme, DCL-NN matches k f 1 , · · · , k f m via kernel distillation procedure. Obviously, the target kernel in this procedure is a key factor in enhancing performance.

Theoretically, using the ensemble kernel

can be a good way to construct a good kernel derived from local feature kernels. The reason is that this ensemble kernel is identical to the kernel induced by the (scaled) concatenation of the local feature maps, i.e.,

In other words, the ensemble kernel has greater expressive power than individual feature kernels, and with a sufficient amount of data, it leads to better performance. We empirically verify that the performance of this ensemble kernel surpasses that of individual feature kernels in Figure 4.

DCL-NN sets this ensemble kernel k as the target kernel and local parties match their local feature kernels k f 1 , · · · , k f m with the kernel k using the public dataset Z . For this purpose, we introduce Centered Kernel Alignment (CKA) [8] as a kernel similarity measure. The CKA between two kernels k 1 and k 2 on the public input distribution ˜ ρ x is given by

where HSIC ( · , · ) is a Hilbert-Schmidt Independence Criterion (HSIC) defined as

and the centered kernel k c i is given by

x 1 , x 2 ∈ X ( i = 1 , 2 ). However, since we have a finite number of samples, we employ the empirical CKA. The empirical CKA between two kernels k 1 and k 2 on inputs { c 1 , · · · , c p } is given by

where

are Gram matrices and ̂ HSIC is an estimator of HSIC defined as

where H := I p -1 p 11 glyph[latticetop] is the centering matrix, I p is a p × p identity matrix, and 1 = [1 , 1 , · · · , 1] glyph[latticetop] is a p -dimensional one vector. During the kernel distillation procedure, the i th local party maximizes ̂ CKA ( k f i , k ) on Z where k is a fixed target kernel given by (4). In practice, we use batching to perform the kernel distillation to reduce computational costs.

Due to the definition of the empirical CKA, it is necessary to calculate the Gram matrix of k over Z . To this end, the i th local party calculates the Gram matrix of k f i over Z and uploads it to the server for i = 1 , · · · , m . Then the server computes the Gram matrix of k by weighted averaging the Gram matrices of local feature kernels k f 1 , · · · , k f m . Since this process only requires communication of feature kernel values, DCL-NN still preserves the privacy of local model information.

While (empirical) CKA is a good metric for kernel matching, it is invariant to scaling, and therefore the local feature kernels resulting from the kernel distillation may have different scales. This affects the degree to which each local training influences during the DCL-KR-like follow-up procedure. To

illustrate this point, consider the following example: Let the feature kernels of two local models f 1 and f 2 be as follows:

After distilling on the public data with consensus predictions, these two models become like

i.e., two models are the same. Nevertheless, a gradient descent update on a data point ( x 0 , y 0 ) with a learning rate η is

for f 2 . Thus, after the gradient descent step we have

for f 1 and and

Hence the scale α affects the collaborative learning procedure. To address this issue, we compute the scale α using the estimator ̂ HSIC. Specifically, at the beginning of the collaborative learning phase, we compute α i = ̂ HSIC ( K i , K i ) where K i is the Gram matrix with respect to the feature kernel of the i th party on Z ( i = 1 , · · · , m ). Then we set the learning rate for the i th party as

where η 0 is a base learning rate. In practice, computing HSIC over all public inputs is costly and unnecessary. Using only a small subset of public inputs is sufficient.

We present DCL-NN in Algorithm 2. Here are some remarks.

The feature kernel distillation procedure requires only one round of two-way communication between the server and the parties.

The collaborative learning procedure follows the same process as in DCL-KR with g 1 , · · · , g m fixed. Note that kernel gradient descent reduces to standard gradient descent since the kernels have finite rank. In this process, if possible, optimization on the public dataset can be performed using the closed-form solution of kernel linear regression instead of gradient descent.

In this work, we apply FedMD for the pretraining of DCL-NN in the experiment. However, DCL-NN is a general algorithm that can use any algorithm for pretraining to obtain good local feature kernels. For example, kernel learning techniques [63] may be applied for pretraining.

Our algorithm can be naturally extended to regression problems with multi-dimensional outputs.

C Details and Further Discussion on Experiments

C.1 Dataset Description

For all datasets, we follow Algorithm 3 to construct non-i.i.d. settings. Note that this procedure is similar to the non-i.i.d. data generation procedure in classification tasks [44, 69].

Algorithm 3 Data Generating Procedure

1: Inputs: the number of parties m , the total number of private data n , the number of public inputs n 0 , the partition of input space A = {A 1 , · · · , A c }

2: Generate the whole private dataset size of n from ρ x ,y and the public inputs size of n 0 from ˜ ρ x .

3: Sample a base private data ratio ( α 1 , · · · , α m ) from Dir ([10 , 10 , · · · , 10]) .

glyph[negationslash]

4: while ∑ m j =1 C ij = 0 for all i = 1 , · · · , c do

5: for party j = 1 , · · · , m do

6: Sample two elements uniformly from the partition A .

7: end for

8: Set C ij = 1 if A i is chosen at the j th party and C ij = 0 otherwise.

9: end while

10: for i = 1 , · · · , c do

11: Put the private data points where inputs are in A i into the datasets of parties with ratio

12: end for

C.1.1 Toy-1D

Let X = [0 , 1] ⊂ R and ρ x be the uniform distribution on X . The space

is the reproducing kernel Hilbert space associated to the kernel k ( x, y ) = min( x, y ) where AC [0 , 1] is the collection of all absolutely continuous functions on [0 , 1] [33, 60]. As mentioned in [33], the covariance operator T k,ρ x has eigenpairs { ( λ i , e i ) } i ∈ N where

Thus, the eigenvalue decay rate s is 1 2 . Set a target function

we have f ∗ 0 = T 1 / 2 k,ρ x g ∗ 0 such that

Then r = 1 . We generate data points from ρ x · ρ y | x such that ρ x is the uniform distribution on X as above and

We divide X into a partition A := {[ i 8 , i +1 8 ] : i = 0 , · · · , 7 } . We follow Algorithm 3 with m ∈ [10 , 20 , · · · , 100] , n = 50 m , and n 0 ≈ n 1 2 r + s (log 10 n ) 3 . With n 0 ≈ n 1 2 r + s (log 10 n ) 3 , we also achieve the same bound (with different prefactors) in Theorem 3.4. In the main experiments, we set ρ x = ˜ ρ x .

From the fact that

C.1.2 Toy-3D

Let X = [0 , 1] 3 ⊂ R 3 and ρ x be the uniform distribution on X . Define a kernel k ( x , y ) = (1 - ‖ x -y ‖ R 3 ) 2 + . The reproducing kernel Hilbert space H k associated to the kernel k is normequivalent to Sobolev space H 2 ( X ) [55] and the eigenvalue decay of T k,ρ x is s = 3 4 [68]. Note that k ′ ( x , y ) = (1 -‖ x -y ‖ R 3 ) 6 + (35 ‖ x -y ‖ 2 R 3 +18 ‖ x -y ‖ R 3 +3) is a kernel and its reproducing kernel Hilbert space is norm-equivalent to Sobolev space H 4 ( X ) . Using the interpolation relation between H 2 ( X ) and H 4 ( X ) gives that

is in H k and the regularity r is 1 . Similarly as before, we generate data points from ρ x · ρ y | x such that ρ x is the uniform distribution on X and

We divide X into a partition A := { [ k 1 2 , k 1 +1 2 ] × [ k 2 2 , k 2 +1 2 ] × [ k 3 2 , k 3 +1 2 ] ⊂ R 3 : ( k 1 , k 2 , k 3 ) ∈ { 0 , 1 } 3 } . Again, we follow Algorithm 3 with m ∈ [10 , 20 , · · · , 100] , n = 50 m , n 0 ≈ n 1 2 r + s (log 10 n ) 3 , and ρ x = ˜ ρ x for kernel machine-based algorithms. For neural network-based algorithms, m = 50 , n 0 = 2500 , and the other configurations are the same.

C.1.3 Real World Datasets

Energy Energy dataset is a real-world tabular dataset from the UCI database [12]. It has 28 input features including measurement time, temperature and humidity of each room, outside temperature, and wind speed. The output is the appliances energy use. We normalize all features, including the output, using MinMaxScaler. There are 12,000 training data points distributed across the parties. We use 6,000 samples as public inputs and 1,000 samples for testing. To construct a non-i.i.d. setting, we set a partition A consisting of 8 subsets, each formed by splitting three normalized variables (measurement time, visibility, and dewpoint) at their midpoints. We apply Algorithm 3 with m = 50 .

RotatedMNIST RotatedMNIST is a dataset derived from MNIST [11]. The task is to predict the rotated angle of a given rotated MNIST image. Each image is 1 × 28 × 28 image, and we normalize all images by their mean and variance. To generate RotatedMNIST, we rotate MNIST images by a random angle between -π 2 and π 2 and use the angle as the label. To construct a large-scale dataset, each image is rotated at multiple angles to generate multiple data instances. For training data, we additionally inject Gaussian noise with a standard deviation of 0 . 2 to each label. We use 200,000 images as the entire training data, 50,000 images as public inputs, and 50,000 images as test data. The whole training input distribution, public input distribution, and test input distribution are uniformly distributed across digits 0 to 9. For example, there are 20,000 rotated images of the digit '4' in the training set. For a non-i.i.d. setting, we partition the data into A where |A| = 10 , based on the digit (0 ∼ 9) and follow Algorithm 3 with m = 50 .

UTKFace UTKFace dataset [71] is an image dataset used for age estimation. Since the image sizes vary, we resize all images to 3 × 128 × 128 and normalize them by their mean and variance for each channel. The labels are normalized to the range [0 , 1] using MinMaxScaler. For training data, we inject Gaussian noise with a standard deviation of 0 . 5 to each label before normalization. We use 12,544 samples for training and 1,039 samples for testing. We have 6,234 public inputs. These three datasets have the same distribution for metadata (gender and race). Based on this metadata, we construct a partition A with |A| = 10 and distribute the training data among 50 parties according to Algorithm 3.

IMDB-WIKI IMDB-WIKI dataset [52] is also an image dataset for age estimation. In experiments, we utilize a clean version [42]. We further resize all images to 3 × 64 × 64 and normalize them as UTKFace. The labels are also normalized to the range [0 , 1] using MinMaxScaler. We use 147,107 images as the entire training data, 36,780 images as public inputs, and 56,087 images as test data. In this dataset, we utilize the triplet (head_roll, head_yaw, head_pitch) as metadata. Both training inputs and public inputs have the same distribution for metadata. We construct a decentralized setting among 50 parties using a partition A based on the metadata, following Algorithm 3. The partitioning is performed by dividing the dataset into regions based on the median values of each metadata variable.

C.2 Implementation

The experiments are implemented in PyTorch. We simulate a decentralized setting on a single deep learning workstation (Intel(R) Xeon(R) Gold 6430 with one NVIDIA GeForce RTX 4090 GPU and 189GB RAM). All DCL-KR implementations take less than 10 minutes per simulation. The non-parallel implementation of DCL-NN for large-scale datasets is completed within 48 hours. With parallel computing, the execution time for the same setup is expected to be reduced to within 2 hours.

C.3 Experimental Setup and Results on Kernel Machine-based Algorithms

C.3.1 Experimental Setup Details and Main Results

In this experiment, we evaluate DCL-KR by comparing its performance against two central models and three baselines. Specifically, we employ DC-NY [66], DKRR-NY-CM [67], and IED [48] as baselines. We also compare DCL-KR with central Kernel Ridge Regression (centralKRR) and central Kernel Regression with Gradient Descent (centralKRGD). As mentioned earlier, we evaluate the performance of these algorithms on Toy-1D and Toy-3D datasets.

There are several hyperparameters for kernel machine-based algorithms: the ridge regularization hyperparameter λ in the ridge regressions and the number of iterations (or communication rounds) in the gradient descent-based regressions. We set λ = C · n -1 2 r + s and T = int ( D · n 1 2 r + s ) which are the optimal choices from theory. We determine the best values for C and D by grid search in our experiments (see Table 4 for the selected hyperparameter values). For centralKRGD and DCL-KR, we set the learning rate η = 0 . 5 which satisfies η ∈ (0 , 1 /κ 2 ) in Theorem 3.4. The number of local iterations E for DCL-KR is set to 5. For DKRR-NY-CM, we modify its Newton-Raphson iteration as shown below due to an instability issue, and we set the learning rate η = 0 . 01 :

The communication round T for DKRR-NY-CM is set to 10 .

To measure performance, we sample data from the distribution presented in Appendix C.1 for each simulation and compute E ‖ ι ρ x ( f i,T -f ∗ 0 ) ‖ L 2 ρ x by averaging the Root Mean Squared Errors (RMSEs) on the test dataset. We conduct 500 simulations for each setting, and the results are summarized in Figure 1.

As shown in Table 1, DC-NY and DKRR-NY-CM have theoretical performance guarantees under the statistically homogeneous condition for a limited number of parties. However, they do not exhibit sufficiently good performance in massively distributed statistically heterogeneous settings. The performance degradation of DKRR-NY-CM appears to be linked to its second-order optimization scheme, which leads to ineffective batching in statistically heterogeneous settings. The performance degradation of DC-NY is expected, given the inherent limitations of divide-and-conquer algorithms.

In contrast, IED demonstrates relatively better performance, despite the strong assumptions underlying its theory. Nevertheless, it still exhibits performance degradation compared to centralized models. DCL-KR, on the other hand, achieves performance comparable to centralized models, validating both the theoretical results and its practical feasibility.

C.3.2 Effect of n 0

As public inputs directly affect the training information sharing, we anticipate that the performance of DCL-KR will vary depending on the number of public inputs n 0 . To examine this effect, we

20

40

60

80

100

The number of parties

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Average of RMSEs

1e

1

=0.5

=1

=4,

=0.5

(a) DCL-KR

20

40

60

80

100

The number of parties

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Average of RMSEs

1e

1

=0.5

=1

=4,

=0.5

(b) IED

2.5

3.0

3.5

4.0

4.5

log

m

2.8

2.6

2.4

2.2

2.0

1.8

Logarithm of Average of RMSEs

=0.5

=1

=4,

=0.5

(c) DCL-KR (log-scale)

2.5

3.0

3.5

4.0

4.5

log

m

2.6

2.4

2.2

2.0

1.8

1.6

Logarithm of Average of RMSEs

=0.5

=1

=4,

=0.5

(d) IED (log-scale)

glyph[negationslash]

measure the performance of DCL-KR on Toy-3D for n 0 ≈ α · n 1 2 r + s (log 10 n ) 3 with various α ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 1 , 2 } . Additionally, we conduct the same experiment with IED, the most competitive baseline, for comparison.

The results are summarized in Figure 2. Consistent with theoretical results, the value of α does not affect the convergence rate of IED and DCL-KR. However, IED displays significant performance variations across different α values. In contrast, DCL-KR achieves its maximum performance when α is not too small (i.e., α ≥ 0 . 3 in Figure 2). This implies that DCL-KR requires fewer public inputs to achieve its maximal performance compared to IED, as predicted by theoretical results.

C.3.3 Effect of ˜ ρ x

glyph[negationslash]

So far, we consider the settings with ρ x = ˜ ρ x . However, Theorem 3.4 covers the general case where ˜ ρ x = ρ x . To verify this, we define the public input distribution in Toy-3D with the following density function (parametrized by β ):

The Radon-Nikodym derivative dρ x d ˜ ρ x satisfies 0 ≤ dρ x d ˜ ρ x ≤ ( 1 β ) 3 . We conduct additional experiments to verify Theorem 3.4 and Corollary A.7, considering the case where β = 0 . 5 (with α = 1 ) and the case where β = 0 . 5 but α = 4 to compensate. The results are provided in Figure 5. In the log-scale plot, the slope represents the convergence rate.

First, we observe that the convergence rate of DCL-KR remains unchanged when β is changed from 1 (i.e., ρ x = ˜ ρ x ) to 0.5. This observation is consistent with Theorem 3.4. Additionally, regarding Corollary A.7, we confirm that DCL-KR achieves performance almost identical to the case of ρ x = ˜ ρ x by increasing n 0 . In contrast, the convergence rate of IED worsens when ˜ ρ x changes, even when n 0 is increased. These experimental results highlight the advantages of DCL-KR in statistically heterogeneous environments.

C.4 Experimental Setup and Results on Neural Network-based Algorithms

C.4.1 Experimental Setup Details

In the experiments on neural network-based collaborative learning algorithms, we evaluate three baselines (FedMD, FedHeNN, KT-pFL) and our algorithm DCL-NN. Note that while KT-pFL is a personalized collaborative learning algorithm, it also performs well in non-personalized settings, so we include it for comparison. Additionally, we evaluate centralized models as ideal cases and standalone models as worst cases. Centralized models are trained using all local data.

We use two tabular datasets, Toy-3D and Energy, and three image datasets, RotatedMNIST, UTKFace, and IMDB-WIKI. The number of parties is set to 50 for all settings. For the tabular datasets, we employ four different fully connected neural networks (FNNs) with a ratio of 30%, 30%, 20%, and 20%. Specifically, There are fifteen 4-layer FNNs with 32 hidden units, fifteen 4-layer FNNs with 64 hidden units, ten 5-layer FNNs with 32 hidden units, and ten 3-layer FNNs with 64 hidden units. Similarly, for the image datasets, we use four different convolutional neural networks (CNNs) with the same ratio of 30%, 30%, 20%, and 20%. For the large-scale image datasets (RotatedMNIST and IMDB-WIKI), we construct 50 local parties using fifteen ResNet-18, fifteen ResNet-34, ten ResNet50 [20], and ten MobileNetv2 [54]. For UTKFace, which is an image dataset with limited data, we utilize four simpler CNN architectures due to the ineffectiveness of knowledge distillation with large underperforming models. The first and third CNNs share a similar architecture, featuring two convolutional layers with batch normalization [22], two max pooling layers, and two fully-connected layers at the end. They differ only in the number of channels. In contrast, the second and fourth CNNs are more complex, featuring four convolutional layers with batch normalization, two max pooling layers, and two fully-connected layers at the end. They also differ in the number of channels. We use the ReLU activation function throughout.

All optimizers used are Adam [26]. 2 One remark is that baseline algorithms only utilize a subset of public inputs through random sampling in each communication round, as performance tends to deteriorate due to overfitting when all public inputs are used in every round. Hyperparameters are tuned via grid search.

Standalone models are trained with cross-validation and early stopping to prevent overfitting. To evaluate centralized models, we first compute the averaged test Root Mean Squared Error (RMSE) from at least 10 simulations for each neural network architecture and then calculate the weighted average of the performances of all architectures according to their ratio. For standalone models, we use the average of the test RMSEs of local models with the hyperparameters listed in Table 5. In the table, RotatedMNIST is abbreviated as MNIST, and IMDB-WIKI is abbreviated as IMDB.

For FedHeNN, we set the number of local epochs to 30 in all experiments. For KT-pFL, we set the number of local epochs to 10, the distillation coefficient to 0.5, and the learning rate of knowledge coefficient to 1e-3. Lastly, for DCL-NN in the main experiment, we set the learning rate for kernel matching to 1e-4 and the number of communication rounds in the collaborative learning phase to 50. We use the closed-form solution to train public data and full-batch gradient descent to train local data in the collaborative learning phase of DCL-NN and utilize FedMD for pretraining in DCL-NN. In the pretraining phase, the hyperparameters are the same as in the FedMD setting, except that the number of communication rounds is 100 for tabular data and 50 for image data. The remaining hyperparameters are presented in Table 6, 7, 8, and 9. For all distillation-based collaborative learning algorithms, we simulate each setting at least 5 times with different initializations.

Communication Efficiency Compared with FedMD and KT-pFL, DCL-NN incurs higher communication costs of O ( n 2 0 ) due to the transmission of the Gram matrix. FedHeNN also utilizes kernel matching but performs it in batches for each communication round. Thus, in scenarios requiring many communication rounds, DCL-NN is more efficient than FedHeNN. However, pretraining also demands more communication cost. We leave the study of communication-efficient methods in DCL-NN for future work.

C.4.2 Effect of Public Inputs

In practice, the public inputs can be sampled from a distribution whose support is disjoint from that of the whole local input distribution. In this case, the assumption of DCL-KR does not hold; however, DCL-NN can still be applicable. To evaluate the performance of DCL-NN under these conditions, we compare the performance of DCL-NN and FedMD when the distribution of public inputs differs, as in Table 10. We use CIFAR10 [28] for public inputs on UTKFace. As shown in Table 10, DCL-NN

does not yield better results in this case. Note that the kernel performance of local feature kernels is improved compared to FedMD, leading us to conclude that the performance degradation of DCL-NN with CIFAR10 is due to the violation of the DCL-KR assumption rather than the ineffectiveness of the kernel distillation procedure.

D Limitations and Future Works

Privacy Benefits of Distillation-based Collaborative Learning Due to its black-box nature, distillation-based information interaction is expected to offer privacy preservation benefits compared to parameter exchange (mainly done in FL) as mentioned in [19]. To the best of our knowledge, there is no rigorous study that discusses the privacy preservation advantages of distillation-based collaborative learning. We hope to see further discussion on this as well.

Public Input Distribution Theorem 3.4 covers the case where the public input distribution ˜ ρ x differs from that of local data inputs ρ x , but at least the support of ˜ ρ x must include the support of ρ x . Therefore, we experimentally observe a performance drop in DCL-NN, a practical extension of DCL-KR, when ˜ ρ x and ρ x have different supports. Enhancing the robustness of DCL-NN in this scenario is considered a promising direction for future work. Our theory does not cover situations where collecting public inputs is difficult. In such cases, a seperate generative model is usually trained to generate public inputs [72]. We leave the theoretical discussion that includes these cases for future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately describe the motivations, theoretical/experimental contributions, and scope of our work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical results and its assumptions are clearly described in Section 3 and Appendix A. Idea of the proof is briefly described in Section 3 and the full proof is provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The proposed algorithms are clearly stated in Algorithm 1 and Algorithm 2. Experimental details such as experimental setting, performance measure, data preprocessing, and hyperparameter setting are also explained in Section 5 and Appendix C. The code is also provided via the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided via the supplementary material. Regarding datasets, the paper contains data source and preprocessing descriptions. The paper also provides sufficient experimental details to reproduce the experimental results. See Section 5 and Appendix C.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details such as data split, hyperparameters, optimizers, and model architectures are provided in Section 5 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results contains 1-sigma error bars and the explanations about the error bars are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Descriptions of computer resources are provided in Appendix C.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors verify that the research is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All data and models used in the paper are credited through citations according to the license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code is provided via the supplementary material with a well-written documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an analysis of a distillation-based collaborative learning algorithm called FedMD from a nonparametric perspective. The method is to adopt an operator theoretic approach to obtain an upper rate for the expected generalization error of local models. Then, the authors propose DCL-KR to achieve a privacy-preserving nearly minimax optimal collaborative learning algorithm with kernel regression in massively distributed statistically heterogeneous environments.\n\nSTRENGTHS:\n+ The idea seems to be interesting.\n+ The first work tries to prove the (nearly) minimax optimality of a privacy-preserving collaborative learning algorithm.\n\nWEAKNESSES:\n- The paper's organization and structure are not clear.  \n- The writing looks like combining different parts of different papers without a clear clue.\n- The paper suddenly gives many assumptions in Section 3 for analysis but does not give any intuition or insight into them.\n- Some important experimental results are in the appendix. I would suggest that the authors extract insight into the experimental findings in the main body.\n- The language used in the paper is not concise.\n\nQUESTIONS:\nThe related work only gives a rough summary of the previous works. When the authors focus on the theoretical analysis, could the authors compare their theoretical bounds to other works?\n\nThere are many important experiments in the appendix, such as C.3.1. Can the authors summarize empirical findings with a better paper structure? \n\nIn Section 3.3, the paper starts with \"To derive theoretical results\" and lists a lot of assumptions without any intuitive understanding behind them. Could the authors give an overview and intuition of theoretical results?\n\nMinor: I feel very confused when reading the paper as it introduces many conclusions/results without too much explanation. I suggest the authors elaborate on the intuitions before going into the details.\n\nLIMITATIONS:\nThe paper presents a discussion of limitations, which is not convincing. For example, \"there is no rigorous study that discusses the privacy preservation advantages of distillation-based collaborative learning\". Actually, the standard technique for privacy is not distillation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a nonparametric version of FedMD, a distillation-based collaborative learning methodology. They also propose a neural network variant of the nonparametric approach as an extension for heterogeneous local neural network models. They provide both theoretical results on the nonparametric approach and experimental results demonstrating efficacy of the neural network extension. The theoretical results prove the nearly minimax optimality of this nonparametric collaborative learning algorithm in massively distributed statistically heterogeneous environments.\n\nSTRENGTHS:\n(1) The authors prove the (nearly) minimax optimality of a privacy-preserving collaborative learning algorithm using kernel regression in massively distributed, statistically heterogeneous environments. (2) Theorem 3.4 addresses a more general setting than prior works [48, 58]. Su et al. [58] only cover r = \\frac{1}{2}​ of Assumption 3.3. Park et al. [48] do not consider Assumption 3.2, which provides a finer result. Furthermore, compared to [48], the required size of public inputs is reduced and the statistical homogeneity condition is dropped. (3) The rate n^{-\\frac{r}{2r+s}} is the minimax lower rate under Assumptions 3.1, 3.2, and 3.3, making DCL-KR nearly optimal in a minimax sense. (4) On the difference between public and local distributions: Theorem 3.4 allows the public input distribution \\tilde{\\rho}_x to differ from the local input distribution \\rho_x​. (5) Finally, The difference between \\rho_x​ and \\tilde{\\rho}_x​ impacts the expected risk upper bound as a multiplication of B_r​, which can be removed by increasing public inputs.\n\nWEAKNESSES:\n(1) The need to perform learning rate scaling to ensure the impact of local iterations is consistent across models introduces additional complexity. The exact method for scaling may vary and might not be trivial to determine or implement. (2) The computation of Gram matrices for CKA involves O(p^2) operations, where p is the number of public data points. This quadratic complexity can become prohibitive as the size of the public dataset increases. (3) The method assumes that the ensemble kernel k = \\sum_{i=1}^{m} \\frac{n_i}{n} k_{f_i} will perform better than individual feature kernels. This assumption might not hold in cases where local models are highly heterogeneous, as the averaged kernel could lose important characteristics of individual models, leading to suboptimal performance. The authors empirically verify the superiority of the ensemble in the appendix.\n\nQUESTIONS:\n(1) How do you handle the computational burden associated with the calculation of Gram matrices for large datasets? (2) How do you handle cases where these representations before the last layers of the neural networks are suitable for kernel matching?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper investigates the theoretical underpinnings and practical implementation of distillation-based collaborative learning (DCL) from a kernel regression perspective. The authors propose DCL-KR, a nonparametric version of the FedMD algorithm, which achieves nearly minimax optimal convergence rates in massively distributed and statistically heterogeneous environments. Building on these theoretical insights, the authors introduce DCL-NN, a practical DCL algorithm designed for neural networks, which incorporates feature kernel matching to align local models. Extensive experiments on various regression tasks demonstrate the superiority of DCL-KR and DCL-NN over existing methods.

STRENGTHS:
1. The paper provides a rigorous theoretical analysis of DCL-KR, proving its nearly minimax optimality in distributed and heterogeneous settings.

2. Experiments are comprehensive. Six datasets are included.

3. The improvement of DCL-NN is significant over the other baselines in some datasets.

WEAKNESSES:
1. The paper lacks a discussion on the privacy risk and communication efficiency of DCL-KR and DCL-NN.

2. The paper is based on distillation-based collaborative learning method that requires public data, which limits the applications on private domains.

QUESTIONS:
1. How does the choice of kernel affect the performance/theoretical guarantee of the proposed approach?

2. Is transferring predictions a common practice in collaborative learning for regression? If not, how does it compare with the other approaches in terms of privacy and efficiency?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper theoretically proves a nonparametric version of the most standard distillation based collaborative learning algorithm (named DCL-KR) is nearly minimax optimal in massively distributed statistically heterogeneous environments.
Extensive experiments demonstrate their theoretical results and show the practical feasibility of DCL-NN.

STRENGTHS:
1. The paper is well-written.
2. The authors conducted extensive experiments to verify the effectiveness of the proposed method.

WEAKNESSES:
Please see Limitations

QUESTIONS:
Please see Limitations

LIMITATIONS:
This is not my field of study and I can only give a Borderline Reject. I will adjust my score based on the scores of other more specialised judges.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
In this paper, the authors perform a study of distillation-based collaborative learning (DCL) in massively distributed statistically heterogeneous environments. In particular, the study focuses on analyzing DCL-KR, a non-parametric version of FedMD, and proves its near-minimax optimality. The authors proposed DCL-NN, a method that leverages kernel matching to implement DCL with heterogeneous neural networks. Experimental results support the theoretical analysis of DCL-KR and demonstrate the effectiveness of DCL-NN.

STRENGTHS:
The paper provides a theoretical analysis of DCL-KR and proposes a practical algorithm for using neural networks in massively distributed statistically heterogeneous environments. The authors perform experiments with both synthetic and real-world datasets to support the theoretical results and evaluate the performance of DCL-NN.

WEAKNESSES:
The paper does not perform any formal privacy analysis. As such, I believe using terms like "privacy-preserving" can be misleading in this context.

QUESTIONS:
Could the authors avoid mentioning privacy as no formal privacy study is conducted?

LIMITATIONS:
The paper unnecessarily emphasizes privacy while admitting that privacy is not formally defined (cf. question). It would be nice to provide more discussion on this aspect or to present some empirical evidence of the claimed benefit (e.g., performing privacy attacks such as membership inference)
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15