Title: HyperPrism: An Adaptive Non-linear Aggregation Framework for Distributed Machine Learning over Non-IID Data and Time-varying Communication Links
Venue: NeurIPS 2024
Accepted: yes
Keywords: Distributed Machine Learning

Time-varying Communication

Non-Linear Aggregation

HyperNetwork
Abstract: While Distributed Machine Learning (DML) has been widely used to achieve decent performance, it is still challenging to take full advantage of data and devices distributed at multiple vantage points to adapt and learn, especially it is non-trivial to address dynamic and divergence challenges based on the linear aggregation framework as follows: (1) heterogeneous learning data at different devices (i.e., non-IID data) resulting in model divergence and (2) in the case of time-varying communication links, the limited ability for devices to reconcile model divergence. In this paper, we contribute a non-linear class aggregation framework HyperPrism that leverages distributed mirror descent with averaging done in the mirror descent dual space and adapts the degree of Weighted Power Mean (WPM) used in each round. Moreover, HyperPrism could adaptively choose different mapping for different layers of the local model with a dedicated hypernetwork per device, achieving automatic optimization of DML in high divergence settings. We perform rigorous analysis and experimental evaluations to demonstrate the effectiveness of adaptive, mirror-mapping DML. In particular, we extend the generalizability of existing related works and position them as special cases within HyperPrism. Our experimental results show that HyperPrism can improve the convergence speed up to 98.63% and scale well to more devices compared with the state-of-the-art, all with little additional computation overhead compared to traditional linear aggregation.

HyperPrism: An Adaptive Non-linear Aggregation Framework for Distributed Machine Learning over non-IID Data and Time-varying Communication Links

Abstract

While Distributed Machine Learning (DML) has been widely used to achieve decent performance, it is still challenging to take full advantage of data and devices distributed at multiple vantage points to adapt and learn; this is because the current linear aggregation paradigm cannot solve inter-model divergence caused by (1) heterogeneous learning data at different devices ( i.e. , non-IID data) and (2) in the case of time-varying communication links, the limited ability for devices to reconcile model divergence. In this paper, we present a non-linear class aggregation framework HyperPrism that leverages Kolmogorov Means to conduct distributed mirror descent with the averaging occurring within the mirror descent dual space; HyperPrism selects the degree for a Weighted Power Mean (WPM), a subset of the Kolmogorov Means, each round. Moreover, HyperPrism can adaptively choose different mapping for different layers of the local model with a dedicated hypernetwork per device, achieving automatic optimization of DML in high divergence settings. We perform rigorous analysis and experimental evaluations to demonstrate the effectiveness of adaptive, mirror-mapping DML. In particular, we extend the generalizability of existing related works and position them as special cases within HyperPrism. For practitioners, the strength of HyperPrism is in making feasible the possibility of distributed asynchronous training with minimal communication. Our experimental results show HyperPrism can improve the convergence speed up to 98.63% and scale well to more devices compared with the state-of-the-art, all with little additional computation overhead compared to traditional linear aggregation.

1 Introduction

The proliferation of edge devices, such as mobile phones, wearable devices and unmanned aerial vehicles, has resulted in an exponential growth in diverse data types (e.g., images, sound and text). Addressing this surge necessitates advanced techniques capable of accurately, quickly, and practically processing vast amounts of data through efficient and scalable algorithms. Distributed Machine Learning (DML) has gained significant traction in recent years as a strategy to minimize data transfer

Shanghai, China

and computational costs by bringing computation closer to the data sources. It has become a natural solution for scaling up machine learning while preserving data privacy in various domains ( e.g. , large language model training [39, 33], autonomous driving [60, 37], military applications [56], web search [29, 30, 28] and recommendation [32, 27, 31]).

While existing DML attempts have made significant progress, two technical barriers remain in realistic scenarios. First , data heterogeneity is one of the most critical concerns. Specifically, in a DML system where mobile devices serve as computing nodes, the data generated is typically non-independent and identically distributed (non-IID), which means the data distribution may be unbalanced across different classes or categories. Consequently, the model may exhibit bias toward the majority class or dominant data patterns, leading to suboptimal performance on the minority class or rare patterns. Second , traditional DML methods usually deliver poorer performance under the time-varying communication condition. Specifically, devices holding critical data may unexpectedly go offline or become out of range during the training, causing changes in the communication links. These interruptions can cause information loss in the aggregation, making local models no longer interchangeable and drifting away from the global model. We can summarize these powers of deviation and drift as ' divergence forces ', which drastically slows down the convergence and significantly impacts the efficiency and effectiveness of the training process in DML.

In order to tackle the above issues, we present a novel decentralized DML framework HyperPrism , which utilizes mirror descent [3] and employs adaptive mapping functions to project models into a mirror space, with both aggregation and gradient steps then carried out in the mapped space. Moreover, HyperPrism leverages the concept of 'weighted power means' (WPM) as the aggregation function, raising the model parameters to the power of p , and uses HyperNetworks [12] to adaptively adjust the power degree p . In summary, the main contributions are summarized as follows:

We study the problem of divergence forces in decentralized DML, where we particularly focus on two technical barriers due to data heterogeneity (i.e., non-IID data) and time-varying communication links . To the best of our knowledge, it is the first work to simultaneously address the challenges of non-IID data and time-varying communication links in realistic DML scenarios.

We propose a non-linear class aggregation DML framework HyperPrism based on Kolmogorov Means, which can also be seen as mapping models to mirror space for aggregation, and we instantiate the means with an adaptive p power function to enhance the convergence speed and scalability. HyperPrism achieves superior performance in its dependence on the number of devices m and the power degree p , upgrading from m √ m to m p √ m . This achievement also reduces the optimality gap [2] from √ m to p √ m .

We conduct rigorous analysis and prove that the loss bound of HyperPrism is O (( m P +2 T ) 1 P +1 ) . In cases where few communication epochs can occur ( i.e. , T ≤ m ), employing a larger value of p yields improvements over traditional linear aggregation. Our theoretical results are consistent with state-of-the-art bounds in distributed gradient/mirror descent and single-device mirror descent.

We carry out comprehensive experiments to assess the performance of HyperPrism framework. The experimental results demonstrate that HyperPrism achieves a remarkable acceleration in convergence speed with improvements of up to 98.63%. Moreover, HyperPrism also shows increased scalability in settings characterized by time-varying communication.

2 Related Work

Decentralized DML with Data Heterogeneous. Numerous studies have addressed non-IID data using linear solutions, such as local fine-tuning of a global model [4, 9, 8, 10], personalization in Federated Learning (FL) as a meta-learning objective [15], knowledge distillation[61], and prototype aggregation [53]. Furthermore, Li et al. analytically demonstrate the limitations of FedAvg on non-IID data [26]. Li et al. propose a variant of FedAvg by incorporating a penalty term in the local objective function [25]. Liu et al. propose an algorithm that capture similarities between clients to compute personalized aggregation weights for personalized FL [36]. Recently, Aketi et al. [1] propose a tracking-based method to mitigate the impact of heterogeneous data distribution in DML without introducing communication overhead. However, these methods, less explored in decentralized DML, are confined by the boundaries of linear aggregation.

DMLwith Time-varying Communication setting. Many DML models and their variants have been proposed to process huge amounts of data locally over time-varying communication settings. Kovalev et al. propose the ADOM and ADOM+ method for decentralized optimization over time-varying networks with projected Nesterov gradient descent, respectively [18, 20]. Koloskova et al. introduce a framework covering local SGD updates and synchronous and pairwise gossip updates on adaptive network topology [17]. De Vos et al. [5] introduce Epidemic Learning in decentralized learning, leading to faster convergence and improved performance by randomly changing topologies. Nedic et al. [44, 42] tackle the DML with topology dynamicity from the consensus perspective. They propose a model aggregation method for agents in a time-varying network topology to collaboratively solve a convex objective function, with a convergence guarantee. Moreover, future generation DML systems [20, 19, 7] also consider these time-varying communication settings as an important research area, where the naturally time-varying connectivity among pairs of mobile devices or mobile devices and edge servers will be dictated by their physical proximity.

In this work, we focus on solving two barriers due to data heterogeneity (i.e., non-IID data) and time-varying communication links and conduct rigorous analysis to show that prior methods suffer from lower accuracy, higher loss, and slower convergence speed over time-varying networks than under a fixed topology. Additionally, their methods use linear averages for aggregation, which our framework encompasses as a particular case.

3 Problem Formulation and Preliminaries

3.1 Problem Formulation

We consider decentralized DML with m devices over time-varying communication links represented by a directed graph G ( t ) = ( V, ε ( A ( t ) )) , where V denote vertices and A ( t ) = [ a ( t ) 11 , a ( t ) 12 , ..., a ( t ) mm ] is the weight matrix of the topology graph. Then the ε ( A ( t ) ) denotes the set of directed time-changing edges between vertexes. In each time interval, we assume that the communication links between devices are symmetrical and restricted, causing random disconnections and reconnections. Each device i has its own private dataset, denoted by D i . Let w = ( w 1 , w 2 , . . . , w m ) denote the collection of all local models, where w i is the model held at device i . Then, Device i constructs its local loss function as f i ( w ) = E ζ i ∼ D i [ F ( w i ; ζ i , G ( t ))] . Then, devices aim to solve the following optimization problem collaboratively:

without sharing local data ( i.e. , without revealing f i ).

3.2 Motivation

In recent years, model merging has developed non-linear aggregation methods. One motivation for non-linear aggregation is the decreased variance in the original parameter space. HyperPrism is designed to combat stale gradients from diverged models with synchronization.

In existing work, synchronization is achieved with an aggregation mechanism, usually a mean or median, with clipping. Inspired by similarities between Mirror Descent [46] and Quasi-Arithmetic Means [16], also known as Kolmogorov Means, of the form f -1 ( 1 n ∑ n k =1 f ( x k ) ) , HyperPrism maps models to the dual domain before averaging to better align with the geometry of the objective function. In implementation, we focus on the special case of ϕ ( w ) = 1 p +1 ∥ w ∥ p +1 , transforming models as w → w p , where HyperPrism therefore replaces linear means with the following weighted power mean:

where a ( t ) ij is the aggregation weights of device j in device i at t round. The WPM is a special case of the general strategy of averaging in a mirror descent dual space, but it is particularly useful due to its

ease of computation and convexity guarantees. In general, the aggregation step of HyperPrism is the following Equation (3):

But we choose to focus on ϕ ( w ) = 1 p +1 ∥ w ∥ p +1 which gives ∇ ϕ ( w ) = w p (termwise exponentiation). This allows for simple run-time computation. For more intuition, in the p = ∞ limit the distributed averaging consensus problem [47] turns into an easier 'distributed maximum problem', which is what allows for HyperPrism's tighter synchronization.

3.3 Distributed Mirror Descent

The phrase 'distributed mirror descent' can have multiple interpretations. In the existing literature, it typically refers to the process of taking local mirror descent steps and then linearly aggregating [23, 48, 6, 59] In the case of HyperPrism, mirror descent is transformed into the distributed algorithm in a different manner. It still takes local mirror descent steps but also uses the aggregation function in Equation (3). This approach proves to be more effective because, under the optimal model w ∗ , distributed mirror descent with Equation (3) remains stable, while the linear aggregation functions do not. In other papers' analysis ( e.g. , [11], [58]), this inaccuracy is not obvious since the dominant terms are often related to communication costs, but in the perfect communication case, using a generalized mean is necessary for exact theoretical convergence.

3.4 HyperNetworks

Hypernetworks (HNs) are deep neural networks used to generate weights of other target networks [12]. HNs can learn the mapping relationship between the embedding vector and the target network, and adaptively generate the target network based on the input. Shamsian et al. apply HNs in federated learning to generate personalized model parameters [50]. Ma et al. present pFedLA [38] using HNs to generate aggregation weights of each model layer in personalized federated learning. Previous works [53, 55] found the different layer parameters of the local model have different impacts on model aggregation. For example, the locally learned feature representations are prone to over-fitting and thus cannot generalize well when each device only has insufficient data. In HyperPrism, the mapping function chosen is a form of the Bregman divergence, specifically ∇ ϕ ( w ) = w p , where p represents the degree of WPM. Our experiments indicate that different functional layers of the model respond differently to the degree of p . Increasing p has a more significant impact on accelerating convergence in linear or fully-connected layers. This observation inspires us to select the appropriate p for different functional layers adaptively. Through the chain rule [38], we demonstrate that HN can effectively correlate the mirror mapping with the objective function, enabling it to discover the optimal p for each device's model parameters in each round.

4 The Design of HyperPrism Framework

4.1 Overview

With the above preliminaries and motivation presented, we now give the design of HyperPrism framework. The framework assumes that each device i maintains its local model denoted by w i ; for simplicity of specification, we use a round-based specification, in which the model holds by device i after t communication rounds is denoted as w ( t ) i .

The framework first chooses the aggregation weights a . Each a ( t ) ij , defined by

̸

where e ( t ) ij = 1 N ( t ) i +1 when i, j are connected, otherwise e ( t ) ij = 0 . N ( t ) i is the number of neighboring devices i at t -th round. a ( t ) ij generated in this way satisfy Assumption 5.1. Note that the connectivity

Device1

Devicej

Dataset

Local Model

HyperNetwork

Class1

EmbeddingVector

Class2

Class3

Local Update

Gradients

Devicei

po

Device2

Aggregate

WPM?

Mirrormap

pe

p4

3

Inversemap

Devicen

Devicem

1

po

p

MirrorSpace

Primal Space

e ( t ) ij is determined by underlying communication systems, which consider both feasibility and security requirements ( e.g. , low probability of detection). HyperPrism applies to generic DML, the parameter server and all-reduce frameworks can be seen as special cases (where all a ( t ) ij values equal 1 m ).

In summary, the core of HyperPrism is that at every round t , device i computes its local model for the next round t +1 as:

where ∇ ϕ is the mapping function with a selected degree of p for each round.

4.2 Adaptive Degree of Power p Mapping

To determine the p of various layers at each round, HyperPrism establishes a relationship between the optimization problem and the degree of p by utilizing hypernetworks, which adaptively choose the optimal p for different layers to achieve the best performance.

Without loss of generality, HyperPrism also decouples the local model into the representation and decision parts. The representation part θ includes components like convolutional and embedding layers. The decision part φ includes components like fully-connected layers. Thus, for device i , we have w i = { w θ,i , w φ,i } . Each device i holds a local hypernetwork HN i and a randomly generated embedding vector v i . Every hypernetwork consists of several fully-connected layers and employs the softmax for the output layer. HN i takes v i and gradient of the local model as input, then output p i = { p θ,i , p φ,i } for w θ,i and w φ,i parts, respectively. Then the mapping function evolves to ∇ ϕ ( w i ) = { ( w θ,i ) p θ,i , ( w φ,i ) p φ,i } . The hypernetwork on device i can be defined as

where ψ i denotes the parameters of HN i . Hence, a new objective function can be derived from the original problem as

HyperPrism can transform the optimization problem for model parameters w i into the HN's v i and ψ i . HNs adaptively output p θ,i and p φ,i based on the input, and simultaneously update both v and ψ by gradient descent at each round. Specifically, the gradient of v i and ψ i can be computed based on the chain rule [38] as

Algorithm 1 HyperPrism Framework.

1: Input: datasets { D 1 , D 2 , . . . , D m } , learning rate η , the number of Rounds T .

2: Output: the final model of all devices after T rounds. w ( T ) = w ( T ) 1 , . . . , w ( T ) i , . . . , w ( T ) m } .

3: Initialize all devices' models w (0) , hypernetworks ψ (0) , and embedding vectors v (0) .

4: for t = 0 to T do

5: for each device i in parallel do

6: Local Update: w ( t ) i ← w ( t ) i -η ∇ f ( t ) i .

7: Compute p ( t ) θ,i and p ( t ) φ,i by HN i ( v ( t ) i ; ψ ( t ) i ) .

8: Compute ∇ ϕ ( w ( t ) i ) ←{ ( w ( t ) θ,i ) p ( t ) θ,i , ( w ( t ) φ,i ) p ( t ) φ,i } .

9: Send ∇ ϕ ( w ( t ) i ) and N ( t ) i to i 's neighbors.

10: Receive ∇ ϕ ( w ( t ) j ) and N ( t ) j from neighbors.

11: Compute weights a ( t ) ij as Equation (4), (5).

12: Aggregate received models using a ( t ) ij .

13: Update w ( t +1) i according to Equation (6).

14: Update v ( t +1) i , ψ ( t +1) i as Equation (11), (12).

15: end for

16: end for

17: return w ( T ) i .

Then, v i and ψ i can be represented as

Algorithm 1 demonstrates the full procedure of HyperPrism. In our real deployment, we adopt the asynchronous protocol.

5 Analytical Results

We rigorously analyze the properties of HyperPrism. We fully introduce all assumptions, analyze the convergence behavior of HyperPrism, and finally compare HyperPrism to previous works.

5.1 Assumptions

Assumption 5.1 (Connectivity) . The network graph G ( t ) = ( V, ε ( A ( t ) )) and the connectivity weight matrix A ( t ) satisfy the following:

A ( t ) is doubly stochastic for all t ≥ 1 ; that is ∑ m j =1 a ( t ) ij = 1 and ∑ m i =1 a ( t ) ij = 1 .

There exists a scale ζ > 0 , such that a ( t ) ij ≥ ζ for all i and t ≥ 1 , if { i, j } ∈ E t .

There exists an integer B ≥ 1 such that the graph ( V, E kB +1 ∪ · · · ∪ E ( k +1) B ) is strongly connected for all k ≥ 0 .

Definition 5.2 (Bregman Divergence) . The Bregman Divergence of a function ϕ is defined as

Note that for ϕ ( x ) = ∥ x ∥ 2 , D ϕ ( x, y ) = ∥ x -y ∥ 2 .

Definition 5.3 (Uniform Convexity) . Consider a differentiable convex function ϕ : R d → R , an exponent r ≥ 2 , and a constant σ > 0 . Then, ϕ is ( σ, r ) -uniformly convex with respect to a ∥·∥ norm if for any x, y ∈ R d ,

Note that for r = 2 , this is known as strong convexity . This assumption also implies that D ϕ ( x, y ) ≥ σ r ∥ x -y ∥ r .

Assumption 5.4 (Smooth Gradient) . Assume that the functions f i are convex with its gradients ∇ f i ( · ) satisfying L -Lipschitz continuity [59], namely:

for all x, y pairs.

This final assumption is used in nearly every mathematical analysis of DML, including [54, 3, 52, 59, 43, 51, 49]. In our analysis, it is used to bound the distance between local models.

5.2 Weighted Power Mean

In our experiments, we focused on the weighted power mean, generated by using ϕ ( x ) = 1 p +1 ∥ x ∥ p +1 which gives ∇ ϕ ( x ) = x p . Such ϕ is uniformly convex as seen in Proposition 5.5, and the rest of the analysis will be generalized to all uniformly convex ϕ .

Proposition 5.5 (Uniform Convexity of Power Functions) . For p ≥ 1 , the function φ p ( x ) = 1 p +1 ∥ x ∥ p +1 is uniformly convex with degree p +1 . This is because ∥∇ φ p ( x ) -∇ φ p ( y ) ∥ = ∥ x p -y p ∥ ≥ 1 2 p -1 ∥ x -y ∥ , and as a corollary we have

Given the function ϕ , HyperPrism instructs local models to take a WPM of received models. Using the Connectivity Assumption (Assumption 5.1), we can bound the distance between local models. Lemma 5.6 (Consensus) . Under Assumption 5.1, for each device i , after t rounds:

where ϑ = ( 1 -ζ 4 m 2 ) -2 , κ = ( 1 -ζ 4 m 2 ) 1 B , G l = 2 L ( max f ( w ( t ) i ) -f ∗ ) and ζ is a constant related to the graph connectivity.

Theorem 5.7 (Convergence Behavior) . Consider a ( 1 2 p -1 , p +1) uniformly convex ϕ and the sequence w ( t ) i under Algorithm 1 with constant step size η . Then, under Assumptions 5.1, and 5.4, if x ∗ is the value that minimizes F ( w ) = ∑ m i =1 f i ( w ) , then

In this Theorem 5.7, the first term uses Lemma 5.6, and is caused by the differences between local models held at different devices. This is the effect of the diverging forces mentioned in earlier sections. Note that ϑ and κ are constants related to the graph connectivity, reflecting the influence of time-varying communication links. The second term represents the error caused by a non-zero learning rate in the Distributed Mirror Descent process, and the third term represents the lingering effects of the initialization. For more proof details, please refer to the Appendix.

Corollary 5.8. The bound on HyperPrism's (Algorithm 1) loss is O ( m p √ ηm + m ηT ) .

The first term becomes O ( m · P √ mη ) because since κ < 1 , the κ t term goes to 0 as t gets large. The second term is O ( m P √ η ) is smaller than the first term, so we drop it. The final term is clearly O ( m ηT ) .

5.3 Comparison to Previous Work

Table 1 summarizes the big O notation convergence rates of f ( w ( t ) ) -f ∗ . Previous works also assume bounded gradients. Our analysis recovers the same bounds as the state-of-the-art in DML and single-device mirror descent with generic uniform convexity assumptions. Nedic [44] is standard gradient descent, and thus has ϕ ( x ) = 1 2 ∥ x ∥ 2 , which is (1 , 2) -uniformly convex, corresponding to p = 1 . Yuan [59] considers distributed mirror descent under strong convexity ( r = 2) equivalent to p = 1 , and based on their analysis, the bound should be O ( ηm 2 + m ηT ) .

6 Evaluation

6.1 Experimental Setup

The experimental platform consists of 8 Nvidia Tesla T4 GPUs, 4 Intel XEON CPUs, and 256GB of memory. All the models and training scripts are implemented in RAY [40] and PyTorch [24].

HyperNetworks Setup. We construct a hypernetwork model comprising three fully-connected layers and two additional output layers activated using softmax. The outputs of the fully-connected layers are fed into each of the two output layers to generate the degree of P for various parts.

Time-varying Communication Links Setup. We employ the NS3 platform [13] to simulate realistic time-varying communication environments consisting of multiple distributed devices. Each device is configured with the WiFi 802.11a protocol and communication among themselves in Ad-Hoc mode. To quantify the degree of connectivity of the communication links, we define the topology density as the ratio of available tunnels to the total tunnels.

Models and Dataset. We use MNIST [22] and CIFAR-10 [21] datasets distributed among devices in non-IID settings. We construct two models based on layer functionalities. The Logistic Regression (LR) [14] model consists solely of linear layers used for the MNIST dataset. On the other hand, the CNN model consists of both convolutional and fully-connected layers used for CIFAR-10.

Non-IID Data Partitioning. To distribute datasets in a non-IID fashion, we employ Dirichlet distribution [35] to allocate all samples among devices. The Dirichlet Degree α is used to control the non-IID degree. The α = 0 . 1 represents the extreme scenario where each device possesses samples from only one class, while α = 10 equals the IID scenario. These distributions reflect a challenging and realistic training environment.

Metrics. We consider two metrics to measure the performance of HyperPrism.

Average Accuracy. We evaluate the performance of the local model per device using a global test set that contains samples with all categories. The average Top-1 accuracy of all devices is calculated in each round to measure overall performance and convergence rate.

Convergence Speed. Wetrack the loss of each round and the number of rounds to investigate the rounds needed to reach the convergence point for specific accuracy and loss.

Baselines. We conduct a comparative analysis of HyperPrism with state-of-the-art methods for DML in non-IID data and time-varying communication links, including p = 1 methods, which is one of the most influential and widely applied works in DML studies. SwarmSGD [41], Mudag [57] and ADOM[20]. To ensure a fair comparison, we made minor adjustments to each baseline method.

p=1 . We define it as a class of methods using a linear aggregation function, including DPSGD [34], and its variants [43, 45]. These methods can be seen as special cases of HyperPrism without the mirror mapping process.

SwarmSGD . We set the number of local SGD updates equal to 1, where the selected pair of devices performs only a single local SGD update before aggregation. It can also be viewed as a special case of HyperPrism where the topology density is very low.

ADOM and Mudag . We set the condition number k to 10, and the number of features d equals the number of classes. The gossip matrix W ( t ) at round t is chosen to be the Laplacian of time-varying communication links divided by its largest eigenvalue.

Hyperparameters. For all experiments, the learning rate and batch size are both fixed at 0 . 01 and 128 . We generate time-varying communication graphs with different sizes and densities and evaluate 100 rounds total. The graph changes every round.

6.2 Experimental Results

0

20

40

60

80

100



0

20

40

60

80

100













1

2

3

4

5

6

7

8

9 101112131415161718192021



83

84

85

86

87

88





LR+MNIST

CNN+CIFAR10

Total (s)

Total (s)

DPSGD

1.141

212.22

8.36

3891.75

SwarmSGD

0.871

90.58

4.53

4134.32

Mudag

0.999

43.95

9.13

2365.70

ADOM

1.055

38.45

8.96

inf

Ours(Without HN)

5.064

30.38(

↓

)

33.43

1974.76(

↓

)

Ours(With HN)

8.345

41.72(

↓

)

14.08

2074.79(

↓

)

(c) The Comparison of Time Cost

Impact of different degrees of p . We present the impact of varying degrees of p in HyperPrism on model performance in Figure 2. In Figure 2(a), it is evident that the model converges more swiftly and attains greater accuracy as p grows larger. Figure 2 (b) demonstrates that accuracy exhibits a significant fluctuation with different p . This underscores the substantial impact of the p value selection on the performance of HyperPrism. These findings highlight the importance of choosing an appropriate value for p in HyperPrism to achieve optimal performance.

Comparison of time cost. In HyperPrism, each hypernetwork contains only 3 linear layers with 64 nodes per layer to ensure a minimal extra computational resource cost. We record the time cost in Figure 2(c). Although HyperPrism does result in a higher time cost per iteration, it notably decreases the total number of rounds required for convergence, thereby reducing the overall time needed to achieve a specific accuracy.

Performance of HyperPrism. To showcase the practicality of HyperPrism, we consider the basic configuration with Dirichlet = 0 . 1 , density = 0 . 5 , and m = 50 devices. The accuracy results for all benchmarks and HyperPrism are presented in Figure 3. The convergence speed is summarized in Table 2, 3, 4. Notably, HyperPrism outperforms all benchmarks across all models. It demonstrated a remarkable superior performance over state-of-the-art baselines with convergence accuracy and convergence speed improvements of up to 4.87% and 98.63%, respectively.

Impact of non-IID. To investigate the impact of the non-IID Dirichlet degree on HyperPrism, we experiment with various α = 0 . 1 , 1 , 10 . The corresponding results are presented in Figure 3, and Table 2. All methods exhibit poorer performance as the non-IID degree becomes more extreme, which aligns with common intuition. However, HyperPrism demonstrates enhanced stability and faster convergence speed, especially at highly non-IID degrees.

















































































































































































Scalability. We further evaluate the performance of HyperPrism with varying scales m ∈ { 20 , 50 , 100 } . The results are summarized in Table 4. It can be noticed that most of the baseline methods exhibit deteriorating performance as the number of devices increases. The ADOM even barely converges under 100 devices. In contrast, HyperPrism is minimally affected by the scale and maintains superior acceleration and model performance.

Communication Graph Densities. To further analyze the impact of connected densities on model performance, we present the performance and convergence speed of HyperPrism with various density ∈ { 0 . 2 , 0 . 5 , 0 . 8 } in Table 3. In the extreme non-IID case, the performance of baselines deteriorates as the communication becomes denser. Particularly, ADOM exhibits significant fluctuations at a density of 0.8. However, HyperPrism maintains better performance across different densities. This can be attributed to the fact that as communication becomes denser, the information exchanged between devices becomes more complex. Given HyperPrism's resilience to non-IID scenarios, it maintains good performance in such cases.

7 Conclusion

In this work, we studied the important problem of divergence forces in decentralized DML, due to data heterogeneity (i.e., non-IID data) and time-varying communication links . We propose a non-linear class aggregation DML framework with adaptive Kolmogorov Means for aggregation to enhance the convergence speed and scalability. HyperPrism achieves superior performance in its dependence on the number of devices m , improving from m √ m to m p √ m , and achieving optimality in the limit p →∞ . We also conduct rigorous analysis and demonstrate that the loss bound of HyperPrism is O (( m P +2 T ) 1 P +1 ) . In cases with few communication epochs ( i.e. , T ≤ m ), employing a larger value of p yields improvements over traditional linear aggregation. Our theoretical results are consistent with state-of-the-art bounds in distributed gradient/mirror descent and single-device mirror descent, all under a general uniform convexity assumption. To verify the effectiveness of HyperPrism, we carry out extensive experiments that demonstrate that HyperPrism achieves a remarkable acceleration in convergence speed with improvements of up to 98.63%. Moreover, HyperPrism shows increased scalability.

Acknowledgments and Disclosure of Funding

Weexpress our gratitude to Chengdong Ni for providing technical support. This work was supported in part by the Shanghai Municipal Education Commission grant Z2024-119, the NSFC grant 62141220, and the Yunnan Key Research Program grant 202402AD080004.

References

Sai Aparna Aketi, Abolfazl Hashemi, and Kaushik Roy. Global update tracking: A decentralized learning algorithm for heterogeneous data. Advances in Neural Information Processing Systems , 36, 2024.

Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization, 2022.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. Advances in Neural Information Processing Systems , 35:10572-10586, 2022.

Martijn De Vos, Sadegh Farhadkhani, Rachid Guerraoui, Anne-Marie Kermarrec, Rafael Pires, and Rishi Sharma. Epidemic learning: Boosting decentralized learning with randomized communication. Advances in Neural Information Processing Systems , 36, 2024.

Thinh T. Doan, Subhonmesh Bose, D. Hoa Nguyen, and Carolyn L. Beck. Convergence of the iterates in mirror descent methods, 2018.

Haizhou Du, Yijian Chen, Xiaojie Feng, Qiao Xiang, and Haoyu Liu. An efficient federated learning framework for multi-channeled mobile edge network with layered gradient compression. Computer Networks , 221:109517, 2023.

Haizhou Du, Chaoqian Cheng, and Chengdong Ni. A unified momentum-based paradigm of decentralized SGD for non-convex models and heterogeneous data. Artificial Intelligence , 332:104130, 2024.

Haizhou Du, Chengdong Ni, Chaoqian Cheng, Qiao Xiang, Xi Chen, and Xue Liu. FedSwarm: An adaptive federated learning framework for scalable aiot. IEEE Internet of Things Journal , 2024.

Haizhou Du and Zhiyuan Yang. FedPrime: An adaptive critical learning periods control framework for efficient federated learning in heterogeneity scenarios. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 125-141. Springer, 2024.

Nima Eshraghi and Ben Liang. Distributed online optimization over a heterogeneous network with any-batch mirror descent. In International Conference on Machine Learning , pages 2933-2942. PMLR, 2020.

Tomer Galanti and Lior Wolf. On the modularity of hypernetworks. Advances in Neural Information Processing Systems , 33:10409-10419, 2020.

Thomas R Henderson, Mathieu Lacage, George F Riley, Craig Dowell, and Joseph Kopena. Network simulations with the NS-3 simulator. SIGCOMM demonstration , 14(14):527, 2008.

David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression , volume 398. John Wiley & Sons, 2013.

Yihan Jiang, Jakub Koneˇ cn` y, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488 , 2019.

Anna Kolesárová. Limit properties of quasi-arithmetic means. Fuzzy Sets and Systems , 124(1):65-71, 2001.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In ICML , pages 5381-5393, 2020.

Dmitry Kovalev, Elnur Gasanov, Alexander Gasnikov, and Peter Richtarik. Lower bounds and optimal algorithms for smooth and strongly convex decentralized optimization over time-varying networks. NIPS , 34, 2021.

Dmitry Kovalev, Anastasia Koloskova, Martin Jaggi, Peter Richtarik, and Sebastian Stich. A linearly convergent algorithm for decentralized optimization: Sending less bits for free! In ICAIS , pages 4087-4095. PMLR, 2021.

Dmitry Kovalev, Egor Shulgin, Peter Richtarik, Alexander V Rogozin, and Alexander Gasnikov. ADOM: Accelerated decentralized optimization method for time-varying networks. In ICML , pages 5784-5793, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. IEEE , 86(11):2278-2324, 1998.

Jueyou Li, Guo Chen, Zhaoyang Dong, and Zhiyou Wu. Distributed mirror descent method for multi-agent optimization with delay. Neurocomputing , 177:643-650, 2016.

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 , 2020.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-IID data. arXiv preprint arXiv:1907.02189 , 2019.

Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. Gs2p: a generative pre-trained learning to rank model with over-parameterization for web-scale search. Machine Learning , pages 1-19, 2024.

Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, and Dawei Yin. Mpgraf: a modular and pre-trained graphformer for learning to rank at web-scale. In 2023 IEEE International Conference on Data Mining (ICDM) , pages 339-348. IEEE, 2023.

Yuchen Li, Haoyi Xiong, Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4437-4448, 2023.

Yuchen Li, Haoyi Xiong, Linghe Kong, Shuaiqiang Wang, Zeyi Sun, Hongyang Chen, Guihai Chen, and Dawei Yin. Ltrgcn: Large-scale graph convolutional networks-based learning to rank for web search. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 635-651. Springer, 2023.

Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, and Guihai Chen. Meta hierarchical reinforced learning to rank for recommendation: a comprehensive study in moocs. In Joint European conference on machine learning and knowledge discovery in databases , pages 302-317. Springer, 2022.

Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical reinforced ranking. IEEE Transactions on Services Computing , 2023.

Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dejing Dou, et al. Coltr: Semi-supervised learning to rank with co-training and over-parameterization for web search. IEEE Transactions on Knowledge and Data Engineering , 35(12):12542-12555, 2023.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In NIPS , page 5336-5346. Curran Associates Inc., 2017.

Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In ICML , pages 6654-6665, 2021.

Jiahao Liu, Jiang Wu, Jinyu Chen, Miao Hu, Yipeng Zhou, and Di Wu. FedDWA: Personalized federated learning with dynamic weight adjustment. In International Joint Conference on Artificial Intelligence , 2023.

Zhonghao Lyu, Yuchen Li, Guangxu Zhu, Jie Xu, H Vincent Poor, and Shuguang Cui. Rethinking resource management in edge learning: A joint pre-training and fine-tuning design paradigm. arXiv preprint arXiv:2404.00836 , 2024.

Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10092-10101, 2022.

Marie Maros and Gesualdo Scutari. Decentralized matrix sensing: Statistical guarantees and fast convergence. Advances in Neural Information Processing Systems , 36, 2024.

Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. USENIX Association, 2018.

G. Nadiradze, A. Sabour, D. Alistarh, A. Sharma, I. Markov, and V. Aksenov. SwarmSGD: Scalable decentralized SGD with local updates. 2019.

Angelia Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine , 37(3):92-101, 2020.

Angelia Nedic, Alex Olshevsky, Asuman Ozdaglar, and John N Tsitsiklis. Distributed subgradient methods and quantization effects. In ICDC'2008 , pages 4177-4184. IEEE, 2008.

Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. TACON , 54(1):48-61, 2009.

Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi. Decentralized gradient methods: does topology matter? In ICAIS , pages 2348-2358. PMLR, 2020.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.

Alex Olshevsky and John N. Tsitsiklis. Convergence speed in distributed consensus and control, 2009.

Shahin Shahrampour and Ali Jadbabaie. Distributed online optimization in dynamic environments using mirror descent, 2016.

Shahin Shahrampour and Ali Jadbabaie. Distributed online optimization in dynamic environments using mirror descent. TACON , 63(3):714-725, 2018.

Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning , pages 9489-9502. PMLR, 2021.

Pranay Sharma, Prashant Khanduri, Lixin Shen, Donald J. Bucci, and Pramod K. Varshney. On distributed online convex optimization with sublinear dynamic regret and fit, 2020.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. On the universality of online mirror descent, 2011.

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8432-8440, 2022.

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. CSUR , 53(2):1-33, 2020.

Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. arXiv preprint arXiv:2306.11867 , 2023.

Ryan Yang, Haizhou Du, Andre Wibisono, and Patrick Baker. Aggregation in the mirror space (AIMS): Fast, accurate distributed machine learning in military settings. MILCOM 2022 - 2022 IEEE Military Communications Conference (MILCOM) , pages 470-477, 2022.

Haishan Ye, Luo Luo, Ziang Zhou, and Tong Zhang. Multi-consensus decentralized accelerated gradient descent. arXiv preprint arXiv:2005.00797 , 2020.

Zhan Yu, Daniel WC Ho, and Deming Yuan. Distributed randomized gradient-free mirror descent algorithm for constrained optimization. IEEE Transactions on Automatic Control , 67(2):957-964, 2021.

Deming Yuan, Yiguang Hong, Daniel WC Ho, and Shengyuan Xu. Distributed mirror descent for online composite optimization. TACON , 66(2):714-729, 2020.

Jiakang Yuan, Bo Zhang, Xiangchao Yan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. Advances in Neural Information Processing Systems , 36, 2024.

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-IID federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10174-10183, 2022.

8 Appendix

8.1 Main Notations

We summarize the main notations in Table 5.

8.2 Impact of WPM to parameters

We illustrate how HyperPrism leverages weighted power mean (WPM) to facilitate more efficient aggregation in Figure 4. By examining the distribution of model parameters, we observe that the traditional linear averaging model (w/o WPM) has approximately 11.65% of parameters lying within the range of [-0.01, 0.01] after 80 rounds. In contrast, when utilizing WPM with degrees of p = 9 and p = 15 , only 1.67% and 1.21% of parameters, respectively, fall within the same range. This indicates that WPM enables the model to effectively preserve a broader range of features. Consequently, HyperPrism can extract more information from the model parameters during aggregation, leading to enhanced performance. These results underscore the effectiveness of WPM in enabling HyperPrism to capture a wider range of features and facilitate a more informative aggregation.

Initial Parameter

Distribution

WPM with P = 15

WPM with P = 9

Without

Mapping

Mapping

with P=9

Mapping

with P=15

Layer =linear,Round =20

Layer =linear,Round =80

Number

1000

Zero Position

800

ZeroPosition

800

600

600

400

400

200

200

Parameter Distribution

Parameter Distribution

Layer =linear,Round=0

Layer =linear,Round =20

Layer=linear,Round =80

Number

1000

Zero Position

Number

1000

1250

750

-ZeroPosition

1000

750

ZeroPosition

500

500

250

250

0

#

Parameter Distribution

Layer =linear,Round =20

Layer=linear,Round=80

Number

1500

-ZeroPosition

1000

1250

Zero Position

1000

750

500

500

250

D

8.3 Proof under Uniform Convexity

To aid analysis, introduce the sequence y ( t ) i . Note that the w ( t ) i and y ( t ) i sequences satisfy

Additionally, we can define

8.4 Proof of Lemma 5.6

Proof. Define a transition matrix Φ( t, s ) = A ( t ) A ( t -1) · · · A ( s +1) A ( s ) . Then, under Assumption 5.1, Corollary 1 in [43] states that

where ϑ and κ are defined in the lemma statement.

We are able to write out a general formula for ∇ ϕ ( w ( t +1) i ) :

as well as ∇ ϕ ( w ( t ) ) :

It is known that for a L -Lipschitz continuous function f , if x ∗ is the optimum of f , then:

Then, ∇ ϕ ( w ( t ) i ) -∇ ϕ ( w ( t ) ) can be bounded by applying the Triangle Inequality and Equation (22):

Then, plugging in k = 0 gives

Finally, shifting t down by 1 gives the desired bound.

8.5 Proof of Theorem 5.7

Firstly, note that under Algorithm 1 and Assumption 5.4, Weighted AM-GM gives:

We may follow the main line of reasoning that proves Theorem 5.7.

8.6 Main Line of Reasoning

Proof. We prove bounds for generic x . Note that w ( t ) = h -1 ( 1 m ∑ m i =1 h ( w ( t ) i )) . Then, we get:

Cauchy's inequality can bound the first term. The third term can be manipulated using Equation (20). Then, combined with f i ( w ( t ) ) ≤ f i ( w ( t ) i ) + G l ∥ w ( t ) -w ( t ) i ∥ , we get

The factor of 2 comes from the 2nd term of Equation (27) added to the error from f i ( w ( t ) ) ≤ f i ( w ( t ) i ) + G l ∥ w ( t ) -w ( t ) i ∥ . Note that the last term is equal to m · ⟨ w ( t ) -w ( t +1) , w ( t +1) -x ⟩ . This is also equal to m ( D ϕ ( x, w ( t ) ) -D ϕ ( x, w ( t +1) ) -D ϕ ( w ( t +1) , w ( t ) )) by the Triangle Inequality for Bregman Divergences. We also substitute Claim 1 to replace the second term, so the value is

But, by uniform convexity, D ϕ ( w ( t +1) , w ( t ) ) ≥ σ ∥ w ( t +1) -w ( t ) ∥ r , and thus this is also

Then, taking the sum over T gives

Dividing through by ηT , substituting σ = 1 2 p -1 and r = p +1 , and substituting x = x ∗ and Lemma 5.6 gives the desired result.

8.7 Weighted Power Mean Skew Correction

Theorem 8.1. Consider positive x i and α i close to 1 m . Then,

This allows us to see the weighted power mean as a way to decrease skew; this theorem is relevant in the setting of Lemma 5.6.

Write α i = 1 m + ϵ · e i where all e i ≤ 1 m . Hold e i constant. Then, for α i ≈ m , it is true that ϵ ≈ 0 . The derivative of the average with respect to ϵ is

and the derivative at ϵ = 0 is ∑ e i x p i p · ( ∑ 1 m x p i ) p -1 p . Thus, we have the following first-order approximation:

But we have ∑ e i x p i ≤ ∑ 1 m x p i , so

The 1 p (∑ 1 m x p i ) 1 p term is bounded, so the error is proportional to O ( ϵ p ) .

8.8 Linear Bound on Power Mean

As a starter, we first prove a useful lemma on two real numbers with different signs. Note that all variables used in this subsection are generic ones not tied to the HyperPrism-based mechanism.

Lemma 8.2 (Two Numbers with Different Signs) . Given any two real numbers x > 0 , y < 0 , an odd integer p ≥ 1 , and 0 ≤ α ≤ 1 , we have

Proof. Denote C = 1 2 p and rewrite the right-hand side (RHS) of Equation (34) as C ( αx + (1 -α ) y ) + (1 -C ) y . We first note that we can scale both x, y by 1 | y | such that y = -1 . As such, we can simplify our proof to only focus on the case of y = -1 . We will use shorthand ∆ , and rewrite as

We then consider two cases.

Case 1: x ≤ 2 p -1 . Since -Cα ( x +1) ≥ -Cα (2 p ) = -α ≥ -1 , by Bernoulli's inequality, we have

Since x +1 2 ≤ x p +1 , which can be verified by casework on x ≥ 1 or x ≤ 1 , then

We then finish this case by taking the power of 1 /p on both sides.

Case 2: x ≥ 2 p -1 . We will denote the expression on the left-hand side and right-hand side of Equation (34) with LHS and RHS, respectively. First note that both sides of the inequality in Equation (34) are -1 when α = 0 and y = -1 . Next, we will show that the derivative with respect to α is always larger for the LHS of Equation (34) when x ≥ 2 p -1 .

The LHS's derivative w.r.t α is clearly constant. Also, the d dα RHS clearly has local maxima at α = 0 , 1 . At α = 0 , we clearly have that since x ≥ 2 p -1 , we have

For α = 1 , since x ≥ 1 2 p ( x +1) -1 ≥ 0 , we have

Thus, for all 0 ≤ α ≤ 1 , d dα LHS ≥ d dα RHS , and they are equal at α = 0 , so we always have LHS ≥ RHS for all 0 ≤ α ≤ 1 when x ≥ 2 p -1 , and we are done.

Next, consider a list of real numbers x i , i = 1 , . . . , m . We let M = min x i and U = max x i . We then prove the following lemma.

Lemma 8.3. Assume a list of non-negative real numbers x i , i = 1 , . . . , m and p ≥ 1 . Given α i , i = 1 , . . . , m such that ∑ m k =1 α k = 1 and α i ≥ 0 , we have

Proof. The lower bound in Equation (42) is a direct result from the power mean inequality. As for the upper bound, we have

where the last step uses the generalized AM-GM inequality.

We now tie these two Lemmas together to prove Theorem 8.4, which to the best of our knowledge is a novel inequality on weighted power mean.

Theorem 8.4 (Linear Bound on Weighted Power Mean) . Assume a list of real numbers x i , i = 1 , . . . , m and α i , i = 1 , . . . , m such that ∑ m k =1 α k = 1 and α i ≥ 0 . For any odd integer p ≥ 1 , we have:

Proof. First, note that if all x i s are positive, Equation (44) holds from the lower bound of Lemma 8.2. Second, if all x i s are ≤ 0 , Equation (44) holds from the upper bound of Lemma 8.3 after flipping all the signs.

For the case where x i s have mixed signs, let α pos , α neg be the sum of the corresponding α s for the negative and positive elements in x , respectively. We have,

where the first inequality comes from the n -variable same sign case in Lemma 8.3, and the second inequality comes from the 2-variable case in Lemma 8.2. Here, each positive x i is weighted with at least 1 2 p α i and each negative x i is weighted with at least ( 1 2 p 2 + 2 p -1 2 p 2 ) α i , which equals to 1 p α i ≥ 1 2 p α i , since α neg ≤ 1 .

Since α i ≥ M ( M < 0 when x i s have different signs), we can turn the extra α i terms into M :

which completes our proof.

In addition to a new lower bound of HyperPrism, we can also prove an upper bound of the HyperPrism at 1 2 p ∑ m i =1 α i x i + 2 p -1 2 p U following similar steps. Note that these bounds are generic and can be applied outside of DML. These bounds are necessary for analyzing the benefits of using a HyperPrism aggregation function in DML because they guarantee that the HyperPrism takes at least some minimum consideration of each x i term. This consideration is crucial because each device's local objective function is unique, and they must all be considered to optimize the overall objective function.

9 Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract provides a concise summary of the paper's key findings and contributions, allowing viewers to quickly grasp the main points.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the additional computational overhead that Hypernetworks may introduce, and we provided some time consumption comparisons. However, further research is needed to investigate whether the computational resource consumption will limit the performance.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide comprehensive definitions of all theories and assumptions and presented complete proofs in the appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed explanations of all experimental settings in the experimental section and uploaded the source code and data for reference.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We utilized an open-source dataset and provided the complete source code for reference.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed description of the experimental setup in the evaluation section.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide comprehensive experimental result and detailed explanations in the evaluation section.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient experimental information to reproduce the experiments, some detailed information are present in appendix due to space limitation.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research in this paper satisfies all of the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research in this paper focuses on foundational research and does not have potential negative societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk with this paper.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and properly respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets introduced in the paper are well documented and provided alongside the assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraditional DML methods are limited to (1) data heterogeneity and (2) time-varying communication links. In this work, they present a non-linear class aggregation framework HyperPrism that leverages Kolmogorov Means to conduct distributed mirror descent with the averaging occuring within the mirror descent dual space. The proposed method can improve the convergence speed up to 98.63% and scale well to more devices compared with the state-of-the-art, all with little additional computation overhead compared to traditional linear aggregation.\n\nSTRENGTHS:\n1.\tThe speedup of convergence is satisfactory.\n2.\tThe theoretical discussion is sufficient.\n\nWEAKNESSES:\n1.\tThe authors claim that they use hypernetworks to predict p (since they use softmax, type might be int), the exponent of the mapping function. There is a little deviation from the common practice of hypernetworks, where they usually output more complex parameter weights, like the weight of classifiers (a 768 * 100 matrix, type is float). I would prefer the authors remove the part of hypernetworks, and simply introduce HN as a simple MLP.\n2.\tWhile HNs act as a crucial part of HyperPrism, the analyses of them seem limited. Since HN has softmax layers, I would be curious about (1) the value set of p and its influence, (2) the variance of p on each distributed machines when given different gradients, (3) the results of preset p (≠ 1) (like directly use the most outputted p of HNs rather than 1). I would suggest these ablation studies be added.\n3.\tWhile the performance of HyperPrism is excellent on the major experiments, the baselines used for comparison seems out of date (before 2021), it would be more reasonable if the authors could provide more recent baselines (in 2022 or 2023).\n4.\tThe results of the experiments are a bit confusing, especially in terms of improvement. For example, in the last column of Table 1, Conv Rds of the proposed method is 13, while the best baseline is 14, how is 85.86% computed ? It is computed with the least performed baseline? It would be better if the authors could provide detailed explanations.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nN.A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses challenges in distributed machine learning (DML) caused by non-IID data and \nunstable communication links. The proposed HyperPrism framework uses mirror descent and adaptive \nmapping to project models into a mirror space for better aggregation and gradient steps. \nIt employs adaptive weighted power means (WPM) for efficient model aggregation, significantly \nimproving convergence speed and scalability. The framework's analytical results show that HyperPrism \neffectively handles decentralized DML challenges, offering a robust solution for edge device data processing.\n\nSTRENGTHS:\na) The paper exhibits notable originality by addressing the dual challenges of non-IID data and time-varying \ncommunication links in distributed machine learning (DML). The HyperPrism framework introduces a novel \ncombination of mirror descent and adaptive weighted power means (WPM) for effective model aggregation. \n\nb) The quality of the research is high, demonstrated through rigorous theoretical analysis and comprehensive \nexperimental validation, providing strong support for the framework's claims. \n\nc) The clarity of the paper is commendable, with well-structured explanations and supportive figures and tables, \nalthough some dense sections could benefit from further simplification. \n\nd) The significance of the work is substantial, as it offers a robust solution to a pressing issue in DML, \nwith implications for improving the efficiency and scalability of edge device data processing. The results \nshowing enhanced convergence speed and scalability are valuable contributions to the field.\n\nWEAKNESSES:\nWhile the paper is strong overall, there are a few areas that could be improved. \n\nFirst, some sections are densely packed with technical details, which might be challenging for readers \nnot deeply familiar with the subject. Simplifying these sections or providing additional explanations \ncould enhance accessibility. \n\nIn addition, the paper could provide more comparative analysis with existing methods to highlight the \nspecific advantages and potential limitations of HyperPrism. However, due to the limited period of rebuttal, \nit is just optional and not necessary. \n\nLastly, a more detailed discussion on the computational overhead and scalability of the proposed framework \nin extremely large-scale settings would be beneficial. Addressing these weaknesses would strengthen the paper \nand its contributions.\n\nQUESTIONS:\nHow does HyperPrism perform compared to other non-linear aggregation frameworks or adaptive learning methods in similar scenarios? \nAre there specific benchmarks or datasets where it excels or underperforms?\n\nLIMITATIONS:\nThe potential limitation of the proposed method is not discussed in this manuscript. It will be of great importance if the authors can give an insight to the potential readers by comparing the pro/cons of the proposed method compared to the latest works in this topic.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HyperPrism, a novel framework for decentralized machine learning (DML) that aims to address the challenges of non-IID data and time-varying communication links. The authors propose a non-linear aggregation method based on Kolmogorov Means and adaptive mapping functions, which they argue improves convergence speed and scalability compared to traditional linear aggregation methods. The paper includes theoretical analysis and experimental results to support their claims.\n\nSTRENGTHS:\n•\t**Novelty:** The use of non-linear aggregation with adaptive mapping functions is a novel approach to simultaneously address the challenges of non-IID data and time-varying communication in DML. \n\n•\t**Theoretical Analysis:** The paper provides a theoretical analysis of the convergence behavior of their proposed approach, which is a valuable contribution. \n\n•\t**Experimental Results:** The experimental results demonstrate somewhat promising improvements in convergence speed and scalability compared to few baseline methods.\n\nWEAKNESSES:\n•\t**Experimental Setup:** The experimental setup could be expanded to include more diverse datasets, models, and settings. The current experiments are limited to MNIST and CIFAR-10 with specific model architectures and the non-IID setting include only two extreme cases ($\\alpha=0.1$ and $\\alpha=10$). This would help to assess the generalizability of HyperPrism's performance improvements. \n\n•\t**Clarity:** The paper could benefit from improved clarity in some sections. For example, in section 3 the authors discussed HyperPrism without first introducing it. The motivation for using mirror descent and Kolmogorov Means was unclear. The connection between these concepts and the challenges of non-IID data and time-varying communication could be made more explicit. \n\n•\t**Hyperparameter Tuning:** The paper does not provide sufficient details on how the hyperparameters for the different methods were chosen. This makes it difficult to assess the fairness of the comparison. \n\n**Others minor issues:**\n\n•\tThere seems to be a missing index $i$ on $w$ in equation 1. \n\n•\tHeterogeneity in referencing in related work section. Some references seem to be typed manually (or at least they are not connected to any entry in bibliography).\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nOverall, the paper addresses an important problem in DML and proposes a novel solution with promising theoretical and empirical results. However, the experimental setup and clarity issues mentioned above make it just fall short of this venue’s bar.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}