HyperPrism: An Adaptive Non-linear Aggregation Framework for Distributed Machine Learning over Non-IID Data and Time-varying Communication Links
Authors: Haizhou Du
Yijian Chen
Ryan Yang
Yuchen Li
Linghe Kong
Venue: NeurIPS 2024 poster (2024)
Keywords: Distributed Machine Learning
Time-varying Communication
Non-Linear Aggregation
HyperNetwork
Primary area: optimization
TL;DR: Introduces Kolmogorov Means As A New Distributed Machine Learning Communication Primitive.

Abstract: While Distributed Machine Learning (DML) has been widely used to achieve decent performance, it is still challenging to take full advantage of data and devices distributed at multiple vantage points to adapt and learn, especially it is non-trivial to address dynamic and divergence challenges based on the linear aggregation framework as follows: (1) heterogeneous learning data at different devices (i.e., non-IID data) resulting in model divergence and (2) in the case of time-varying communication links, the limited ability for devices to reconcile model divergence. In this paper, we contribute a non-linear class aggregation framework HyperPrism that leverages distributed mirror descent with averaging done in the mirror descent dual space and adapts the degree of Weighted Power Mean (WPM) used in each round. Moreover, HyperPrism could adaptively choose different mapping for different layers of the local model with a dedicated hypernetwork per device, achieving automatic optimization of DML in high divergence settings. We perform rigorous analysis and experimental evaluations to demonstrate the effectiveness of adaptive, mirror-mapping DML. In particular, we extend the generalizability of existing related works and position them as special cases within HyperPrism. Our experimental results show that HyperPrism can improve the convergence speed up to 98.63% and scale well to more devices compared with the state-of-the-art, all with little additional computation overhead compared to traditional linear aggregation.

adaptively choose HyperPrism: An Adaptive Non-linear Aggregation Framework for Distributed Machine Learning over non-IID Data and Time-varying Communication Links
Haizhou Du Shanghai University of Electric Power Shanghai, China
Yijian Chen Shanghai University of Electric Power Shanghai, China
Ryan Yang Massachusetts Institute of Technology MA, USA
Yuchen Li
Linghe Kong
Shanghai Jiao Tong University Shanghai, China
Shanghai Jiao Tong University Shanghai, China
Abstract
While Distributed Machine Learning (DML) has been widely used to achieve decent performance, is still challenging to take full advantage of data and devices distributed at multiple vantage points to adapt and learn; this is because the current linear aggregation paradigm cannot solve inter-model divergence caused by (1) heterogeneous learning data at different devices (i.e., non-llD data) and (2) in the case of vareingying communication links, the limited ability for devices reconcile model divergence. In this paper, present non-linear class aggregation framework HyperPrism that leverages Kolmogorov Means to conduct distributed mirror descent with the averaging occurring within the mirror descent dual space; HyperPrism selects the degree for a Weighted Power Mean (WPM), a subset of the Kolmogorov Mcans, each round. Moreover, HyperPrism different mapping for different layers of the local model with a dedicated hyper- network per device, achieving automatic optimization of DML in high divergence settings. We perform rigorous analysis and experimental evaluations to demonstrate the effectiveness of adaptive, mirror-mapping DML. In particular, generalizability of existing related works and position them as special cases within HyperPrism. For practitioners, the strength of HyperPrism is in making feasible the possibility of distributed asynchronous training with minimal communication. Our experimental results show HyperPrism 98. 98.63% and scale well to more devices compared with the state-of toee----tthhrttt. all with little additional computation overhead compared to traditional linear aggregation.
extend the
improve the convergence speed up
Introduction
The proliferation of edge devices, such as mobile phones, wearable devices and unmanned aerial vehicles, has resulted in an exponential growth in diverse data types (c.g.. images, sound and text). Addressing this surge necessitates advanced techniques capable of accurately, quickly, and practically processing vast amounts of data through efficient and scalable algorithms. Distributed Machine Learning (DML.) has gained significant traction in recent years as strategy to minimize data transfer
"Email: duhaizhou @ shicp.edu..r enu.en.
These authors are the same contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and computational costs by bringing computation closer to the data sources. It has become a natural solution for scaling up machine learning while preserving data privacy in various domains (e.g.. large language model training [39 33 autonomous driving 60) 37 military applications [56]. web
search [2 30 28 and recommendation [2
While existing DML attempts have made sigmificant progress, two technical barriers remain in realistic scenarios. First, data heterogeneity is one of the most critical concerns. Specifically, in a DML system where mobile devices serve as computing nodes, the data generated is typically non-independent and identically distributed (non-IID), which means the data distribution may be unbalanced across different classes or categories. Consequently, the model may exhibit bias toward the majority class or dominant data patterns, leading suboptimal performance on the minority class or rare patterns. Second, traditional DML methods usually deliver poorer performance under the time-varying communication condition. Specifically, devices holding critical data may unexpectedly go offline or become out of range during the training, causing changes in the communication links. These interruptions can cause information loss the aggregation, making local models no longer interchangeable and drifting away from the global model. We can summarize these powers of deviation and drift as "divergence forces", which drastically slows down the convergence and significantly impacts the efficiency and effectiveness of the training process in DML.
In order to tackle the above issues, we present a novel decentralized DML framework HyperPrism. which utilizes mirror descent (3 and employs adaptive mapping functions to project models into a mirror space, with both aggregation and gradient steps then carried out in the mapped space. Moreover, HyperPrism leverages the concept of "weighted power means" (WPM) as the aggregation function, raising the model parameters to the power of p. and uses HyperNetworks [12 to adaptively adjust the power degree p. In summary, the main contributions are summarized as follows:
We study the problem of divergence forces in decentralized DML, where we particularly focus on two technical barriers due to data heterogeneity (i.e., non-llD data) and time-varying communication links. To the best of our knowledge, it is the first work to simultaneously address the challenges of non-IID data and time-varying communication links in realistic DML scenarios.
We propose non-linear class aggregation DML. framework HyperPrism based on Kol- mogorov Means, which can also be seen as mapping models to mirror space for aggregation, and we instantiate the means with an adaptive p power function to enhance the convergence speed and scalability. HyperPrism achieves superior performance in its dependence on the number of devices and the power degree p. upgrading from m m to z/m. This achievement also reduces the optimality gap 2 from v to t/m.
We conduct rigorous analysis and prove that the loss bound of HyperPrism is O((m==))r Pf1). In cases where few communication epochs can occur (i.e., T < m), employing a larger value of p yields improvements over traditional linear aggregation. Our theoretical results are consistent with state-of-the-aar bounds in distributed gradient/mirror descent and single-device mirror descent.
We carry out comprehensive experiments to assess the performance of HyperPrism frame- work. The experimental results demonstrate that HyperPrism achieves remarkable acceler- ation in convergence speed with improvements of up to 98.63%. Moreover, HyperPrism also shows increased scalability in settings characterized by time-varying communication.
2 Related Work
Decentralized DML with Data Heterogeneous. Numerous studies have addressed non-IID data using linear solutions, such as local fine-tuning of a global model [4] 8 [00, personalization in Federated Learning (FL) as a meta-learning objective [15]. knowledge distillation and prototype aggregation [53 Furthermore, Li et al. analytically demonstrate the limitations of FedAvg on
non-lID data 226] Li et al. propose a variant of FedAvg by incorporating penalty term in the local objective function [25] Liu er al. propose an algorithm that capture similarities between clients to compute personalized aggregation weights for personalized FL (36). Recently, Aketi et al. 1 propose a tracking- based method to mitigate the impact of heterogeneous data distribution in DML without introducing communication overhead. However, these methods, less explored in decentralized DML, are confined by the boundaries of linear aggregation.

---

DML with Time-varying Communication setting, Many DML models and their variants have been proposed to process huge amounts of data locally over varying communication settings. Kovalev el al. propose the ADOM and ADOM+ method for decentralized optimization over time-varying networks with projected Nesterov gradient descent, respectively [8 20] Koloskova el al. introduce framework covering local SGD updates and synchronous and pairwise gossip updates on adaptive network topology [17]. De Vos et al. (5) introduce Epidemic Learning in decentralized learning, leading to faster convergence and improved performance by randomly changing topologies. Nedic et al.
tackle the DML with topology dynamicity from the consensus perspective. They propose model aggregation method for agents in a time-varying network topology to collaboratively solve a convex objective function, with a convergence guarantee. Moreover, future generation DML systems [20] [9] also consider these time- varying communication settings as important research area, where the naturally time-varying connectivity among pairs of mobile devices or mobile devices and edge servers will be dictated by their physical proximity.
In this work, we focus on solving two barriers due to data heterogeneity (i.e., non-IlD data) and time-varying communication links and conduct rigorous analysis to show that prior methods suffer from lower accuracy, higher loss, and slower convergence speed over varying networks than under fixed topology. Additionally, their methods use linear averages for aggregation, which our framework encompasses as a particular case.
3 Problem Formulation and Preliminaries
3.1 Problem Formulation
We consider decentralized DML with m devices over time-varying communication links represented by a directed graph G(t) = (A()((()))))))))))))))))))))(< where V denote vertices and (() [a "12 .... 0mm] is the weight matrix of the topology graph. Then the ((4()) denotes the set of directed time-changing edges between vertexes. In each time interval, we assume that the communication links between devices are symmetrical and restricted, causing random disconnections and reconnections. Each device i has its own private dataset, denoted by D,. Let = (W1, W2,,... Wm. denote the collection of all local models, where w; is the model held at device i. Then, Device i constructs its local loss function as f:w = (. C. G(t))). Then, devices aim solve the following optimization problem collaboratively:
min F(w)=
Σ f((w), 1=1
weRd
without sharing local data (i.e, without revealing fi).
3.2 Motivation
In recent years, model merging has developed non-linear aggregation methods. One motivation for non-linear aggregation the decreased variance in the original parameter space. HyperPrism is designed to combat stale gradients from diverged models with synchronization.
In existing work, synchronization is achieved with aggregation mechanism, usually mean or median, with clipping. Inspired by similarities between Mirror Descent [46) and Quasi-Arithmetic
Means [16), also known as Kolmogorov Means, of the form f-1 (
> fxxx). HyperPrism
maps models to the dual domain before averaging to better align with the geometry of the objective
function. In implementation, we focus on the special case of s(w) I+I
- || transforming
models as w + wº, where HyperPrism therefore replaces linear means with the following weighted power mean:
W
where (1) the aggregation weights of device j in device at round. The WPM is special case of the general strategy of averaging in mirror descent dual space, but it is particularly useful due to its
3

---

ease of computation and convexity guarantees. In general, the aggregation step of HyperPrism is the following Equation 33:
w [
))(w)"))
(3)
p+1 w ||+ which gives VΦ(w)= w" (termwise exponentiation). This allows for simple run-time computation. For more intuition, in the p xo limit the distributed averaging consensus problem [47] turns into an easier "distributed maximum problem", which is
But we choose to focus on d(w) =
what allows for HyperPrism's 's tighter synchronization.
3.3
Distributed Mirror Descent
The phrase "distributed mirror descent" have multiple interpretations. In the existing literature, it typically refers to the process taking local mirror descent steps and then linearly aggregating [23] 48 6 [59] In the case of HyperPrism, mirror descent is transformed into the distributed algorithm in different manner. It still takes local mirror descent steps but also uses the aggregation function in Equation 3 This approach proves to be more effective because, under the optimal model w*, distributed mirror descent with Equation (3) remains stable, while the linear aggregation functions do not. other papers' analysis (e.g., M. () this inaccuracy is not obvious since the dominant terms are often related to communication costs, but in the perfect communication case, using generalized mean necessary for exact theoretical convergence.
3.4 HyperNetworks
Hypernetworks (HNs) are deep neural networks used to generate weights of other target networks 12]
HNs learn the mapping relationship between the embedding vector and the target network, and adaptively generate the target network based on the input. Shamsian et al. apply HNs in federated
learning to generate personalized model parameters [50]. Ma el al. present pFedLA (38] using HNs to generate aggregation weights of each model layer in personalized federated learning. Previous works 53 [55] found the different layer parameters of the local model have different impacts on model aggregation. For example, the locally learned feature representations are prone to wett-fitting and thus cannot generalize well when each device only has insufficient data. In HyperPrism, the mapping function chosen is a form of the Bregman divergence. specifically Vo(w) w", where represents the degree of WPM. Our experiments indicate that different functional layers of the model respond differently to the degree of p. Increasing p has more significant impact on accelerating convergence in linear or fully-con conle-cco layers. This observation inspires us to select the appropriate for different functional layers adaptively. Through the chain rule [8), we demonstrate that HN can effectively correlate the mirror mapping with the objective function, enabling to discover optimal p for each device's model parameters in each round.
4
The Design of HyperPrism Framework
4.1
Overview
With the above preliminaries and motivation presented, we now give the design of HyperPrism framework. The framework assumes that each device maintains its local model denoted by w; for simplicity of specification, we use round- based specification, in which the model holds by device i after í communication rounds is denoted as u
The framework first chooses the aggregation weights a. Each a defined by
aj min
(t)
(
(0)
(4)
d
(t)
1 Σ
(0)
(5)
where ei
N1 when i are connected, otherwise e
= 0. (1(0) is the number of neighboring
devices at t-th round. a,y generated in this way satisfy Assumption
5.1
Note that the connectivity

---

Deviee
Deviee
Dataset
Local Model
HyperNetwork
Class
Embodding Voctor
Class 2
Class
Local Update
Gradlenis
Device
P9
P-
Devize 2
WPM Ggaie
Mirror
Pa
Pr
Devize
Pa
P+
Mirror Space
Primall Space
Figure I: The Overview of HyperPrism Framework. Device i executes local update on its own datasct. @ Device i adaptively generates degrees of po and P. for the representation and decision parts by hypernetworks. @ Device i maps the model the mirror space by raising the parameters with the degrec of power pe and Pp. respectively. @ Device communicates with its neighbors exchanging local models. @ Device aggregates received models mirror space with WPM. Device inverses the model primal space by the degree of power pa and pp, then finishes the round.
requirements (e.g., low probability of detection). HyperPrism applies to generic DML, the parameter (t) eij is determined by underlying communication systems, which consider both feasibility and security
server and all-reduce frameworks can be seen as special cases (where all di
(1) values equal )
In summary, the core of HyperPrism is that at every round t, device i computes its local model for the next round +1 1 as:
m
(t-1)
[
-
where VΦ is the mapping function with a selected degree of p for each round.
4.2 Adaptive Degree of Power p Mapping
To determine the p of various layers at each round, HyperPrism establishes relationship between the optimization problem and the degree of p by utilizing hypernetworks, which adaptively choose the optimal p for different layers to achieve the best performance.
Without loss of generality, HyperPrism also decouples the local model into the representation and decision parts. The representation part 0 includes components like convolutional and embedding layers. The decision part yp includes components like fully-connected layers. Thus, for device i, we have w; {wa,, ). Each device i holds local hypernetwork HN, and a randomly generated embedding vector v, Every hypernetwork consists of several fully-connected layers and employs the softmax for the output layer. HN, takes i and gradient of the local model as input, then output p; = {po, } ) for wa,; and W., parts, respectively. Then the mapping function evolves to Vo(w;) = {(oo,))o. (Wpp)) Po.c). The hypernetwork on device can be defined as
pt HN: sht).
where v/s denotes the parameters of HN,, Hence, a new objective function can be derived from the original problem as
min F(w) = Σ (
(8)
HyperPrism can transform the optimization problem for model parameters w, into the HN's and v/z. HNs adaptively output po,, and Pe..i based on the input, and simultaneously update both u and v by gradient descent at each round. Specifically, the gradient of i; and a/; can be computed based on
the chain rule (38] as
Vo,fi = (V,, wi) V Vi.
(9)
V Vo,fs = (V. W. V C. f.
(10)
5

---

Algorithm HyperPrism Framework.
1: Input: datasets (Di; D2...., Dm.. learning rate the number of Rounds T.
2: Output: the final model all devices after T rounds. w(T) Li w (T)
um )
3: Initialize all devices' models w("), hypernetworks v5(0), and embedding vectors v(0)
4: for t = 0 to T do
5: for each device in parallel do
6: Local Update:
w(t) (e) (t)
wi
(()) < wi
7V f
7: Compute Pa. and Pip by HN, (1)))
Compute Vo(w,") (0) + (Wwg pbu, (w p3) Pn).
8:
Send Vo(w!)) and N²(") to i's neighbors.
9:
Receive Vo(w)) ( and N,(0) from neighbors.
10:
(1) Compute weights aiy as Equation
11:
12:
Aggregate received models using dig
Update wi (t+1) according to Equation 6
13:
Update vf ((tt1), vi v,(++1)) as Equation
15:
end for
end for
return w!")
17:
Then, U, and t/ can be represented as
(t+1) -1 - (( 1110) ((+1) w/! n/
(11)
( v. 5
(12)
Algorithm Tantthmmmemmme demonstrates the full procedure of HyperPrism. our real deployment, we adopt the asynchronous protocol.
5
Analytical Results
We rigorously analyze the properties of HyperPrism. We fully introduce all assumptions, analyze the convergence behavior of HyperPrism, and finally compare HyperPrism to previous works.
5.1
Assumptions
Assumption 5.1 (Connectivity). The network graph G(() (V, (V,)()()) and the connectivity weight
matrix ( satisfy the following:
(1) doubly stochastic for all 1; that Σ Dij
(t)
1 and Σ dij
(t)
There exists scale 0, such that
(t) d C for all and 1, {i,j) Et.
There exists an integer B > 1 such that the graph (V, ExB41 U U... E(++)) is strongly connected for all 2
Definition 5.2 (Bregman Divergence). The Bregman Divergence of function is defined
Ds(2,v) o(x) o(y) (Vo(y), v).
(13)
Note that for ((x) | (1, y) || y/
Definition 5.3 (Uniform Convexity). Consider differentiable convex function RR R, an
exponent > 2, and constant 0. Then, á is (a, -uniformmyy convex with respect to ||-|| norm if for any e Rd
(xx) 2 )(v) + (Vo(v), x 4) + o | yll.
(14)
r

---

Note that for r = 2, this is known as strong convexiry. This assumption also implies that Dz : 2 "/l".
Assumption 5.4 (Smooth Gradient). Assume that the functions fi are convex with its gradients
(-) satisfying L-Lipschitz continuity [59], namely:
V (z()) fi(y)ll s L| vll.
(15)
for all x, y pairs.
This final assumption is used in nearly every mathematical analysis of DMI., including In our analysis, is used to bound the distance between local models.
54
52 59
43 51 49
5.2
Weighted Power Mean
In our experiments, we focused on the weighted power mean, generated by using ((x) 1
p+1 || ||
which gives Vε(x)) xº. Such á is uniformly convex seen in Proposition 5.5 and the rest of the analysis will be generalized to all uniformly convex Φ. Proposition 5.5 (Uniform Convexity of Power Functions). For 2 the function (p((x)
- ||=|| | is uniformly convex with degree p + 1. This is because [[)pp Vup(y))l = 1|
ye1 2--1 1 ---/l yll. and as a corollary we have
(2, v) 2
y/
2F-1 Pp - yy)
(16)
Given the function o, HyperPrism instructs local models take WPM of received models. Using the Connectivity Assumption (Assumption) 5.1h. we can bound the distance between local models.
Lemma 5.6 (Consensus). Under Assumption 5.7 for each device i, after t rounds:
(Vo(
V ( )| mnGi 2n))))
(17)
S 0(
where I- -(1 -1-15
(1-
c Im?
G 2L mmax f(w rr) and Ç is constant
1m?
K
related to the graph connectivity.
Theorem 5.7 (Convergence Behavior). Consider -p-r,p+17 uniformly convex o and the sequence
w; (1) under Algorithum| with constant step size n. Then, under Assumptions 5.1 and
5.4 if x is the
value that minimizes F (w) = f((w), then
min (w)(t)) F(=")]
4mGi
1
mnGi Vo(w ( 1- + 2n)11)
(18)
G + m Do(x", w()) + m p 2
P
Tf
In this Theorem 5.7 the first term uses Lemma 5.6 and is caused by the differences between local models held at different devices. This is the effect of the diverging forces mentioned in earlier sections. Note that i and x are constants related the graph connectivity, reflecting the influence of time-varying communication links. The second term represents the error caused by a non-zero learning rate in the Distributed Mirror Descent process, and the third term represents the lingering effects of the initialization. For more proof details, please refer to the Appendix.
Corollary 5.8. The bound on HyperPrism's (Algorithm 7 loss O(m 7/mm
The first term becomes 0(m {/mr)) because since 1, the x² term goes second term is ((m (/T) is smaller than the first term, we drop it. The final term is clearly O m
t gets large. The
Corollary 5.9. For = O( etV TP the upper bound becomes O(m 7/mm + m
77 0
+1
m
T
7

---

5.3
Comparison to Previous Work
Table I: Convergence Rates in terms of p. and T
Framework
error
optimal
Recovered
HyperPrism
O(m Vrm + m
O P
NA
Bubeck 3 (m 1, 2)
O(n .
Yes
Srebro 52 (m = 1)
On1 + 77))
O
Yes
Nedic 44 (GD
2)
O m
Yes
Yuan 59
2MD)
O m7
mym
Yes
Table summarizes the big O notation convergence rates of (( Previous works also assume bounded gradients. Our analysis recovers the same bounds the state-of-the tttthhe-rtttt in DML and single-device mirror descent with generic uniform convexity assumptions. Nedic 44 is standard gradient descent, and thus has 6(x) = ||<=||² which is (1,2)_uniformly convex, corresponding to
p 1. Yuan [59] considers distributed mirror descent under strong convexity (r = 2) equivalent to
p = 1, and based on their analysis, the bound should ( + 77
Evaluation
6.1 Experimental Setup
The experimental platform consists of 8 Nvidia Tesla T4 GPUs, 4 Intel XEON CPUs, and 256GB of memory. All the models and training scripts are implemented RAY 44) and PyTorch [24].
HyperNetworks Setup. We construct a hypernetwork model comprising three fully-connected layers and two additional output layers activated using softmax. The outputs of the fully-connected layers are fed into each of the two output layers to generate the degree of P for various parts.
Time-varying Communication Links Setup. We employ the NS3 platform [3 to simulate realistic vireiigrying communication environments consisting of multiple distributed devices. Each device is configured with the WiFi 802. la protocol and communication among themselves in Ad-Hoe mode. To quantify the degree of connectivity of the communication links, we define the topology density as the ratio of available tunnels to the total tunnels.
Models and Dataset. We use MNIST (22 and CIFAR-10 [2] datasets distributed among devices in non-IID settings. We construct two models based on layer functionalities. The Logistic Regression (LR) 14] model consists solely of linear layers used for the MNIST dataset. On the other hand, the CNN model consists of both convolutional and colleceonnected layers used for CIFAR-10. 10.
Non-IID Data Partitioning. To distribute datasets in a non-IID fashion, we employ Dirichlet distribution 35 to allocate all samples among devices. The Dirichlet Degree is used to control the non-IID degree. The a = 0.1 represents the extreme scenario where each device possesses samples from only one class, while and realistic training environment.
10 equals the IID scenario. These distributions reflect a challenging
Metrics. We consider two metrics to measure the performance of HyperPrism.
Average Accuracy. We evaluate the performance of the local model per device using a global test set that contains samples with all categories. The average Top-1 accuracy of all devices calculated each round to measure overall performance and convergence rate.
Convergence Speed. We track the loss of each round and the number rounds to investigate the rounds needed reach the convergence point for specific accuracy and loss.
Baselines. We conduct a comparative analysis of HyperPrism with state-of-the-art methods for DML in non-IID data and vime-varying communication links, including p = 1 methods, which is of the most influential and widely applied works in DML studies. SwarmSGD [41], Mudag [57] and ADOM (20.. To ensure a fair comparison. we made minor adjustments to each baseline method.
p. We define it as a class of methods using linear aggregation function, including DPSGD R4,, and variants 43 45). These methods can be seen as special cases of HyperPrism without the mirror mapping process.

---

SwarmSGD. We set the number of local SGD updates equal to 1. where the selected pair of devices performs only a single local SGD update before aggregation. can also be viewed as special case of HyperPrism where the topology density is very low.
ADOM and Mudag, We set the condition number k to 10, and the number of features d equals the number of classes. The gossip matrix (tt) at round is chosen to be the Laplacian of time-varying communication links divided by its largest eigenvalue.
Hyperparameters. For all experiments, the learning rate and batch size are both fixed at 0.01 and 128. We generate vareing communication graphs with different sizes and densities and evaluate 100 rounds total. The graph changes every round.
6.2 Experimental Results
LR+MNIST
CNN&CIEARTO
Rds
Total
DPSGD
8.36
Pr6
SvamSGDD
0.871
4.53
4134.32
Madis
43.05
9.13
ADOM
L,55
38:45
8.96
inf
Maximum Point
(WisoWt HN)
Ours(With HIN)
3.345
5.064
41.721.1
30.38r,1
14.08
33.43
2074.79(
1974.76()
Round
Degre
(a) Accuracy Curve
(b) Max Accuracy
(c) The Comparison of Time Cost
Figure 2: The impact of different and time cost.
Impact of different degrees of p. We present the impact of varying degrees of p in HyperPrism on model performance in Figure 2 In Figure 2 a), is evident that the model converges more swiftly and attains greater accuracy as p grows larger. Figure 2 (b) demonstrates that accuracy exhibits a significant fluctuation with different p. This underscores the substantial impact of the value selection on the performance of HyperPrism. These findings highlight the importance of choosing an appropriate value for p in HyperPrism to achieve optimal performance.
Comparison of time cost. In HyperPrism, each hypemetwork contains only 3 linear layers with 64 nodes per layer to ensure a minimal extra computational resource cost. We record the time cost in Figure c)) Although HyperPrism does result in higher time cost per iteration, it notably decreases the total number of rounds required for convergence, thereby reducing the overall time needed to achieve specific accuracy.
Performance HyperPrism. To showcase the practicality of HyperPrism. we consider the basic configuration with Dirichlet = 0.1, density = 0.5, and m = 50 devices. The accuracy results for all benchmarks and HyperPrism are presented in Figure The convergence speed summarized in Table 2 4 Notably, HyperPrism outperforms all benchmarks across all models. It demonstrated remarkable superior performance over state-of-the -he-ttr-rrt tht-art baselines with convergence accuracy and convergence speed improvements of up to 4.87 % and 98.63%, respectively.
Impact of non-IID. To investigate the impact of the non-llI) Dirichlet degree HyperPrism, Figure and the non-IlD degree becomes more extreme.
experiment with various a = 0.1, 1, 10. The corresponding results are presented in
Table All methods exhibit poorer performance which aligns with common intuition. However, HyperPrism demonstrates enhanced stability and faster convergence speed, especially at highly non-lID degrees.
Accumryy
Accunicy
ADOM
Dlpectrii
AIXOX
Hypertrier
AbooM
Hypertier Modag
ADOM
Hypertrier
SWamSSCD
Mudag
SwamSGD
SwamSCD
SwamSGD
Modag
DPSGD
100
Round
Round
Round
Round
(a) LR, 10
(b) LR, Dirichlet=10 (c) CNN. Dirichlet=0.1 (d) CNN. Dirichlet=10
Figure 3: The Impact of non-lII Degrees

---

Table 2: Comparison Different non-IID -ID Degree
LR MNIST
CNN a Cifar-1
Method
Dirichlet 0.1
Diricheer =1
Diricher 10
Dirichler
0.1
Dirchler
Dirichler
10
Max Ace
Come: Rds
Max Acc
Cony, Rds
Max Ace
Com: Rds
Max Ace
Cone: Rds
Max Acc
Cony, Rds
Max Acc
Coev, Rdk
SwarSGGD
84.77
164
57.01
S0.74
18
18.24
298
01.21
150
72.22
14
DPSGD
71:51
186
77.00
173
71,50
131
18.04
187
43.3
282
69.2
92
Madag
8664
44
58.03
20.15
83
64.75
70.79
ADOM
50.58
11
90.88
10
50.06
23
10.07
11,45
86.
12.32
t5f
90.61
6
90.30
90.60
20.45
86
53.26
169
75.74
15
Ours
(1 70%)
96.777)
16.005)
(1 95.93%)
26.65%) (4 (0.866
13.333)
20.69%)
(4 900055)
(.433%)
85.80%)
Table 3: Comparison on Different Connection Densities
MNIST
CNN
Method
Dannsity 0.2
Famnsty -0.5
Denstyy
Temsity
Dirnsity
Density
Max
Couw, Rds
Cons, Rds
Max
Con:. Rds
Max
Come,
Coow. Rds
Max ACC
Cons. Rds
SwarMGGD
82.58
162
63.75
104
62,79
114
17.57
212
18:24
198
17.5
201
DPSGD
21.37
239
71.01
245
71.05
252
16.22
364
18.04
157
19.00
110
10.17
ADDM
00.1
90.58
00.97
55
10.11
11:96
125
11.71
inf
90.58
00.61
90.00
5
20.17
107
20
65
21. :8
64
Ous
60%) 48.4%)
767)
97.9671
20.52%)
98.0110)
24.35%
70.906)
13.36%
08%)
(1.66.
(..115%)
Table 4: Comparison on Different Device Numbers
MNSST
CNN Cilar
Method
Wurld-Sire
20
World Size 50
World- Size
100
World Size
20
World Size
50
World Size
100
Max ACC
Com., Ril
Max ACC
Comy Rei
Max ACC
Com RdS.
Max ACC
Coay: Rds
Mix ACC
Coow. Rds
Max ACC
Corr: Rds
SwumSGDD
$4.74
93
83.75
194
81,54
116
22.58
132
15.21
198
14.00
201
DPSGD
50.65
137
71.57
22.66
15:01
157
Mudag
88.68
10
96.4
44
84.13
81
30,68
31
20.16
56
11,46
ADOM
20
06.77
11:36
fx.
10:11
60.
01
90.75
20:45
19.32
Ours
11 16%) ( 105 1013)
(1 26 70:11
97
9093
20.
7903)
(1 10.1000]
38
8335
(4 (0.022))
(1 13:35/5
[1 58 08%)
(1 31.78
(1.6775)
Scalability. We further evaluate the performance of HyperPrism with varying scales m e (20, 50, 100). The results are summarized in Table 4 can be noticed that most of the base- line methods exhibit deteriorating performance as the number of devices increases. The ADOM even barely converges under 100 devices. In contrast, HyperPrism is minimally affected by the scale and maintains superior acceleration and model performance.
Communication Graph Densities. To further analyze the impact of connected densities model performance, we present the performance and convergence speed of HyperPrism with various densily E {0.2, 0.5, 0.8) in Table In the extreme non-IID case. the performance of baselines dete- riorates as the communication becomes denser. Particularly, ADOM exhibits significant fluctuations density of 0.8. However, HyperPrism maintains better performance across different densities. This can be attributed to the fact that as communication becomes denser, the information exchanged between devices becomes more complex. Given HyperPrism's resilience to non-IID scenarios, maintains good performance in such cases.
7 Conclusion
In this work, we studied the important problem of divergence forces in decentralized DML, due to data heterogeneity (i.e., non-lII data) and time-varying communication links. We propose non-linear class aggregation DML framework with adaptive Kolmogorov Means for aggregation to enhance the convergence speed and scalability. HyperPrism achieves superior performance in its dependence on the number of devices
improving from m m/m to m f/m, and achieving optimality in the limit p + 0o. We also conduct rigorous analysis and demonstrate that the loss bound of HyperPrism O((m T-T).
T ) ). In cases with few communication epochs (i.e., T < m), employing larger value of yields improvements over traditional linear aggregation. Our theoretical results are consistent with state-of-the-ar bounds in distributed gradient/mirror descent and single-device mirror descent, all under a general uniform convexity assumption. To verify the effectiveness of HyperPrism, we carry out extensive experiments that demonstrate that HyperPrism achieves a remarkable acceleration in convergence speed with improvements of up to 98 98.63%. Moreover, HyperPrism shows increased scalability.
10

---

Acknowledgments and Disclosure of Funding
We express our gratitude to Chengdong Ni for providing technical support. This work was supported in part by the Shanghai Municipal Education Commission grant 22024-119, the NSFC grant 62141220, and the Yunnan Key Research Program grant 202402AD080004,
References
[1] Sai Apama Aketi, Abolfazl Hashemi, and Kaushik Roy. Global update tracking: A decentralized learning algorithm for heterogeneous data. Advances in Neural Information Processing Systems, 36, 2024.
[2] Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization, 2022.
[3] Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and TrendyO in Machine Learning, -33-4):231-357,201
[4] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. Advances in Neural Information Processing Systems, 15:10572-10586 2022.
[5] Martijn De Vos, Sadegh Farbadkhani, Rachid Guerraoui. Anne-Marie Kermarrec, Rafael Pires, and Rishi Sharma. Epidemic learning: Boosting decentralized learning with randomized communication. Advances in Neural Information Processing Systems, 36, 2024.
[6] Thinh T. Doan, Subhonmesh Bose, D. Hoa Nguyen, and Carolyn L. Beck. Convergence of the iterates in mirror descent methods, 2018.
[7] Haizhou Du, Yijian Chen, Xiaojie Feng, Qiao Xiang, and Haoyu Liu, An efficient federated learning framework for multi-channeled mobile edge network with layered gradient compression. Computer Networks, 221:109517, 2023.
[8] Haizhou Du, Chaoqian Cheng, and Chengdong Ni. A unified momentum- based paradigm of decentralized SGD for non-convex models and heterogeneous data. Artificial Intelligence, 332:104130, 2024.
[9] Haizhou Du, Chengdong Ni, Chaoqian Cheng, Qiao Xiang, Xi Chen, and Xue Liu, FedSwarm: An adaptive federated learning framework for scalable aiot. IEEE Internet of Things Journal, 2024.
[10] Haizhou Du and Zhiyuan Yang. FedPrime: An adaptive critical learning periods control framework for efficient federated learning in heterogeneity scenarios. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 125-141. Springer, 2024.
[11] Nima Eshraghi and Ben Liang. Distributed online optimization over a heterogeneous network with any-batch mirror descent. In International Conference on Machine Leurning, pages 2933-2942. PMLR, 2020.
[12] Tomer Galanti and Lior Wolf. On the modularity of hypernetorkss. Advances in Neural Information Processing Systems, 33: 13:10999-10419, 2020.
[13] Thomas R Henderson, Mathieu Lacage, George F Riley, Craig Dowell, and Joseph Kopena. Network simulations with the NS-3 simulator. SIGCOMM demonstration, 14(14):527, 2008.
[14] David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression, volume 398. John Wiley & Sons, 2013.
[15] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personalization via model agnostic meta learning. arXiv preprint (90i::900.1220 2019.
[16] Anna Kolesárová. Limit properties of quasi-arithmetic means. Fuzzy Sets and Systems, 124(1):65-71, 2001.
11

---

[17] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In JCML, pages 5381-5393,, 2020.
[18] Dmitry Kovalev, Elnur Gasanov, Alexander Gasnikov, and Peter Richtarik. Lower bounds and optimal algorithms for smooth and strongly convex decentralized optimization over time-varying networks. NIPS, 34, 2021.
[19] Dmitry Kovalev, Anastasia Koloskova, Martin Jaggi, Peter Richtarik, and Sebastian Stich. A linearly convergent algorithm for decentralized optimization: Sending less bits for free! In ICAIS, pages 4087-4095. PMLR, 2021.
[20] Dmitry Kovalev, Egor Shulgin, Peter Richtarik, Alexander V Rogozin, and Alexander Gasnikov. ADOM: Accelerated decentralized optimization method for time-varying networks. In ICML, pages 5784-5793, 2021.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. IEEE, (6:11))227--233 2324, 1998.
[23] Jueyou Li. Guo Chen, Zhaoyang Dong, and Zhiyou Wu. Distributed mirror descent method for multi-agent optimization with delay. Neuroconputing, 177:643-653 2016.
[24] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006. /5704, 2020.
[25] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020.
[26] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-IID data. arXiv preprint arXiv: arXiv:1907.022 1907.02/8/ 2019.
[27] Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. Gs2p: a generative pre-trained learning to rank model with wverparrameterization for web-scale search. Machine Learning, pages 1-19, 2024.
[28] Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen, Shuaiqiang Wang, and Dawei Yin. Mpgraf: modular and pre-tr graphformer for learning to rank at web-scale. In 2023 IEEE International Conference on Data Mining (ICDM), pages 3393348. IEEE, 2023.
[29] Yuchen Li, Haoyi Xiong. Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: supervise pre-training for web search over heterogeneous learning to rank data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4437 4448, 2023.
[30] Yuchen Li, Haoyi Xiong, Linghe Kong, Shuaiqiang Wang, Zeyi Sun, Hongyang Chen, Guihai Chen, and Dawei Yin. Ltrgen: Large-scale scale graph convolutional networks- based learning to rank for web search. In Joint European Conference Machine Learning and Knowledge Discovery in Dutabases, pages 635-651. Springer, 2023.
[31] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Dejing Dou, and Guihai Chen. Meta hierarchical reinforced learning to rank for recommendation: a comprehensive study in mooes. In Joint European conference machine learning and knowledge discovery in databases, pages 302-317. Springer, 2022.
[32] Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical reinforced ranking. IEEE Transactions Services Computing, 2023.
12

---

[33] Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang. Guihai Chen, Dejing Dou, al. Coltr: aeii-superviee learning rank with co-training and over-paramettrrizaton for web search. IEFE Transactions Knowledge and Data Engineering, 35(12) 12555, 2023.
[34] Xiangru Lian, Ce Zhang. Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In NIPS, page 5336-5346. Curran Associates Inc., 2017.
[35] Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In ICML, pages 6654-6665, 2021.
[36] Jiahao Liu, Jiang Wu, Jinyu Chen, Miao Hu, Yipeng Zhou, and Di Wu, FedDWA: Personalized federated learning with dynamic weight adjustment. In International Joint Conference Artificial Intelligence, 2023.
[37] Zhonghao Lyu, Yuchen Li, Guangxu Zhu, Jie Xu, H Vincent Poor, and Shuguang Cui. Re- thinking resource management in edge learning: A joint pre-training and fine-tuning design paradigm. arXiv preprint arXiv:2404.0000 00836, 2024.
[38] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10092-10101, 2022.
[39] Marie Maros and Gesualdo Scutari. Decentralized matrix sensing: Statistical guarantees and fast convergence. Advances in Neural Information Processing Systems, 36, 2024.
[40] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging Al applications. USENIX Association, 2018.
[41] G. Nadiradze, A. Sabour, D. Alistarh, A. Sharma, I. Markov, and V. Aksenov. SwarmSGD: Scalable decentralized SGD with local updates. 2019.
[42] Angelia Nedic. Distributed gradient methods for convex machine learning problems networks: Distributed optimization. IEEE Signal Processing Magazine, 37(3):92-101, 2020.
[43] Angelia Nedic, Alex Olshevsky, Asuman Ozdaglar, and John N Tsitsiklis. Distributed subgradi- ent methods and quantization effects. In ICDC'2008, pages 4177-4184. IEEE, 2008.
[44] Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent opti- mization. TACON, 54(1):48-61, 2009.
[45] Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi. Decentralized gradient methods: does topology matter? ICAIS, pages 2348-2358. PMLR, 2020.
[46] Arkadij Semenović Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
[47] Alex Olshevsky and John N. Tsitsiklis. Convergence speed in distributed consensus and control, 2009.
[48] Shahin Shahrampour and Ali Jadbabaie. Distributed online optimization in dynamic environ- ments using mirror descent, 2016.
[49] Shahin Shahrampour and Ali Jadbabaie. Distributed online optimization in dynamic environ- ments using mirror descent. TACON, 63(3):714-725, 2018
[50] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik, Personalized federated learning using hypernetworks. In International Conference Machine Learning, pages 9489-9502. PMLR, 2021.
[51] Pranay Sharma, Prashant Khanduri, Lixin Shen, Donald J. Bucei, and Pramod K. Varshney. On distributed online convex optimization with sublincar dynamic regret and fit, 2020.
13

---

[52] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. On the universality of online mirror descent, 2011.
[53] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432-8442 2022.
[54] Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey distributed machine learning. CSUR, 53(2):1-33, 2020.
[55] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. arXiv preprint arXiv:2306. 11677, 2023.
[56] Ryan Yang, Haizhou Du, Andre Wibisono, and Patrick Baker. Aggregation in the mirror space (AIMS): Fast, accurate distributed machine learning in military settings. MILCOM 2022 2022 IEEE Military Communications Conference (MILCOM), pages 470-477, 2022.
[57] Haishan Ye, Luo Luo, Ziang Zhou, and Tong Zhang. Multi-consensus decentralized accelerated gradient descent. arXiv preprint arXiv:2005.0079) 2020.
[58] Zhan Yu, Daniel WC Ho, and Deming Yuan. Distributed randomized gradient-fres mirror descent algorithm for constrained optimization. IEEE Transactions on Automatic Control, 67(2):957-964, 2021.
[59] Deming Yuan, Yiguang Hong, Daniel WC Ho, and Shengyuan Xu. Distributed mirror descent for online composite optimization. TACON, 66(2):114 729, 2020.
[60] Jiakang Yuan, Bo Zhang, Xiangchao Yan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. Advances in Neural Information Processing Systems, 36, 2024.
[61] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-llD federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174-10183, 2022.
14

---

8 Appendix
8.1
Main Notations
We summarize the main notations in Table
Table 5: Summary of Main Notations
Symbol
Description
The size of edge devices
The total round
The number round
The learning
G(
The time-varying network topology graph consisting of V and The set of vertexes (i.c, edge devices)
(A(0))
The of directed time-changing edges between vertexes
Ar
The weight matrix the topology
The aggregation weights of device in device round
D.
The private dataset of device
The of all local models The model held at device i
(ww
Device i's local loss function
P(w)
F(w) > fi(w)
The aggregation step of device t:w (1) Vo) - ( Vo((f)))
o(w)
Define the function 6(w) p+1 ||w||p+
Ve(w)
mapping function Vo(w) w"
The adaptive degree the WPM
%
(c)
N
when i, connected, otherwise
N7
The number of devices neighboring device f-th round Vo((w,) = [(wr, (W..))
0,
HNN vb2
The Hypernetwork on device i
The parameter of HN; A scale related to connectivity Smoothness of gradient / Upper bound on gradient derived from The Bregman Divergence
Da(2,)) v)
function (f)
(a, ruuiffoolyy convex
Vol. (9) Σ Oi. Vol(n
(0 Vol- E Vo(u:"))
0,K
Constants related to graph connectivity, containing
(t,
A transition matrix (tt, s) A(t)A(ff 1)... + 1)A(s)
k
round number
8.2 Impact of WPM to parameters
We illustrate how HyperPrism leverages weighted power mean (WPM) to facilitate more efficient aggregation in Figure By examining the distribution of model parameters, we observe that the traditional linear averaging model (w/o WPM) has approximately 11.65% of parameters lying within the range of [-0.01, 0.01] after 80 rounds. In contrast, when utilizing WPM with degrees p and 15, only 1.67% and 1.21% of parameters, respectively, fall within the same range. This indicates that WPM enables the model to effectively preserve a broader range of features. Consequently, HyperPrism can extract more information from the model parameters during aggregation, leading to enhanced performance. These results underscore the effectiveness WPM in enabling HyperPrism to capture wider range of features and facilitate more informative aggregation.
15

---

Layer linetr, Round - 20
Layer linear, Round
Number
Zem Pounse
Zam Frenoo
Without Mapping
55 5 8888888338
www.-tttttt--ooti Parameter Distribution
Parameter Distribution
Layer lineer: Round 0
Laver linear, Round 20
Lover lincor. Kound 80
Number
Number
Zar Poutee
Zan Fration
Mapping with P-9
Parameter Distribution
Parameter Distribution
Parameter Distribution
Initial Parameter Distribution
linear, Round 20
linear, Round
Number
Mapping with P=15
Parameter Distribution
Parameter Distribution
Figure 4: WPM's impact on parameter distribution.
8.3 Proof under Uniform Convexity
To aid analysis, introduce the sequence y
(t) Note that the w!t)
and (d) sequences satisfy
m
Voly. Σ
ai V
(c)
Vo(w))")
(19)
Volw (o(w((+)))
(") v.l(..
(20)
Additionally, can define
V
Vo(w | Σ (0()
(21)
m
8.4 Proof of Lemma 5.6
Proof. Define transition matrix (1, 8) = A(L)A(11) A(s+1)(())) Then, under Assumption
5.1 Corollary 1 in [43 states that
[b(,))] 1 < OK'
(22)
where d and x are defined in the lemma statement.
We are able write out general formula for Velw!" (1+11))
V (++1)
o (t, k3]s (( (+))
(23)
-7 ((o(") {
kl;
+=*+1 1=1
as well as Vo(T()))
m
volw!") (0) = > m
Vo(
volwg(t) (b))
m
j=1 m
(24)
(fl(W/11) 1)
V (w (1)
m
16

---

It is known that for L-Lipschitz continuous function f, if x is the optimum of f, then:
2(((()))
Then, Volu!") Vo(wo!)) can be bounded by applying the Triangle Inequality and Equation (2)
Vo(w((tl) ( - Un (t- 13 (2(
jm1
(25)
8x 11 1,1 + 2nG4i.
j=1
Then, plugging in h 0 gives
((x(1) m
V Vdlw, (I + mnGi Σ (((- + 21/Gi)
(26)
m
S 0(x())
Σ
\ + mnG:
2nGi)
1 1 n
Finally, shifting down by gives the desired bound.
8.5 Proof of Theorem 5.7
Firstly, note that under Algorithm and Assumption 5.4 Weighted AM-GM gives:
(r 1)- orr-Im (mn)Grr + m - [1(()) 7(0+1)1p
2r. (7) opr-1m (mnGiirr (m - |1 g(tti19?))
(mnGir) (| g(ttiip))
(mn)i) || - 7(1+1111.
We may follow the main line of reasoning that proves Theorem 5.7
8.6 Main Line of Reasoning
Proof. We prove bounds for generic æ. Note that (T(1) = --- Σm h(w!")). Then, get:
(f(lw()) (1) f((z)] Σ V fV w0?" (0(0)=20) =z)
(n fio (1) v0(") (()))
(27)
m
m
(VVii()) wlth (12))
(1 V (w,(")) (1-1)) - x).
+
17

---

Cauchy's 's inequality can bound the first term. The third tern can be manipulated using Equation (2
Then, combined with f(((()) - f((w! + Gi|lo!
/ -w,""
get
m
77 (F(w()) F(x)) - (2 Σ nt: 111o 1
+ mnGGi
| - w(+
m
(oly V w(++1) x).
The factor of 2 comes from the 2nd term of Equation (7 added to the error from f((w()) 2
fu(w(") Gi|l - w -w:" ||. Note that the last term is equal to m (wo() - m((+1) )((++) - æ). This is also equal to m (z, w (ax w((++)) D& (t((11)) ()))) by the Triangle Inequality for Bregman Divergences. We also substitute Claim to replace the second term, so the value
T
(2
G Imm
T
(28)
(mn)Gi) + moll w(++1)1
+ m( & (x, : (()) Dz ml++1) D (To((+1) m()))).
But, by uniform convexity, D& (z7((1)1) T 2 o a| - wht) I|r and thus this also
S (2 Σ G |10 (t) - (l(0)l + r- -- v om (mr)Gr)
(29)
m( mmDo(II, (1))) Dola, w((+)))).
Then, taking the sum over T gives
T-1
T-1
(((((t)
2 F(x)] - Σ
G (ml! - ((()
1=0
1=0
1=1
1
T
(mnGi)"
r
om
ml mmmDDa Do (2, w. (0)) Do(r, w()))
(30)
T-1
2mn
Tm
(nGi) + Dola, w()
Dividing through by n/L' substituting - and r = p 1, and substituting r x" and Lemma 5.6 gives the desired result.
8.7
Weighted Power Mean Skew Correction
Theorem 8.1. Consider positive I; and as close to m
1 Then,
Σ
m
m
O
max (
1
(31)
m
p
This allows us see the weighted power mean as a way to decrease skew; this theorem is relevant the setting of Lemma 5.6
18

---

Write a; = 1 + e where all e; v 1 Hold constant. Then, for ar 22 m, it is true that R2 0.
The derivative of the average with respect to is
Eea
E m of e (Eea)))
p
Σ Eqg
and the derivative at =
Thus, we have the following first-order approximation:
p. m =,)))
m
E E
Σ C3Ti
Di
+
(32)
i==1
m
p
x
p-1
m
But we have Σ e, S Σ 1 L
5o
Σ
m
Zj
+
1
Σ m
(33)
m
p
The ( = term is bounded, so the error is proportional to Og+)
8.8 Linear Bound on Power Mean
As starter, we first prove useful lemma two real numbers with different signs. Note that all variables used in this subsection are generic ones not tied to the HyperPrism- based mechanism.
Lemma 8.2 (Two Numbers with Different Signs). Given any two real numbers 0. an odd integer p 2 1, and 0 S a S 1, we have
(axa (1 (1-ou a))")
1/8 z
a x + (1
a
yy.
(34)
2p
2p
Proof. Denote C and rewrite the right-hand side (RHS) of Equation (34 as Claæ + a)v) + - C)y. We first note that we can scale both x, y by by| such that =-1. 1. As such, we can
2p
simplify our proof to only focus on the case of y = -1 We will use shorthand A, and rewrite as
(C (ar + (1 2 a)w) + (1-C) -)'
(35)
= + -1
(1- -11-Co - Ca(l + 1".
We then consider two cases.
Case 1: < 2p 1. Since -C(( 11 -C((pp) = -a 2 -1, by Bernuuli's inequality, we have
+ 1))) = -1 pCo(x + 1). - S x + 1, which can be verified by casework on x 2 1 or r 1 1, then
(36)
Since
A < 11 a + 1) = oz + (1 -
(37)
We then finish this case by taking the power of 1/p on both sides.
Case 2: > 2p 1. We will denote the expression on the left-hand side and right-hand side of Equation (34) with LHS and RHS, respectively. First note that both sides of the inequality in Equation (34) are -1 when = 0 and y = -1. Next, we will show that the derivative with respect a is always larger for the LHS of Equation (4 when 2p 1.
LHS = +1 da
(38)
RHS = pC(z + 1). (Ca(z + 1) 1)p-1
da
(39)
1 (2 1) (Ca(æ + 1) 1
19

---

The LHS's derivative w.r.t is clearly constant. Also, the d RHS clearly has local maxima at 0, At 0, we clearly have that since x 2 2p we have
da
d LHS a- x" xp++ +1
da
(40)
(r 1) 1y-11
RH -
For 1, since
2 1 (x + - 1 2 0, we have
2p
d
LHS = x" +12 +120.000 (x)
da
(10++1)
((+
1
p-1
(41)
2p
de R11S|
Thus, for all < a < 1, da LHS 2 de RHS, and they are equal at a = 0, so we always have LHS 2 RHS for all 0 S a S 1 when 2p 1, and we are done.
Next, consider a list of real numbers F,, = I...., m. We let M min H; and U max We then prove the following lemma.
Lemma 8.3. Assume list of non-negative real numbers Ia I,..., m and p 2 Given ag. i i= 1, , such that Σ 0% = 1 and a; 2 O, we have
1/r
Σ
Σ
p P-lu
U.
(42)
P
Proof. The lower bound in Equation the upper bound, we have
42
is a direct result from the power mean inequality. As for
1/v
m
1/2
Σ ouo?
Σ
(43)
1/P
=UP-1
UP-1
U,
p
where the last step uses the generalized AM-GM inequality.
We now tie these two Lemmas together to prove Theorem 8.4 which the best of our knowledge is a novel inequality on weighted power mean.
Theorem 8.4 (Linear Bound on Weighted Power Mean). Assume a list of real numbers i 1. 1....,m m and a,, 1,..., m such that Σ ag = 1 and a, 2 0. For any odd integer 2 1, we have:
Σ
0ga (1 )MM
(44)
2p
2p
Proof. First, note that if all .z,S positive, Equation 44 holds from the lower bound of Lemma 8.2 Second, all x;S are < 0, Equation (44) holds from the upper bound Lemma 8.3 after flipping all the signs.
For the case where v,s have mixed signs, let Oppos; Oney be the sum of the corresponding as for the negative and positive elements in respectively. We have,
20

---

0,24
E app +
Σ )xop)
pos
neg
2(Qpos
Oi
22)a
Oneg
ai
P M)
Opon
Oneg
pas
neg
Opos
Oli
Opos
as Ws + p p- M)
(45)
2p
appax
2p
aneg
pos
2p
Jai
)as
2p
2p
2pd
pos
+
P
2p
Oneg
+
(2p
2p²
()(p
1)
)M,
where the first inequality comes from the n-variable same sign case in Lemma 8.3 and the second inequality comes from the 2-variable case Lemma (8.2] Here, each positive 2; weighted with
at least
2p a4 and each negative x; is weighted with at least 2n1² 2p² ag which equals to
2p-1
ao Z
2p Oli since Oneg S
Since a, M (M < 0 when 2,5 have different signs), turn the extra as terms into M:
m
2y M,
Σ
022 +
(46)
2p 1=1
2p
which completes our proof.
In addition to new lower bound HyperPrism, we can also prove an upper bound of the HyperPrism 2p 2p-1U following similar steps. Note that these bounds generic and can be applied outside of DML. These bounds are necessary for analyzing the benefits of using HyperPrism aggregation function in DML because they guarantee that the HyperPrism takes at least some minimum consideration of each x; term. This consideration is crucial because each device's local objective function is unique, and they must all be considered to optimize the overall objective function.
2p Ei= Ox32;
21

---

Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract provides a concise summary of the paper's key findings and contributions, allowing viewers to quickly grasp the main points.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discussed the additional computational overhead that Hypernetworks may introduce, and we provided some time consumption comparisons. However, further research is needed to investigate whether the computational resource consumption will limit the performance.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide comprehensive definitions of all theories and assumptions and presented complete proofs in the appendix.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed explanations of all experimental settings in the experimen- tal section and uploaded the source code and data for reference.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We utilized an open-source source dataset and provided the complete source code for
reference.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed description of the experimental setup in the evaluation
section.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide comprehensive experimental result and detailed explanations in the evaluation section.
8. Experiments Compute Resources
22

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient experimental information to reproduce the experiments, some detailed information are present in appendix due to space limitation.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect. with the NeurlPS Code of Ethics /ttpa///oooooooao///////o////aaa httpa//pps.......oooo.////cccela
Answer: [Yes]
Justification: Yes, the research in this paper satisfies all of the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research this paper focuses on foundational research and does not have potential negative societal impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk with this paper.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in the paper are properly credited, and the licenses and terms use are explicitly mentioned and properly respected.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are well documented and provided alongside the assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
23


Reviews:
[{"review_id": "AhBaHTtkaL", "reviewer": "Reviewer_qJvT", "summary": "This paper presents HyperPrism, a novel framework for decentralized machine learning (DML) that aims to address the challenges of non-IID data and time-varying communication links. The authors propose a non-linear aggregation method based on Kolmogorov Means and adaptive mapping functions, which they argue improves convergence speed and scalability compared to traditional linear aggregation methods. The paper includes theoretical analysis and experimental results to support their claims.", "questions": "See weaknesses.", "limitations": "Overall, the paper addresses an important problem in DML and proposes a novel solution with promising theoretical and empirical results. However, the experimental setup and clarity issues mentioned above make it just fall short of this venue’s bar.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\t**Novelty:** The use of non-linear aggregation with adaptive mapping functions is a novel approach to simultaneously address the challenges of non-IID data and time-varying communication in DML. \n\n•\t**Theoretical Analysis:** The paper provides a theoretical analysis of the convergence behavior of their proposed approach, which is a valuable contribution. \n\n•\t**Experimental Results:** The experimental results demonstrate somewhat promising improvements in convergence speed and scalability compared to few baseline methods.", "weaknesses": "•\t**Experimental Setup:** The experimental setup could be expanded to include more diverse datasets, models, and settings. The current experiments are limited to MNIST and CIFAR-10 with specific model architectures and the non-IID setting include only two extreme cases ($\\alpha=0.1$ and $\\alpha=10$). This would help to assess the generalizability of HyperPrism's performance improvements. \n\n•\t**Clarity:** The paper could benefit from improved clarity in some sections. For example, in section 3 the authors discussed HyperPrism without first introducing it. The motivation for using mirror descent and Kolmogorov Means was unclear. The connection between these concepts and the challenges of non-IID data and time-varying communication could be made more explicit. \n\n•\t**Hyperparameter Tuning:** The paper does not provide sufficient details on how the hyperparameters for the different methods were chosen. This makes it difficult to assess the fairness of the comparison. \n\n**Others minor issues:**\n\n•\tThere seems to be a missing index $i$ on $w$ in equation 1. \n\n•\tHeterogeneity in referencing in related work section. Some references seem to be typed manually (or at least they are not connected to any entry in bibliography).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R1-1: Experiments are limited to MNIST and CIFAR-10 and the non-IID setting includes only two extreme cases.\n\nResponse: The MNIST and CIFAR-10 are the most commonly used datasets in the DML field, and recent works such as [R1] have also chosen these datasets for verification. Moreover, due to space limitations, some experimental results have not been presented in figures, including various Non-IID settings (α = {0.1, 1, 10}), topology density, number of devices, etc. The proposed HyperPrism demonstrates superiority in both convergence speed and accuracy under various settings. Please refer to Tables 1, 2, and 3 in the experimental\nsection for details (lines 281).\n\nR1-2: The motivation for using mirror descent and Kolmogorov Means.\n\nResponse: Model Merging is a growing field, which in recent years has developed nonlinear aggregation methods of their own. This constitutes the ultra-low update frequency update solution, with one update post-training. For ultra-high frequency updates, i.e., once per gradient, linear aggregation is optimal. However, for this vast swathe of space between these frequencies, other solutions should emerge.  One motivation for using nonlinear aggregation is the decreased variance in the original parameter space. Our system is designed based on the thesis that combining gradients from different models is dangerous if the gradient is computed at vastly different sets of parameters. Thus, the main problem is synchronizing these sets of parameters. Kolmogorov Means let you tune this, for example, with weighted power means of $p=11$, all parameters go immediately to values near the maximum value 25 (by calculating).  Meanwhile, in DML, the primary challenges are data heterogeneity and time-varying communication links. Traditional linear aggregation struggles to address the model divergence stemming from these issues, which hurts performance. The proposed HyperPrism maps models to a dual domain to better align with the geometry of the objective function. It introduces a specific mapping function, $\\phi(w) = \\frac{1}{p+1} \\lVert w\\rVert^{p+1}$, transforming models as $w \\rightarrow w^p$. Then, the Kolmogorov Means method is applied to achieve nonlinear aggregation in the form of Weighted Power Mean (WPM), enabling HyperPrism to capture a broader array of features, thus making it particularly suitable for scenarios with data heterogeneity and time-varying communication links. We illustrate how HyperPrism leverages WPM to facilitate more efficient aggregation in the Appendix; please refer to section 8.4 for details (lines 451-459).\n\nR1-3: How the hyperparameters for the different methods were chosen to ensure fairness.\n\nResponse: We use the same basic hyperparameters, such as model structure, learning rate, batch size, optimizer, etc., for all baselines. For methods with unique hyperparameters, we also made targeted adjustments to ensure a fair comparison between all methods. Details are presented in the Baselines section (lines 266-275).\n\n[R1] Martijn De Vos, Sadegh Farhadkhani, Rachid Guerraoui, Anne-Marie Kermarrec, Rafael Pires, and Rishi Sharma. Epidemic learning: Boosting decentralized learning with randomized communication. Advances in Neural Information Processing Systems, 36, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TqiyxEAk1V", "reviewer": "Reviewer_5NnV", "summary": "This paper addresses challenges in distributed machine learning (DML) caused by non-IID data and \nunstable communication links. The proposed HyperPrism framework uses mirror descent and adaptive \nmapping to project models into a mirror space for better aggregation and gradient steps. \nIt employs adaptive weighted power means (WPM) for efficient model aggregation, significantly \nimproving convergence speed and scalability. The framework's analytical results show that HyperPrism \neffectively handles decentralized DML challenges, offering a robust solution for edge device data processing.", "questions": "How does HyperPrism perform compared to other non-linear aggregation frameworks or adaptive learning methods in similar scenarios? \nAre there specific benchmarks or datasets where it excels or underperforms?", "limitations": "The potential limitation of the proposed method is not discussed in this manuscript. It will be of great importance if the authors can give an insight to the potential readers by comparing the pro/cons of the proposed method compared to the latest works in this topic.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "a) The paper exhibits notable originality by addressing the dual challenges of non-IID data and time-varying \ncommunication links in distributed machine learning (DML). The HyperPrism framework introduces a novel \ncombination of mirror descent and adaptive weighted power means (WPM) for effective model aggregation. \n\nb) The quality of the research is high, demonstrated through rigorous theoretical analysis and comprehensive \nexperimental validation, providing strong support for the framework's claims. \n\nc) The clarity of the paper is commendable, with well-structured explanations and supportive figures and tables, \nalthough some dense sections could benefit from further simplification. \n\nd) The significance of the work is substantial, as it offers a robust solution to a pressing issue in DML, \nwith implications for improving the efficiency and scalability of edge device data processing. The results \nshowing enhanced convergence speed and scalability are valuable contributions to the field.", "weaknesses": "While the paper is strong overall, there are a few areas that could be improved. \n\nFirst, some sections are densely packed with technical details, which might be challenging for readers \nnot deeply familiar with the subject. Simplifying these sections or providing additional explanations \ncould enhance accessibility. \n\nIn addition, the paper could provide more comparative analysis with existing methods to highlight the \nspecific advantages and potential limitations of HyperPrism. However, due to the limited period of rebuttal, \nit is just optional and not necessary. \n\nLastly, a more detailed discussion on the computational overhead and scalability of the proposed framework \nin extremely large-scale settings would be beneficial. Addressing these weaknesses would strengthen the paper \nand its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R2-1: Some sections are densely packed with technical details, which might be challenging for readers.\n\nResponse: To the best of our knowledge, HyperPrism is the first non-linear aggregation DML framework that combines mirror gradient descent and hypernetwork techniques. Therefore, a comprehensive explanation of the technical and theoretical details would be necessary to help readers clearly understand our approach. We will make our best efforts to provide concise overviews and technical explanations in the revised version to increase readability. \n\nR2-2: Discussion on the computational overhead and extremely large-scale settings.\n\nResponse: We have conducted some experiments to compare the computational time, as presented in Section 8.3 of the Appendix (lines 446-450). The results show that HyperPrism requires more time per training round than the baseline methods. However, it significantly reduces the total number of rounds required for convergence, thereby shortening the overall time to reach a specific accuracy target. Regarding extremely large-scale settings, we conducted experiments with device sizes of {20, 50, 100} and the results show that HyperPrism is minimally affected by the number of devices and maintains excellent acceleration and model performance. Theoretically, larger scale settings will still maintain this robustness to the growth in the number of devices. Please refer to Table 3 and Section 8.9 for more details.\n\nR2-3: How does HyperPrism perform compared to other non-linear aggregation frameworks or adaptive learning methods in similar scenarios?\n\nResponse: We are the first step in this new non-linear aggregation field, meaning there is almost no similar approach available for comparison. However, the non-linear aggregation mechanism in HyperPrism is performed via a specific mapping function $\\phi(w) = \\frac1{p+1} \\lVert w\\rVert^{p+1}$. According to the theory of mirror descent, it can be replaced by any convex and smooth function. This observation opens up an interesting direction for our future work, we plan to explore the use of other unique mapping functions for nonlinear aggregation, aiming to further enhance the performance and capabilities of DML systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0mb7V2fqf", "reviewer": "Reviewer_uEh3", "summary": "Traditional DML methods are limited to (1) data heterogeneity and (2) time-varying communication links. In this work, they present a non-linear class aggregation framework HyperPrism that leverages Kolmogorov Means to conduct distributed mirror descent with the averaging occuring within the mirror descent dual space. The proposed method can improve the convergence speed up to 98.63% and scale well to more devices compared with the state-of-the-art, all with little additional computation overhead compared to traditional linear aggregation.", "questions": "See weaknesses", "limitations": "N.A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe speedup of convergence is satisfactory.\n2.\tThe theoretical discussion is sufficient.", "weaknesses": "1.\tThe authors claim that they use hypernetworks to predict p (since they use softmax, type might be int), the exponent of the mapping function. There is a little deviation from the common practice of hypernetworks, where they usually output more complex parameter weights, like the weight of classifiers (a 768 * 100 matrix, type is float). I would prefer the authors remove the part of hypernetworks, and simply introduce HN as a simple MLP.\n2.\tWhile HNs act as a crucial part of HyperPrism, the analyses of them seem limited. Since HN has softmax layers, I would be curious about (1) the value set of p and its influence, (2) the variance of p on each distributed machines when given different gradients, (3) the results of preset p (≠ 1) (like directly use the most outputted p of HNs rather than 1). I would suggest these ablation studies be added.\n3.\tWhile the performance of HyperPrism is excellent on the major experiments, the baselines used for comparison seems out of date (before 2021), it would be more reasonable if the authors could provide more recent baselines (in 2022 or 2023).\n4.\tThe results of the experiments are a bit confusing, especially in terms of improvement. For example, in the last column of Table 1, Conv Rds of the proposed method is 13, while the best baseline is 14, how is 85.86% computed ? It is computed with the least performed baseline? It would be better if the authors could provide detailed explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R3-1: The difference between Hypernetwork and MLP. \n\nResponse: You make an excellent point about HN and MLP. The HN itself is basically a simple MLP. However, in terms of parameter selection, there are fundamental differences between the two. First, the MLP learns a direct mapping from the determined input to the target values, treating the selection process as a prediction task. While HN models the parameter selection process as a learning problem. It is trained as a meta-model to predict the optimal parameters, based on an embedding representation of the local models. Second, the MLP only optimizes the model itself to find the best set of model parameters. In contrast, HN allows the model and the\nmodel embedding to be jointly optimized. The embedding vector has also been updated to better ”represent” the local models. So regardless of whether the HN outputs integers or matrices, the optimization process is quite distinct from simply optimizing an MLP. Moreover, the HN gradients are calculated based on the gradients of the local models (as shown in Equation 9), giving HyperPrism a better ability to capture the relationships between the loss function and the optimal P. Recent works such as [R1; R2] also considered similar HN-based optimization approaches, showing the potential benefits over MLP-based parameter selection.\n\nR3-2: The influence of P.\n\nResponse: Due to the space limitations, some of the ablation studies are only presented in the Appendix. In Section 8.2 (lines 437-444), we use the preset fixed P to study how various P values (${P \\in \\mathbb{N} \\mid 1 \\le P \\le 21}$) impact the model performance. The experimental results clearly emphasize the importance of choosing the appropriate $P$. This insight also inspires us to jointly optimize the selection of $P$ along with the performance of the model, and adaptively select the optimal $P$ during the training process.\n\nR3-3: The baselines seem out of date.\n\nResponse: Although existing baseline models may not be the most recent, they are still recognized as powerful and effective methods in extreme scenarios of data heterogeneity and time-varying communication links, and have been used as baselines in recent works[R3]. We choose these baselines to ensure that our method can be fairly compared with industry standards and highlight the innovativeness and advantages of our approach in utilizing non-linear aggregation to accelerate DML training. We believe these innovations can bring new\ninsight into the DML domain, and we will seriously consider exploring more novel baseline models in future work.\n\nR3-4: The confusion improvement in experimental results.\n\nResponse: We apologize for any confusion caused by the performance comparisons in the results. The percentage improvements reported are all compared to the D-PSGD method, which is one of the most influential and widely applied works in DML studies. It is important to note that our proposed method not only demonstrated superior performance over D-PSGD, but also outperformed more recently published methods like ADOM and Mudag with convergence accuracy and convergence speed improvements of up\nto 4.87% and 86.36%, respectively, still demonstrating significant advantage in convergence speed. This underscores the significance and contributions of our work, as it advances brand-new insights in this research area.\n\n[R1] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10092–10101, 2022.\n[R2] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pages 9489–9502. PMLR, 2021.\n[R3] Adel Nabli and Edouard Oyallon. Dadao: Decoupled accelerated decentralized asynchronous optimization. In International Conference on Machine Learning, pages 25604–25626. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their careful reviews and constructive suggestions. Our responses to the main issues are below：

Meta-review: This paper proposes a novel way of non-linear aggregation in distributed machine learning, with the use of a hyper network. The advantage of this method is the reduction of communication rounds especially in the presence of non-IID data and time-varying communication links, thereby improving the convergence speed. The convergence of the algorithm has been analyzed theoretically and its performance has also been validated in experiments. It would be useful if the theoretical convergence results can be discussed further, such as by comparing the convergence rate with linear aggregation algorithms and explaining whether/how the convergence bound captures the effect of non-IID data and time-varying communication links. In addition, as mentioned by the reviewers, the presentation of this paper could be improved.
Decision: Accept (poster)