2019 International Symposium on Nonlinear Theory and Its Applications
NOLTA2019, Kuala Lumpur, Malaysia, December 2-6, 2019.

New Rules for Choosing Values of Consensus Weights inDistributed Training of Neural Networks
Kazuaki Harada, Tsuyoshi Migita and Norikazu Takahashi
Graduate School of Natural Science and Technology, Okayama University
3–1–1 Tsushima-naka, Kita-ku, Okayama 700–8530, Japan
Email: harada@momo.cs.okayama-u.ac.jp, migita@cs.okayama-u.ac.jp, takahashi@cs.okayama-u.ac.jp
Abstract—A framework for distributed training of neural networks was recently proposed. In this framework,
each neural network computes its solution based on the partial knowledge about the training data and then updates thesolution by communicating with each other using some hyper parameters known as the consensus weights. In this paper, we propose new rules for choosing the values of theconsensus weights, and show experimentally that the proposed rules can achieve faster convergence.

rules is that larger weights should be assigned to those NNs
communicating with many NNs. We show experimentallythat the proposed rules not only are convergent like the conventional rules but also achieve a faster convergence speed.
2. Distributed Training of NNs
In this section, we review the distributed training of NNs
proposed by Scardapane and Di Lorenzo [5].
2.1. Problem Formulation

1. Introduction
Distributed optimization over multiagent networks has
attracted a great deal of attention in the past two
decades [1–4]. A typical situation considered in the literature of distributed optimization is that an objective function
expressed as the sum of ﬁnite functions is given to agents,
but each agent knows only one of them. In this situation,
agents have to ﬁnd an optimal (or a locally optimal) solution cooperatively through local optimization for their ownfunctions and local interaction between neighboring agentsto achieve a consensus. Many algorithms for distributed
optimization have been proposed so far, and proven to converge under certain assumptions.
Recently, Scardapane and Di Lorenzo [5] proposed aframework for distributed training of multiple neural networks (NNs) by extending the NEXT algorithm [2] developed for the distributed optimization. In this framework, multiple NNs with the same structure try to ﬁnd acommon set of parameter values that minimizes the loss
for a given training dataset, but the dataset is divided into
nonoverlapping subsets and each subset is given to only one
NN. Thus NNs cooperatively solve the problem through local optimization and local interaction. Scardapane and Di
Lorenzo [5] proved the convergence of their algorithms under some assumptions on the loss function, the graph representing the interaction among NNs, the consensus weights
used in local interaction, and so on, and showed someexperimental results. However, their assumptions on theconsensus weights seem too strong. If these assumptions
can be relaxed, we can choose the values of the consensus
weights more ﬂexibly to increase the speed of convergence.
In this paper, we propose new rules for choosing the values of the consensus weights. The basic idea behind these

Suppose that a given set S of training data is divided
into I nonoverlapping subsets, and each subset is assigned
to one of I NNs that communicate with each other. The
Ni
i-th subset of S is denoted by Si = {(xi,m , di,m )}m=1
wheredxi,m ∈ R and di,m ∈ R for m = 1, 2, . . . , Ni . Note thatthe i-th NN only knows Si and cannot access S j ( j , i).
We assume that all NNs have the same structure, that is,they have the same number of layers, the same number ofneurons in each layer, only one neuron in the output layer,
the same activation functions, and so on. Then the outputof the i-th NN for the input vector x ∈ Rd can be expressed
as f (wi ; x) where wi ∈ RQ is the vector composed of all
adjustable parameters.
Assumption 1 The NN model f (w; x) satisﬁes the following conditions: i) f is continuously diﬀerentiable withrespect to w, and ii) f is Lipschitz continuous with respect
to w, that is, there exists a Lipschitz constant L ≥ 0 suchthat ∥∇w f (w̃; x) − ∇w f (ŵ; x)∥2 ≤ L ∥w̃ − ŵ∥2 where ∥·∥2represents the Euclidean norm.
The distributed training of NNs is formulated as the following optimization problem:P I P I
minimize U(w) = 1I i=1
j=1 g j (wi ) + r(wi )(1)subject to w1 = w2 = · · · = wI
where w = (w1T , w2T , . . . , wIT )T ∈ RQI ,NjXg j (wi ) =ℓ d j,m , f (wi ; x j,m )m=1

with ℓ being a loss function, and r(wi ) is a regularization
term. Note that the i-th NN cannot compute g j (wi ) ( j , i)

- 17 -

because g j (·) is determined by S j and the i-th NN cannot
access it.
Assumption 2 The optimization problem (1) satisﬁes thefollowing conditions: i) ℓ is convex, continuously diﬀerentiable, and Lipschitz continuous, ii) r is either a) convex,
continuously diﬀerentiable and Lipschitz continuous, or b)
nondiﬀerentiable convex function with bounded subgradients, and iii) U is coercive, that is, lim∥w∥→∞ U(w) = ∞.
2.2. Communication among NNs
Throughout this paper, we assume for simplicity thatthe communication among NNs is time-invariant, though
the time-varying case is considered in [5]. It is thus expressed as a time-invariant directed graph G = (V, E)
where V = {1, 2, . . . , I} is the set of vertices, each ofwhich corresponds to an NN, and E ⊆ V × V is the setof directed edges. E contains ( j, i) if and only if i , j
and the i-th NN can receive information from the j-th
NN. The in-neighborhood and out-neighborhood of vertex i ∈ V are deﬁned as Niin = { j | ( j, i) ∈ E} ∪ {i} andNiout = { j | (i, j) ∈ E} ∪ {i}, respectively.
2.3. Algorithm
In the distributed training of NNs [5], each NN iterates
alternating two operations: local optimization and consensus update. Let wi [n] be the solution obtained by the ith NN in the n-th iteration, and π̃i [n] be an estimate ofPI
j=1, j,i ∇wi g j (wi [n]) (details will be explained later). In
the operation of local optimization, each NN ﬁrst solves
the optimization problem:
minimize g̃i (wi ; wi [n]) + π̃i [n]T (wi − wi [n]) + r(wi ) (2)where g̃i (wi ; wi [n]) is a strongly convex surrogate functionof gi (wi ) at wi = wi [n]. In this paper, we use a simple
surrogate function obtained from gi by linearizing it aroundwi = wi [n] and adding a proximal regularization term:g̃i (wi ; wi [n]) = gi (wi [n]) + ∇gi (wi [n])T (wi − wi [n])τ+ ∥wi − wi [n]∥22(3)2
where τ is a positive constant. Let w̃[n] be the solution of(2). Each NN then computes zi [n] byzi [n] = wi [n] + α[n](w̃i [n] − wi [n])(4)where {α[n]}∞n=0 is a sequence determined by α[n] = α[n −1](1 − ϵα[n − 1]) with α[0], ϵ ∈ (0, 1]. In the operation ofconsensus update, each NN computes wi [n + 1], yi [n] andπ̃i [n + 1] byX
wi [n + 1] =ci j z j [n],(5)j∈Niinyi [n + 1] =X

ci j y j [n] + ∇gi (wi [n + 1]) − ∇gi (wi [n]),j∈Niin(6)π̃i [n + 1] = Iyi [n + 1] − ∇gi [n + 1],(7)

Algorithm 1 Distributed Optimization for Problem (1)Input: gi , r : RQ → R, G, {ci j } j∈Niin , τ, α[0], ϵ
Output: wi ∈ RQ
1: Set wi [0] to a randomly chosen vector. Set yi [0] ←∇gi (wi [0]), π̃i [0] ← Iyi [0] − ∇gi (wi [0]) and n ← 0.
2: If the stopping condition is satisﬁed then return wi [n]and stop. Otherwise go to Step 3.
3: Set w̃i [n] to the optimal solution of (2) with (3) andcompute zi [n] by (4).
4: Compute wi [n + 1], yi [n + 1] and π̃i [n + 1] by (5), (6)and (7), respectively.
5: Set n ← n + 1 and go to Step 2.

respectively, where ci j are nonnegative constants which wecall the consensus weights in this paper. We should notePthat yi [n] and π̃i [n] are estimates of 1I Ij=1 ∇wi g j (wi [n])PI
and j=1, j,i ∇wi g j (wi [n]), respectively (see [5] for details).
A pseudocode of the algorithm is shown in Algorithm 1.
Proposition 1 (Simpliﬁed version of Proposition 2 in [5])Suppose that i) Assumptions 1 and 2 hold, ii) G = (V, E)
is strongly connected, iii) the consensus weights ci j arepositive and satisfy the following conditions:X
ci j = 1, ∀i ∈ {1, 2, . . . , I},(8)
j∈NiinXc ji = 1,∀i ∈ {1, 2, . . . , I},(9)j∈Nioutiv) {α[n]}∞is chosen so that α[n] ∈ (0, 1] for all n
P n=0
and ∞
α[n]
= ∞. Then all limit points of the sen=0
quence {(w1 [n]T , w2 [n]T , . . . , wI [n]T )T }∞n=0 generated by
Algorithm 1 are stationary solutions of (1).
3. How to Choose Values of Consensus Weights
In this section, we focus our attention on how to choose
the values of consensus weights ci j . We hereafter assume
for simplicity that the communication among NNs is symmetric, that is, (i, j) ∈ E if and only if ( j, i) ∈ E.
3.1. Conventional Rules
Simple rules often used in consensus algorithms [1] aregiven by
1/(δi + s), if j ∈ Niin \ {i},(10)ci j =  s/(δi + s), if j = i,where δi is the degree of vertex i in G and s ∈ {0, 1}. Notethat we have two rules here depending on the value of s.
Eq.(10) clearly satisﬁes (8) in both cases where s = 0 ands = 1, and hence the right-hand side of (5) represents thearithmetic mean of {z j [n]} j∈Niin when s = 1 and that of- 18 -

{z j [n]} j∈Niin \{i} when s = 0. However, Eq.(10) does not satisfy (9) in general. Therefore we cannot use Proposition 1
to show the convergence.
Scardapane and Di Lorenzo [5] did not use (10) but theMetropolis-Hasting rule:
if j ∈ Niin \ {i},1/(max{δi , δ j } + 1),
ci j = P1 − j∈N in 1/(max{δi , δ j } + 1), if j = i,i
(11)in their experiments. It is easy to see that (11) satisﬁes ci j =c ji , (8) and (9). Therefore, by Proposition 1, all limit pointsof the sequence of {(w1 [n]T , w2 [n]T , . . . , wI [n]T )T }∞n=0 generated by Algorithm 1 are stationary solutions of (1).
Sun et al. [3] claimed that (8) is not needed to prove theconvergence and proposed the following rules:
1/(δ j + s), if j ∈ Niin \ {i},ci j = (12) s/(δ j + s), if j = i,where s ∈ {0, 1}1 . It is easy to see that (12) satisﬁes (9) inboth cases where s = 0 and s = 1. In particular, when s =0, the values of ci j for j ∈ Niin are inversely proportional tothe degrees of vertices j.
3.2. Proposed Rules
Among various conditions on the consensus weights in
Proposition 1, (8) is reasonable because it makes the righthand side of (5) a convex combination of {z j [n]} j∈Niin . On
the other hand, the role of (9) is not clear, though it isneeded to the proof [5] of Proposition 1. In fact, it has
been proven [4, 6] that some distributed optimization algorithms converge even if (9) is violated. Taking this fact into
account, we propose the following rules:

P
in


δ j /( j∈Niin \{i} δ j + s), if j ∈ Ni \ {i},(13)ci j = P s/( j∈N in \{i} δ j + s), if j = i,iwhere s ∈ {0, 1}. Like (10), Eq.(13) satisﬁes (8) but neither
ci j = c ji nor (9) in general. The diﬀerence between (10)and (13) is that ci j for j ∈ Niin \ {i} take the same value inthe former while the values of ci j for j ∈ Niin \ {i} are proportional to the degrees of vertices j in the latter. The idea
behind (13) is that vertices with higher degrees may havemore information than those with lower degrees. It is thus
expected that NNs reach a consensus faster by assigninglarger weights to NNs with high degrees.

Figure 1: Communication among 10 NNs.
4.1. Setup
We use two datasets called Boston (or Boston House
Prices) and Wisconsin (or Breast Cancer Wisconsin)2 ; theformer is a regression problem and the latter a classiﬁcation
problem. Their characteristics are summarized in Table 1.
In all experiments, the original data are normalized so thatboth inputs and outputs lie in the interval [0, 1]. Also, eachdataset is divided into 10 nonoverlapping subsets and they
are assigned to 10 NNs that can communicate with eachother according to the undirected graph G shown in Fig. 1.
Each NN has three layers: the input layer with d neurons, the hidden layer with 10 neurons and the output layer
with a single neuron. The output neurons of NNs for
Boston dataset have the sigmoid activation function andall other neurons have the hyperbolic tangent activation
function. We use the same loss functions and the regularization term as [5], that is, the squared loss functionℓ(a, b) = (a − b)2 for Boston dataset, the cross-entropy loss
function ℓ(a, b) = −a log(b)−(1−a) log(1−b) for Wisconsin
dataset, and the squared regularization term r(w) = λ2 ∥w∥22
for both datasets. The parameters of NNs are initialized byrandom numbers drawn from the standard normal distribution. The hyper parameters are set as τ = 101.5 , λ = 10−1 ,α[0] = 0.001 and ϵ = 0.01 for Boston dataset, and τ = 10,
λ = 10−0.5 , α[0] = 0.0001 and ϵ = 0.01 for Wisconsin
dataset. These values are determined based on preliminary
experiments. All algorithms are implemented in Python
3.6.6 and executed on a PC with Intel Core i5-4590 and
8GB RAM.
4.2. Results

4. Numerical Experiments
In order to clarify the relationship between the rule forchoosing the values of the consensus weights and the convergence property of the distributed training, we perform
experiments using some benchmark datasets.
1 Strictly speaking, they proposed only the rule (12) with s = 0.

Table 1: Datasets used in experiments.
PI
DatasetTypedj=1 N jBostonRegression
13
506
Wisconsin Classiﬁcation 9689

We ﬁrst compare the seven rules presented in the previous section in terms of the convergence speed of the objective value of (1). The time evolution of the objective value
is shown in Fig. 2, where the rules (10), (11), (12) and (13)2 Both datasets are available at UCI Machine Learning Repository(https://archive.ics.uci.edu/ml/datasets.html).

- 19 -

Simple (s=1)
Simple (s=0)
MH
DIP (s=1)
DIP (s=0)
DP (s=1)
DP (s=0)150

Simple (s=1)
Simple (s=0)
MH
DIP (s=1)
DIP (s=0)
DP (s=1)
DP (s=0)102

disagreement

objective value

200

100

50

100
-2

10

10-4
-6

10

0

0

50

100

150

200

250

300

epoch

350

400

450

10-8

500

0

50

100

150

(a)

200

250

300

epoch

350

400

450

500

450

500

(a)
104

Simple (s=1)
Simple (s=0)
MH
DIP (s=1)
DIP (s=0)
DP (s=1)
DP (s=0)

Simple (s=1)
Simple (s=0)
MH
DIP (s=1)
DIP (s=0)
DP (s=1)
DP (s=0)102

disagreement

objective value

104

103

0

10

10-2
10-4
10-6

2

10

0

50

100

150

200

250

300

epoch

350

400

450

10-8

500

0

50

100

150

200

250

300

epoch

350

400

(b)

(b)

Figure 2: Time evolution of the objective value of (1) for(a) Boston dataset and (b) Wisconsin dataset.

Figure 3: Time evolution of the average disagreement for(a) Boston dataset and (b) Wisconsin dataset.
References

are denoted by Simple, MH (Metropolis-Hastings), DIP(Degree Inversely Proportional) and DP (Degree Proportional), respectively. It is easy to see that all rules decrease
the objective value in a similar way for both datasets andthere is no signiﬁcant diﬀerence among seven rules.
We then compare the seven rules in terms of the convergence speed of the average disagreement: D[n] =
1 PI
1 PI
i=1 wi [n] − I
j=1 w j [n] 2 where I = 10. The timeIevolution of the average disagreement is shown in Fig. 3.
As one can see, DP is the fastest, Simple is the second, MH
is the third, and DIP is much slower than these three rules.
This indicates that the proposed rules allow NNs to quickly
reach a consensus. It is also important to mention that DPwith s = 0 is faster than that with s = 1, and the same
holds true for Simple. This indicates that each NN should
exclude its own solution when computing the weighted average of the solutions in its neighborhood.

5. Conclusions
We have proposed new rules for choosing values of consensus weights in the distributed training of neural networks, and shown experimentally that the proposed rules
can achieve consensus faster than conventional rules. Evaluating the performance of the proposed rules on various
training datasets, and theoretical analysis of the convergence property are future problems.

[1] V. D. Blondel, J. M. Hendrickx, A. Olshevsky and
J. N. Tsitsiklis, “Convergence in multiagent coordination, consensus, and ﬂocking,” Proceedings ofthe 44th IEEE Conference on Decision and Control,
pp.2996–3000, December 2005.
[2] P. Di Lorenzo and G. Scutari, “NEXT: In-network
nonconvex optimization,” IEEE Transactions on Signal and Information Processing over Networks, vol.2.
no.2, pp.120–136, June 2016.
[3] Y. Sun, G. Scutari and D. Palomar, “Distributed
nonconvex multiagent optimization over time-varying
networks,” Proceedings of the 50th Annual Asilomar Conference on Signals, Systems, and Computers,
2016.
[4] A. Nedić and J. Liu, “On convergence rate ofweighted-averaging dynamics for consensus problems,” IEEE Transactions on Automatic Control,
vol.62, no.2, pp.766–781, February 2017.
[5] S. Scardapane and P. Di Lorenzo, “A framework forparallel and distributed training of neural networks,”
Neural Networks, vol.91, pp.42–54, July 2017.
[6] N. Takahashi and K. Kawashima, “A simple suﬃcientcondition for convergence of projected consensus algorithm,” IEEE Control Systems Letters, vol.2, no.3,
pp.537–542, July 2018.

- 20 -