20

Deep learning without data aggregation from nodes
Asynchronous consensus algorithm on distributed networks

Abstract
While use of massive data benefits on training DNN models, aggregating all data into one physical location (e.g. a clouddata center) may not be possible due to data privacy concerns from consumers. For example, according to EU GDPR, it ispreferable to minimize data transmission between processing nodes.
Our goal is to construct training algorithms to obtain a global DNN model that can be adapted to all data, even when
individual nodes only have access to different subsets of the data. We assume that this algorithm is allowed tocommunicate autonomously between nodes, exchanging information such as model variables or their update difference,but data are prohibited from being moved from node they reside on.
Now, several platformers provide advanced services by aggregating/monopolizing data. However, we aim to create asociety where data ownership belongs to individual and can be used for a variety of services while protecting data privacy.

Background, goal

Asynchronous consensus algorithm

Background: We are entering an era of distributed
aggregation of due to data volume, privacy protection andlegal regulations (e.g. GDPR).

Proposed algorithm: A training algorithm is constructed toobtain a global model by asynchronously exchanging primalmodel variables and Lagrangian dual variables among nodes.
Goal: To obtain a global DNN model without data aggregation (this enables to work on arbitrary network structure).
(where asynchronous communication among nodes, such asmodel variable exchange, is allowed).
Edge computing(composed of small data centers)

HealthcareMobility

Smartphone

FactoryHomeOfficeUpdate direction modified tomove towards a global model

Gradient

Global model

Consensus force
Node #1
Node #2
Node #3
Node #V

Proposed algorithm

Node #i(Primal-dual)
variables are updatedProblem
U

Problem: When the data at each node is statisticallyheterogeneous, a global DNN model cannot be obtained byjust minimizing each nod cost function.

X
Variables areexchanged

Approach: We solve problem by minimizing sum of costfunctions under a consensus constraint that all nodemodels are identical with each other.

X
Node #j

Update time

Data sets are placed across V nodes (x1,…,xV). Model variables are
updated (i) such that minimizes sum of cost functions (ΣFi) (ii) under aconsensus constraint that models are identical among V nodes (s.t....)

U

Node #1

U U U U X U U U U X U U U X U U U U U U U U X

Node #2U U U X U U U U U U U X U U U U U U U UNode #3U U U U U U X U U U U U U U U U U U X U U U UNode #VU U U U XUX U U U U U U X U U XReferences[1] T. Sherson, R. Heusdens, and B. Kleijn, Derivation and analysis of the primal-dual method of multipliers based on monotone operator theory,
IEEE transactions on signal and information processing over networks 5, 2 （2018）, 334–347.
[2] K. Niwa, N. Harada, G. Zhang, B. Kleijn, Edge-consensus learning: deep learning on P2P networks with nonhomogeneous data, submitted toKDD 2020.

Contact
Kenta Niwa Email: cs-openhouse-ml@hco.ntt.co.jp
Communication Science Laboratory, Media Intelligence Laboratory
オープンハウス 2020