Federated Inference through Aligning Local Representations and Learning a Consensus Graph
Authors: Tengfei Ma
Trong Nghia Hoang
Jie Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: Federated inference
local latent representation
feature alignment
graph structure learning
Gumbel softmax

Abstract: Machine learning is faced with many data challenges when applied in practice. Among them, a notable barrier is that data are distributed and sharing is unrealistic for volume and privacy reasons. Federated learning is a recent formalism to tackle this challenge, so that data owners can develop a common model jointly but use it separately. In this work, we consider a less addressed scenario where a datum consists of multiple parts, each of which belongs to a separate owner. In this scenario, joint efforts are required not only in learning but also in inference. We study \emph{federated inference}, which allows each data owner to learn its own model that captures local data characteristics and copes with data heterogeneity. On the top is a federation of the local data representations, performing global inference that incorporates all distributed parts collectively. To enhance this local--global framework, we propose aligning the ambiguous data representations caused by arbitrary arrangement of neurons in local neural network models, as well as learning a consensus graph among data owners in the global model to improve performance. We demonstrate effectiveness of the proposed framework on four real-life data sets including power grid systems and traffic networks.


Under review as a conference paper at ICLR 2022
FEDERATED INFERENCE THROUGH ALIGNING LOCAL REPRESENTATIONS AND LEARNING A CONSENSUS GRAPH
Anonymous authors Paper under double-bling review
ABSTRACT
Machine learning is faced with many data challenges when applied in practice. Among them, notable barrier that data are distributed and sharing unreal- istic for volume and privacy reasons. Federated learning is a recent formalism to tackle this challenge, so that data owners can develop common model jointly but use it separately. In this work, we consider a less addressed scenario where a da- tum consists of multiple parts, each of which belongs to a separate owner. In this scenario, joint efforts required not only in learning but also in inference. We study federated inference, which allows each data owner to learn its own model that captures local data characteristics and copes with data heterogeneity. On the top is a federation of the local data representations, performing global inference that incorporates all distributed parts collectively. To enhance this local-global framework, we propose aligning the ambiguous data representations caused by arbitrary arrangement of neurons in local neural network models, as well as learn- ing a consensus graph among data owners in the global model to improve perfor- mance. We demonstrate effectiveness the proposed framework on four real-life data sets including power grid systems and traffic networks.
INTRODUCTION
Machine learning models become increasingly data hungry as the promise of deep learning continues to realize. More and more applications grow scale thanks to the availability of distributed data across devices and organizations. Federated learning (McMahan et al.) [017] Yang al. 2019 Li
ct al. 2019, Kairouz et al., 2019) emerges as formalism that allows data owners to collaboratively train common model by using one's own data without sharing. Such formalism is poised to resolve the challenges of expensive data communication and the risk of privacy violation, in light of policies such as the General Data Protection Regulation (Albrecht) 2016).
An issue less addressed by federated learning is the inference process. In fact, inference therein is trivial: once the common model is learned, each data owner retains copy and applies it local data, independently of other owners. However, such a scenario is not the only one how data are distributed in practice. In this work, we consider the following scenario: datum has multiple parts, each of which belongs to a separate owner. Then, the inference must be collectively performed by all participating owners, since none of them alone possesses the entire information.
Vertical federated learning studies such a scenario (Hardy et al., 2017 Hu et al., 2019 Chen et al., 2020). This concept is figuratively named through cutting the data matrix vertically along the feature axis, rather than the data axis. From the sporadic literature addressing in this scenario, methods gen- erally introduce model parameters distributed with data parts and optionally global parameters that reside in central server. All parameters are learned jointly, causing however practical drawback- expensive coordination (even synchronization) is required among data owners and the central server (if present).
Consider live example- the national electricity grid, over which thousands of phasor measurement units (PMUs) have been deployed to monitor the grid condition (Smartgrid.gov) PMU measure- ments, as time series data, are owned by several parties. These data may be used to train machine learning models that identify grid events (e.g., fault, oscillation, and generator trip). Such an event

---

Under review as a conference paper at ICLR 2022
detection system relies collective series measurements the same time window but distributed across different data owners. To minimize coordination among owners and maximize autonomy, it is more desirable if each maintains model of their own and does not participate joint training.
In this work, we propose a local-global model framework that maintains data owner autonomy while staying effective in global inference. Therein, each data owner trains local model with its data part; the training is independent and incurs no coordination. In the deep learning terminology. the local models produce data representations for input data. Then, a central server takes these representations as input and trains model for global inference.
We term the scenario where inference is collectively performed by data owners but training incurs little coordination among them, federated inference, to distinguish federated learning and vertical federated learning. The local-global framework we propose for this scenario, however, bears two technical challenges. One is the ambiguity of local data representations, because feature dimensions can be arbitrarily permuted without changing the local model. Another challenge how the global model leverages innate interactions of local data missed by independent local models.
We resolve the first challenge through aligning the feature dimensions across all local represen- tations. We resolve the second challenge through employing graph neural networks as the global inference model, where the graph corresponds to the explicit or implicit relational structure of the data owners. When such graph is not present, we treat the combinatorial graph structure random variable the Bernoulli distribution and optimize the distribution parameters as well.
We summarize the contributions of this work follows.
We formalized federated inference, less addressed scenario of machine learning with distributed data, where inference is conducted jointly by data owners without data sharing and coordinated training (Section 2].
2. We propose inference framework that consists of autonomous local models and central model that digests local data representations and produces a global output (Section 33.
We address the ambiguity challenge of this framework through aligning local representations (Section 4 and address the missing of local model interactions through employing a graph neu- ral network in the central global model. We further propose to simultaneously learn the graph structure if not present (Section(5).
We study approaches to latent alignment and approaches to graph structure learning and develop
theoretical insights into these approaches (Theorems о As byproduct, more efficient Bernoulli sampling method icdf is proposed to sample graphs for structure learning.
5. We demonstrate experiments with four real-life data sets including power grids and traffic net- works and show the effectiveness of the proposed framework (Section 6)
2 PROBLEM SETTING
Federated inference refers to a machine learning scenario where both training and inference are con- ducted on distributed data collectively by data owners. Each owner enjoys data and model autonomy but subject to centralized coordination that produces a global prediction. This scenario stands in contrast to federated learning, whose inference process is local and separate among owners.
Formally, we use a superscript to index data owners. Let æ be a datum with label y and let x' be the part of datum the äth owner possesses; that is, (x x? ,-..,.") x²) with owners. The problem
to learn a model
y f(x) "",)
collectively with all data parts and to perform inference jointly by all owners. Additionally, owners share neither data models with each other for, e.g., privacy reasons. Moreover, owners do not participate joint training, which often incurs expensive coordination.
Similarity to federated learning. Federated inference shares the defining data characteristics of federated learning, first coined in McMahan et al. (2017): distributed, non-lID, and unbalanced. Data distributed among owners but not shared. In fact, data may be heterogeneous. For example, time series measures of the power grid may have different attribute dimensions and may be under different sampling frequencies. As result, data size may vary significantly among owners.
2

---

Under review as a conference paper at ICLR 2022
Dissimilarity to federated learning. The root of the differences between these two concepts is the constituent of one datum (data point). In federated learning, a data point the basic unit data and thus all owners learn a common model but use it separately. On the other hand, federated inference is concerned with data split in parts across owners. All parts of a data point contribute to the inference collectively.
Similarity and dissimilarity to vertical federated learning. Both concepts are concerned with the split of datum across owners. However, approaches taken for vertical federated learning differ substantially from ours because of joint training among owners. From the less prolific literature, two lines of work are noted. One takes the data matrix literally, by assuming tabular data and studying linear models, where model parameters have natural correspondence to the data parts (Hardy et al. 2017 Nock al. 2018 Heinze et al., 2014 20T6). Often, these approaches are hard to generalize to complex data and/or many owners. Another line of work uses local-globa model framework similarly as we do but jointly trains these two parts, incurring expensive communication and creating dependence of local models (Hu et al. 2019 Chen et al. 2020 In contrast, allow data owners to train their models independently, maintaining local model autonomy.
Data example. Let consider the power grid. Figure [ pictorially illustrates PMU measure- ments distributed across data owners. A panel of time series corresponds to a specific time window and the series collectively represent one data point, which the event detection sys- tem classifies. In this simplified illustration, each data owner possesses one series recorded by one PMU; but in practice they may own different amounts of PMUs (and thus series). Moreover, the series may differ in length be- cause of varying sampling frequencies; and the series are multivariate with possibly different number of variates. All these variations con- tribute to data heterogeneity, which necessitates the construction of separate local models. Note that an event does not cascade over the en- tire grid, some local models may report event whereas others report normal, resulting in con- flicting opinions. consensus (global) model is responsible to resolve the conflict. Addition- ally, missing data may occur.
Dala
owner
Anr
owner
ownnr
ownnrr
mmm
owner
generator error
transfomer error
line error
Figure 1: Federated inference: global label is predicted collectively based local data from multiple owners. Local data may be heteroge- neous and missing data may occur.
FEDERATED INFERENCE FRAMEWORK
As such, the proposed framework for federated inference consists of local models f and global model g. such that their composition is the sought f denoted in (T). Each data owner possesses local model trained with its data, independently of other owners. This way, data sharing is invoked and privacy is minimal concern. However, because the local models lack a global vision and may be conflicting, central (global) model is key coordinating the local opinions for final prediction. To maintain autonomy, local models are frozen once pretrained and will not join the training of the global model. Data owners send local data representations for global model training (and inference). In other words, the global model queries neither the raw data nor the local models from data owners. As long as owners agree send the less decipherable representations to the central server, global inference can be made.
centralized server
Local models. We treat a neural network except the final output layer as a feature extractor, which produces the representation h of an input x'; and treat for simplicity the output layer as logistic regression. That is, local model f reads:
(xx') Noftmax(W'h b²) where h embedding(x").
Note that the approach proposed by Hu et al. 2019) assumes parameters for the global model. Were global parameters present, gradient communication inevitable.
3

---

Under review as a conference paper at ICLR 2022
latent dimension alignment
latent representation
graph neural network
owner 2
owner 3
W
label
owner
owner 5
Independent local models
Global model
Figure 2: Federated inference framework. Local models are trained independently and separately from the global model. The algorithm is summarized in Algorithm in supplement Section
We interchangeably use "representation", "embedding", and "latent vector" mean h². These h are assumed to have the same shape across i, although x² can have different shapes and the embedding function can have different architectures cope with data heterogeneity. simple example of the embedding function is fully connected layer h' ReLU(U"x + cc);; but an arbitrarily complex function applicable.
Global model. The global model is a function of all local representations:
y = g(h .,.....,h")
(3)
example g is a fully connected layer, followed by mean pooling and another fully connected layer:
softmax (W1 1 E. Re + bb) + +b)
(4)
Challenges. Two considerations are pertinent to this framework. First, when the latent dimensions have semantic meaning (e.g., when the local models are trained to yield disentangled representa- tions (Higgins et al., 2018)), each latent feature of local representations may not match, because an arbitrary permutation of the latent dimensions does not change local model. Second, naive mean pooling as in (4) may miss the interdependencies between local data, leading to a less well per- forming global model. Such interdependencies naturally occur in the power grid example because of the physics of an electricity network. Hence, in subsequent sections, we use latent alignment to address the first problem and graph neural network to address the second one. Incorporating these two components, we show the full, proposed framework in Figure 2 and Algorithm (supplement Section
ALIGNING LOCAL REPRESENTATIONS
For the global model to be meaningful, the feature dimensions of the local representations ht' should match. For example, in ( all h multiply the same weight matrix Wo; in other words, each element of h' corresponds of the elements will destroy the correspondence. That is, even if the local models are fixed, the arbitrary arrangement of the feature dimensions of the latent vectors causes ambiguity of what an optimal global model can be built.
one input neuron of the initial fully connected layer. Permutations
Mathematically, let us use vector p to denote permutation and place superscript whenever necessary. The ith local model 2 can be equivalently written as
f((x') = softmax(W' p'"" [p"]+++]++++ where h'pp'= embeddiggaan
(5)
for any permutation p² as long as the embedding function is able to produce permuted h [p"] be casily satisfied if the embedding function is
under the same input / Such requirement fully connected layer (i.e.. h[p] = ReLU(W[p, [p. + b(p])). fact, is satisfied by most neural networks well. In the supplement, give another example: the GRU (Cho et al. 2014).

---

Under review as a conference paper at ICLR 2022
Hence, propose to align the feature dimensions across all local vectors A to disambiguate the ambiguity. This proposal amounts to modifying the global model (3) to the following:
y - (P
(6)
where P an alignment matrix for each data owner
Two approaches of defining P exist. The first approach is soft alignment, which treats each Pt free parameter matrix optimize. may be square or rectangle, the latter case indicating a change of the number of features.
The second approach is hard alignment, which treats each p a permutation matrix. Learning permutation matrices is challenging, however, because they correspond to combinatorial structures and are unsuitable for gradient-based training. We follow Mena al.] (2018); [mmami Ranka (2018) and relax P by doubly stochastic matrix, which can be differentiably parameterized by the Sinkhorn-KKmn algorithm (Sinkhorn Knopp 1967). Specifically, starting from a nonnegative square matrix Kg and column vectors ro co of matching lengths, define the sequence
Cj+1 = 1 2 (K) r,) and ry+1 @ (Kocz), for j = 1,... Then, under mild condition, Ky == diag(r,) Ko Ko deag(e,) converges doubly stochastic matrix. We truncate the sequence the 7 th step and treat Kr an approximation of pa
Despite the advocation by Mena et al. 2018); Emami & Ranka 2018), we obtain the following convergence result of Sinkhorn-Knopp. which reveals free lunch.
Theorem (informal), Under a condition of Ko, there exists a positive integer J and constant
JJ such that for all j
KT1 K
-
where 02 is the second largest singular value of the limit of Kj-
For a formal statement and the analysis, see supplement Section D and Theorem 5 The result suggests that for a desirable limit being permutation matrix, whose d₂ = 1, the error (a 25) does drop. In practice, to expect an approximate permutation matrix, 02 and the convergence is exceedingly slow. The practical usefulness of 7 depends on the learned quality of Ko-
The soft and hard alignment approaches have pros and cons. The hard approach maintains the correspondence of each feature dimension of the latent vectors while the soft approach does not. Maintaining the dimension correspondence is an advantage. especially for local models that pro- duce disentangled latent representations (Higgins et al. 2018), because each feature dimension is equipped with semantic meaning that controls certain aspect of the data. the other hand, the soft approach is more straightforward and the hard approach is based on an algorithm that barely converges. practice, we observe that neither approach decisively outperforms the other feder- ated inference.
5 LEARNING A CONSENSUS GRAPH
The example global model performs a naive averaging for the local representations. Since data owners are often interconnected, a expressive model exploits their relational interactions to improve inference (Battaglia et al. 2018). To this end, we propose to use a graph neural network (GNN) (Zhang al. 22200] Wu et al.) 22221) to process the latent representations.
Many GNNs are applicable; we focus on GCN (Kipf & Welling 2017) for its simplicity. Let A be the graph adjacency matrix and let H be the matrix of aligned local representations:
-P""
H
1PP"
Traditionally, GCN was designed for node classification, but we modify it slightly for purpose as
T = softmax (11TA) Ā. ReLU(AHW)) wi).
(8)
5

---

Under review as a conference paper at ICLR 2022
where Á is normalization of A (see (Kipf & Welling 2017) for details) and Wo and W1 are weight matrices. The modification is the inclusion of # as pooling before output. Modulo this modification, the formula (8) is standard one used in the literature, with the bias terms omitted. It is interesting to note the equivalence of GCN ( and the graph-agnostic model 44 when À is replaced by the identity matrix (omitting bias terms).
In GCN, corresponds to the consensus graph among local owners as graph nodes. If such graph is not present, it is possible to learn one such that (8) still outperforms 4 In this case, we treat A as a random variable of the matrix Bernoulli distribution, where the success probabilities
free parameters to learn. Formally, the elements A.j are independent and each follows Ber(O where 0, denotes the corresponding probability (Kipf al. 2018; Shang et al. [022]. Then, the global model g has Wo. WI, the ps's, as well as 0, as parameters. Following Franceschi al. (2019); Shang et al. (202T), we formulate the training loss as an expectation over A's distribution and draws a sample to obtain an unbiased estimate of the loss as well as the gradient, in each stochastic optimization step.
The central challenge of this approach is that A (and hence also the loss) not differentiable with respect to 0. A popular remedy is the Gumbel softmax reparameterization trick (Jan et al.) 2017] Maddison et al. 2017). In what follows, for simplicity of exposition, we treat a scalar rather than matrix. The Gumbel trick works in the following manner. Let Cat(T) be the categorical distribution with probability vector π and let g, of the same shape as π, be a vector variable whose elements are iid ~ (O, 1). Then, the vector random variable
(oftmmx(((lgg) π + g)/T), T>0
admits a distribution converging to Cat(T)) when T + O. Hence, to sample Ber(O) approximately but differentiably, it suffices let [0, 0] and use y/1 the sample.
In order to obtain one Bernoulli sample, the Gumbel trick requires to sample the Gumbel distribution twice. We consider alternative that samples any appropriate distribution only once.
Definition 1. Let F be the edf of an arbitrary continuous probability distribution. Sample s from this distribution and let
2 = sigmoid - s)/T), T> O.
(10)
We call this method iedf.
The name icdf is owing to the use of F-1. The reader should not confuse this method with the inverse transform method for sampling a random variable with a particular edf F. Here, we use any F to sample (approximate) Bernoulli distribution. The following result qualifies to be an approximate Bernoulli variable. The proof, as well as those of subsequent theorems, is given in the supplement.
Theorem 2. For all T > O, E (0,1), and E [0, 1). if the distribution with cdf F is finitely supported on [a, b), then
(0 if t < sigmoid ((F2 - b)/T);
1)(=
1 if t > (F
(11)
(1 F() + log((=--1111)
otherwise.
On the other hand, if the distribution is not finitely supported (i.e., oo and/or b +0o), (I) still holds because either. or both, of the first two cases will not be invoked. As a consequence, the distribution of converges to Ber(f) as
0.
is imperative to understand the rate of convergence of y1 (Gumbel trick) and that of = (iedf method). While one may take the usual convergenee-in-diistbbuuuis approach, the complex forms of the edf (e.g., (TT)) render the analysis difficult. Instead, we take the convergence-in in-mmennnnnnnn approach and calculate Bias(x) E[x] O. We derive the following result.
Theorem 3. When T small,
Bias(y)) = --00 01 Bias(z) F (
O(T*),
(12)
(13)
Moreover, when F is the edf of a normal variable ~ (0, then
Bias(2) 2) 2 erl 1 1)e (erl (20 11)² O(77).
(14)

---

Under review as a conference paper at ICLR 2022
Theorem suggests that the iedf method converges equally fast does the Gumbel trick (both the order of )))))). On the other hand, the biases depend on 8. Thus, one cannot set temperatures independently of the desired probability 8, to equate the two biases. In practice, is tunable hyperparameter and we use the same tuning range to fairly compare the Gumbel trick and icdf. The rationale is justified in supplement Section
We conclude this section by stressing the advantage of the proposed iedf method for differentiably sampling graphs for global model training: it requires fewer random number generations than does the Gumbel trick, saving time and memory.
6 EXPERIMENTS
In this section, demonstrate comprchensive experiments to show that federated inference can be effectively conducted by using the proposed framework.
Data sets. We four real-life, time series data sets. Two PMU data collected from multiple data owners of the U.S. power grid. For proof concept, we smooth out heterogeneity and prepare homogeneous data sets. Such a preprocessing is sufficient test the proposed techniques under minimal impact of the complication by the otherwise diverse local models. Since the PMU data sets are proprictary, we also use two public, traffic data sets (Li et al. 2018) for experimentation. A summary of these data sets is given in Table and the processing details are given in the supplement.
Table 1: Data sets.
10-
PEMS-BAY
PMU-C
METR-LA PEMS-BAY
PMU-B
PMU-C
densiyy
Data samples
2856
4343
4853
1884
# Data owners
207
325
43
188
Series length
12
12
30
30
# Features
1
1
2
Classes
2
2
0.8 entropy
0.5
Missing data?
no
n1
yes
yes
Given graph?
yes
yes
no
no
Figure 3: Distributions of prediction en- tropy across local models.
Experiment setting. All local models are LSTM (Hochreiter & Schmidhuber, 1997) with the same hyperparameters, but pretrained separately by using local data. The local models are not fine-tuned in the training of the global model. Each data set is split randomly for haining/validationnessnngg See the supplement for further details.
Conflicting local predictions. We first show that local models do not produce consistent predic- tions, which justifies the effort of training a global model and performing federated inference. For each datum, compute the entropy of the predicted labels and summarize the entropies for all data into a distribution, plotted in Figure 3 Recall that the lower the entropy. the more consistent the local predictions. The figure, however, shows that a substantial amount of entropies is away from zero, suggesting that local predictions are inconsistent.
Effectiveness of the proposed framework. We make two sets of comprehensive comparisons to evaluate the effectiveness of the proposed framework. The first set, as outlined in Table compares with a number of straightforward baselines (A-F) and methods outside the federated inference setting (A and K). This set contains several alignment strategies for local models: (G) no alignment; ()) soft alignment; and (K) hard alignment. straightforward variant between G and H, where common weight matrix W is used for all local models, serving Methods G to use graph structure learning (icdf method) as the global model.
alternative to alignment.
One sees that methods A to D, either lacking local model or global model, perform poorly as expected. Methods E to H perform better than A to D, but they lack a proper alignment of the local models and hence outperformed by methods I and that perform alignment. Between the two strategics, neither decisively wins over the other. The advantage of soft alignment its simplicity and that of hard alignment is the preservation of neuron correspondence. Finally, method K (end-to-

---

Under review as a conference paper at ICLR 2022
Table 2: Effectiveness of latent alignment in graph-based global model. Superscript numbers are standard deviations. Note that A and K are not applicable to the federated inference setting.
METR-LA
PEMS-BAY
PMU-B
PMU-C
Fl
AUC
F1
AUC
F1
AUC
F1
AUC
A: Common model
.255 .00
.334 000
.360 000
.286 000
Local model + majority voting
.114.000
.089 000
.291 0000
.182 000
C: Local model + binary threshold
.692-000
.639-00 000
D: Best local model
.528-000
.702-000
.553 -000
.792 -000
.370-000
.692 000
.324-000
.618
E: Local model + MLP (Eqn
.768-00
.957 004
.738 .738-012 012
.935 001
.391 .003
.727 006
.342 .342-008 D08
.636-010
F: Local model concatenation
8244000
.91 001
1854-003
.79
.386 005
.693 084
.389 018
.698 010
Local model + icdf (no align.)
.798 009
.963 004
.755 009
.943 001
.387 003
.734 015
.380 006
.658-005 005
H: Local model + shared W icdf
881-0009
9966.001
.747 .747-009
.941 .004
.387 0D6
.725 010
.368
.660-008
I: Local model + soft align. + icdf
8835-010
.975 001
.60 DuG
.980 0022
.390-008
.734 008
444-027
.693-011 011
J: Local model + hard align. + icdf
839-000
973 0u1
.855.0us
996-401 401
.390-004
.737-016
.404 DIG
.686-008 008
K: end-to-end
.25
.973 -02
.823 006
.972
.382 007
.717
.392-000 020
.683
Table 3: Comparison of global models. Some references of rows are with respect to Table 2
METR-LA
PEMS-BAL BAY
PMU-B
PMU-C
Fl
AUC
Fl
AUC
F1
AUC
F1
AUC
No align
No graph
.768 009
957 004
.738 012
935-001
.391 003
.727-000 006
.32
008
.636.910
Given graph
.763 020
.957 007
.742 024
.942-00 005
Gumbel
.785 .785.uu uus
.959 0os
.751 008
.942 001
.387 001
.730-917
.381
.925
-658-90G 00G
iedf (row G)* * .798 uve
.963 004
.755 U0g
.943 001
.387-008
.734-013 -11
.380 00G
.658-0u5
Soft align
No graph
833-010
.975 .95-0001
844 008
99770001
.388-001
.736-015
.386-088
-694-005
Given graph
8888
.97-0001 001
884 003
.977-003
Gumbel
834
.955 001
863 014
.980
.390
003
.733
.435
.693 007
iedf (row 1)*
.855 010
.975 001
860 005
.980 002
.390 008
.734 008
.44 027
.693 011
Hard align
No graph
825
.971 .003
847 008
.976 001
.387
.736 0077
.372-004 004
.674-015
Given graph
824
.97 .002
848 010
.923.02
Gumbel
8337.013
.973 002
850 009
.976 001
.391 004
.732-014
.410 016
.687-004
iedf (row)) J)*
8099006
.973 001
.85 008
.976 001
.390 004
.737-016
.16.
.686-00 008
end training) performs worse than method J (separate training). The result is not surprising, because joint training compromises the optimality of local data representations separately obtained by each local model. We also note this method is outside the setting of federated inference and generally cannot be used unless data owners agree share data.
Comparison of global models. The other set of comparisons. outlined in Table 3 extends each alignment strategy (including no alignment) to the role of graphs in the global model: not using graph, using the given graph, and learning a graph (by using either the Gumbel trick or the iedf method). The numbers in the table suggest that within each alignment strategy, graph structure learning significantly improves the classification. The performance of the Gumbel trick and that of iedf is highly comparable.
Quality of learned permutations. For hard alignment, investigate the learning of the permu- tation matrices. According to Theorem [ of of the limit of K, dictates the convergence speed. Since we do not know the limit, we compute of of Kr and summarize them in Table 4 for some local model in each data set, under several involved methods. One sees that all values close to 1, suggesting that the convergence indeed rather slow, agreeing with theory. Note that some values greater than 1 because Ky is not strictly doubly stochastic (owing to slow convergence).
In Figure fl we visualize Ky for some local model in each data set. The plots clearly show patterns of permutation matrix: there is and only significant value per row and per column. Because of the slow convergence, we attribute the desirable results of K (at a small T) to the success of the learning of Ko. Note also interestingly that learned permutation may be the identity mapping.
8

---

Under review as a conference paper at ICLR 2022
Table 4: Examples of squared second singular value, 02. of Kr-
METR-L.A
PEMS-BAY
PMU-C
Hard align
No graph
1.007
1.000
1.000
1.000
Given graph
1.010
1.000
Gumbel
1.007
1.002
1.001
1.019
icdf
1.000
1.000
1.000
1.025
local model
local model 48
local model 24
loral model 72
10
00
(a) METR-LA
(b) PEMS-BAY
(c) PMU-B
(d) PMU-C
Figure 4: Examples of learned permutation matrices (K7). All from Method H.
Comparison of Gumbel softmax and iedf. Prior results suggest that these two approaches for differentiably sampling the Bernoulli distribution perform equally well. An advantage of iedf is its lower computational cost. To demonstrate this advantage, we design a mini-benchmark that high- lights the sampling and gradient computation and minimizes the effect of irrelevant complications (such as permutation and GNN). To this end, we generate samples (x, E R",", A {0, 1)"xm, where y/- Az; + noise, and use the samples to learn A through differentiable parameterization. 3 shows the time and memory consumption at a fixed number of learning epochs. As sanity check, the running time scales nicely O(n²) expected (while the memory consumption is complicated; it does not follow (n²²) because of memory management in Python). Overall, one clearly sees the lower computational cost of the icdf method.
Vi E IR") for some
We also report the time and memory consumption for the experiments on the four data sets; see Table in the supplement. The results well agree that the iedf method is more economic.
64.0
Gumbel
Gumbel
16.0
512.0
icdr
256.0
1.0
128.0
0.25
64.0
32
64
128
(a) Time in seconds
(b) Memory in MB
Figure 5: Time and memory consumption as the matrix size (n, horizontal axis) increases.
7 CONCLUSIONS
In this paper, we study federated inference, less addressed scenario of machine learning with distributed data that require collective inference. This scenario is contrast to federated learning, where inference is local and requires no joint efforts. We motivate the practicality of federated inference by using a power grid example and propose a local- globgll model framework for it. Two important components of the framework
the alignment of the data representations produced by local models and the learning the global model by using graph neural network. Comprehensive experiments suggest the feasibility of federated inference and the effectiveness of the framework.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jan Philipp Albrecht. How the GDPR will change the world. European Data Protection Law Review, 2016.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. Preprint arXiv: 1iXiv11866011 1806.01261.. 2018.
Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. VAFL: method of vertical asynchronous federated learning. In ICML Workshop, 2020.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.
Patrick Emami and Sanjay Ranka. Learning permutations with Sinkhorn policy gradient. Preprint arXiv:1805.0 0,0010, 2018
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In ICML, 2019.
Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thoree Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. Preprint arXiv:1711.10677 2017.
Christina Heinze. Brian McWilliams, Nicolai Meinshausen, and Gabriel Krummenacher. LOCO:
Distributing ridge regression with random projections. Preprint 14Xx:1400.3469, 2014.
Christina Heinze, Brian McWilliams, and Nicolai Meinshausen. DUAL-LOCO: Distributing statis- tical estimation using random projections. In AISTATS, 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. Preprint arXiv:1812.02230, 2018
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780) 1997.
Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2nd edition, 2012.
Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. FDML: A collaborative machine learn- ing framework for distributed features. In KDD, 2019.
H. V. Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M. Pa- tel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Commun. ACM, 57(7)866-94, 2014.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with Gumbel-softmax, In ICLR, 2017.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G.L. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi. Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateck Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Se- bastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma,
10

---

Under review as a conference paper at ICLR 2022
Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Preprint 11X1v:1912.04977,200
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In ICML, 2018.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Philip Knight. The Sinkhom Sinohom-Knopp algorithm: Convergence and applications. SIAM Journal on Matrix Analysis and Applications, 30(1 (1:6:::2772 2008.
Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-boe based neural DAG learning. In ICLR, 2020.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. Preprint arXiv:1907.09993. 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In ICLR, 2018.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Commuiictionn-fficceentt learning of deep networks from decentralized data. In AISTATS, 2017.
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with Gumbel-Sinkhorn networks. Preprint arXiv: 100.:1802.08665, 2018
Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Giorgio Patrini, Guillaume Smith, and Brian Thorne Entity resolution and federated learning get federated resolution. Preprint arXiv:1803.00 803.04035, 2018.
Gabriel Peyré and Mareo Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning. (155-6):555-600 2019.
Chao Shang, Jie Chen, and Jianbo Bi. Discrete graph structure learning for forecasting multiple time series. In ICLR, 2021.
Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matri- ces. Annals of Mathematical Statististics, 35(2):876 879, 1964.
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matri- ces. Pacific J. Math., 21(2):343-348 1967.
Smartgrid.gov. Recovery act: Synchrophasor applications in transmission systems. Webpage actp:p/rrrrryaatptror.raaa http:aasso__tttccccocooo.o.....
retrieved in Jan 2021. impacts/ catpacts/pppiicattiia ions htw.:aaaaoaooocoooooooooooooeeee
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.
Federated learning with matched averaging. In ICLR, 2020.
Alan Geoffrey Wilson. The use of entropy maximising models, in the theory of trip distribution, mode split and route split. Journal Transport Economics and Policy, 3(1):108- 126, 1969.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In KDD, 2020.
11

---

Under review as a conference paper at ICLR 2022
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4- 24, 2021.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology. 10(2), 2019.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In ICML, 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, and Trong Nghia Hoang. Statistical model aggregation via parameter matching. In NeurlPS, 2019a.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In ICML, 2019b.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. Transactions on Knowledge and Data Engineering, 2020.
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Con- tinuous optimization for structure learning. NeurlPS, 2018.
12

---

Under review as a conference paper at ICLR 2022
A
RELATED WORK
The concept of federated learning first coined by McMahan et al. (2017) and it has attracted surging interests since. A form of distributed optimization, federated learning is faced with data challenges beyond conventional assumptions and puts communication efficiency and data privacy as
primary concerns. Recent surveys (Yang et al. 2019 Li et al. 2019 Kairouz et al. 2019) compre- hensively study the subject, review systems and infrastructures, and suggest open problems.
The typical setting of federated learning is that data sets across owners share the same feature space but differ in samples. Besides this horizontal partitioning of the data matrix, a vertical partitioning was studied by HHrdy et al. (2017); Nock et al. (018); Heinze et al. (2014, 2016), wherein features are split across owners instead. This setting bears resemblance to our federated inference scenario, but crucial distinction is that existing methods for vertical federated learning all perform joint training. the referenced work, to preserve privacy, encrypted data or randomly projected data are communicated among data owners well as a central coordinator. Such an approach incurs demanding communication for many owners. Recently, [Cen et al. (2020) study a different model, whose parameters are distributed among owners well corresponding to an owner bears resemblance to our local models; but they are not local models since they are not independently trained by using local data. Another work along similar direction is conducted by Hu et al. (209)), but the global model has no parameters; it is merely a sum of the local outputs followed by activation (e.g., sigmoid for classification).
central server. The part of the model
Our framework learns parameter matrices to align local representations. Such alignments similarly appear in model fusion, where number of models are fused together into a common model through aligning model parameters (Yurochkin et al. 2019a). In the context of deep learning, if the neural networks come from the same model family, their weights can be matched layerwise, even the numbers of weights are different (Yurochkin et al. 2019b Wang et al. (020). The referenced work treats the problem as a bipartite graph matching, where the cost matrix is inferred from maximum a posteriori estimation. Then, the Hungarian algorithm (Kuhn) 1955) is applied to find the matching. In our work, instead we treat the permutation alignment as a differentiable parameterization with the help of Sinkhorn-Knopp (Sinkhorn & Knopp, 1967 Mena et al. 2018 Emami & Ranka 2018), so that it can learned end-to-end with other parameters of the global model.
Our framework also advocates learning graph of data owners in the global model. Graph structure learning appears under various contexts. One field of study is probabilistic graphical models and casual inference, whereby directed acyclic structure is Iearned. Gradient-based approaches in this context include [Zheng et al. 2018); Yu et al. (2019): Lachapelle et al. (020)) On the other hand, a general graph may still be useful without resorting causality. Recent approaches supporting GNN- based modeling include Kipf et al. (2018); Franceschi et al. (2019); Wu et al.] (2020); Shang et al. (202I), wherein graph structure simultaneously learned together with the GNN parameters. The Gumbel trick (Jang et al.) 2017); Maddison et al. (017)) is frequently used for differentiable parameterization, but in this paper we study more economic alternative parameterization, icdf.
B PERMUTATION AMBIGUITY EXAMPLE FOR GRU
In section 3 discuss that can arbitrarily permute the latent representations while keeping local model fixed. Here, we give another example- the GRU. Let x = {œ1, .2..... .,, .rr] be an input sequence. The embedding function h embedding(z) implemented as GRU reads:
function h GRU((Ze))a.....)
2: hn = 0
3:
for = 1.... T do
4:
2; sigmoid(W.n, w. + U-h4-1 bz)
5:
Tt = sigmoid (W, + Uph111 + br)
6:
Te tanh(W,.. n., + Un(r, @ 11-1) bn)
7:
h, = (1 - 21) 2 4-1 + 21 2 n4
8:
end for
9:
return h hr
10: end function
13

---

Under review as a conference paper at ICLR 2022
One can arbitrarily permute the elements of h through manipulating the GRU parameters properly. To achieve hi[p] embedding(2; p),
the gate outputs and bias vectors (Zt, rt, nyy leg Uu bys bn) will need be permuted accordingly (zı[p), rx [p]. nx[p], hy[p). by[p), [p]. [n(p));
the weight matrices attached to the input (W,, W,, Wn) will need to have their rows (i.e., output neurons) permuted [P. :], W. W,, [p, [p, :)); and
the weight matrices attached to the hidden states (U,, U,, U2) will need to have both their rows and columns permuted (U=[P. U. Pl).
C FEDERATED INFERENCE FRAMEWORK
The framework is summarized in Algorithm
Details are elaborated in Sections 4 and
Algorithm Federated Inference Framework
1: function TRAINING( (<+)) j (x")),.... (xx"), Uÿ :=1........))
2: Each data owner trains local model f with its local data part {(xt))}s and label (us)s
3: Each data owner sends all local data representations ((h');}, to the central server
4: Central server trains global model y s(P² p2 ,..., , p*!")) by using {(h'););
gathered from data owners and labels {uy}y. Here,
The global model is (8), where the loss is taken over the distribution of Å, whose entries sampled by using (10;;
Each alignment matrix pa is either arbitrary parameter matrix (soft alignment), doubly stochastic matrix Ky computed by (7) using 7 steps (hard alignment).
5: end function
6: function INFERENCE(2* ,.,,,"""'
7: Each data owner evaluates its local model with x' to obtain h' and sends to the central server
8: Central server evaluates the global model by taking all ht input and produces prediction
9: end function
D FURTHER DETAILS OF SECTION 4 (PARAMETERIZATION OF PERMUTATION)
The permutation matrix P is a combinatorial object and unsuitable for gradient-based training. We relax P by doubly stochastic matrix, which nonnegative and whose row sums and column sums are all 1. A benefit of such a relaxation is that differentiable computational procedure exists for doubly stochastic matrices. The procedure, called the Sinkhorn Sinkhorn-Knopp algorithm (Sinkhorn & Knopp 1967), was recently used by Mena et al. (2018) and Emami & Ranka (2018) in other contexts. Informally speaking, starting from a nonnegative square matrix Ko- repeatedly normalize its rows and columns by their sums until convergence. The limit is doubly stochastic matrix.
In fact, Mena et al. (2018) and Emami Ranka 2018 not start from free Kg but parameterize it as exp(M/T), where M is free parameter matrix and > 0 is a hyperparameter. Such parameterization follows more closely Sinkhorn's 's algorithm (Sinkhorn, 1964) for computing the optimal transport (Peyré & Cuturi 2019) under an entropic regularization (Wilson) 1969), where supposed to anneal to zero. However, this parameterization not imperative and is not the only option. For simplicity of exposition, treat Ky a free parameter.
To understand the convergence this method, we rewrite the algorithm equivalently as
Start with To = 1, for j = 0, 1,.., Cj+1 = (KTr;) and r2+1 Ø (Koc++1).
(15)
Here, the cy's 's and ry 's correspond to the column sums and sums of the iterating matrix in the above informal description. Let diag(r) () a diagonal matrix whose diagonal is x. The following
result is proved by (Sinkhorn & Knopp 1967).
14

---

Under review as a conference paper at ICLR 2022
Knopp). If Ku is nonnegative, then necessary and sufficient condition that the iteration [55 converges is that Ky has total support. Let positive vectors c, and T, be the limits of {c₁} and try j. respectively. The matrix diag(r.) diag(c.) is doubly stochastic.
Theorem 4 -inop)
This result, however, does not give the convergence speed. In order to obtain a tight one, we fol- low Knight (2008) and perform minor modification of (5)
Start with To = Co = 1, for j = 0, L,.,..,, Cj-1 = 1 @ r,) and ry+1 = @ (Kocz),
This modification changes nothing essential of the algorithm, since it is equivalent to applying (15)
0 Ko] Ko o We define Kj- diag(ry) diag(c,) and derive the following result. Its proof is
on
given in Section
Theorem 5. If Ky is nonnegative and irreducible, then the iteration 7 will converge linearly.
Let positive vectors c, and r. be the limits of {ez} and {ry}. respectively. Then, K. diag(r.) diag(c.) ) is the limit of Kj and is doubly stochastic. Furthermore, there exist a pos- itive constant C' and positive integer J such that for all j 2 J.
1 K21 -
|||=co sC( 25502- with Tj =
c [3
c
(16)
where 02 is the second largest singular value of K..
A few important points of Theorem 5 are noted. First, the convergence (16) says that the row sums and column sums of Kj approach 1 at a speed O(o2²), Second, the iteration (7) preserves the irreducibility of the Ký and thus of the limit K,. Then, by the Perron-Frobenius Theorem (Homn & Jobnson 2012), the doubly stochastic matrix K, has a simple eigenvalue 1, which coincides with the spectral radius as well as the dominant singular value. Thus, the second singular value, 02. must be < 1. The farther away it is from 1, the faster the iteration converges.
Unfortunately, Theorem is not applicable to permutation matrix, because such a matrix is not irreducible. However, the theorem is practically useful under perturbation theory. After all, the limit K. is hardly an exact permutation matrix; at best, is e-close. Note that all singular values of a permutation matrix P is 1. If K. is different from P by e in the spectral norm, then according to Weyl's Theorem (Horn& & Johnson 2012). 02 2 1 c.
The above analysis indicates that the Sinkhorn Knopp algorithm 7 will be extremely slow to con- verge e is small. A natural question is, is this parameterization useful? We argue positively. In practice, we run the procedure 7 with only few steps, T. The iterate Ky can still be reasonably close to a permutation matrix, when the starting point Ko is sufficiently good. In essence, we rely on the training of the global model to find good starting point and the procedure 7 to normalize its values to [0, 1]. One may further use an entropy regularization
Σ (K ()) + (x7
to push the values toward the two extremes, 0 or 1.
D.1 PROOF OF THEOREM 5
Applying the iteration formula 7. it is straightforward see that
K.T - = (cz @ (j+1) -1 and K, - = = (0 - 1.
We expand the left equation to
(cj 2 (+41) - 1 (c) c. + c. - (3+1) 6 cj+1
and similarly for the right equation. Then,
c - Cxl c KT 171-1 = ([gg
- cj+1 r. - rj+1
c++1
K21 K
-r. r.
ri+t
Applying the triangle inequality and the property ||. 2 v/1 S || min |us/. we obtain
KT1-2 Ký1-1
9 r. - r.
c.
||
g++ co
rS+ - r- 1)) /TTHL
s
(17)
15

---

Under review as a conference paper at ICLR 2022
where 7y+1 denotes the absolute minimum of the elements of cj+1 and ry+1-
We will need the following result from Knight (200)).
Theorem 6. If Ko is nonnegative and irreducible, then there exists a positive integer such that for all j J,
C++1 [5+1
c. T+
9
c. r.
Theorem states that T S oîT, for all j 2 J. Because d2 < 1, Tÿ is monotonically decreasing once j 2 J. Moreover, because Cj and ry positive vectors for all j. when j 2 J, the elements of Cj and r; are bounded away from zero. Let T > 0 be the infimum of the elements all cj and r when j J: (17 becomes
K 1 1 K21
T T T
Thus, applying the relation
T)+1 S TgTT, we obtain
K K
(1 + o2)T
(1 + oz 0z)02
2T-
20--
which concludes the proof by taking 1/7.
E PROOFS AND ADDITIONAL RESULTS OF SECTION SAMPLING)
5
BERNOULLI
E.I DISTRIBUTION GUMBEL SOFTMAX
As preliminary, we consider the first entry y1 of the random variable defined in ( for the Gumbel softmax parameterization. Note that for any T * O, y1 is only approximately binary; the possible values of y/1 in fact span the entire interval [0, 1]. We derive the following edf for y1. Recall that for notational simplicity, 0 denotes a scalar rather than a matrix.
Theorem 7, For all t > 0, E (0, 1). and t E [0, 1). we have
t t=(1- 0)
Pr(11 f t= 0) (1-
(18)
Proof. We first consider the case 0 < 1. Through simple algebraic manipulation, we obtain that 81 is equivalent to
91 92 log 1-t log 1-6
(19)
Let 91 log(- log u) and Ø2 log(- log v). where and are independent and ~ (1 1).
Then, is equivalent to
MM where M 87 (1
(1 - t)=0
Therefore, by recalling that and are uniform in [0, 1J3, we note that the probability that u happens is the double integral
Pr(c u )²²= = [0=-/
f( dvdu.
This integral is nothing but
( u M du 1 + M
M
which completes the proof of 18
The cases of L obviously hold by continuity.
16

---

Under review as a conference paper at ICLR 2022
E.2 PROOF OF THEOREM
We first consider the case when the distribution with cdf F is finitely supported on (a, bj. Through simple algebraic manipulation, we obtain that Σ is equivalent to M where M == F- (0) + - b)/T), we see that M > b and thus such & can never
log((--~ 1). If / < sigmoid(( occur. Similarly, if sigmoid((F=) a)/T), we see that M a. which indicates that s 2 M always happens. Otherwise, when t is within the two extremes, the probability that 2 M happens F(M), concluding the proof of (TT).
The statement of the theorem regarding the case when the distribution is not finitely supported is obviously true.
To show that the distribution of converges to Ber(ø), let us first consider the scenario when the distribution with edf F is finitely supported. The edf of z (see (TT) is always continu- ous but it has three segments connected by two joints: 4 sigmoid 22 = (0) - a)/T). When т + O, the joint t - and the joint t2 + and thus the middle segment has wider and wider support converging to [0, ll. Hence, it suffices to consider only the middle segment. Further, with an analogous argument for other scenarios, it is also true that it suffices to consider only the third case of (II).
b)/7) and
In this case, for any fixed t < and when T + O, we have r log(t" 1 - 1) + and thus Pr(2 S A. Mcanwhile, r log(t-1 - 1) is undefined. However, we know by definition that Pr(= I 1) = 1. Hence, the continuous distribution of converges to degenerate distribution Pr(2 1) 1 A and Pr(= 1) = 1. This is the edf of Ber(
- F(F (0)
cannot push
because then the limit of
E.3 PROOF OF THEOREM 3
By the definition of bias, we have
Bias( x) = E [ 0 where [[x] = / tdPr(x =t)=1 1- L/ Pr(x St) dt.
Therefore, for Gumbel softmax,
(5(1 -0
=1 1
00 t= 0) (1 t) dl,
and for icdf with any F.
Bias(2) 2) = 1' F(F-1(B) + +log(t 1)) dt A.
We now prove Theorem 3 in few parts.
Proof of 13 Let (0) and perform a change variable m = -log(t" 1). Then,
Bias(z) = / (f(s+Tm)) F())] dt
{ [F(s + rm) - F()]
[
e
(1 emiz dm.
We perform Taylor expansion of around s and obtain
(())(s)
F(s + Tm) - F(s)
n!
Therefore,
F(n)
00
m m
Bias( 2)
=
dm (1 + em
n!
numl
Each integral term is finite and the odd terms vanish because the integrands are odd functions. Thus, for small T, we are left with
(s)
m²
Bias(2)
(1 + emyz dm + O
2
00
17

---

Under review as a conference paper at ICLR 2022
The definite integral evaluates to n²
therefore conclude the proof.
Proof of (14 Equation
(4 is straightforward by substuting
2
erf (20 1), (art (20-1)1
F"
2
σ 22
0²VT
into 13
Proof of 12 3. To simplity notation, let B = 0/(1 - 0) and perform a change of variable m
log(t
Then,
(7(1 + ( di dt
t²(1 - 0)
en
dm.
Be
1 Be (1 em
Denote ((T, m) = [] + -em Treating h a function of T and performing Taylor expansion
around zero, we obtain
h(m) (0. m)
h(T, =
n!
R=0
Therefore,
(1(1-0) t -) 0) (1-
T 00
e
dt
((20 (0.m)
(1 + em dm.
n!.
n=0
-oc
In moment, we will show that for all
h (0) (0, m) C. where C₁ is independent of
(20)
Suppose that (20) holds. Then, each integral term is finite and the odd terms vanish, because the integrands are odd functions. Therefore, for small we are left with
Le tT(1 - +1 t)rd
(=(1-0) di Co L
e
72
m
(1 em+ dm + C2
(1 + emp dm O(74).
By calculating
Co = 11(00 m) = [1 + -
C2 = 11,( ()) = -0(1 - - 20),
e
10
m²e
n²
emyz dm =
(1 + em) dm
3
conclude that
70 - (1 20)
Bias (y))
+ +(7))
6
remains to prove 20). We suppress the argument on m and write g(T) = 1 + me and h(T) -( By Faà di Bruno's 's formula,
(n)
n
(-1)-k
(0)
Bri (0)) ... (/ ((n)
g(())
g(0)++1
r=0
where Bn, is the Bell polynomial. Clearly, g(0) 1+ 8 and (0) Bm" for all 0. Hence, By,k multiple of m.. Therefore, (((m) (0) is multiple of m..
E.4 ADDITIONAL RESULT REGARDING THE BIAS
Theorem 3 states results for a small temperature т. The purpose is to understand the limiting be- havior of the bias. Here, give additional result for any 0. It states that the biases of the two sampling approaches have the same sign. This result is a nontrivial extension of Theorem and requires different proof technique.
18

---

Under review as a conference paper at ICLR 2022
Theorem 8. For any O,
Bias(y:) 0 when 0 Moreover, if F" (x) (that is, the pdf) is even and is increasing when x < 0, then
Bias(y1) when 2.
2³ Bias(y:) when 2.2
(21)
Bias(z) (z) > 0 when 0 < 2: Bias(z) 0 when 0 Bias(z) 0 when 0 > 2
(22)
We prove Theorem two parts.
Proof of 21 Consider
Bias(y:) I(" g(t, 0) where g(t,0) 1--0
t1 0) 17 - 0) + (1 1)=0"
With a bruteforce calculation, we have
[11 1)" 17)=0(1 0) -0)
gtt, + o(1 0, [tt(1 8) + (1 -) (1) +
All terms on the right-hand side are positive, except 20. Therefore, when 0 < 1, ,(1, 0) + g(1
1, 0) > 0 and hence
Bias(yy) = S(² g(t, 0) + g(1 t. 0) dt > 0.
2
The other cases (0 > 1/2 and = 1) are similarly proved.
Proof of 22 Consider
Bias(z) = L h(t, 0) dt 0 where h(t, F(F-((0) + + Tlog(t - 1)).
We have
h(1 - 0, = ((F
To simplify notation, let F (0) and log(( 1) a. Then, htt, F(s a) and 1(1 1,0) = F(s - a). Let us first consider the case 8 < 0 and a > 0. We see that
f F'mm dm. a-a
F(s+ a) - F(ss)
('mm) dm and F(s) - F(s a)
For any b> 0, if s 0, then by monotonicity, (' b) (' b). On the other hand, if - b 2 0, then ('ss + b) (' b) (' (/ss-b),, b). In both cases, the right integral is always smaller than the left integral. In other words.
F(s + a) + F(s - a) > 2F())
In fact, the above inequality is also established when
and 0. Therefore, whenever < 0,
S h(t, 0) dt -t//// h(t, 0) + h(1 f,
dt >
F(F dt = 6.
2
In other words, Bias(=) > The other cases (s = F 10) and = F=1(()) = 0) are similarly
proved.
F TUNING GUIDANCE FOR TEMPERATURE
Theorem 3 suggests that the iedf method converges equally fast does the Gumbel trick (both on the order of O(?²)). On the other hand, the biases depend on 9. Thus, one cannot set temperatures independently of the desired probability 0, to equate the two biases. In practice, is tunable hyperparameter and guidance on the tuning range called for.
To this end, we use a subscript to distinguish the two temperatures Ty for the Gumbel trick and n for the icdf method-and write, based on 12) and (44 and ignoring the high order terms,
Bias(v1)
T²0² r()) where r(0) erf erf-1(20
v=0(1 0)(20 1)
Bias(2)
T,
(20 1)e-(et) (22 1])
19

---

Under review as a conference paper at ICLR 2022
Note that r(0) is symmetric around 0 2.2 is concave, attains maximum when 0 2. and attains minimum when 0 0, 1. Hence, if Tg 7, and o v2, the bias of the Gumbel trick is (approximately) smaller than that of the icdf method. On the other hand, for a > 22, there exist 01 < 02 such that 0ー2 = =0 For example, when 2.5, on the interval E [o.01, 0.99], the bias of the Gumbel trick (approximately) greater than that of the icdf method.
r(02) and that (ia)(yı) 2 Bias(=), whenever 0 E [0], 02].
Based on the foregoing, a practical guide is use the same tuning range of for the icdf method as for the Gumbel trick. A small change of a (c.g., v2 versus 2.5) will entirely flip the landscape of the bias comparison between the two methods. Because the tuning range is much wider than the change of o, for simplicity suffices to fix o
G DATA SET DESCRIPTION AND PREPROCESSING
METR-LA and PEMS-BAY. These are traffic data sets (MIT licensed) used by Li et al. (2018) The former was collected from loop detectors in the highway of Los Angles, CA (Jagadish et al. 2014) and the latter was collected by the California Transportation Agencies Performance Measure System. Both data sets recorded several months of data the resolution of five minutes. The network graphs are available, which were constructed by imposing a radial basis function on the pairwise distance of sensors certain cutoff.
The data sets were originally prepared for forecasting tasks and hence no labeling information exists. We adapt the data for classification. Specifically, we split the time series the hour, forming hourly windows. We label each window as whether or not it corresponds to rush hour. For proof of concept, we specify 07:00-10:00 and 16:00-19:00 as rush hour and the others non-rush hour. We note that in the original data sets, one of the attributes is time. We remove this attribute to avoid triviality and retain only the speed attribute.
The specification of rush hours may not be highly accurate, but it is sensible practice to cope with the nonexistence of laheling information. Intuitively, the signal of rush hour comes from reduced traffic speed, but not every location of the network experiences traffic jam. Hence, the diverse traffic patterns inside the same time window under a single label causes nontrivial challenges for local models to discern. Therefore, the need of global consensus model is justified and it fits well the federated inference scenario.
PMU-B and PMU-C. These are proprietary data sets coordinately provided by multiple data owners of the U.S. power grid. No personally identifiable information is present. The suffixes B and C indicate the interconnects of the grid. The data sets come with thousands of annotated grid events spanning a period of two years; they form the classification labels. Many variables (attributes) of the grid condition are recorded; we select only the voltage magnitude and the current magnitude, because they appear to be the strongest signals for event detection based on domain knowledge, and also because more data are available for these two variables. The grid topology is not available.
For each event, we select a one-second window from the three-minute window that covers the ap- proximate annotated event time, based on the largest z-score. We retain a sampling frequency of 30Hz, even though some data are 60Hz. Furthermore, large amount of data are missing in the raw data. We impute the series by using a rame int eapolalt (method 'linear', limit t. rectior = 'both') ) from the Python pandas package. This way, a windowed series complete if ever has raw data. Even so, many series are entirely empty, which corresponds to the scenario illustrated by Figure [
Classes these two data sets are rather skewed. For PMU-B, we remove a class that consists of only one data point and for PMU-C, we combine classes that contain fewer than 24 data points into a single class.
H EXPERIMENT DETAILS
The experiments are conducted on one x86 node of a computing cluster with one K40 Nvidia GPU, The compute node has eight Intel cores and 128GB memory.
20

---

Under review as a conference paper at ICLR 2022
For each data set, perform 70/10/20 random split for training, validation, and testing, respec- tively.
For local models, use LSTM with the same hyperparameters: hidden layer, hidden dimension 16, and maximum number of epochs parameters afterward.
200. We pretrain the local models and freeze their
We train each global model for maximum 500 epochs and use early stopping according to the validation loss, with patience of 50 epochs.
For the GNN global model, we use a 2-layer GCN with skip connections. The hidden dimension is set at 8 and select the learning rate from {0.01, 0.001). For missing data, impute the node features by using zero.
For the learning of the permutation matrix, parameterize Ko M where the squaring is el- ementwise. On data sets METR-LA, PEMS-BAY, and PMU-C, M is initialized using xavier uni- form, and for PMU-B it is initialized by identity matrix with small Gaussian noise. The num- ber of Sinkhorn-Kang Knopp steps is T = 10 and the reguoay-rggularization strength is selected from {0, 0.00,.1000 0.1}.
For the learning of the consensus graph, the temperature 7 in the Gumbel-softmax method is tuned from f0.1, 0.013. The parameter 7 in the iedf method is tuned from the same range, based on the analysis after Theorem
I FURTHER EXPERIMENT RESULTS
Extending the last experiment in Section 6 Table 5 summarizes the time and memory consumption during the training of global models on the four data sets. The results indicate that the iedf method is more economic than the Gumbel-softmax method.
Table Time and memory consumption in global model training (five epochs). Time is in seconds and memory is in MB.
METR-LA LA
TEMS-BAY
PMU-B
PMU-C
Time
Memory
Time
Memory
Time
Memory
Time
Memory
Gumbel trick
87.89
832,38
270.52 157.93
1896.11 1167.19
42.40
348.39 322.59
84.89 54.07
1119.13 894.63
icdf method
79.69
568.2
30.16
21


Reviews:
[{"review_id": "oHRaRPO4F78", "reviewer": "Reviewer_hLWJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper is well-written.\n2. The proposed method is practical and technique soundness with insight discussion.\n3. The proposed setting is explained well.\n4. The main technique contribution comes from the server model design that is to tackle the data heterogeneity, neuron/feature alignment and consensus graph learning. \n\n\nWeaknesses\n\n1. It will be controversial to name the proposed method as federated inference. Most federated learning methods are designed to serve each user or participant in a distributed system, e.g. smartphone users in cross-device FL, or a banking organisation in cross-silo FL. However, this paper’s setting is to train a global model for server use only while each participant works for data collecting and pre-processing (representation learning). Thus, this setting is more suitable to be categorised as distributed machine learning rather than federated learning. \n2. The training of each local model is independent without cross-client collaboration or server-guided coordination. Given the limited resources of each client, it will cause possible practical challenges to training a good model with desired performance.\n3.  The experiments are based on datasets for power grid and traffic network with time-series inputs that are unusual benchmarks in FL. \n4. Despite the proposed new setting, the proposed method has very limited novelty from a technique perspective. Especially, there is no technical novelty from a graph neural network perspective.\n", "summary_of_the_paper": "The paper aims to train a GCN-based multivariate time series classifier by leveraging the pre-processed data (representation per instance) from multiple sources (data owners). The overall model training is decomposed into two steps. First, the client extracts representations using its pre-trained model. Second, the server aligns client-wise representation, and then trains a GCN-based classifier to leverage the structural information across clients.  ", "summary_of_the_review": "The paper is a combination between dynamic graph neural networks and federated settings. The proposed method is a practical solution for learning a model in a distributed system with topology information. However, the alignment of federated settings is unconvincing, and the contribution from graph neural networks is also very limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QKpnPGk7M3M", "reviewer": "Reviewer_ePJ1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed pipeline is interesting and addresses the relevant problem of heterogeneity in FL. This being said, there are to my opinion important issues that should be addressed:\n\n- The definition of federated inference is not very clear. According to the example depicted in Figure 1, the problem here analyzed consists in FL with heterogeneous data, where heterogeneity arises from different data dimensions within- and between- clients, and different sample size owned by each client. Is this actually the case? \n\n- If this is the case, would simple data standardization and resampling techniques to format the data prior to FL already be sufficient to account for the heterogeneity? My feeling is that a benchmark with simple pre-processing and data imputation approaches is needed to fully appreciate the value of the pipeline proposed in this work.\n\n- The sharing of the data representation extracted from the local models raises important privacy concerns, as it may leak important information about the clients. This is a strong limitation of this work, which should be at least acknowledged in the paper.\n\n- Concerning the sharing of the local representation, the proposed approach could be simply being reformulated as applying dimensionality reduction on the clients’ data (e.g. PCA), and train a centralized model on the top of the latent representations. In what the proposed framework would improve this simple baseline?\n\n- The pipeline requires the further training of centralized models to provide the alignment and consensus graph. It is not clear whether the assessments of Table 2 and 3 are made on the data used for training such a models, or on an independent hold-out dataset. \n\n- Given the complexity of the proposed frameowrk, it would be useful to include a benchmark on controlled scenarios, for example using synthetic data. \n\n- Section 6. The sentence “we smooth out heterogeneity and prepare homogeneous data sets” should be clarified. How is the homogenization performed in a FL setting? Is there any information leaked across clients?\n", "summary_of_the_paper": "This work is motivated by the problem of federated inference, which arises when clients own different parts of a datum, and learning must be collectively performed on the complete data representation. This application is presented as an extension of vertical federated learning. The proposed approach is based on the training of local models at the clients’ level. From the local models, an opportune data representation is extracted and shared in a centralized setting, which is then used to perform global inference. To cope with the mismatch of data representations across clients, a centralized alignment  procedure is proposed, which provides the input for a graph neural network encoding the relational structure among clients. Experiments are provided for the specific application of modeling real-life time series datasets from data owners of the US power grid, showing that the proposed local-global analysis pipeline outperforms both local and global training.\n", "summary_of_the_review": "The proposed framework is novel and addresses a relevant problem of FL.\nThe formulation of the paper seems ad-hoc with respect to the proposed application, as well as the related experiments. There are also  concerns on the privacy leaked during the sharing of the local representations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OcIkUmSA3Y", "reviewer": "Reviewer_ETbu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper made a list of interesting contributions, including formulating a new research problem, developing a complete solution with a few technical innovation. The paper is easy to follow and the results are promising. \n\nHowever, I have the following concerns of the paper: \n\n1. The problem of federated inference is not well motivated: 1)  For federated learning, privacy concern is a major issue, each individual party only shares the gradient to the center server. In this case, the local party will share an embedding; and this may raise the concern on the data privacy. Especially, it seems the local models are known or being the same from each party (the parameters maybe different). I hope there are some discussion on the privacy aspects. 2) In addition, since the local models are pretrained separately by using the local data, the main problem can be considered as a classification problem with  a number of features f(h_1, h_2, ... h_n), where h_i have the same name of dimension. This naturally leads to model F (local mode+concatenation), which indeed shown to perform quite well against other advanced models (especially on AUC). 3) The applications of the problem seems to be limited; besides the time series data in power grid, are there any other real world applications?\n\n2. Lacking proper baselines: a simple solution seems to be adding MLP (two/three layers) \\sigma( W_i^\\prime \\sigma (W_i h_i)), and then  perhaps concatenate them  for the final aggregated output. In fact, since model F is already performing well, it worth looks into more to see how much it can be improved, and what are the issues of this direction. Those one or two extra layers seem to be able to capture the alignment issue or provide similar functionality.\n\n3. Why the GCN is helpful in this situation? The rational for using GCN is not very clear (Paragraph 1 in Section 5 may need to be extended). And is it a novel contribution to use GCN this way? Since this is a rather simple modification of GCN, I wonder if similar attempt has been done for graph classification (classifying the subgraphs). \n\n4. The technical contributions from Theorem 1 and Theorem 2/3 are very different research problems, and quite disconnected. Are those indeed the main problems for this federated inference problem? \n\n5. The performance of the newly proposed models seems to be relatively small comparing the baseline models, such as model F (table 2). \n\n\n\n\n\n\n", "summary_of_the_paper": "This paper introduces a new problem, called federated inference, and proposed a framework of solutions including local representation alignment and learning a consensus graph. The technical contribution mainly comes from a theorem (Theorem 1) on the convergence of learning permutation matrices, and a new approach (icdf) for parameter inference of matrix Bernoulli distribution. The results demonstrate the proposed approaches are useful in improving the prediction accuracy. \n", "summary_of_the_review": "Overall, I feel the paper may need some additional work to better connect different technical points, and provide discussions on the necessities of the sophisticated methodology. \n\n===================================\nAfter reading the responses and other reviewers’ comments, I will keep my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BGPv_nzWnOE", "reviewer": "Reviewer_oPhp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n(1) The paper proposed an interesting framework to perform \"federated inference\" to address the scenario which vertical federated learning also studies. This framework can save the expensive coordination among data owners and the central server.\n(2) The authors leverage a bunch of techniques to address the hardship of federated inference.\nCons:\n(1)\tThe studied topic is interesting, but the technical ideas have the high risk of leaking privacy of local datasets, due to the upload of new representations of the local data.\n(2)\tBecause of the uploaded representations local datasets, attacks may be performed to infer the local datasets and destroy the privacy protection target.\n(3)\tIn addition, the local representations may be cannot be well learnt by isolating them at first, and thus the alignment and global model training can not be guaranteed as federated learning. In fact, from the results of concatenation method, which has close results as the proposed complex one. The two-stage solution (first representation of local data, and then align representations and global model training) can not sufficiently prove its effectiveness.\n(4)\tIn essence, the technical idea is the first local representations, and then fusion of these representations for decision. Although the authors terms this as “federated inference”, I would like the authors to more care about the “federated” term. In fact, an encryption technique can be adopted for each local data to encode the data and then make the prediction, it is not so clear for the advantage or necessity of federated inference. The authors can further clarify this point.\n(5)\tOther comments:\n(a)The authors should specify methods A and K in Table 2.\n(b) There are some similarities between this work and the work of Hu et al. [1] and Chen et al. [2], the authors can introduce a more details of above two papers in the Related Work. \n(c) Vertical federated learning also studies such a scenario, I consider the authors should compare the methods of Hu et al. [1] and Chen et al. [2] in the experiments, either in terms of performance or cost.\n(d) The current experimental verifications are not so solid and well justified the technical ideas. I would like the authors to consider the weakness/comparisons mentioned in (1)-(4) to confidently prove the proposed method.\n[1] Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. FDML: A collaborative machine learning framework for distributed features. In KDD, 2019.\n[2] Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. VAFL: a method of vertical asynchronous federated learning. In ICML Workshop, 2020.\n", "summary_of_the_paper": "The authors proposed a local-global framework to perform “federated inference” in a less addressed scenario where a datum consists of multiple parts, each of which belongs to a separate owner. Different from federated learning, to perform “federated inference”, joint efforts are required only in inference stage. To enhance the proposed framework, the authors propose two alternative alignment methods and a consensus graph learning among the new representations of different clients. In the experiments, both the theoretical analysis and empirical results are presented to support the effectiveness of the proposed framework. The main technical idea is the isolated learning of representation of local data and then aligning them to make the prediction. But this learning paradigm may heavily hurt the privacy of federated learning, since the new representations may leak the local data, and alignment of multiple representations may also lead to inference attack of these data.", "summary_of_the_review": "The paper considers a less addressed scenario where a datum consists of multiple parts, each of which belongs to a separate owner and proposes a local-model framework to perform “federated inference” in this scenario.  The main technical idea is the isolated learning of representation of local datasets and then aligning them to make the prediction. But this learning paradigm may heavily hurt the privacy of federated learning, since the new representations may leak the local data, and alignment of multiple representations may also lead to inference attack of these data. Furthermore, the validation and justification of the proposed methods are still not so mature for a top-tier conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to compute local representations on device, which are then shared between clients using an alignment mechanism. Reviewers did appreciate the value of the topic and several contributions, but unfortunately consensus is that it remains below the bar, even after the discussion phase. Concerns remained on privacy and motivational positioning with FL, and lack of simpler baselines, even after the author feedback.

We hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject