Merging Models Pre-Trained on Different Features with Consensus Graph
Authors: Tengfei Ma
Trong Nghia Hoang
Jie Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Network
Probabilistic Methods

Abstract: Learning global models effectively on private and decentralized datasets has become an increasingly important challenge of machine learning when applied in practice. Federated Learning (FL) has recently emerged as a solution paradigm to address this challenge. In particular, the FL clients agree to a common model parameterization in advance, which can then be updated collaboratively via synchronous aggregation of their local model updates. However, such strong requirement of modeling homogeneity and synchronicity across clients makes FL inapplicable to many practical learning scenarios that cannot afford such requirements. For example, in distributed sensing, a network of heterogeneous sensors sample from different data modalities of the same phenomenon. Each sensor thus requires its own specialized model. Local learning therefore needs to happen in isolation but inference still requires merging the local models for better performance. 

To enable this, we investigate a feature fusion approach that extracts local feature representations from local models and incorporates them into a global representation to train a more holistic predictive model. We study two key aspects of this feature incorporation. First, we develop an alignment algorithm that draws accurate correspondence between feature components which are arbitrarily arranged across clients. Next, we propose learning a consensus graph that captures the high-order interactions between these feature components, which reveals how data with heterogeneous features can be stitched together coherently to train a better model. The proposed framework is demonstrated on four real-life data sets including monitoring and predicting power grids and traffic networks.

Under review as a conference paper at ICLR 2023
MERGING MODELS PRE-TRAINED ON DIFFERENT FEATURES WITH CONSENSUS GRAPH
Anonymous authors Paper under double-blind review
ABSTRACT
Learning an effective global model on private and decentralized datasets has be- come increasingly important challenge of machine learning when applied in practice. Federated Learning (FL) has recently emerged as a solution to address this challenge. In particular, the FL clients agree common model parameteri- zation in advance, which can then be trained collaboratively via synchronous ag- gregation of local model updates. However, such a strong requirement of model- ing homogeneity and synchronicity across clients makes FL inapplicable to many practical scenarios. For example, in distributed sensing, network of heteroge- neous sensors sample from different data modalities of the same phenomenon. Each sensor thus requires its own specialized model. Local learning therefore hap- pens in isolation but inference still requires merging the local models to achieve consensus.
To enable isolated local learning and consensus inference, we investigate feature fusion approach that extracts local feature representations from local models and incorporates them into a global representation for holistic prediction. We study two key aspects of this feature fusion. First, we use alignment to correspond feature components which are arbitrarily arranged across clients. Next, we learn a consensus graph that captures the high-order interactions among data sources or modalities, which reveals how data with heterogeneous features can be stitched together coherently to achieve a better prediction. The proposed framework is demonstrated on four real-life data sets including power grids and traffic networks.
INTRODUCTION
To improve the scalability and practicality of machine learning applications in situations where train- ing data are becoming increasingly decentralized and proprietary, Federated Learning (FL) (McMa- han et al., 2017; Yang et al., 201 9a; Li et al., 2019; Kairouz et al., 2019) has been proposed as a new model training paradigm that allows data owners to collaboratively train common model without having to share their private data with others. The FL formalism is therefore poised to resolve the computation bottleneck of model training on single machine and the risk of privacy violation, in light of recent policies such as the General Data Protection Regulation (Albrecht, 2016).
However, FL requires strong form of homogeneity and synchronicity among the data owners (clients) that might not be ideal in practice. First, it requires all clients to agree in advance to common model architecture and parameterization. Second, it requires clients synchronously communicate their model updates to common server, which assembles the local updates into a global learning feedback. This is rather restrictive in cases where different clients draw observations from different data modality of the phenomenon being modeled. It leads to heterogeneous data complexities across clients, which in turn requires customized forms of modeling. Otherwise, en- forcing common model with high complexity might not be affordable to clients with low compute capacity; and vice versa, switching to a model with low complexity might result in the failure to unlock important inferential insights from data modalities.
variant of FL (Hardy et al., 2017: Hu et al., 2019; Chen et al., 2020), named vertical FL, has been proposed address the first challenge, which embraces the concept of vertically partitioned data. This concept is figuratively named through cutting the data matrix vertically along the feature axis, rather than the data axis. Existing approaches generally maintain separate local model parameters

---

Under review as a conference paper at ICLR 2023
distributed across clients and global parameters on central server. All parameters are then learned jointly, causing however a practical drawback:
Coordination overhead among clients and the central server, such as engineering protocols that enable multiple rounds of communication (i.e., synchronicity) and coordination effort (i.e., homogeneity) to converge universal choices of models and training algorithms, would be required, which can be practically expensive depending on the scale of the application.
To mitigate both constraints homogeneity and synchronicity' satisfactorily, ask the following question and subsequently develop an answer to it:
Can we separate global consensus prediction from local model training?
As shown later in our experiments, we will address this question in a real-world context of the national electricity grid, over which thousands of phasor measurement units (PMUs) were deployed to monitor the grid condition and data were recorded in real-time by each PMU (Smartgrid.gou PMU measurements, as time series data, are owned by several parties, each of which may employ different technologies leading to heterogeneous recordings under varying sampling frequencies and measured attributes. These data may be used to train machine learning models that identify grid events (e.g., fault, oscillation, and generator trip). Such an event detection system relies on collective series measurements at the same time window but distributed across owners. Using VFL. build a common model on such decentralized and heterogeneous data is plausible but not practical, because of a lack of autonomy that facilitates coordination among the owners.
To resolve the challenge, we instead introduce a feature fusion perspective to this setting, which aims to minimize coordination among clients and maximize their autonomy via local-global model framework. Therein, each client trains customized local model with its data modalities. The train- ing is independent and incurs no coordination. Once trained, local feature representations of each client can then be extracted from the penultimate layer of the corresponding local models. Then, central server collects and aggregates these representations into a more holistic global representa- tion, used train a model for global inference. There are two technical challenges that need to be addressed to substantiate the envisioned framework.
C1. There is an ambiguity regarding the correspondence between components of local feature representations across different clients. This ambiguity arises because local models were trained separately in isolation and there is no mechanism to enforce that their induced feature dimensions would be aligned. As a matter of fact, it is possible to permute the induced feature dimensions without changing the prediction outcome. Thus, if two models are trained separately, they might end up looking at the same feature space but with permuted dimensions.
C2. There are innate local interactions among subsets of clients that need be accounted for. Naively concatenating or averaging the local feature representations accounts for the global interaction but ignores such local interactions, which are important to boost the accuracy of global prediction shown later in our experiments.
To address CI, note that the feature dimension alignment problem is discrete in nature; furthermore, there is no direct feedback to optimize for such alignment. To sidestep this challenge, we develop neuralized alignment layer whose parameters are differentiable and can therefore be part of larger network, including the feature aggregation and prediction layers, which can be trained end-to-end via gradient back pack-propagation (Section 4). To address C2, we employ graph neural networks as the global inference model, where the graph corresponds to the explicit implicit relational structure of the data owners. As such a graph might not be given in advance, we treat the combinatorial graph structure as a random variable of product of Bernoulli distributions whose (diffcrentiable) param- eters can also be optimized via gradient-baad based approach (Section 5). The technical contributions of this work are summarized below.
1. We formalize a feature fusion perspective for distributed learning, in settings where data is ver- tically partitioned. This an alternatiee view VFL but as elaborated above, more applicable when iterative training synchronicity is not possible among clients (Section 2).
Note that in our case, synchronicity requires co-training among clients which is weaker constraint than its usual meaning of further requiring clients to synchronize their updates per iteration.
2

---

Under review as a conference paper at ICLR 2023
2. We formulate a federated feature fusion (F²) framework that consists of a network of pre- pre-trained local models and a central model that collects and fuses the local feature representations (induced from these pre-trained models) generate a global model with better predictive performance (Sec- tion 3). This is achieved via addressing C1 (Section 4) and C2 (Section 5) above.
3. We demonstrate experiments with four real-life data sets, including power grids and traffic net- works, and show the effectiveness of the proposed framework (Section 6).
2 PROBLEM SETTING AND RELATED WORK
Federated Feature Fusion (F*) is a new but more practical setup for VFL (Hu et al., 2019; Chen et al., 2020); it aims to enable collaboration between data owners that possess private access to different sets of features describing the same set of training data points. However, unlike VFL which require clients to synchronize their training processes (Yang et al., 2019b; Li et al., 2021; Fu et al., 2021; Cheng et al., 2021; Hu et al., 2019; Diao et al., 2021) in multiple iterations of communication, F8 allows data owners to train their own local models in isolation and only requires one round of communication in which local feature representations induced from the heterogeneously pre-trained local models are shared with a trusted server for feature fusion.
Remark. Previously, similar ideas on extending federated learning to accommodate clients with heterogeneous models (Tan al., 2022b;a; Lin et al., 2020; Chen al., 2022) has been proposed but are still restricted to horizontal settings: Local models still need to operate on the same feature space and cannot trained isolation which consequently require multiple rounds of communication and potentially incur extra coordination overhead.
Thus, to emphasize on the novelty of our setting and solution significance, we further review and discuss the formulation of VFL and 1 below, which argues with concrete, real-life examples why the F setting is more practical and how this practicality would entail significant technical challenges that necessitate new solutions in Sections 4 and 5.
Federated Learning with Vertically Partitioned Data. From a data perspective, the decentralized nature of data in VFL is transposition to that of the traditional horizontal federated learning (HFL) (McMahan et al., 2017). Instead of owning the same set of features for different sets of data points as in HFL, the data owners in VFL now own different sets of features for the same set of data points; and they share a common label set of these data points.
From the existing literature, two lines of work are noted. One takes the data matrix literally by assuming tabular data and studying linear models - where model parameters have natural corre- spondence to the data parts (Hardy et al., 2017; Nock al., 2018; Heinze et al., 2014; 2016). Often, these approaches are hard to generalize to complex data with many owners. Another line of work advocates the use of models with modular structure in which separate parts of the model are respon- sible to locally aggregate different sets of local features owned by different owners; and global parameterization is used to combine these local features. This is similar in spirit to F3 but require clients to synchronize the training processes of their assigned model parts, which incurs expensive communication and creates dependence among the clients (Hu et al., 2019; Chen et al., 2020).²
Mathematically, for each datum x with label yk. let x be the feature set of the datum that the i-th owner possesses. That is, X (x .x...., ..... with n data owners. VFL computes:
m
minimize L (w, 0) w.0
l
( (x):0)) , 02 (x2: 02) ..... On (xf: On) ; w) yk
(I)
where each di (x}: 0.) is (learnable) local embedding of x parameterized by a separate parameter vector 0, owned by the i-th owner, g( Øz...., o. w) is an aggregation function parameterized by w and f a prediction loss, e.g. the cross-entropy loss for classification or la loss for regression. The loss in Eq. (1) is averaged over all training data points x1, X2,.... Xm-
Federated Feature Fusion. The setting of F is similar to VFL, except that the data owners share neither data nor models with each other to ensure higher degree of privacy compliance, which is
Note that the approach proposed by Hu et al. (2019) assumes parameters for the global model. Were global parameters present, gradient communication inevitable.
3

---

Under review as a conference paper at ICLR 2023
often the more practical setting in industry the example on power grid at the end of this section. For this reason, the VFL minimization task in Eq. (1) above is changed to
1 =
g (1k. hg. hg: ))::).
minimize L (ww)
where hi of (x)) with of arg ming, &s (x)(x,), (, yx) which characterizes the locally optimal feature representation obtained in isolation by the i-th owner. As such, Eq. (2) only requires one round of communication where (hi k. are communicated to trusted server. Prior that, each data owner can freely learn their own feature representation model (x(x)) with different parame- terization and architecture, catering towards their own compute capacities and data representation. This avoids forcing the data owners participate in a joint training scheme which often requires expensive coordination and is not practical. However, in exchange for this practicality, two key challenges arise. First, as local models are separately trained, the correspondence between compo- nents of induced feature representations across local models become ambiguous since there is no mechanism enforce their alignment. Second, for the same reason, there are potential innate local interactions among subsets of clients and a naive concatenation or averaging of their corresponding feature presentations will likely ignore such interactions, resulting in decreasing performance. These correspond to high-level challenge C1 and C2 in Section I which will be addressed in Sections and as our key technical contributions.
Data Example. Let us consider the power grid monitoring task as an example. Figure pictorially illustrates PMU measurements dis- tributed across data owners. A panel of time series corresponds to specific time window and the series collectively represent one data point, which the event detection system clas- sifies. In this simplified illustration, each data owner possesses one series recorded by one PMU; but in practice they may own different amounts of PMUs (and thus series). More- over, the series may differ in length because of varying sampling frequencies; and the series are multivariate with possibly different number of variates. All these variations contribute to data heterogeneity, which necessitates the con- struction of separate local models. Note that if event does not cascade over the entire grid, some local models may report event whereas others report normal, resulting in conflicting opinions. A consensus global model is respon- sible resolve the conflict. Additionally, miss- ing data may occur.
Data owner
Data owner 2
mmm
Data
Anre
owner
Data owner 4
Data owner 5
~mma
Data
mm
m
owner 6
generator error
transformer error
line error
Figure 1: Federated Feature Fusion: A global pre- diction produced collectively based on set of global features which are the result of fusing lo- cal feature representations supplied by the data owners. These feature representations from locally trained models on raw local data which might be heterogeneous.
induced
3 FEDERATED FEATURE FUSION FRAMEWORK
As detailed above, the proposed framework for Federated Feature Fusion consists of local models o; and global feature fusion model such that their composition minimizes the loss in Eq. (2). Each data owner i possesses a local model trained with its data, independently of other owners. This way, no data sharing is invoked and privacy is of minimal concern. However, because the local models lack a global vision and may be conflicting, central (global) model is key coordinating the local opinions for final prediction. To maintain autonomy, local models are frozen once pre- pre-trained and will not join the training of the global model. Data owners send local data representations to centralized server for global model training (and inference). In other words, the global model queries neither the raw data nor the local models from data owners. As long as owners agree to send the less decipherable representations the central server, global inference can be made.
Local Models. We treat a neural network except the final output layer as a feature extractor, which produces the representation hj of input fragment x and treat for simplicity the output layer as
4

---

Under review as a conference paper at ICLR 2023
latent dimension alignment
latent representation
graph neural network
owner 2
owner 3
W
label
owner
owner 5
Independent local models
Global model
Federated Feature Fusion Framework. Local models are trained independently and sepa- rately from the global model. The algorithm summarized in Algorithm
Figure 2:
logistic regression. That is, a local model gi (xx.) reads:
9. (x))
softmax (W; h + b.) where h = di (xx)
(3)
Hereafter, will interchangeably representation, embedding, and latent vector mean h These hg 's assumed to have the same shape across i, although x can have different shapes and the embedding function can have different architectures to cope with data heterogeneity. simple example of the embedding function is fully connected layer h ReLU(U, x c,); but an arbitrarily complex function also applicable.
Global Model. The global model g melds together all local representations to generate a prediction:
yk
yk
9 (h.. he.............................. h::w).
which parameterized by w. For example, the parameterization w = (Wo, W bo, bi can characterize fully connected layer followed by mean pooling and another fully connected layer:
g (hg. h w) = softmax
Wi
ReLU (Wo h + bo) + b (5)
1=1
Thus, given a particular parameterization w, we can substantiate Eq. (4) above and plug it into Eq. (2). The optimal value for w can then be achieved by solving the corresponding minimiza- tion task therein. However, designing the form of w is highly non-trivial and is in fact tied to the previously mentioned challenges C1 and C2, which we further elaborate below.
Challenges. Two considerations are pertinent to the design of w. First, when the latent dimensions have semantic meaning c.g. when the local models are trained to yicld disentangled representa- tions (Higgins et al., 2018) each latent feature of the local representations may not match, because an arbitrary permutation
the latent dimensions does not change a local model. Second, a naive mean pooling as in (5) may miss the interdependencies between local data, leading less well per- forming global model. Such interdependencies naturally occur in the power grid example because of the physics of electricity network. Hence, in subsequent sections, dress the first problem and graph neural network to address the second one. Incorporating these two components, we show the full proposed framework in Figure 2 and Algorithm 1, We now discuss the solutions to these challenges in Sections and 5 below.
latent alignment ad-
4 ALIGNING LOCAL REPRESENTATIONS
For the global model to be meaningful, the feature dimensions of the local representations h should be aligned under the same feature space. For example, in (5), all h 's multiply the same weight matrix Wo: in other words, each element of h; corresponds to one input neuron of the initial fully connected layer. Permutations of the elements will destroy the correspondence. That is, even if

---

Under review as a conference paper at ICLR 2023
Algorithm Federated Feature Fusion (F3)
1: function TR TRAINING ((( yk (==)>==))=>
2: Each data owner trains local model g² with its local data part (x)= Ma)/=11-
3: Each data owner sends its local data representations (h [ k== to the central server - Eq. (3)
4: Central server learns VI ( P?h?.... P"h") via Eq. (7).
5:
Here, the global model is (8), where the loss taken over the distribution of Å.
6: Entries of are sampled using (9).
7: Each alignment matrix P is learnable arbitrary parameter matrix.
8:
end function
9:
function INFERENCE(XI:-.. Xm where X = -x,))..,.;))
10:
Each data owner evaluates its local model with x to obtain h and sends to server.
11:
Server evaluates takes (hr as input and produces prediction via Eq. (5).
12:
end function
the local models are fixed, the arbitrary arrangement the feature dimensions of the latent vectors causes ambiguity of what optimal global model can be built.
Mathematically, let us use vector p to denote the index (column) permutation of vector (matrix).
Then, the ith local model (3) can be equivalently written
go (x)
softmax (W: [s)
hy [P.] + b. where hj. de (xk)
for any permutation P; as long as the embedding function is able to produce a permuted h [P:] under the same input xf- Such requirement can be easily satisfied if the embedding function is fully connected layer such h[p] ReL.U(W[p. (p;:) networks as well. In Appendix B, we give another example: the GRU (Cho et al., 2014).
b(p))- In fact, it is satisfied by most neural
Hence, propose align the feature dimensions across all local vectors h; to disambiguate the ambiguity. This proposal amounts to modifying the global model (4) to the following:
g (Pi h P2 h P.. hf)
Vk
where P, is alignment matrix for each data owner i, implementing the (manual) index column permutation above in linear algebra. We can then treat each P; as a free parameter matrix to opti- mize. may be square rectangle, the latter case indicating change of the number of features. We also show an alternative hard alignment by parametrizing P; a permutation matrix in Appendix H.
5 LEARNING A CONSENSUS GRAPH
The example global model (5) performs a naive averaging for the local representations. Since data owners are often interconnected, a more expressive model exploits their relational interactions to improve inference (Battaglia et al., 2018). To this end, we propose to use a graph neural network (GNN) (Zhang et al., 2020; Wu et al., 2021) to process the latent representations.
A. Modeling Consensus Graph GCN with Latent Graph. Many GNNs are applicable; we focus GCN (Kipf & Welling, 2017) for its simplicity. Let A be the graph adjacency matrix and let H₃ be the matrix of aligned local representations:
H.
Traditionally, GCN was designed for node classification so we modify slightly for our purpose,
(LITA. RoLU (AH.Wo) W,
yk
Ük
softmax
(8)
where À is a normalization of see (Kipf Welling, 2017) for details and Wo and W, are weight matrices. The modification is the inclusion of 1 pooling before output. Modulo this

---

Under review as a conference paper at ICLR 2023
modification, the formula (8) is standard one used in the literature, with the bias terms omitted. It is interesting to note the equivalence of GCN (8) and the graph-agnostic model (5) when A is replaced by the identity matrix (omitting bias terms).
In GCN, A corresponds the consensus graph among local owners as graph nodes. If such graph is not present, it is possible to learn one such that (8) still outperforms (5). In this case, we treat A a random variable of the matrix Bernoulli distribution, where the success probabilities parameters to learn. Formally, the elements Aij are independent and each follows Ber(O.y), where Bi denotes the corresponding probability (Kipf al., 2018; Shang et al., 2021). Then, the global model has Wo. W1, the P,'s, as well as 0, as parameters. Following Franceschi et al. (2019); Shang al. (2021), we formulate the training loss as an expectation over A' 's distribution and draws sample A to obtain an unbiased estimate of the loss as well as the gradient.
free
B. Differentiable Graph Sampling via Re-parameterization. However, the central challenge of this approach is that the sample A is not differentiable with respect to the corresponding Bernoulli bias Ozy. which in turn makes the training loss non-differentiable with respect to 0. To sidestep this difficulty. we propose the following reparameterization, which presents learnable (differentiable) transformation of a sample drawn from a continuous distribution to a discrete Bernoulli sample. This transformation is detailed in Definition I below, which is followed by Theorem I showing the distributional convergence of this transformation to the desired Bernoulli distribution.
Definition 1. Let be the CDF of an arbitrary continuous probability distribution. Sample from this reference distribution and let
())-)))))))
sigmoid
t T>0
We call this the ICDF re- re-parameterization which is named after the use of inverse cumulative F-!
Theorem 1. For all r 0, A E 10, and t [0, 1]. if the distribution with CDF F is finitely supported on [a, b). then
if < (F
Pr(: St)
if > sigmoid((=-10)) - (F-F((F)(0)) + r 1og(((1111)) otherwise.
(10)
On the other hand, if the distribution is not finitely supported (i.e., = -o and/or b = +xx), Eq. (10) still holds because either (or both) of the first two cases will not be invoked. As a conse- quence, the distribution of z converges to Ber(O)
- 0.
Discussion. We note that an alternative to the above can be achieved via using the Gumbel softmax reparameterization (Jang et al., 2017; Maddison et al., 2017) which also features differentiable relaxation of the categorical distribution (in this case, the Bernoulli distribution) that approximates it asymptotically. However, in order to obtain one Bernoulli sample, the Gumbel trick requires to sample the Gumbel distribution twice. Instead, our proposed reparameterization only requires sampling from the reference distribution only once. We also show that the ICDF re-pe e--arameterization converges as fast as the Gumbel softmax. Both approaches have asymptotic convergence rate on the order of O(72) as shown in Section D. Empirically, we also show that ICDF induces marginally better performance than Gumbel softmax. This is why we prefer ICDF to Gumbel in our work.
6 EXPERIMENTS
In this section, we demonstrate comprehensive experiments to show that federated feature fusion (F3) can be effectively conducted by using the proposed techniques in Sections 4 and 5.
Datasets. We use four real-life, time series datasets. Two are PMU data collected from multiple data owners of the U.S. power grid. For proof of concept, we smooth out heterogeneity and prepare homogeneous data sets. Such a pre-processing is sufficient to test the proposed techniques under minimal impact of the complication by the otherwise diverse local models. Since the PMU data sets are proprictary, we also use two public, traffic data sets (Li et al., 2018) for experimentation. A summary of these data sets is given in Table and the processing details given in the supplement.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Datasets.
10-
METR-LA PEMS-BAY
PMIL-B
METR-LA
PEMS-BAY
PMU-B
PMU-C
# Data samples
2856
4343
4853
1884
# Data owners
207
325
43
188
Series length
12
12
30
30
Features
01
# Classes
2
2
prediction entropy
Missing data?
no
n0
yes
yes
Given graph?
yes
yes
Figure 3: Distributions of prediction en- tropy across local models.
Experiment Setting, All local models are LSTM (Hochreiter & Schmidhuber, 1997) with the same hyperparameters, pre-trained separately by using local data. The local models are not fine-tuned in the training of the global model. Each dataset is split randomly for See the supplement for further details.
Confticting Local Predictions. We first show that local models do not produce consistent predic- tions, which justifies the effort of training global model and performing federated inference. For each datum, compute the entropy of the predicted labels and summarize the entropies for all data into a distribution, plotted in Figure 3. Recall that the lower the entropy. the more consistent the local predictions. The figure, however, shows that substantial amount of entropies is away from zero, suggesting that local predictions are inconsistent.
Effectiveness of Federated Feature Fusion. We make two sets of comprehensive comparisons to evaluate the effectiveness of the proposed framework. The first set, as outlined in Table 2, com- pares F with number of non-graph baselines (A-G) and including (a) vanilla federated learning baseline (A) which requires both model homogeneity and training synchronicity among clients that are not admitted our setting: (b) set of standard ensemble strategies (B-F) combine the local models such as voting, binary thresholding, best local model, meen-pooling via Eq. (5), as well as an advanced fusion model via simplified Set Transformer with layers and 4 heads (Lee al., 2019) (F); and (c) a vertical federated learning baseline (G) via feature concatenation. This set also con- tains several variants our proposed federated feature fusion model, featuring an ablation study of the effectiveness of our model components: (H) F without alignment; (K) F with learnable align- ment; and (J) F with partially tied parameter among local models which require local models to standard FL to collaboratively learn a common feature aggregation matrix Vi [n] : W W. readers can recall the role of W, in Eq. (3) among all local models. Baseline (K) is thus an alternative to alignment which comes with the cost of imposing strong homogenization though not as strong as (A) among local models despite the different nature of their local data. All variants of F (HI-K) use the ICDF re-parameterization to learn the graph structure.
From Table 2, we observe that baselines (A-D), either lacking necessary localized models or a holis- tic global model, perform significantly worse than the other baselines (including our F variants, the ensemble via mean pooling baseline and vertical federated lcarning). On the other hand, basclines (E-H) perform better than (A-D) but lacking a proper alignment of local models or imposing strong form of homogenization among local models to sidestep alignment, they are expectedly out- performed by baseline (K) that performs alignment.
We also compare with two variants of our final model K in the vertical federated learning setting. uses the same local and global models K but allows gradients be sent back to local clients, thus local models can be updated. It achieves similar performance as K but leads to much more communication cost with multiple rounds of gradient messages. M assumes no local pretrained models and all local and global models are trained jointly from scratch. Its performance is much worse than K and L - explains the merit of pretrained local models. Another typical VFL baseline (G) with pretrained local models and simple concatenation based global model is also inferior.
Impact of Learning Graph. Our next set of experiments, as outlined in Table 3, demonstrate the impact of leamning graph that characterizes the innate local interactions among subsets of clients, following our challenge statement C2 in the introduction, on both alignment and non-alignment baselines. This provides ablation studies the isolated impact of having specifie graph learning
8

---

Under review as a conference paper at ICLR 2023
Table 2: Effectiveness of latent alignment in graph-based global model. Superscript numbers are standard deviations. Note that baseline (A) and (H) are not applicable to the federated feature fusion where (local) model homogenization and training synchronicity are not allowed.
METR-LA
PEMS-BAY
PMU-B
PMU-C
AUC
F1
AUC
AUC
AUC
A: Federated Learning
.25-000
33-000
.36000
.99 000
B: Majority Voting
.11 -11-000
.09 000
.29 000
.18 000
C: Binary Thresholding
.69.000 000
.64 64-000
D: Best Model Selection
500
.70 .70-000
.55 000
.79 000
.37 000
.69 000
.32 000
.62 000
E: Mean Pooling Eq. (5)
.77-009
.96
.74-012
.93 001
.38 .38-009
.71 9D6
.34-00s
.64-020
F: Transformer
.78 023
.4-011
.72 U44
.93 U27
.39.003
.70-009
.40 U63
.67 0058
G: Concatenation
883 008
.97 002
.80 066
.96 028
.39 008
.71 71-036
.40 025
.68 040
1S with no alignment
.80 009
.96 004
.75 009
.94 001
.39 .39.003 003
.73
.40 020
.66 018
J: F' with parameter tying
.22 U09
.97 UD1
.75
.94 U04
.39.
.72.u10
.37-012
.66.008 00g
Id with alignment
.83 010
.97 001
.86 005
.8 002
.39 39.008
23-008
U16
.72-003
L: VFL w. graph/alignment
.83 .83.012
.7 001
.86 014
.98 002
.39 UDG
.74 009
.45 015
.73.003
M: VFL w.o. pretrained local
.77-02
.4 921
.77 014
.95 006
.34 .14014
.69 .69-012
.35 008
.65 014
Table 3: Impact of learning graph across different alignment settings. with respect to Table 2.
Some references of rows are
METR-LA
PEMS-BAT
PMU-B
PMU-C
AUC
F1
AUC
AUC
AUC
No Align
No Graph
.768 7688009
955-9004
.738 012
.935 001
.381 381-005
.711 006
.42 UDs
.636 910
Given Graph
.763 020
.957-007
.742 024
.42 005
Is-NN Graph
.715-016
.9520004
.695 013
.934-004
.3720001
.711-013
416-016
.68 .68-014
ICDF
.798 .798-009 0000
.963 004
.755 009
.943 001
.387.003
.734
.403 020
.663 018
No Graph
813-009
9070-002
.846 008
.977 001
.386.009 000
.725
.386 0DS
.694 005
Align
Given Graph
828 027
901-001
.854 003
.977 001
k-NN Graph
803 020
.968 002
.855 003
.973 002
.378.002
.718
.418
007
.702 009
ICDF
.835 .835.010
.975 001
.860 005
.980 002
.390 008
.734 108
.451 015
.725 003
component. In particular, for each alignment setting, we demonstrate the impact performance with (a) not using a graph; (b) using graph given by the domain experts; (c) learning the graph structure using a k-NN baseline (Fatemi et al., 2021); and (d) Iearning the graph structure using ICDF. The reported numbers suggest that regardless of whether the model performs alignment, graph learning always improves performance.
Remark. The i--N baseline (k: 10) implemented following the description in (Fatemi al., 2021). Specifically, during training. we generate local graph for each batch for node features X via symmetrization of À k--N D-NN(MLPP) (MTP(X))) which (1) feeds the node features through MI.P neural block; and (2) draws an edge between each node and its nearest neighbors where the neighborhood is defined using the cosine similarity on the space of MLP-projected feature vectors.
7 CONCLUSIONS
In this paper, we study federated feature fusion, which presents a less addressed scenario of federated learning where data owners or clients need to customize their own local models to accommodate different sets of (federated) features. Unlike federated learning, the clients need to learn their own model separately in isolation and only communicate their local feature representations afterwards. We motivate the practicality of federated feature fusion scheme with power grid example and propose a local-ge -llobal model framework for it. Two important components the framework are the alignment of the data representations produced by local models and the learning of the global model by using a graph neural network. Comprchensive experiments suggest the feasibility of federated feature fusion and the effectiveness of the framework.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jan Philipp Albrecht. How the GDPR will change the world. European Data Protection Law Review,
2016.
Peter Battaglia, Jessica B. Hamrick, Vietor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. Preprint arX:v: 1Xx::11866 1806.00012220 261, 2018.
Fengwen Chen, Guodong Long, Zonghan Wu, Tianyi Zhou, and Jing Jiang. Personalized federated learning with graph. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22). 2022.
Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. VAFL: method of vertical asynchronous federated learning. In ICMI. Workshop. 2020.
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems, 36(6); 87-98, 2021.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. 2014.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. The International Conference Learning Repre- sentations (ICLR), 2021.
Patrick Emami and Sanjay Ranka. Learning permutations with Sinkhorn policy gradient. Preprint arXiv:1805.07010, 2018
Bahare Fatemi, Layla El Asri, and Seyed Mchran Kazemi. Slaps: Self-supervision improves struc- ture learning for graph neural networks. Advances in Neural Information Processing Systems, 34: 22667 22681, 2021.
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In ICML, 2019.
Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. Vf2boost: Very fast vertical federated gradient boosting for cross-enterprise learning. In Pro- ceedings of the 2021 International Conference on Management of Data, pp. 563-576, 2021.
Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorn. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic eneryption. Preprint arXiv:1711.10677 2017.
Christina Heinze, Brian McWilliams, Nicolai Meinshausen, and Gabriel Krummenacher. LOCO: 2014.
Distributing ridge regression with random projections. Preprint
Christina Heinze, Brian McWilliams, and Nicolai Meinshausen. DUAL-LOCO: Distributing statis-
tical estimation using random projections. In AISTATS, 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. Preprint arXiv: 18:118883 812.02230, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. FDML: A collaborative machine learn- ing framework for distributed features. In KDD, 2019.
10

---

Under review as a conference paper at ICLR 2023
H. V. Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou. Jignesh M. Pa- tel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Commun. ACM, 57(7):86-94 94, 2014.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softtax. In ICLR, 2017.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafacl G.L. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adria Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi. Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Se- bastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Preprint 19I.:1911..04977, 2019.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang. Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In ICML, 2018.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, ((1-2):83-97 1955.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural DAG learning. In ICLR, 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- permutation-invariant neural networks. In Interna- tional conference on machine learning, pp. 3744-3753. PMLR, 2019.
Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. Label leakage and protection in two-party split learning. arXiv preprint arXiv:2102.08504 2021.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. Preprint arXiv:1907.09693, 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural net- work: Data-driven traffic forecasting. In ICLR, 2018
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effcceee learning of deep networks from decentralized data. In AISTATS, 2017.
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snock. Learning latent permutations 2018.
with Gumbel-Sinkhorn networks. Preprint
Richard Nock, Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Giorgio Patrini, Guillaume Smith, and Brian Thome. Entity resolution and federated learning get a federated resolution. Preprint arXiv: 1803880380000 04035, 2018.
11

---

Under review as a conference paper at ICLR 2023
Chao Shang, Jie Chen, and Jianbo Bi. Discrete graph structure learning for forecasting multiple time series. In ICLR, 2021.
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matri- ces. Pacific J. Math., 21(2):343-348, 1967.
Smartgrid.gov. Recovery act: Synchrophasor applications in transmission systems. retrieved in Jan 2021.
Webpage
http://www........oo//eeeeeeee.e html..
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fed- proto: Federated prototype learning across heterogeneous clients. In AAAI Conference Artifi- cial Intelligence, volume 1, pp. 3, 2022a.
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. arXiv preprint arXiv:2209. 10083, 2022b.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICIR, 2020.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In KDD, 2020.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions Neural Networks and Learning Systems, 32(1):4-24, 2021.
Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 10(2), 2019a.
Shengwen Yang, Bing Ren, Xuhui Zhou, and Liping Liu. Parallel distributed logistic regression for vertical federated learning without third-party coordinator. arXiv preprint 2019b.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In ICML, 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, and Trong Nghia Hoang. Statistical model aggregation via parameter matching. In NeurlPS, 2019a.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In JCML, 2019b.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. Transactions on Knowledge and Data Engineering, 2020.
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Con- tinuous optimization for structure learning. NeurlPS, 2018.
12

---

Under review as a conference paper at ICLR 2023
A
RELATED WORK
The concept of federated learning first coined by McMahan et al. (2017) and it has attracted surging interests since. A form of distributed optimization, federated learning is faced with data challenges beyond conventional assumptions and puts communication efficiency and data privacy as primary concerns. Recent surveys (Yang et al., 2019a; Li et al., 2019; Kairouz et al.. 2019) comprehensively study the subject, review systems and infrastructures, and suggest open problems.
The typical setting of federated learning is that data sets across owners share the same feature space but differ in samples. Besides this horizontal partitioning of the data matrix, a vertical partitioning was studied by Hardy et al. (2017); Nock et al. (2018); Heinze et al. (2014; 2016), wherein fea- tures are split across owners instead. This setting bears resemblance our federated feature fusion scenario, but crucial distinction is that existing methods for vertical federated learning all perform joint training. In the referenced work, to preserve privacy, encrypted data or randomly projected data are communicated among data owners as well as central coordinator. Such an approach incurs de- manding communication for many owners. Recently, Chen et al. (2020) study a different model, whose parameters are distributed among owners well central server. The part of the model corresponding to an owner bears resemblance to our local models; but they are not local models since they are not independently trained by using local data. Another work along similar direction is conducted by Hu et al. (2019), but the global model has no parameters; it is merely a sum of the local outputs followed by activation (e.g., sigmoid for classification).
Our framework learns parameter matrices to align local representations. Such alignments similarly appear in model fusion, where number of models are fused together into a common model through aligning model parameters (Yurochkin et al., 2019a). In the context of deep learning, if the neural networks come from the same model family, their weights can be matched layerwise. even the numbers of weights are different (Y (Yurochkin et al., 2019b; Wang et al., 2020). The referenced work treats the problem as a bipartite graph matching, where the cost matrix is inferred from maximum a posteriori estimation. Then, the Hungarian algorithm (Kuhn, 1955) is applied to find the matching. In our work, instead we treat the permutation alignment as a differentiable parameterization with the help of Sinkhorn-Knopp (Sinkhorn & Knopp, 1967; Mena et al., 2018; Emami & Ranka, 2018), so that it can be learned end-to-end with other parameters of the global model.
Our framework also advocates learning graph of data owners in the global model. Graph structure learning appears under various contexts. One field of study is probabilistic graphical models and casual inference, whereby directed acyclic structure is Iearned. Gradient-bris basen approaches in this context include Zheng et al. (2018); Yu et al. (2019); Lachapelle et al. (2020). On the other hand, a general graph may still be useful without resorting causality. Recent approaches supporting GNN- based modeling include Kipf et al. (2018); Franceschi et al. (2019); Wu et al. (2020); Shang et al. (2021), wherein graph structure simultaneously learned together with the GNN parameters. The Gumbel trick (Jang et al., 2017; Maddison et al., 2017) is frequently used for differentiable parameterization, but in this paper we study more economic alternative parameterization ICDF.
B PERMUTATION AMBIGUITY EXAMPLE FOR GRU
In Section 3, we discuss that one can arbitrarily permute the latent representations while keeping local model fixed. Here, we give another example - the GRU. Let x = {œ1, Ø2...... FT be an input sequence. The embedding function h embedding(2) implemented as GRU reads:
function h G)U((z))E...)
2: Anp = 0
3:
for = 1...., T do
4:
2; sigmoid(W,ng .x + U_h4-1 + bz)
5:
Tt = sigmoid (Wr + Uph1-1 + br)
6:
Te tanh(W,.. n., + Un(r, @ 11-1) bn)
7:
h, = (1 - 21) 2 4-1 + 21 O n4
8:
end for
9:
return h hr
10: end function
13

---

Under review as a conference paper at ICLR 2023
One can arbitrarily permute the elements of h through manipulating the GRU parameters properly. To achieve ht[p] = embedding(2; p),
the gate outputs and bias vectors (Zz, Tt. 7y, he, bus br, bn)) will need be permuted accordingly (zr[p), rx[p]. ne[p]. h,[p]. ba[p], [p]. [p]:
the weight matrices attached to the input (W,, W,, W.) will need to have their rows (i.e., output neurons) permuted (W, pp :. [x, =)); and
the weight matrices attached the hidden states (Uz, Uy, Un) will need have both their rows and columns permuted (Us[P. pl, U+IP, Un(p, pl).
C PROOFS AND ADDITIONAL RESULTS OF THEOREM 1, SECTION 5
C.I DISTRIBUTION OF GUMBEL SOFTMAX
The Gumbel softmax reparameterization trick (Jang al., 2017; Maddison et al., 2017) works the following manner. Let Cat( be the categorical distribution with probability vector π and let of same shape as π,, vector variable whose elements are i.i.d. ~ Gumbel 1). Then,
softmax (- (log π + g) +
y
T T>0
(11)
admits distribution converging Cat(T)) when + O. Hence, sample Ber(() approximately but differentiably, it suffices to let π = [0, 0 and use y/1 as the sample.
As preliminary, we consider the first entry 8/1 the random variable y defined in (11) for the Gumbel softmax parameterization. Note that for any т * 0. is only approximately binary; the possible values of y/1 in fact span the entire interval [0, 1]. We derive the following CDF for y1. Recall that for notational simplicity, 0 denotes scalar rather than a matrix.
Theorem 2. For all 0, 0 (0, 1),
and E [0, 1), we have
t= - 0) (7 0) (1-1)
Pr(v1 5))
(12)
Proof. We first consider the case 0 < t << 1. Through simple algebraic manipulation, we obtain that y1 is equivalent to
0 0
91 - 92 S
T llog
log
(13)
1-t
Let 91 log(- log u) and 92 log( - log v). where and independent and 21(0, 1). Then, (13) is equivalent to
t"
M where
M
(1 t
Therefore, by recalling that and are uniform in [0, 1J2, we note that the probability that S AM happens is the double integral
Pr(v 2 wM J.= dodu.
This integral is nothing but
M M du
1 M'
which completes the proof of (12). The cases of 0 or obviously hold by continuity.
14

---

Under review as a conference paper at ICLR 2023
C.2 PROOF OF THEOREM 1
We first consider the case when the distribution with cdf F is finitely supported on (a, bj. Through simple algebraic manipulation, we obtain that S is equivalent to M where M == F- (0) + log(t= 1). If f < (F - b)/T), we see that M > b and thus such s can never T)/T), we see that M a, which indicates that s 2 M
occur. Similarly, ift sigmoid((F= always happens. Otherwise, when i is within the two extremes, the probability that 2 M happens F(M), concluding the proof of (10).
The statement of the theorem regarding the case when the distribution is not finitely supported is obviously true.
To show that the distribution of converges to Ber(O), let us first consider the scenario when the distribution with edf F is finitely supported. The edf of = (see (10)) is always continu- ous but it has three segments connected by two joints: & sigmoid((F- 22 = sigmoid((F - a)/T). When т + O, the joint t - and the joint t2 + and thus the middle segment has wider and wider support converging to [0, ll. Hence, it suffices to consider only the middle segment. Further, with an analogous argument for other scenarios, it is also true that it suffices to consider only the third case of (10).
6)/T) and
In this case, for any fixed t < and when T + O, we have T log(t 1 - 1) + 0 and thus Pr(2 S O. Meanwhile, r Flo(( - 1) is undefined. However, we know by definition that Pr(= I 1) = 1. Hence, the continuous distribution of converges to a degenerate distribution Pr(z 1) 1 and Pr(z 1) = 1. This is the CDF of (er
- F( (0)
cannot push
because then the limit of
D TUNING GUIDANCE FOR TEMPERATURE: T
Our tuning guidance for the temperature 7 is motivated from a asymptotic convergence comparison between ICDF and Gumbel re-parameterization, which is featured in the theorem below.
Theorem 3. When 7 is small,
Bias(yn) = 72220(1 - 0)(1 20) + O(T*),
(14)
Bias(z) = - 2 (F + 0(7*).
(15)
Moreover, when F is the CDF of normal variable ~ N(),
Bias(2) =
602 3 erf (20 1) (erf ((2011)) + O(74).
(16)
Its formal proof is detailed later in Appendix E.
Theorem 3 suggests that the ICDF method converges equally fast as does the Gumbel trick - both on the order of O(r2). On the other hand, the biases depend on 0. Thus, one cannot set temperatures T independently of the desired probability 0 to equate the two biases. In practice, 7 is a tunable hyper-parameter and guidance the tuning range is therefore necessary.
To begin, we use subscript to distinguish the two temperatures - Tg for the Gumbel trick and r, for the ICDF method and write, based on (14) and (16) and ignoring the high order terms,
Bias(y))
T20² 7 - where r(f) erf-1 1)e- (orf 1(28-1)) 13)=
- V=)(1-0)(22 -
Bias(z)
Note that r(0) is symmetric around 0 }. is concave, attains maximum when 0 }. and attains minimum 0 when 0 = 0, 1. Hence, if T3 a and a v2, the bias of the Gumbel trick is (approximately) smaller than that of the icdf method. On the other hand, for > v2, there exist 01 0g such that o - r(0)) r(02) and that Bins(yı) R Bias(z), whenever 0 [0, 0g). For example. when o x 2.5, on the interval 0 E [0.01, 0.99], the bias of the Gumbel trick is (approximately) greater than that of the iedf method.
15

---

Under review as a conference paper at ICLR 2023
Based on the foregoing, practical guide is the same tuning range of T for the ICDF method as for the Gumbel trick. A small change of a (e.g., v2 versus 2.5) will entirely flip the landscape of the bias comparison between the two methods. Because the tuning range is much wider than the change of o, for simplicity it suffices to fix =.
E PROOF OF THEOREM 3 AND ADDITIONAL RESULTS
By the definition of bias, we have
where E[æ] [
tdPr(s td)r(ss()==1 =1 st)= [
Bias x:) [[x] 0
Pr(x S 1) dt.
Therefore, for Gumbel softmax,
0 17 0) + (1-11))) 1)=0 dt,
and for icdf with any F.
Bias(z) = S F(F (() log(t 1)) dt 0.
We now prove Theorem 3 in few parts.
Proof of (15). Let F-1(() (8) and perform change variable m log((- 1). Then,
- Bias(=) - 1/- [F(s+ rm) rm) F())] di [ [F(s rm) - F())]
c
(1 + emm dm.
We perform Taylor expansion of J' around and obtain
F() n!.
F(s + Tm) - F(s) (
T m
n=
Therefore,
00
(()
m (1 + empz dm
Bias( 2) =
n!
n=1
Each integral term is finite and the odd terms vanish because the integrands are odd functions. Thus, for small т, we are left with
(s)
m² dm + O(7*).
Bias(z)
2
(1 + em em+
The definite integral evaluates to 3 we therefore conclude the proof.
Proof of (16). Equation (16) is straightforward by substuting
erf 1 (20 (orf (arf-1(2001 1))-
F*))
02 0²
into (15).
Proof of (14). To simplify notation, let B = 0/(1 0) and perform change of variable m log(t 1). Then,
[ (=(1-0) - (1-t)=8
(1 dt
+ Be (1+ em dm.
00 1 Be
Denote h(T, m) [1 -1 Treating h function of and performing Taylor expansion around zero, we obtain
(nn) (0, m)
((T,m)
m!!
7=0
16

---

Under review as a conference paper at ICLR 2023
Therefore,
(-(1-0) - 0) (1 0) + (1 1)r0
00 7
e
dt
(0,
(1 enjs dm.
n!
n=0
moment, we will show that for all n,
h (1)))) (0, m) C, where C₁ is independent of m.
(17)
Suppose that (17) holds. Then, each integral term finite and the odd terms vanish, because the integrands are odd functions. Therefore, for small we are left with
(-11 (1--0 di Co J 01 (1 -
L
-2 (1 + emys dm + C2
e
m
2 (1 emp dm + O(T²).
By calculating
Co = h(0, m)
[1 + 3 - 0,
C2 = =10 m) = 0(1 0) 20),
e
100
m²e"
+ em) dm π²?
em²z em+ dm 1,
we conclude that
72 - (1 20)
Bias( (y1)
+ +(T4)
6
It remains to prove (17). We suppress the argument on m and write g(T) = 1 + 3e and h(T)
g(r)_1_ By Faà di Bruno's formula,
(n)
(-1) g(0)++ Br,
(0)
..... 14
where Br, is the Bell polynomial. Clearly, (()) ++ 8 and g (0) Bm" for all 0. Hence, Bu,k multiple of m". Therefore, ((()) (0) is multiple of m".
E.1 ADDITIONAL RESULT REGARDING THE BIAS
Theorem 3 states results for small temperature The purpose is to understand the limiting be- havior of the bias. Here, we give an additional result for any t > 0. states that the biases of the two sampling approaches have the same sign. This result is nontrivial extension of Theorem 3 and requires a different proof technique.
Theorem 4. For any 5 >
Bias(y1) = 0 when 0 = 2 Bias( < when 0 >> 1 Bias(y1) > 0 when < },
(18)
Moreover, if F* (x=) (that is, the pdf) is even and increasing when æ < then
Bias(2) (2) > 0 when f < 2 Bias(z) = 0 when 2 Bias(2) < when 1 2"
(19)
We prove Theorem 4 in two parts.
Proof (18). Consider
Bias(y1) f(' g(t. 0) dt where gtt, 11-
t (1 0)
t² - 0) + (1 t)=0
With a brute-force calculation, we have
-11-1)-+77001 - - -
g(1 0) + 9(1 1,
[ET(1 - 0) + (1 - 0)=0[((1 ()= 01 +1
All terms the fight-hand side are positive, except 28. Therefore, when 0 2/2 g(t 0) + 9(1 ł, 0) > 0 and hence
dt > 0. Bias(y1) = f- g(l, 0) + g(1 1. 0)
2
17

---

Under review as a conference paper at ICLR 2023
The other cases (0 2/2 and A 22) are similarly proved.
Proof of (19). Consider
Bias(2) = S/¹ 1((,0) d) where 1((40) (() + + +og(( - 1)).
We have
h(1 0,,) = logg(t= 1)).
To simplify notation, let (0) s and log(t=1 - = a. Then, /(t, = F(s + a) and h(1 1, F(s a). Let first consider the case 8 < 0 and O. We that
F(s + a) - F(s) = ('(m) dm and F(s) - F(s - a) f ('mm) dm.
For any b > O, if s + b < 0, then by monotonicity, (' + b) > ('ss - b). On the other hand, if
b 2 0, then (' + b) (- (s b). In both cases, the right integral always smaller than the left integral. In other words.
F(s + a) + F(s - a) > 2F(s)-
In fact, the above inequality is also established when 8 < and < O. Therefore, whenever <
J. O(t,0) L- hit, 0) +h(11 -0 0,
dl > / F(F dt 0.
2
That is, Bias(z) > O. Other cases (s F-) > 0 and s = (0( = 0) are similarly proved.
E.2 EMPIRICAL COMPARISON BETWEEN GUMBEL AND ICDF RE- RE-PARRAMETERIZATTON
Extending the last experiment in Section 6, Table summarizes the time and memory consumption during the training of global models the four data sets. The results indicate that our developed ICDF re- parameterization is more economic than the Gumbel-Softmax approach.
Table 4: Time and memory consumption of F (five epochs) with respect to ICDF and Gumbel- Softmax re- prrrammtterrzztionn Time in seconds and memory is in MB.
METR-LA
PEMS-BA BAY
PMU-B
PMU-C
Time
Memory
Time
Memory
Time
Memory
Time
Memory
Gumbel-Softmax
87.89
832.38
270.52
1896.11 11
42.40
348.39 322.59
84,89 54.07
1119.13 894.63
ICDF
79.69
568.24
157.93
1167.19
30.16
F DATA SET DESCRIPTION AND PREPROCESSING
METR-LA and PEMS-BAY. These are traffic data sets (MIT licensed) used by et al. (2018). The former was collected from loop detectors in the highway of Los Angles, CA (Jagadish et al., 2014) and the latter was collected by the California Transportation Agencies Performance Measure System. Both data sets recorded several months of data at the resolution of five minutes. The network graphs are available, which were constructed by imposing radial basis function on pairwise distance of sensors a certain cutoff.
The data sets were originally prepared forecasting tasks and hence no labeling information exists. We adapt the data for classification. Specifically, we split the time series the hour, forming hourly windows. We label each window as whether or not it corresponds to rush hour. For proof concept, specify 07:00-10:11 in the original data sets, one of the attributes is time. We remove this attribute to avoid triviality and retain only the speed attribute.
and 16:00-10 1::00-9::0000 rush hour and the others non-rush hour. We note that
The specification of rush hours may not be highly accurate, but is a sensible practice to cope with the nonexistence of labeling information. Intuitively, the signal of rush hour comes from reduced
18

---

Under review as a conference paper at ICLR 2023
traffic speed, but not every location of the network experiences traffic jam. Hence, the diverse traffic patterns inside the same time window under a single label causes nontrivial challenges for local models to discern. Therefore, the need of global consensus model is justified and it fits well the federated inference scenario.
PMU-B and PMU-C. These proprietary data sets coordinately provided by multiple data owners of the U.S. power grid. No personally identifiable information is present. The suffixes B and C indicate the interconnects of the grid. The data sets come with thousands of annotated grid events spanning a period of two years; they form the classification labels. Many variables (attributes) of the grid condition are recorded; we select only the voltage magnitude and the current magnitude, because they appear to be the strongest signals for event detection based on domain knowledge, and also because more data are available for these two variables. The grid topology is not available.
For each event, we select a one-second window from the three-minute window that covers the ap- proximate annotated event time, based on the largest z-score. We retain a sampling frequency of 30Hz, even though some data are 60Hz. Furthermore, large amount of data are missing in the raw data. We impute the series by using pandas Dataa rame. inttineee interpolatte (method 'linear', 1 Limit rdireiioo. = 'both') ) from the Python pandas package. This way, a windowed series is complete if ever has raw data. Even so, many series are entirely empty. which corresponds to the scenario illustrated by Figure 1. Classes in these two data sets are rather skewed. For PMU-B, we remove a class that consists of only one data point and for PMU-C, we combine classes that contain fewer than 24 data points into single class.
G EXPERIMENT DETAILS
The experiments are conducted on one x86 node of a computing cluster with one a100 NVIDIA GPU. The compute node has eight Intel cores and 128GB memory. For each data set, we perform 70/10/20 random split for training, validation, and testing, respectively.
For local models, use LSTM with the same hyperparameters: one hidden layer whose hidden dimension is 16 and the maximum number of epochs = 200. We pre-train the local models and freeze their parameters afterward. We train each global model for maximum of 500 epochs and use early stopping according to the validation loss, with a patience of 50 epochs. For the GNN global model, we use a 2-layer GCN with skip connections. The hidden dimension set at 8 and we select the learning rate from (0.01, 0.001}. For missing data, we impute the node features by using zero.
H SOFT AND HARD FEATURE ALIGNMENT
Feature alignment can be achieved in two manners. The first approach is soft alignment, which treats each P a free parameter matrix to optimize. Such an alignment softens the one-to-one cor- respondence in the permutation constraint; i.c., each feature in the source can have weighted correspondence to each of the features in the target. That is the way we used in the main paper.
An alternative approach is hard alignment, which treats each pi a permutation matrix. Learning permutation matrices is challenging, however, because they correspond to combinatorial structures and are unsuitable for gradient-based training. We follow Mena et al. (2018); Emami & Ranka (2018) and relax Ps by a doubly stochastic matrix, which can be differentiably parameterized by the Sinkhorn-Knopp algorithm Sinkhorn & Knopp (1967). Specifically, starting from nonnegative square matrix Ko and column vectors To = = 1 of matching lengths, define the sequence
Cj+1 =1=100 (K) ry) and ry+1 1
Ø (Kuey), for j = 0, 1....
(20)
Then, under mild condition, K3 == diag(r,) , Ko diag(e,) converges to doubly stochastic matrix. We truncate the sequence at the Tth step and treat Ky as an approximation of pt.
Despite the advocation by Mena et al. (2018); Emami & Ranka (2018), we obtain the following convergence result of Sinkhorn -noop, which reveals no free lunch.
19

---

Under review as a conference paper at ICLR 2023
Table Different approaches to feature alignment.
METR-LA
PEMS-BAY
PMU-B
PMU-C
AUC
F1
AUC
AUC
soft alignment
.835
.975
.600
.900
.390
.734
.451
.725
hard alignment
.839
.973
.855
.76
.39
.737
.429
.721
local model
local model 48
local model 24
loral madel 72
06
18
(a) METR-LA
(b) PEMS-BAN BAY
(c) PMU-B
(d) PMU-C
Figure 4: Examples of learned permutation matrices (KT). The plots clearly show patterns of permutation matrix: there is one and only one significant value per row and per column. Because of the slow convergence, we attribute the desirable results of KT small 2) to the success of the learning of Ko. Note also interestingly that learned permutation may be the identity mapping.
Theorem (informal). Under a condition of Ko, there exists positive integer and constant C. such that for all j J.
| s CC ?())
K,1
where d2 is the second largest singular value of the limit of Kj.
Since this is not the focus of this paper, we omit the rigorous analysis of this theorem. The result suggests that for a desirable limit being permutation matrix, whose 02 1, the error ( does not drop. In practice, to expect for an approximate permutation matrix, 02 = and the convergence exceedingly slow. The practical usefulness of (20) depends the learned quality of Ko-
The soft and hard alignment approaches have pros and cons. The hard approach maintains the correspondence of each feature dimension of the latent vectors while the soft approach. Maintaining the dimension correspondence is an advantage, especially for local models that produce disentangled equipped with
latent representations Higgins al. (2018), because each feature dimension semantic meaning that controls a certain aspect of the data. On the other hand, the soft approach more straightforward and the hard approach is based on algorithm that barely converges. In practice, we observe that two approaches decisively similar performance. Due to space limitation we took the simpler approach and presented only the soft version in the main paper, but here we list the results for both approaches in Table 5, and we also visualize the hard alignment matrix learned on each dataset to help readers understand the feature alignment (Figure 4).
20


Reviews:
[{"review_id": "zFOJs5hYTB", "reviewer": "Reviewer_2ATz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers vertical federated learning settings. It addresses the homogeneity and synchronicity problems by proposing a federated feature fusion (F^3) framework and learning a consensus graph among local owners. Experiments on four real-world time series datasets demonstrate the effectiveness of F^3.", "summary_of_the_review": "Please refer to the comments in Strengths and Weaknesses. ", "strength_and_weaknesses": "Strength: \n\n1. The methodology part is written in a clear way and well addressed the main claims in the introduction part. \n\n2. It is an interesting idea to study the VFL problem from the feature fusion perspective.\n\n\n\nWeaknesses: \n\n1. The relation between the two mentioned constraints and VFL is not explicit. As illustrated in the Introduction, the main constraints considered in this paper are the homogeneity and synchronicity problem and the main setting is VFL. However, I do not see any discussion on the constraints and VFL. There should be more discussion on how the constraints hinder the performance of existing VFL methods. \n\n2. It is better to involve more components, e.g., loss functions, model parameters, and communication progress, in Figure 2 to make it clearer.\n\n3. It concerns me whether it is secure to directly transmit data representations from the data owner to a central server, because there are some techniques that can reconstruct raw samples from their representations. \n \n4. It is not persuasive enough to use representation alignment and graph convolution. Section 4,5 do not relate them to VFL or existing VFL methods, and it seems that the two techniques are designed for a certain type of dataset. \n\n5. There lacks a comparison to existing FL methods, especially the following works that utilize the graph to model the relationships among clients, e.g. graph-based aggregation [1], address the model heterogeneity problems, e.g. distillation-based fusion [2], gradient-wise aggregation [3], prototype-wise aggregation [4], and/or fuse representations output by various pre-trained models, e.g. pre-trained model-based aggregation[5].\n \n[1] Personalized Federated Learning With A Graph. In IJCAI 2022.\n\n[2] Ensemble Distillation for Robust Model Fusion in Federated Learning. In NeurIPS 2020.\n\n[3] HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. In ICLR 2020.\n\n[4] FedProto: Federated prototype learning across heterogeneous clients. In AAAI 2022.\n\n[5] Federated Learning from Pre-Trained Models: A Contrastive Learning Approach. arXiv 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q0x27RuLf5", "reviewer": "Reviewer_6rxh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes 'federated feature fusion' for application in the vertical federated learning paradigm using a combination of feature alignment and consensus graph. The technique aligns feature representations from pre-trained client models using a learnable alignment function and aggregates using a graph convolution network that learns the relational interactions of the different features to aggregate the global model. The algorithm is tested on four real-world time series datasets using LSTM models. Experimental results show good performance and are accompanied ablation studies. ", "summary_of_the_review": "The work proposes to use a graph convolution network to learn relational interaction between clients and uses an alignment technique to align the feature representations. The authors show good experimental results and ablation studies. Overall the work is a good contribution to federated learning. ", "strength_and_weaknesses": "The combination of feature alignment and graph consensus is a novel technique. The work has excellent ablation studies that concretely show the effectiveness of each aspect of the algorithm. One thing I want to note is that I think the datasets are uncommon in the FL community and it would be helpful to add brief descriptions of what the features and class labels are in the main text instead of the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aTU-KrShZlI", "reviewer": "Reviewer_Szi9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a problem in federated learning where local models are independently trained, but are aggregated to perform inference. This setting is a more practical setting than vertical federated learning (VFL) as it does not require expensive collaboration. To solve two challenges in this setting, feature misalignment and finding interdependency, the authors propose $F^3$. $F^3$ uses trainable modules to align the features generated from local models. $F^3$ then leverages graph neural network (GNN) and graph structure learning to capture interdependencies between local models. Experimental results on spatio-temporal data show the effectiveness of $F^3$. ", "summary_of_the_review": "My main concerns on this paper lies in 1) Lacking discussion to a related work CNFGNN with similar model design, and 2) Some ambiguity regarding technical correctness (ref. weaknesses). I recommend a weak reject at this stage. I am open to raising the recommendation if the authors clarify my doubts. ", "strength_and_weaknesses": "Strengths: \n- The studied problem is valid and novel. VFL does have the problem of high synchronization cost, while in practice, parties may indeed own heterogeneous features. Moreover, the authors justify the motivation using real-world examples clearly. I appreciate that. \n- The paper is clearly written and well-organized. I am able to understand the motivation, key challenges, design components, etc. of this work. \n\nWeaknesses: \n- Missing related work. Although this paper tackles the setting of independent training and joint inference, the formulation of federated training over time-series data with an underlying graph is similar to CNFGNN (Meng et al. 2021). The authors should discuss the relationship of this work and CNFGNN, especially over the model design. From my perspective, both CNFGNN and $F^3$ use separate temporal feature extractor, and use GNN to aggregate the temporal features. \n- Not clear why the $\\mathbf{P}$ does feature alignment. From my perspective, the matrices $\\mathbf{P}$ are not regularized (e.g. requiring local features to carry similar distributions) , and without specific parameterization (e.g. permutation matrices). Therefore, whether and why $\\mathbf{P}$ actually does feature alignment are at doubt. The effectiveness of introducing $\\mathbf{P}$ can easily come from introducing additional trainable parameters, instead of feature alignment. Some justifications are needed. \n- How to train the global GCN and graph structure learning is not clear. I assume that different parties own different labels, as indicated by Section 2, \"some local models may report event whereas others report normal\". Therefore, it seems that to train the global GNN, local labels have to be somehow passed to the global server. However, this seems to violate the privacy requirement of FL. Some justifications are needed on this point. \n- Efficiency advantage over asynchronous VFL is not clear. The motivation of $F^3$ instead of VFL is that VFL requires synchronization, which is not efficient. However, there are works on asynchronous VFL (e.g. FDML (Hu et al. 2019)) that breaks the synchronization requirement at the cost of some feature staleness. The efficiency advantage of $F^3$ over asynchronous VFL is thus not clear and requires justification. \n\nC. Meng et al. Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling. KDD 2021\n\nY. Hu et al. FDML: A Collaborative Machine Learning Framework for Distributed Features. KDD 2019\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K5_FjqC512Q", "reviewer": "Reviewer_iTHt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study how to learn an effective global model on private and decentralized datasets. Toward this end, the authors propose to use consensus graph to fuse feature representations for a final prediction. The experimental results on four real-life datasets are used to validate the effectiveness of the proposed framework. ", "summary_of_the_review": "The paper has its own merits but many necessary analysis and comparisons are missing. The comparisons against existing vertical federated learning need to be carefully conducted. The efficiency improvements need to be supported by empirical studies.  The convergence and privacy preserving analysis are needed. More details are listed in the Strength And Weaknesses section. ", "strength_and_weaknesses": "Strength:\n1. The studies problems are important and practical. \n2. The challenges of the studied setting are well summarized. The idea of consensus graph is practical when iterative training synchronicity is not possible.\n\nWeakness:\n1. The studies problem still lie in the vertical federated learning and the related work on the vertical federated learning does not cover the enough works, e.g, [1-4].\n2. The convergence analysis and privacy preserving analysis of federated learning algorithms are important and both of them are missing in the paper\n3. In the experiments, the paper need to be clearly compared with existing efforts in vertical federated learning.  The baseline G is a federated learning method which shows strong performance. First, the details of this method need to be included in the paper. Second, the good performance of the simple federated leanring baseline raise one question: if  the challenges regarding existing vertical federated learning in the introduction are over-claimed. \n4. The analysis regarding  communication and training efficiency is missing. As demonstrated in the introduction, \"Expensive coordination and synchronization among clients and the central server would be required, still imposing a strong constraint on synchronicity\". To support the claim, the improvement in the synchronicity need to be provided and the comparison against  Vertical Federated Learning is also needed.\n\n\n\n[1] Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. 2021. SecureBoost: A Lossless Federated Learning Framework. IEEE Intell. Syst. \n[2] Fangcheng Fu, Yingxia Shao, Lele Yu, Jiawei Jiang, Huanran Xue, Yangyu Tao, and Bin Cui. 2021. VF2Boost: Very Fast Vertical Federated Gradient Boosting for Cross-Enterprise Learning. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD 2021. ACM, 563–576\n[3] Oscar Li, Jiankai Sun, Xin Yang, Weihao Gao, Hongyi Zhang, Junyuan Xie, Virginia Smith, and Chong Wang. 2022. Label Leakage and Protection in Two-party Split Learning. In International Conference on Learning Representations, ICLR 2022\n[4] Shengwen Yang, Bing Ren, Xuhui Zhou, and Liping Liu. 2019. Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator. CoRR abs/1911.09824 (2019). arXiv:1911.09824", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject