Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new problem, called federated inference, and proposed a framework of solutions including local representation alignment and learning a consensus graph. The technical contribution mainly comes from a theorem (Theorem 1) on the convergence of learning permutation matrices, and a new approach (icdf) for parameter inference of matrix Bernoulli distribution. The results demonstrate the proposed approaches are useful in improving the prediction accuracy. 

main_review: The paper made a list of interesting contributions, including formulating a new research problem, developing a complete solution with a few technical innovation. The paper is easy to follow and the results are promising. 

However, I have the following concerns of the paper: 

1. The problem of federated inference is not well motivated: 1)  For federated learning, privacy concern is a major issue, each individual party only shares the gradient to the center server. In this case, the local party will share an embedding; and this may raise the concern on the data privacy. Especially, it seems the local models are known or being the same from each party (the parameters maybe different). I hope there are some discussion on the privacy aspects. 2) In addition, since the local models are pretrained separately by using the local data, the main problem can be considered as a classification problem with  a number of features f(h_1, h_2, ... h_n), where h_i have the same name of dimension. This naturally leads to model F (local mode+concatenation), which indeed shown to perform quite well against other advanced models (especially on AUC). 3) The applications of the problem seems to be limited; besides the time series data in power grid, are there any other real world applications?

2. Lacking proper baselines: a simple solution seems to be adding MLP (two/three layers) \sigma( W_i^\prime \sigma (W_i h_i)), and then  perhaps concatenate them  for the final aggregated output. In fact, since model F is already performing well, it worth looks into more to see how much it can be improved, and what are the issues of this direction. Those one or two extra layers seem to be able to capture the alignment issue or provide similar functionality.

3. Why the GCN is helpful in this situation? The rational for using GCN is not very clear (Paragraph 1 in Section 5 may need to be extended). And is it a novel contribution to use GCN this way? Since this is a rather simple modification of GCN, I wonder if similar attempt has been done for graph classification (classifying the subgraphs). 

4. The technical contributions from Theorem 1 and Theorem 2/3 are very different research problems, and quite disconnected. Are those indeed the main problems for this federated inference problem? 

5. The performance of the newly proposed models seems to be relatively small comparing the baseline models, such as model F (table 2). 







summary_of_the_review: Overall, I feel the paper may need some additional work to better connect different technical points, and provide discussions on the necessities of the sophisticated methodology. 

===================================
After reading the responses and other reviewers’ comments, I will keep my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work is motivated by the problem of federated inference, which arises when clients own different parts of a datum, and learning must be collectively performed on the complete data representation. This application is presented as an extension of vertical federated learning. The proposed approach is based on the training of local models at the clients’ level. From the local models, an opportune data representation is extracted and shared in a centralized setting, which is then used to perform global inference. To cope with the mismatch of data representations across clients, a centralized alignment  procedure is proposed, which provides the input for a graph neural network encoding the relational structure among clients. Experiments are provided for the specific application of modeling real-life time series datasets from data owners of the US power grid, showing that the proposed local-global analysis pipeline outperforms both local and global training.

main_review: The proposed pipeline is interesting and addresses the relevant problem of heterogeneity in FL. This being said, there are to my opinion important issues that should be addressed:

- The definition of federated inference is not very clear. According to the example depicted in Figure 1, the problem here analyzed consists in FL with heterogeneous data, where heterogeneity arises from different data dimensions within- and between- clients, and different sample size owned by each client. Is this actually the case? 

- If this is the case, would simple data standardization and resampling techniques to format the data prior to FL already be sufficient to account for the heterogeneity? My feeling is that a benchmark with simple pre-processing and data imputation approaches is needed to fully appreciate the value of the pipeline proposed in this work.

- The sharing of the data representation extracted from the local models raises important privacy concerns, as it may leak important information about the clients. This is a strong limitation of this work, which should be at least acknowledged in the paper.

- Concerning the sharing of the local representation, the proposed approach could be simply being reformulated as applying dimensionality reduction on the clients’ data (e.g. PCA), and train a centralized model on the top of the latent representations. In what the proposed framework would improve this simple baseline?

- The pipeline requires the further training of centralized models to provide the alignment and consensus graph. It is not clear whether the assessments of Table 2 and 3 are made on the data used for training such a models, or on an independent hold-out dataset. 

- Given the complexity of the proposed frameowrk, it would be useful to include a benchmark on controlled scenarios, for example using synthetic data. 

- Section 6. The sentence “we smooth out heterogeneity and prepare homogeneous data sets” should be clarified. How is the homogenization performed in a FL setting? Is there any information leaked across clients?

summary_of_the_review: The proposed framework is novel and addresses a relevant problem of FL.
The formulation of the paper seems ad-hoc with respect to the proposed application, as well as the related experiments. There are also  concerns on the privacy leaked during the sharing of the local representations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to train a GCN-based multivariate time series classifier by leveraging the pre-processed data (representation per instance) from multiple sources (data owners). The overall model training is decomposed into two steps. First, the client extracts representations using its pre-trained model. Second, the server aligns client-wise representation, and then trains a GCN-based classifier to leverage the structural information across clients.  
main_review: Strengths

1. The paper is well-written.
2. The proposed method is practical and technique soundness with insight discussion.
3. The proposed setting is explained well.
4. The main technique contribution comes from the server model design that is to tackle the data heterogeneity, neuron/feature alignment and consensus graph learning. 


Weaknesses

1. It will be controversial to name the proposed method as federated inference. Most federated learning methods are designed to serve each user or participant in a distributed system, e.g. smartphone users in cross-device FL, or a banking organisation in cross-silo FL. However, this paper’s setting is to train a global model for server use only while each participant works for data collecting and pre-processing (representation learning). Thus, this setting is more suitable to be categorised as distributed machine learning rather than federated learning. 
2. The training of each local model is independent without cross-client collaboration or server-guided coordination. Given the limited resources of each client, it will cause possible practical challenges to training a good model with desired performance.
3.  The experiments are based on datasets for power grid and traffic network with time-series inputs that are unusual benchmarks in FL. 
4. Despite the proposed new setting, the proposed method has very limited novelty from a technique perspective. Especially, there is no technical novelty from a graph neural network perspective.

summary_of_the_review: The paper is a combination between dynamic graph neural networks and federated settings. The proposed method is a practical solution for learning a model in a distributed system with topology information. However, the alignment of federated settings is unconvincing, and the contribution from graph neural networks is also very limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a local-global framework to perform “federated inference” in a less addressed scenario where a datum consists of multiple parts, each of which belongs to a separate owner. Different from federated learning, to perform “federated inference”, joint efforts are required only in inference stage. To enhance the proposed framework, the authors propose two alternative alignment methods and a consensus graph learning among the new representations of different clients. In the experiments, both the theoretical analysis and empirical results are presented to support the effectiveness of the proposed framework. The main technical idea is the isolated learning of representation of local data and then aligning them to make the prediction. But this learning paradigm may heavily hurt the privacy of federated learning, since the new representations may leak the local data, and alignment of multiple representations may also lead to inference attack of these data.
main_review: Pros:
(1) The paper proposed an interesting framework to perform "federated inference" to address the scenario which vertical federated learning also studies. This framework can save the expensive coordination among data owners and the central server.
(2) The authors leverage a bunch of techniques to address the hardship of federated inference.
Cons:
(1)	The studied topic is interesting, but the technical ideas have the high risk of leaking privacy of local datasets, due to the upload of new representations of the local data.
(2)	Because of the uploaded representations local datasets, attacks may be performed to infer the local datasets and destroy the privacy protection target.
(3)	In addition, the local representations may be cannot be well learnt by isolating them at first, and thus the alignment and global model training can not be guaranteed as federated learning. In fact, from the results of concatenation method, which has close results as the proposed complex one. The two-stage solution (first representation of local data, and then align representations and global model training) can not sufficiently prove its effectiveness.
(4)	In essence, the technical idea is the first local representations, and then fusion of these representations for decision. Although the authors terms this as “federated inference”, I would like the authors to more care about the “federated” term. In fact, an encryption technique can be adopted for each local data to encode the data and then make the prediction, it is not so clear for the advantage or necessity of federated inference. The authors can further clarify this point.
(5)	Other comments:
(a)The authors should specify methods A and K in Table 2.
(b) There are some similarities between this work and the work of Hu et al. [1] and Chen et al. [2], the authors can introduce a more details of above two papers in the Related Work. 
(c) Vertical federated learning also studies such a scenario, I consider the authors should compare the methods of Hu et al. [1] and Chen et al. [2] in the experiments, either in terms of performance or cost.
(d) The current experimental verifications are not so solid and well justified the technical ideas. I would like the authors to consider the weakness/comparisons mentioned in (1)-(4) to confidently prove the proposed method.
[1] Yaochen Hu, Di Niu, Jianming Yang, and Shengping Zhou. FDML: A collaborative machine learning framework for distributed features. In KDD, 2019.
[2] Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. VAFL: a method of vertical asynchronous federated learning. In ICML Workshop, 2020.

summary_of_the_review: The paper considers a less addressed scenario where a datum consists of multiple parts, each of which belongs to a separate owner and proposes a local-model framework to perform “federated inference” in this scenario.  The main technical idea is the isolated learning of representation of local datasets and then aligning them to make the prediction. But this learning paradigm may heavily hurt the privacy of federated learning, since the new representations may leak the local data, and alignment of multiple representations may also lead to inference attack of these data. Furthermore, the validation and justification of the proposed methods are still not so mature for a top-tier conference.
review_title: null

Meta-review: The paper proposes to compute local representations on device, which are then shared between clients using an alignment mechanism. Reviewers did appreciate the value of the topic and several contributions, but unfortunately consensus is that it remains below the bar, even after the discussion phase. Concerns remained on privacy and motivational positioning with FL, and lack of simpler baselines, even after the author feedback.

We hope the detailed feedback helps to strengthen the paper for a future occasion.
Decision: Reject