Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors consider applying Byzantine-tolerant federated learning for online GPR and proposed a new algorithm with various modules for IoT applications. Theoretical supports on the proposed algorithms are also provided. 

strengths_and_weaknesses: Strengthes:
1. A novel Byzantine-tolerant federated GPR algorithm for online streaming data. 
2. Theoretical performance analyses on the posterior mean and variance subject to attacks. 

Weaknesses:
1. The main ideas and techniques rely on reference [24] to certain extent.
2. Attack is perhaps too simple.
3. Comparison with other methods is insufficient, for instance, 
(a) Gaussian process with differential privacy, https://arxiv.org/abs/2106.00474
(b) Reference [23] 
4. The following reference also discussed about the malicious protection and privacy issue of distributed GP for IoT applications, which may be mentioned in the manuscript:
https://ieeexplore.ieee.org/abstract/document/9250516
questions: 1. Do assumptions 2 and 3 hold even for the SE kernel? This is not justified in the paper. 
2. Are the bounds tight in Theorem 1 and 2? 
limitations: 1. Insufficient algorithm comparison.
2. Insufficient experimental evaluations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper explains the Byzantine-tolerated local approximation Gaussian process (GP) model for streaming data. In the GP framework, distributed learning scenario is used to scale the costly GP to big data. Besides, this local approximation method is used in federated learning when security requirements should be satisfied. The paper proposes a solution to divide the agent set into Benign and Byzantine agents sets. By specifying the Byzantine agents, the model excludes them and uses the benign agents to reduce the negative effects of the attack. The authors evaluate the proposes model on synthetic and real world data sets to show the capacity of their solution. 
strengths_and_weaknesses: Strengths:

The paper correctly defines the problem. It explains the model, restrictions, related assumptions, and the main problem by addressing all related issues. 

Weaknesses: 
The paper has some shortcomings, and I will mention them here:

1- Agent-based local GPR: 

The size of the local input data in an agent is not determined. When time rises (i.e., t goes to infinity), how the model handles the data assigned to each agent? In streaming data, the size of the input data increases continuously. Since the local models use the nearest neighborhood GP regression, how many streaming inputs should be selected for NNGP? 
The size of the data inputs raises, so the model needs a clearly defined strategy that determines the local partitions' scope, the number of neighborhood points in NNGP, and related complexity. The computational cost of a GP has a cubic dependency on the sample size (in the local approximations, cubically depends on the subsample sizes). Since the sample size increases when time t raises, the model needs a predefined mechanism to deal with complexity. Moreover, the complexity O(t) mentioned in line 146 only shows the search cost in NN and not the cost of the training of the NNGP. 


2- Cloud-based aggregated GPR

The paper's main contribution is this part, where they proposed a solution to find Byzantine agents. The paper considers the agents with the smallest and largest local means and variances at each test point as potential Byzantine agents. It is not supported by a theoretical analysis in the paper and is a heuristic trick. Consider a simple case that all Byzantine agents reported small values as local predictions while the higher mean values are more proper. In this case, steps 4 to 7 in Algorithm 3 exclude a small number of Byzantine agents, while some high-quality agents are also excluded. 

There are some related selection methods in the literature. For instance, the Gaussian graphical model can select the benign agents using the precision matrix between agents' predictions. The agent selection method in (https://arxiv.org/abs/2102.01496.pdf)  uses the interactions between agents in the related graphical model, determines those agents whose predictions are not close to the other agents' predictions, and excludes them for the final aggregation. Indeed, the ensemble technique in (https://arxiv.org/pdf/2010.08873.pdf) can divide the agents set into two clusters, Byzantine and benign agents, using spectral clustering and the precision matrix of the local predictions. Besides, the importance of an agent in the original GPoE (https://arxiv.org/pdf/1410.7827.pdf) and robust Bayesian committee machine (https://arxiv.org/pdf/1502.02843.pdf) can also help to select benign agents. They proposed the difference in the differential entropy between the prior and the local posterior distribution. 

3-  Byzantine-tolerant PoE 

The generalized product of experts (GPoE) with equal weights (Deisenroth et al. 2015) can provide conservative predictions. This model averaging approach uses the agents' weights to improve the overconfident predictions of traditional PoE. However, its prediction quality is generally lower than the Bayesian committee machine family, mainly when disjoint partitioning is used. It would be better if the authors considered the other newer aggregation methods. 

4- Agent-based fused GPR

This part is somewhat unclear. When the local predictions have been combined using PoE, what is the reason for this step? It should be explained that can it change the final aggregation? The model compares the variance of the GPoE and local variances.
The reason for comparison is unclear, and also, the paper does not describe why this comparison and related replacement (steps 3 and 5 in Algorithm 4) are essential? 
questions: Please check the points mentioned in "Strengths And Weaknesses.". Indeed, it has been confirmed that disjoint partitioning captures the local features more accurately and outperforms random partitioning (see reference [23] in the paper); why did the authors use random partitioning in the experiments? 

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies consider Byzantine-tolerant federated learning for streaming data using Gaussian process regression (GPR). A cloud and a group of agents aim to collaboratively learn a latent function where some agents are subject to Byzantine attacks. The paper develops a Byzantine-tolerant federated GPR algorithm, which includes three modules: agent-based local GPR, cloud-based aggregated GPR and agent-based fused GPR.


strengths_and_weaknesses: Strengths:

1. Develops a Byzantine-tolerant federated GPR algorithm. The algorithm includes three modules: agent-based local GPR, cloud-based aggregated GPR, and agent-based fused GPR.
2. The predictive mean of the cloud-based aggregated GPR is theoretically guaranteed to be restricted within a neighborhood of the target function provided that Byzantine agents are less than one-quarter of all the agents.
Experiments on a synthetic dataset and two real-world datasets are conducted to evaluate the proposed algorithm.

Weakness:
1. Technical contribution: the paper is a combination of existing methods on Byzantine attacks, federated GPR, and online learning, the technical challenges of such a combination, however, have not been identified, so the technical contribution of this work is not clear.
2. Related work: the paper does not include discussions with related work, there has been a sequence of works on Byzantine resilient secure federated learning, federated learning, online learning, and data streams. However, the difference between this work and theirs are not clearly stated.
3. Experiments: this paper targets data streams, but in experiments, there is no study in terms of data streams; for example, to simulate streaming data with concept drift. The measures should also include efficiency, as well as computation and memory costs.
questions: 1. What are the technical challenges and contributions?
2. Is there a section of related work?
3. How to address the streaming data challenge, e.g., concept drifting?
4. How to simulate and test a data stream in experiments?
limitations: See the above questions.
review_title: null

Meta-review: metareview: This paper proposes a novel Byzantine-robust method for performing Gaussian process regression in a Federated learning setup. The contributions include the proposed method (novel), an accompanying theoretical analysis, and experiments.

The reviewers were generally favorable about this paper, finding that the approach is novel, and appreciating the theoretical guarantees. Several criticisms were raised in the initial reviews and these were resolved in the post-rebuttal discussion. While the paper may have some limitations, there was consensus that the paper should be accepted. When preparing a camera ready version, please take care to improve the presentation and address common concerns that came up in the reviews, such as more clearly describing the relationship to previous work, and establishing the relationship between the attack model considered in this paper and that of previous work.
confidence: Less certain
Decision: Accept