Robust Decentralized VFL Over Dynamic Device Environment

Authors: Surojit Ganguli, Zeyu Zhou, Christopher Brinton, David I. Inouye
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Robustness, Vertical Federated Learning

Abstract:
Robust collaborative learning on a network of edge devices, for vertically split datasets, is challenging because edge devices may fail due to environment conditions or events such as extreme weather. The current Vertical Federated learning (VFL) approaches assume a centralized learning setup or assume the active party or server cannot fail. To address these limitations, we first formalize the problem of VFL under dynamic network conditions such as faults (named DN-VFL). Then, we develop a novel DN-VFL method called **M**ultiple **A**ggregation with **G**ossip Rounds and **S**imulated Faults (MAGS) that synthesizes faults via dropout, replication, and
gossiping to improve robustness significantly over baselines. We also theoretically analyze our proposed approaches to explain why they enhance robustness. Extensive empirical results validate that MAGS is robust across a range of fault rates—including extreme fault rates—compared to prior VFL approaches.

Decision: reject
Reviewer scores: 6, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730386716818

thread_type:
official_review

<thread_object>
creator: reviewer_dsse

<content>
summary:
The algorithm presented in this paper is designed to address the robustness of Vertical Federated Learning (VFL) in dynamic network environments, especially when devices may fail due to faults caused by environmental factors or extreme weather. The algorithm is called MAGS (Multiple Aggregation with Gossip Rounds and Simulated Faults), which improves robustness by simulating faults during training, replicating the data aggregator, and utilizing gossip propagation rounds.

strengths:
The MAGS algorithm proposed in this paper is mainly compared with several existing Vertical Federation Learning (VFL) methods, including traditional VFL, PD-VFL, MVFL and its variants such as MVFL-G4 and CD-MVFL-G4. These methods have different capabilities in dealing with device failures, especially in distributed environments such as edge computing where devices may fail due to a variety of reasons (such as extreme weather, power problems, etc.) to fail, which in turn affects the reliability and performance of the whole system. MAGS enables the system to maintain good performance in spite of high equipment failure rates by simulating failures and multiple rounds of rumor propagation.

weaknesses:
The algorithm proposed in this paper has several disadvantages or limitations that need further clarification. The first one is due to the fact that MAGS needs to perform additional rounds of rumor propagation, which may incur more computational and communication overhead. In addition, there seems to be a limitation in the applicability of the MAGS algorithm, based on this paper it is known that although MAGS excels in dealing with high failure rates, it may not be advantageous compared to other VFL methods in the case of lower failure rates. How can a better algorithm be chosen for a specific equipment failure rate in a practical application? It seems that this paper does not provide quantitative selection criteria.

questions:
1.In Algorithm 1 proposed in this paper, the message interaction between clients is based on the multi-round Gossip protocol. However it seems that Algorithm 1 will put higher requirements on communication and computation resource consumption compared to the single round Gossip usually used in decentralized optimization. Can the authors avoid using the multi-round Gossip protocol? 
2.Another important topic of concern to the reviewer is that the paper does not quantitatively analyze the effect of the number of Gossip rounds on the performance of the algorithm under different network structures. It makes the results of this paper lack theoretical interpretability. In addition, can the authors further explore how to automatically adjust the optimal number of Gossip rounds according to different network structures?
3.Could the authors describe the differences and connections between the two types of failures leading to dynamic networks considered in the paper and the Byzantine attacks (malicious server) and denial-of-service attacks (communication link corruption) that are common in centralized or decentralized federated learning literature?
4.The MAGS approach is presented in the paper and its superiority is shown in the experiments. However, the theoretical analysis section could be more detailed as to why the MAGS approach improves robustness. It is recommended to add a mathematical derivation of the convergence analysis of the MAGS algorithm and how the algorithm enhances robustness by simulating faults, replicating data aggregators, and gossip propagation.
5.The effect of network structure on performance can be further explained, especially how MAGS performs under different network topologies, such as complete graph, ring, star, etc., and whether the parameters or design ideas need to be adjusted.
6.In accordance with this paper, MVFL is the optimal model when trained in a fault-free scenario. This means that in scenarios where high failure rates are not anticipated, MAGS may not be the most efficient approach. Can the authors discuss the reasons for that phenomenon? Can the algorithm be modified so that it achieves optimal performance under different failure rate conditions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570390782

thread_type:
official_review

<thread_object>
creator: reviewer_8kjt

<content>
summary:
This paper attempts to improve the robustness of the (Vertical) Federated Learning framework, where data features are spread across different clients. It proposes a method called MAGS (Multiple Aggregation with Gossip Rounds and Simulated Faults) to overcome scenarios of device failures and communication reliability.  The authors propose combining three strategies to enhance fault tolerance and resilience: (1) fault simulations (dropout) during training, (2) aggregation redundancy using multiple clients as data aggregators, and (3) Gossip rounds for reducing prediction variability and improving resiliency.

The authors claim to formalize the integration of these strategies into a framework that they define as Dynamic Network Vertical Federated Learning (DN-VFL). They introduce a dynamic risk metric to assess model performance and propose three propositions. Proposition 1 establishes a lower bound on the dynamic risk of their approach. Proposition 2 highlights the benefits of averaging prediction using gossip rounds by comparing the risk of ensembled predictions across multiple clients versus that of a single prediction by an individual client. Proposition 3 claims to establish a bound on the difference between the ensemble and the local predictions.

Empirical experiments were conducted on various datasets, including MNIST and StarCraftMNIST, among several others. A key aspect of these studies was to evaluate the performance of the MAGS approach under device and communication faults. The authors compared performance with the Vanilla VFL approach and VFL variations, including party-wise and communication-wise dropouts. Consensus algorithms were also studied.

strengths:
The paper attempts to formalize the combination of three strategies, which have traditionally been used in isolation, to improve the robustness of the VFL framework. The authors formalize some aspects of their approach and performs a reasonably comprehensive experimental study.

weaknesses:
My primary concern with this work is that, by allowing peer-to-peer communication between devices, the framework diverges from the original design and motivation of Vertical Federated Learning. VFL was specifically devised for environments where direct communication between parties is impractical or restricted due to privacy, regulatory, or technical limitations. By introducing peer-to-peer communication through gossip rounds, client-based aggregation, and ensembling, the proposed framework theoretically violates the foundational principles of federated learning. In fact, the framework now resembles a multi-agent system, which allows direct interactions among agents to achieve consensus.  This seems to be a topic that is closely related and yet there is no mention of it in the literature review.  Can the authors explain how their MAGS framework maintains the privacy guarantees of VFL while introducing peer-to-peer communication? Are there specific use cases where this trade-off is justified?

Proposition 1 highlights risk reduction when using multiple aggregators, which seems to be an intuitive results considering that the MAGS framework increases redundancy. Similarly, Proposition 2 highlights a reduction in variance which is a natural byproduct of ensembling. Proposition 3 also seems to be a simple and direct adaptation of a well-known consensus result in distributed averaging. Could the authors clarify how these propositions provide new insights or guarantees specific to the DN-VFL setting that are not immediately obvious from existing results in other domains?

questions:
Can the authors explain how their MAGS framework maintains the privacy guarantees of VFL while introducing peer-to-peer communication? Are there specific use cases where this trade-off is justified?

Could the authors clarify how these propositions provide new insights or guarantees specific to the DN-VFL setting that are not immediately obvious from existing results in other domains?

Could the authors explain how MAGS differs from a multi-agent system (MAS)? While MAS commonly involves multiple agents collaborating for decision-making, similar configurations can also be applied to Machine Learning. Can authors discuss any specific advantages that MAGS might have over traditional MAS approaches in this context? Alternatively, are there any VFL-specific constraints that MAGS addresses that a general MAS approach might not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670125579

thread_type:
official_review

<thread_object>
creator: reviewer_a5wu

<content>
summary:
This paper studies decentralized vertical federated learning over a dynamic device environment, which the authors define as faults in changing client graphs. The authors develop a novel DN-VFL that brings together dropout, replication, and gossiping and show its outperformance over baselines via empirical results.

strengths:
* The paper identifies an important issue in the VFL setup, where fault can occur when the clients or the communications betwwen different clients stochastically drop out.
* The paper provides extensive experiments on different datasets.
* The design of algorithms is intuitive, and the derivations seem correct.

weaknesses:
* The most obvious weakness is the presentation of the paper. The authors should try to shorten the overview of each section and go straight to the point. For example, the overview of Section 3 is just a wordy replication of what is coming next. It might be helpful to use bullet points to highlight the contributions.
* The related work is not thorough enough. Except asynchronous decentralized learning, the authors have to consider a topic called client unavailability problem in federated learning, where clients may (stochastically) drop out, e.g., in [1,2,3]. Specifically, they should talk about the connections and differences between them and this work.
* There are many contents that seem to be out of the space in Section 2. For instance, different client selection strategies have been talked about in great detail, which should belong to the development of algorithms instead.
* Following the problem formulation, there is no motivating example to illustrate that a dynamic network can lead to significant performance degradation to the classic VFL algorithms. Although it might be intuitive to conjecture that is the case, evidence is needed.
* Some of the definitions are missing. In Section 3.1, the authors talk about adopting Communication-wise Dropout. Yet, no formal definitions are given. 
* It is hard to parse the different components directly from Algorithm 1. No elaborations on Algorithm 1 have been given; neither do the authors talk about the connections in Sections 3.2 and 3.3.
* The most confounding part is that the authors greatly emphasize the case when there are no active aggregators. In that case, a trivial solution is to skip and move on to the next round. In addition, Proposition 1 is a lower bound, meaning that the second term, which depends on the probability of an empty active client set, can be ignored.
* In Proposition 2, it is also unclear what the dynamic risk of an ensemble leads to in the sense that no matching upper bound for risk with faults is proved.







**References.**
[1] Gu, X., Huang, K., Zhang, J., & Huang, L. (2021). Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems, 34, 12052-12064.

[2] Wang, S., & Ji, M. (2023). A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging. arXiv preprint arXiv:2306.03401.

[3] Wang, S., & Ji, M. (2022). A unified analysis of federated learning with arbitrary client participation. Advances in Neural Information Processing Systems, 35, 19124-19137.

questions:
* Can the authors provide numerical or theoretical justifications that the dynamic device environment will lead to detrimental effects?
* What is the difference between Definition 2 and Definition 3? It seems that Definition 3 subsumes Definition 2 although with different probabilities.
* Which definition (2 or 3) do the authors consider when proving their theoretical results?
* Can the authors provide formal definitions of communication-wise dropout?
* Can the authors outline the connections between the pseudocode Algorithm 1 and their development of methods?
* Regarding experiment setups, the authors compare their serverless methods with VFL methods with a server while also including baseline communication networks. Can authors elaborate on the experiment setup for VFL methods with a server?
* Why are there NAN results in Table 3 and Table 8 (in Appendix) for VFL and PD-VFL? Can the authors elaborate the difficulty in not being able to produce meaningful results for VFL and PD-VFL?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731009030537

thread_type:
official_review

<thread_object>
creator: reviewer_qxxf

<content>
summary:
This work considers the problem of collaborative inference in vertical federated learning (VFL). In VFL, all clients have the same datapoints, but different subsets of features for each datapoint. Faults in VFL can disrupt both training and inference due to the need for client communication during inference. This work formalizes the problem statement of faults in VFL -- both during the computation in a node, and during communications between nodes.

strengths:
The paper proposes the notion of dynamic risk, where the expectation of loss is also computed over the stochasticity of faults. In the proposed algorithm, the language model heads of the models at each node are trained by simulating faults via dropouts -- which introduces robustness to faults during test time. The work also proposes several heuristics such as multiple VFL, where the data aggregator is replicated to prevent catastrophic faults with a single point-of-failure. Moreover, gossip rounds o ensemble the predictions from multiple data aggregators reducing the prediction variance across devices. (Unlike vanilla VFL, in DN-VFL, the clients are allowed to act as data aggregators and communicate with each other). Inference is done in two steps: Firstly, each client makes a prediction. This is followed by an "entity" collection these predictions, and subsequently making a final prediction. 

This approach seems like a sound and well-defined approach to the problem of faults in VFL.

weaknesses:
My major concern (which prevents me from giving a higher score) is that it is not completely clear to me what are the technical challenges that were overcome and novelty of the proposed approach. It majorly seems like some orthogonal approaches are combined to give the proposed approach. While this is not a negative thing per se, it would be highly appreciated if some discussions were added regarding the choices made, which will help clarify some things. For example:

1. Multiple data aggregators have been used in prior works with conventional federated learning too. It falls under the umbrella of "semi-decentralized federated learning" Prior works consider communication failures between data aggregators as well. Is there any reason why it was not considered in this work? This can perhaps be mitigated by using multiple rounds of gossip, and/or optimizing the collaboration weights during the data aggregation process. 

2. It was not clear to me -- is the idea of training classifier heads with dropout to simulate test-time faults novel to this paper, and has not been considered in prior work? If so, how scalable is this approach, i.e., is it possible to train the classifier heads for robustness by simulating faults on very large networks (or must it be done in a clustered fashion)?

questions:
In addition, I have some more clarification questions:

1. Proposition statements need to be more precise -- eg., in Prop. 1, what is "aggregator specific risk"?

2. More implementation details are needed -- what is the predictor used at each client?

I would be more than happy to reconsider my score contingent on other reviews, and the authors' responses (especially the concerns in the Weaknesses section).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236802729

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all

comment:
We sincerely thank the reviewers for their valuable feedback. Reviewer QXxf found our approach to be sound and well-defined for addressing the problem of faults in VFL. Reviewers a5wU and 8kJT appreciated the extensiveness of our experimentation. Additionally, Reviewer a5wU noted that we have highlighted a critical issue in the VFL setting and acknowledged the correctness of our derivations. In the sections below we try to address the points that were raised by each reviewer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732590906365

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Uploaded PDF with modifications

comment:
We uploaded an updated version of the paper with the following modifications:

- Updated Proposition 1. Instead of using “aggregator specific risk” we denote it by “risk of a predictor” and this is also highlighted in equation 1

- We trimmed down primarily the overview to section 3 and highlighted the contribution of this section using bold letters.

- In section 3, we referenced Algorithm 1 in section 3.2 and 3.3

All the new additions in the paper are highlighted in blue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732591074863

thread_type:
official_comment

<thread_object>
creator: area_chair_yr9j

<content>
title:
Response

comment:
Dear Reviewers,

The authors have provided their rebuttal to your questions/comments. It will be very helpful if you can take a look at their responses and provide any further comments/updated review, if you have not already done so.

Thanks!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper considers a vertical federated learning setting, distributed training with features distributed across compute nodes, where nodes may fail. The reviewers complained about clarity and discussions of relevant literature. There are also questions regarding effectiveness of the proposed system, which persists after the discussions.

Based on the reviews and comments, I recommend rejection.