Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to speed up cross-silo federated learning, in which different silos need to have multiple peer-to-peer communication/synchronization at each round. Following previous works, this paper proposes a new efficient communication topology based on multigraph. Extensive experiments on multiple datasets validate the effectiveness of the proposed method.
strength_and_weaknesses: **Strength**
- The proposed topology design based on multigraphs is novel. It may have great impact on future research on this topic.
- The experiments are kind of comprehensive, which include three different datasets and five different communication networks.


**Weakness**
- Clarity on key insights
    - While I feel the proposed algorithm in this paper looks interesting and novel, and the experimental results are promising, I found that it is hard to get the key ideas behind the algorithm. If I understand correctly, the benefits of the proposed algorithm lie in two aspects: (1) at each round, the activated graph is just a subgraph of the overlay, thus there are less communication links; (2) for the isolated nodes, we will not let them continue performing local training. Instead, they will average with the latest available neighbor models. This can help to improve the convergence. Unfortunately, these two benefits are not clearly stated in the paper. And I believe there might be more benefits hidden in the paper. Even for an expert, it may be difficult to effectively get what are the key insights behind the algorithm.
- Clarity on technical details: I found that the authors may omit a lot of technical details or explanations.
    - It is unclear to readers how you derived equation 4. It is hard to interpret it without any explanations. For example, what does it mean by "$d_k(i,j) = d_k(i,j)$" on the first line. Do you update $d_k(i,j)$ based on its previous values at $k-1$, $k-2$?
    - The authors mentioned that in order to overcome the problem of DPASGD, they propose some new techniques. However, how and why these techniques can overcome the problem are unclear.
    - Related to the above point, it would be better to clearly state why DPASGD has the problem. The authors mentioned that DPASGD will terminate when a node is isolated. As far as I know, this is not true, as long as the expected graph over all rounds is connected. Then, DPASGD won't have the divergence problem.
    - How do you measure the cycle time? If you want to measure the real time, then you must have the same number of machines as the number of silos. On some networks, there is about 80 silos. Does that mean you ran experiments on a cluster of 80 machines? If not, the reported numbers will be just simulation results. The authors should provide full details on how they simulate the numbers.
    - It is unclear how you implemented the MATCHA algorithm. Basically, MATCHA has a hyper-parameter to tune the communication time per round. How did you choose this hyper-parameter?
- Incomplete study: I felt there are still many questions remaining. The authors are supposed to address them in order to make the paper complete and self-contained.
   - The authors just stated the steps of the multi-graph parsing algorithm. But we should also know its parsing properties. For example, as the authors mentioned, "Using our algorithm, some states will contain isolated nodes." It would be better to show either theoretically or empirically how the exact number of states have isolated nodes in different type of graphs. Does this number depend on the number of edges, maximal degree, or something else? Answering these questions would help us to better understand the benefits of the proposed algorithm and when is proper to apply it.
   - Similarly, we readers know little things about the DPASGD++ algorithm. The authors claim that it ensures model convergence. However, this claim is not supported in the main paper. Ideally, one should have a convergence analysis to support this claim. The authors provided one in the Appendix, which is very weak. It says DPASGD++ is equivalent to DPASGD when all states are strong edges and all states are weak edges. This conclusion is obvious and is not interesting. If it is equivalent to DPASGD, then there is no need to have a new algorithm. Therefore, the interesting part is something in between. What happens when some states are strong and others are weak. Without a convergence analysis, it is possible that DPASGD++ can just diverge in some cases.

clarity,_quality,_novelty_and_reproducibility: This paper introduces some novel techniques but many places are lack of clarity as I mentioned in the last section.
summary_of_the_review: I like the proposed algorithm and it seems that it can be an important baseline in the future. But I do think the current draft is not ready to be published. Both the key insights and technical details are incomplete and unclear. The authors are supposed to perform more studies around the proposed algorithm. More details can be found in the weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new multigraph topology for cross-silo federated learning in a decentralized network. Specifically, the proposed method extends the overlay graph with weak connections based on the delay time and improved DPASGD algorithm to process isolated nodes for time reduction.
strength_and_weaknesses: Strength. 
1. Using multigraph topology in federated learning is a novel idea.
2. This paper is well written.

Weaknesses:
1. The targeting scenario is very narrow. It requires a cross-silo federated setting with a peer-to-peer network and a graph topology. 
2.  It is unclear In the main result of table 1, accuracy results should also be provided to show that the proposed algorithm could achieve comparable accuracy with less training time.
3. Below paper could be discussed in the related work. It is a similar idea to graph-based aggregation on federated learning [1]. 

[1] Personalized Federated Learning with A Graph

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well organized.

Quality: The proposed method is well analyzed and discussed with detailed experiments.

Novelty: The paper uses multigraph to model a very scarce scenario on federated learning. 

summary_of_the_review: The paper is organized very well. The paper’s targeting problem is very scarce, thus the contribution to the federated learning community is limited. The novelty is sourced from the combination of multigraph and federated learning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers designing dynamic network topologies to improve the training time and performance of the cross silo FL training. The authors propose an algorithm and verify its effectiveness through empirical evaluation.
strength_and_weaknesses: - Strength: This is an interesting problem, especially the dynamic topology part which is under-explored.
- Weakness: 
   - The paper is poorly written and to some extent not self contained. Many of the pieces necessary for developing the method/algorithm are defined or presented without context. This makes the paper difficult to follow. As an example I was not able to follow why eq. 4 holds and the explanations around it was definitely not sufficient. 
  - In algorithm 1, it is not clear what is the exact motivation behind the algorithm, what are the constraints the extracted topologies need to satisfy, and how do the authors know that it does exactly what is designed to do. It is presented just as is without much context. The same is also true (to a lesser extent) about Algorithm 2.
  - It is not clear if such dynamic topological changes would still allow the overall optimization algorithm to converge to a unified solution over the network? In fact, the interplay between the optimization method and dynamical topology which seems to be an important aspect of such algorithms is not even discussed in the paper. How do the algorithms scale with the number of participating silos? do global models converge? what are the conditions needed for the model to converge correctly? And are they satisfied by the proposed algorithms?
  - In the experiments, the results only show the train loss and train accuracy, while for ML models the test loss and test accuracy are more important. Also, regarding the distributed training procedure it is not clear what train loss and train accuracies are reported? Are these measured at the nodes (with local models) and then averaged? or are they with the global model? Why there are no error bars reported? Are the results averaged over multiple runs?
clarity,_quality,_novelty_and_reproducibility: See the above comments
summary_of_the_review: Overall, the presentation of the paper makes it difficult to comprehensively judge the quality of the paper and the proposed method. In order to improve the paper, I strongly suggest the authors improve the presentation, make the paper more self-contained and give more details about the algorithm designs, limitations and the interplay between the optimization and dynamic topology. Moreover, a more rigorous experimental section is also needed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper attempts to improve distributed learning in cross-silo setting. In this regard, the authors propose a new dynamic communication topology for cross-silo federated learning which is claimed to reduce the training time and improve performance. The key reason for the efficiency is the existence of "isolated nodes" in the dynamic topology which allows model aggregation to happen without waiting for other nodes. Experiments are conducted on five simulated distributed networks and three datasets to show effectiveness of the proposed method. All reviewers agree that studying dynamic topology is interesting, but feel like current work is somewhat incomplete. We thank the authors for detailed response, but unfortunately it still left many questions remained. The paper can be made more self-contained by providing further analysis (either theoretical or empirical) of the two algorithms presented in the paper. In particular, emergence of isolated nodes is crucial for success of the proposed method, but it is not analyzed like what are the guarantees or expected number of states that have isolated nodes in different type of graphs. Moreover, effect of these isolated nodes on convergence is not studied, only a very weak result is provided in the appendix. Adding the study of interplay between the optimization method and dynamical topology which seems to be an important aspect of proposed method would make the paper very interesting and useful to the community. Making the baselines, like MATCHA, stronger by adapting them to studied setting will more strongly showcase the improvement of the proposed and further strengthen the paper. Finally, for some reviewers/readers improving writing by providing a detailed motivation might be helpful.
justification_for_why_not_higher_score: - Missing in-depth study about key properties of the proposed method, like number of isolated nodes and its effect on convergence
- Many unanswered questions of reviewers, like "How do the algorithms scale with the number of participating silos? do global models converge? what are the conditions needed for the model to converge correctly? And are they satisfied by the proposed algorithms?"
Decision: Reject