Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes P2PRISM, a novel secure aggregation protocol for P2PL, with theoretical analysis for convergence and good empirical results.
strength_and_weaknesses: Strength:
1. The paper provides an insightful discussion about the difference between Byzantine-tolerant machine learning in FL and P2PL.
2. This paper proposes P2PRISM, a novel secure aggregation protocol for Byzantine-tolerant P2PL.
3. Theoretical analysis for the convergence of P2PRISM is provided.
4. Empirical results show the the proposed method outperforms the basline

Weakness:
1. I would recommend using larger CV datasets such as cifar-10 instead of mnist, but it is not a bug issue.
2. The general idea of using using local updates for validation and Byzantine tolerance is actually not that novel. For example, the following 2 papers (zeno/zeno++ for short):
Xie, C., Koyejo, O., & Gupta, I. (2019). Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance. ICML. https://proceedings.mlr.press/v97/xie19b.html
Xie, C., Koyejo, O., & Gupta, I. (2020). Zeno++: Robust Fully Asynchronous SGD. ICML.  https://proceedings.mlr.press/v119/xie20c.html
Especially for zeno++, it uses the inner-product between the update on the validation server and the candidate update to calculate a score for Byzantine tolerance, which sounds very similar to the general idea of P2PRISM, though the formula of the scores are different.
Although zeno/zeno++ are originally for centralized learning, I think it's trivial to apply them to P2PL. Basically you just need to treat each worker as a server that uses the local training data to compute the validation score for zeno/zeno++.
I think that zeno/zeno++ should be a better baseline compared to trimmed mean, since zeno/zeno++ are the techniques closet to P2PRISM.
I highly recommend to at least give a detailed discussion about zeno/zeno++ in this paper. Experiments using modified zeno/zeno++ in P2PL as the baselines are also recommended.
clarity,_quality,_novelty_and_reproducibility: The algorithm itself is novel, but the general idea of using local updates for validation and Byzantine tolerance is not that novel. 
summary_of_the_review: This paper proposes P2PRISM, a novel secure aggregation protocol for P2PL, with theoretical analysis for convergence and good empirical results. Some important baselines are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes the effect of malicious nodes in P2PL environment. It proposes a defense mechanism P2PRISM for reviving the nodes from the attack. Experiments are detailed and extensive and showcase the advantage of the proposed work.
strength_and_weaknesses: strengths
-the paper proposes P2PRISM, a defense against malicious attacks in P2PL environment
-based on the direction of updates, the approach uses its own local update as the reference to detect parameter-wise direction flip in the updates received from its neighbors to identify malicious ones
-experiments are detailed and extensive

weakness
-there are many parameters used 
-the reputation update mechanism seems naïve and could be compromised


clarity,_quality,_novelty_and_reproducibility: -the paper is in general well written, though references to FL seems more in the beginning than the intended contribution
-the paper introduces flip score computation to identify malicious nodes
-the paper shows experimental details that help reproducibility

summary_of_the_review: The paper shows adequate contribution is addressing an important problem of malicious nodes in P2PL environments. The approach has been extensively evaluated to verify the paper’s claims
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new secure aggregation protocol P2PRISM for decentralized peer-to-peer learning, against malicious adversaries, in various network topologies. 
strength_and_weaknesses: Strength: This paper considers a fairly interesting setting where no centralized trusted server is presented to aggregate the model updates as in federated learning settings. And it illustrates the possible malicious byzantine attacks that prior works cannot handle. Their protocol is clearly presented.

Weakness:
- The intuition behind the construction is not well presented. Especially the reward/penalize part. 
- A big portion of technical novelty may go to the flip-score proposed by Sharma et al. (2021).
- They assume that their way of using flip-score should trim out the anomalies. This does not look straight to me. Neither does the later convergence analysis. It deserves a better explanation or formal proof.

Other thing:
- Running title says for ICLR 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity is lacking in this paper as discussed in the previous weakness section. Though an interesting protocol is proposed.  The efficacy of the novelty could not be evaluated.
summary_of_the_review: It might be better to emphasize the technical novelty of the proposed protocol and provide more intuition about why it's designed in this way. I think this paper needs improvement before acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses:  The paper proposes P2PRISM, a novel secure aggregation protocol for P2PL. The authors indicate vulnerabilities under malicious attacks in the P2PL setting,  as well as a provably robust algorithm. Empirical results show the the proposed method outperforms the (non-robust) baseline.

The paper could be better positioned w.r.t. related, and novelty should be discussed in comparison with papers brought up by the reviewers; the experimental comparison added certainly looks promising. The paper could also use polishing, along the lines indicated in the reviews.
justification_for_why_not_higher_score: Please see above.
justification_for_why_not_lower_score: NA
Decision: Reject