Kati8/ICLR_2023_2025 huggingface dataset:
authors: Riadh Ben Chaabene
authors: Darine Ameyed
authors: Fehmi Jaafar
authors: Mohamed Cheriet
reviews: 
Comments: This paper presents a defense mechanism for FL against data poisoning attacks, specifically focusing on data-flipping attacks in AVs. The approach combines anomaly detection with robust aggregation techniques, using statistical outlier detection and model consistency checks to filter out compromised updates. Experimental results on benchmark datasets show that the method effectively improves FL robustness, preventing nearly a 15% accuracy drop in the global model and reducing the attack success rate, even at a 20% poisoning level. This solution enhances FL system security against adversarial threats in sensitive data-driven fields.
Strengths: Strengths:
+ This work offers a defense mechanism for FL against data poisoning attacks, specifically focusing on data-flipping attacks in AVs.
Weaknesses: Weaknesses:
- The novelty of this paper needs to be further improved.
- The experimental results of this paper are not convincing.
- More poisoning attack baselines and advanced defenses need to be included.
- The writing quality of this paper needs another round of polishing.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: The paper introduces a defense against label flipping attacks in federated learning in the context of applications for autonomous vehicles. The defense combines anomaly detection and robust aggregation to mitigate the effect of potential attacks by filtering out malicious model updates. The experimental evaluation against label flipping attacks in benchmark datasets show that the method is capable of mitigating the effect of the attack.
Strengths: + The paper proposes a defense against label flipping attacks in the context of applications for autonomous vehicles, which is a relevant challenge for the security and robustness of distributed learning techniques such as federated learning. 
+ The authors present a novel defense integrating PCA and MCSVM for detecting malicious label flipping attacks in federated learning, which effectively enhances the robustness of the algorithm during training against this type of attack.
Weaknesses: + The novelty and the contribution of the method compared to existing defenses in the research literature is unclear. Many defenses are already capable of defending against the type of attacks explored in the paper. In this sense, the authors did not discuss these aspects in the paper and did not compare against any other competing method in the experiments. 
+ The settings for the experiments are relatively trivial. The authors just tested in an environment with IID data partitions and with a very reduced number of participants (just 3). As mentioned before, there is no comparison with other existing methods in the research literature. 
+ Although the paper aims to focus on autonomous vehicle applications, the paper really restricts to computer vision applications. It would be interesting to analyze more applications typical for autonomous vehicles (e.g., LiDAR, sensing, etc.). The way the paper is presented differs little from the majority of the papers in this area, which often consider similar computer vision benchmarks (e.g., CIFAR) for their experiments.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper evaluates a label-flipping attack in federated learning and proposes a simple outlier detection approach using PCA and SVM to filter out compromised updates.
Strengths: I cannot identify any strength of the paper.
Weaknesses: The paper has many typos. 

Both label-flipping attacks and outlier detection methods have been well-studied in federated learning. See, for example, "Defending against the Label-flipping Attack in Federated Learning" by Najeeb Moharram Jebreel et al. The paper does not seem to propose anything new. 

The FL setting and the threat model are poorly described and involve multiple unreasonable assumptions. What do you mean by training FL models without the adversarial settings? How can you ensure that there is no adversarial device during training? Also, what is the testing dataset D_test? Where does it come from, and why is it disjoint from the participants' datasets? In FL, the data is typically provided by participants. 

The evaluation considers a very small setting with three devices and iid data distribution. The results are from being convincing.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This paper proposes a defense for Federated Learning (FL) against data poisoning attacks, especially in autonomous vehicles. The authors use PCA and SVM to detect label flipping attacks and try to build a robust FL framework.
Strengths: + The problem worths detailed investigation. 

+ The structure of the draft is good.
Weaknesses: - Techniques: 
 1. Label Flippinp: Label flipping is the easiest data poisoning, there are many poisoning attacks like backdoor or model poisoning attack, this threat seems to be quite weak as of now. 
 2. No Baselines for a comparative study
 3. Insufficient experiments
No ablation studies and more settings like Non-IID.

- Writing: 
1. The writing needs thorough revision, and the current form read more like an experimental report.  
2. Please split the experiment and proposed framework.
3. Don't introduce too much basic knowledge in your proposed method.
4. Please give a formalized/mathematical description on your method, it is unlikely to understand the proposed method in detail from pure text descriptions.
5. Some statements are unclear. What is the k in 191?
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: This paper aims to resist data poisoning attacks (in particular data-flipping) faced by autonomous vehicles (AVs) in the federated learning (FL) setting. The authors propose a novel defense mechanism that combines anomaly detection with robust aggregation techniques. They employ statistical outlier detection and model-based consistency checks to filter out compromised updates before they affect the global model.
Strengths: 1. Interesting to explore the poisoning attacks in the FL setting, for ensuring the security of AVs.
2. The logic of the proposal is easy to follow, although there are a lots of grammar issues and typos.
3. Experiment results to some extent support their conclusion.
Weaknesses: The contributions of the paper are not solid enough, and the challenges are not adequately highlighted. It appears that the authors merely combine existing technologies into different scenarios.

On one hand, while the authors use a label-flipping attack to implement targeted data poisoning in federated learning (FL), they do not clarify how this attack differs from existing ones or specify the challenges it poses in the FL context.

On the other hand, in their proposed defense mechanism, they set a detection agent in each autonomous vehicle (AV) to filter out attacking data. However, this defense mechanism is independent of the federated learning setting, meaning that existing defense strategies could be directly applied here for protection.

The relevance to AVs is not clear. It is difficult to find the connection between the proposals and AVs, despite the authors providing Figure 1 to illustrate the FL for label-flipping attacks on AVs.

Additionally, no comparisons are provided. A comparison of attacks and defenses should be included to strengthen the contributions. Without this, it is challenging to assess the effectiveness of the proposed attack and defense mechanisms.

Finally, there are too many unexpected grammar issues and typos that make the current version far from acceptable.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: The authors propose a novel defense strategy that combines PCA and SVM to detect and mitigate these attacks. This approach aims to filter out malicious updates by identifying statistical anomalies in model updates before they impact the global model. Their experiments, conducted using various datasets, demonstrate that this defense mechanism successfully maintains model accuracy and integrity, even with significant data poisoning levels.
Strengths: By focusing on label-flipping in AVs, this paper addresses a specific and underexplored threat in FL. The experimental design is robust, testing on multiple datasets and using meaningful metrics to demonstrate the effectiveness of the defense. Clear explanations of PCA and SVM integration, along with structured result presentation and well-contextualized related work, enhance the paper’s readability and situate its contributions within the broader FL literature.
Weaknesses: 1) The study’s scope is restricted to only three participants due to computational limitations, which limits the generalizability of its findings. Scaling up to a larger number of clients would better reflect real-world FL systems in autonomous vehicles.

2) The focus on label-flipping attacks is a good starting point, but broader testing on other attack types, such as backdoor attacks or noise injection, would make the proposed defense mechanism more comprehensive.

3) Although the paper provides context on existing methods, many comparisons with other recent FL defense strategies are missing, such as hierarchical FL or advanced anomaly detection models. It would be better to Include a comparison of metrics like detection accuracy, computational cost, and resilience against adversarial attacks.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The paper proposes new defense method on against data poisoning attacks in federated learning, especially for image classification models in autonomous vehicles. The method uses Principal Component Analysis (PCA) and Multi-class Support Vector Machine (SVM) classifiers together to offer a strong way to defend against label flipping attacks in federated learning. Experiments show the performance.
Strengths: 1. Again data poisoning attack in FL and Autonomous Vehicle is important.
2. The proposed method (PCA+SVM) can be used for detecting outliers.
3. Experiments are provided.
Weaknesses: 1. PCA+SVM has been well studied and has been extended to FL settings. The contribution of the paper is unclear.
2. Experiments do not have comparison methods.
3. It does not have experiments on simulated autonomous vehicles.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2