Towards Practical Defect-Focused Automated Code Review
Authors: Junyi Lu
Lili Jiang
Xiaojia Li
Jianbing Fang
Fengjun Zhang
Li Yang
Chun Zuo
Venue: ICML 2025 spotlightposter (2025)
Keywords: Automated Code Review
Merge Request Analysis
Large Language Models (LLMs)
Defect Detection
Evaluation Metrics for Code Review
Code Context Extraction
Multi-Agent LLM Collaboration
Primary area: applications
TL;DR: This work presents an end-to-end approach to automated code review that goes beyond snippet-level generation and text-similarity metrics, achieving significant gains over existing baselines in real-world, industry-scale codebases.

Abstract: The complexity of code reviews has driven efforts to automate review comments, but prior approaches oversimplify this task by treating it as snippet-level code-to-text generation and relying on text similarity metrics like BLEU for evaluation. These methods overlook repository context, real-world merge request evaluation, and defect detection, limiting their practicality. To address these issues, we explore the full automation pipeline within the online recommendation service of a company with nearly 400 million daily active users, analyzing industry-grade C++ codebases comprising hundreds of thousands of lines of code. We identify four key challenges: 1) capturing relevant context, 2) improving key bug inclusion (KBI), 3) reducing false alarm rates (FAR), and 4) integrating human workflows. To tackle these, we propose 1) code slicing algorithms for context extraction, 2) a multi-role LLM framework for KBI, 3) a filtering mechanism for FAR reduction, and 4) a novel prompt design for better human interaction. Our approach, validated on real-world merge requests from historical fault reports, achieves a 2× improvement over standard LLMs and a 10× gain over previous baselines. While the presented results focus on C++, the underlying framework design leverages language-agnostic principles (e.g., AST-based analysis), suggesting potential for broader applicability.

Towards Practical Defect-Focused Automated Code Review
Junyi Lu 12f Lili Jiang
Xiaojia Li
Jianbing Fang 42 Fengjun Zhang'
Li Yang' Chun Zuo
Abstract
(MCR) is widely used in open-source (Rigby et al., 2008; 2014; Rigby & Bird, 2013) and industrial settings (Sadowski et al., 2018; Shan al., 2022), typically involving: (A) code submission, (B) reviewer examination, (C) feedback, and (D) developer revisions.
The complexity of code reviews has driven ef- forts to automate review comments, but prior ap- proaches oversimplify this task by treating it as snippet-level code-to-text generation and relying on text similarity metrics like BLEU for evalu- ation. These methods overlook repository con- text, real-world merge request evaluation, and defect detection, limiting their practicality. To address these issues, explore the full automa- tion pipeline within the online recommendation service of a company with nearly 400 million daily active users, analyzing industry-grade C++ codebases comprising hundreds of thousands of lines of code. We identify four key challenges: capturing relevant context, @ improving key bug inclusion (KBI), 4 reducing false alarm rates (FAR), and integrating human workflows. To tackle these, we propose code slicing algo- rithms for context extraction, a multi-role LLM framework for KBI, ② a filtering mechanism for FAR reduction, and a novel prompt design for better human interaction. Our approach, validated real-world merge requests from historical fault reports, achieves 2x improvement over standard LLMs and a 10x gain over previous baselines. While the presented results focus on C++, the un- derlying framework design leverages language- agnostic principles (e.g., AST-based analysis), suggesting potential for broader applicability.
Despite its benefits, MCR is labor-intensive and time- consuming (Yang et al., 2016), driving research toward automated review comment generation. Existing meth- ods-whether retrieval-based (Gupta & Sundaresan, 2018; Siow et al., 2020; Hong et al., 2022)
feep-learning-drive (Tufano et al., 2021; 2022; Li et al., 2022b;a; Lin et al., 2023; Lu et al., 2023)-oftee often frame it as a snippet-level code-to-text task. However, this oversimplification diverges from the core goal of reviewers: detecting defects (Bacchelli & Bird, 2013) (see Section A). Furthermore, current eval- uations rely excessively textual similarity metrics (e.g., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004)), which fail to measure real-world effectiveness (Lu et al., 2025).
Challenges. To address these limitations, we investigate a full code review pipeline within a real-world online service (Figure 1). Our system integrates with an internal DevOps platform, generating review reports, filtering comments, and aligning them with code lines. A detailed description of this real-world workflow integration, designed for seamless adoption by developers, is provided in Appendix B. This deployment reveals four key challenges (Appendix C):
Capturing Proper Code Context: Effective review requires analyzing dependencies beyond the immediate diff hunk (e.g., variable declarations or method calls). However, exces- sively long inputs degrade LLM performance, necessitating efficient context extraction.
1. Introduction
Improving Key Bug Inclusion (KBI): The goal of automated review is to detect critical defects, yet existing methods rely on textual similarity metries, which fail to measure defect detection capability. More robust evaluation methods, such as Key-Bug Inclusion (KBI), are needed.
Code review is essential for improving code quality and detecting defects (Fagan, 2002). Modern Code Review
Work done during the intemssip tenure Kuaishou Tech- nology. 'Laboratory of Precise Computing, Institute of Software, Chinese Academy of Sciences, Beijing. China Uniiversity of Chi- nese Academy of Sciences, Beijing. China Kuaishou Technol- ogy, Beijing, China 'Independent Researcher Sinosoot Company Limited, Beijing, China. Correspondence Li Yang <yan- gli2017 @iscas.ac.cn>.
Reducing False Alarm Rates (FAR): Generative models of- ten produce irrelevant or overly strict comments (e.g., nit- picks, hallucinations). burdening developers. A robust fil- tering mechanism is required to reduce false positives and enhance signal-to-noist ratio.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Human- Human-Centric Workflow Integration: Practical review tools

---

Towards Practical Defect-Focused Automated Code Review
</>
B
6
O
Code Author Merge Request
Check User White List
Check File
Auto Review Workflow
Notice the Participunts
Comment Website
Attach Comments Fill Lines
Figure 1. The code review automation pipeline integrated into the online service.
must seamlessly integrate into developers' workflows, ensur- ing comment alignment with code lines while minimizing cognitive overhead. Existing solutions often overlook this critical usability aspect.
necessary comments. In 2022, company reports showed that 30% of severe P1+ incidents (asset losses exceeding $350,000) and 24.04% of P4+ incidents stemmed from low- level faults due to inadequate reviews. Even in 2024, change- related core failures accounted for 67% of incidents, with code change-related graded incidents comprising 19.54%, highlighting the urgent need for effective automated review tools. These tools help ensure thorough, compliant reviews, reducing defect risks.
Our Approach. address these challenges, we propose: static analysis system using code slicing to extract relevant context. @ A multi-role LLM framework with chain-of chain-of-thought reasoning to enhance defect detection. ? A filtering mechanism to eliminate false positive nitpicks and hallucinations. A line-aware prompt design for pre- cise comment placement.
To understand reviewer needs, we surveyed a super reviewer group. summarizing findings in Section D. Background on code slicing and multi-role systems, key techniques in our work, is introduced in Sections and F.
Evaluation. We validate our framework on real-world sys- tem failures, including historical core dumps and fault re- ports that caused significant financial losses. We evaluate it using multiple open-source LLM engines, demonstrating 2x performance improvement over standard LLM methods and a 10x improvement over prior baselines. An ablation study further confirms the contribution each component, highlighting the impact of code slicing, multi-role reasoning, and filtering mechanisms.
3. Proposed Approach
3.1. Overview
Figure 2 illustrates our decoupling process of code review automation architecture: 1) Code Slicing: Extracting code from the diff hunk within repository context (Section 3.2); 2) Multi-role Code Review System: Employing a multi-role system to conduct reviews and compile the results (Section 3.3); 3) Redundancy Comment Filter Mechanism: Filter- ing redundant and irrelevant comments to avoid nitpicks and hallucinations (Section 3.4); 4) Line Number Local- ization: Ensuring precise identification of code lines where issues occur (Section 3.5). To evaluate the automation, we construct a dataset from historical fault reports, simulating real- world merge requests that introduced defects (Section 3.6).
Contributions. Our key contributions include being the first to: ! Repository-Level and Rerge-t Granularity: Elevating automated code review from snippet-level tasks to repository-wide and merge-request (pull-request) granular- ity. @ Integration with Real-Word World DevOps Workflows: Deploying automation into a practical online review system with more practical and objective evaluation metrics beyond text similarity. O Validation Industry-Scale Defects: Demonstrating effectiivness real-world, high-impact fail- ures in industry-level codebases instead of synthetic test data. + Codd--Review-Orientte LLM Framework: De- signing a specialized framework leveraging code slicing, multi-role collaboration, and filtering mechanisms, achiev- ing substantial improvement in code review performances.
3.2. Code Slicing
Previous work used method-level or diff-level code snippets as independent inputs. However, new code integrated into larger codebase during reviews, and understanding the structural context is crucial. We developed a code slicing process that integrates multiple slicing strategies, selectable based on the analysis needs. To avoid redundant slices, we use a caching mechanism to enhance efficiency.
2. Background: Code Review Automation
Automating code review is crucial for maintaining software quality by identifying critical bugs early. The goal is to detect severe issues in new merge requests and provide
The pseudo code our slicing algorithms presented in
2

---

Towards Practical Defect-Focusee Automated Code Review
Action: Merge Request Triggered
Section
Code Slicing
Multi-role Code Review System
Output: Code Review Report
Section 3.5
Section
Line Number Localization
Redundancy Comment Filter Mechanism
Figure An overview of our automated code review workflow.
Section G. Initially, the repository is cloned, and the merge request commit is checked out. A static analysis tool is then applied to generate abstract syntax trees (ASTs), which serve the foundation for our slicing process. Based on data dependencies and control flow analysis, one or more of the following four optional slicing algorithms may applied: 1) Original Diff: The basic code diff without trans- formations, capturing essential changes the commit. 2) Locates the smallest parent function con- taining the changes, providing functional context. 3) Left Flow: Tracks the flow of all left-hand values (L-values) in the function and control structures, focusing on the lifecycle of variables. 4) Full Flow: Extends Left Flow by tracing right-hand values (R-values) and collecting the signatures of callee functions, offering coverage of variable usage and modifications.
3.4. Redundancy Comment Filter Mechanism
LLMs often produce an overwhelming number of comments, many of which either nitpicks hallucinations. To mit- igate this issue, implemented a Redundancy Comment Filter Mechanism to reduce the number of irrelevant com- ments.
Our filtering mechanism, integrated within the multi-role system (Section 3.3), operates by answering three key ques- tions for each comment: Q1: Is this comment nitpick? Typical nitpicks include excessive code comments, handling unnecessary edge cases, overly complex error handling. Q2: Does the comment identify fake problem (i.e., non-existent bug)? For example, if the comment flags a function call to a known reliable internal library, null pointer checks are considered irrelevant. Q3: How critical is the issue identified by this comment? Minor issues, like miss- ing comments, less severe than potential core dumps or infinite loops.
Parent Function:
3.3. Multi-role Code Review System
Our multi-role code review system involves four key roles: Reviewer, Meta-Reviewer, Validator, and Translator. These roles collaborate to enhance the accuracy and efficiency of the review process. The system design is illustrated in Figure 3, and we detail the roles and their processes below.
Each question is rated on scale from to 7, with in- dicating a nitpick, fake problem, or minimal issue, and 7 indicating a severe and real issue. The scoring scale (1 to 7) inspired by other related work (McAleese et al., 2024). We chose this scale to enable fine-grained and manage- able distinction. These scores form the basis of the filtering process throughout the review workflow.
Reviews each code snippet generated by the code slicing algorithm (Section 3.2) and provides detailed comments on potential issues in predefined format. Meta-Reviewer:
Reviewer:
Aggregates comments from multiple Re- viewers, filtering and sorting them based predefined thresholds. It merges common issues across reviews. O Validator: Validates and refines the merged comments, re- scores them, and ensures that only comments exceeding a certain threshold are retained. Translator: Translates the final comments into the required language for multinational teams, ensuring proper formatting for direct integration into the development environment. Each role is integrated with Chain-of- -hought technique, as detailed in Section H.
Coarse Filtering and Sorting by Reviewer. During the review process, the Reviewer L.LMs score each comment based on QI-Q3. Comments with QI or Q2 scores of 4 or below are discarded. This specific threshold was established heuristically enhance interpretability and has been vali- dated by developer feedback during internal piloting. The remaining comments are then sorted based their Q3 score and truncated to the Top-N comments.
Fine Filtering and Sorting by Meta- Reta--Reveewer The Meta- Reviewer further refines the filtered comments by merging those flagged by multiple Reviewers and removing com- ments mentioned by only one Reviewer.
3

---

Towards Practical Defect-Focusee Automated Code Review
Filee, Sont Troccation
Tracetrck to Code
Filter
Truncation
Toccluck Cude
Seet & Iomeation
Tracehack to Code
Mata Reviower
Fileer
Trandasier
Reviewers
Vilidatoos
Figure 3. The multi-role system for automating code review.
Validation and Re-scoring by Validators. Validators then re-score the comments by revisiting the original code snip- pets and applying the same Q1-Q3 criteria. A secondary filter is applied, ensuring that only the most relevant and critical comments proceed to translation and integration into the development platform.
3.6. Offline Validation
To systematically assess the performance of system, we developed dataset curated from the company's 's fault report platform. Each case in this dataset corresponds to an issue that resulted in actual company losses. For each reported fault, we trace back to the merge request that introduced the fault and its subsequent fixing merge request. Using these, we generate ideal reference comments containing details such as affected files, specific lines of code, fault location, root cause, suggested fix, example code, and issue category. The motivation for conducting such validation is illustrated in Section I.
Integration with the Multi-role System. The filtered com- ments processed by the remaining multi-role compo- nents, including translation (if necessary) and final submis- sion to the development platform. This multi-stage process ensures that the delivered comments are both relevant and concise, minimizing redundancy and false alarms. The heuristic approach threshold definition described herein was chosen to prioritize generalizability, interpretability, and mitigate overfitting in this study. While providing bust baseline, exploring adaptive or machine-learned thresh- olds remains a valuable direction for future enhancement to achieve more nuanced filtering.
4. Evaluation Design
4.1. Research Questions
We define the following research questions (RQs) to guide our evaluation, whose detailed illustrations are in Section J:
3.5. Line Number Localization
RQI: How does the overall performance of our framework compare with previous works?
A key challenge overlooked in prior work is the precise localization of comments within the code. Unlike code sum- marization tasks, code reviews require pinpointing specific lines of code where issues are identified. Without this in- formation, developers face inefficiencies in verifying and addressing comments. For example, the change-involved function has 94.54 lines of code in average based on our statistics, missing line localization can result in significant delays for developers.
RQ2: How do code slicing algorithms impact the perfor-
mance of the framework?
RQ3: How do the different components of the multi-role
system impact the performance of our framework?
RQ4: How does the redundancc comment filter mechanism address nitpicks and hallucinations?
RQ5: How does the representation of line number position information impact overall performance and line number localization success rate?
We propose a code formatting approach inspired by Aider(Gauthier, 2024), tailored for code review tasks. As shown in Table 1, the format includes an operation label (indicating whether line is kept, added, or deleted), the line number, and the code content. For non-contiguous code lines, ellipses are used to indicate omissions.
4.2. Dataset and Studied Models
The primary goal of code review is to prevent problematic code from being merged into the target branch. To simulate real-world code review scenarios, we collected data from company's 's core framework team, which is responsible for the production code of the short video recommendation core service. This data was gathered using fault reports recorded on an online platform. These cases come from four reposito- ries and involve total 4,090 developers. By analyzing these reports, we traced the merge requests (MRs) that introduced
Table 1. Code formatting with line position information.
linenumber (kept line)
Ropresents that unchanged.
Jinemmmeer| ( (deldte code line)
Indicate tines that have hean removed.
inneenberr [added code line)
Marks newly added lines.
Indicates the omission of non-esemnial lines.

---

Towards Practical Defect-Focusee Automated Code Review
the issues and examined the specific commits to reproduce the code snapshots. The detailed statistics presented in Section K.
Table 2. Overall performance comparison of framework using different LLM engines and baseline models. LLM engines marked with are quantized. "Val" indicates if Validator role was used.
Our framework supports muitiple LLM engines. To mitigate security risks, we only studied open-source models that can be deployed locally. We exclusively selected large instructed models due to the complex coman-insttuutionn -based tasks in our workflow. The final list of models includes: LLaMA-3.1 (70B), Qwen2 (72B), Command R+ (104B), Mistral- large- 2407 (123B), and LLaMA3.1 (40SB). The reasons for not selecting other models are outlined in Section L.
Model
Val
KBIt
FAR,
CPG,
FAR44
CPl,1
Beseliner
CodeReviower
0.00
97-78
0.00
OCTS
2:22
97.38
2.32
90.91
3.57
LLaMA-
2.22
97.62
2.30
92.86
3.39
DISCOREV
Ours (Lef Flowe)
1LaMA3.1
20.00
17.52
66. 54
(70B)
W 2:22
37.04
4.29
86.67
4.17
Qwen2
Weo
40.00
91.21
14.42
63.55
23. 29
(728)
26.67
90.63
13.87
$1.02
Command
wio 20:00
92.99
10.38
76.06
11.78
(1038)
W
4.44
85.00
6.86
62.50
7.95
Mastral 2407
Wlo 26.07
91.07
13.38
74.85
25. 89
4.3. Metrics
(1258)
26.67
87.07
17.41
68.18
29.01
LLaMA3.1
wlo 31.11
17.51
(405B)
W
20.00
75.37
22:07
43.52
29.54
In accordance with the real-world developer expectations discussed in Section D, we evaluate performance at the merge request (MR) level using four metrics, with their formal definitions provided in Section M:
Ours (Fall Flow)
LLaMAS.11
w/o
13.33
88.22
12:51
61.67
19.78
(70R)
0.00
46.67
0.00
Qwen2
42.22
14.86
53.91
23.30
(72B)
28.89
91.73
12.86
T9.U6
24.28
Command R+
WAo
15.56
94.22
8.43
77.14
18.51
(HIBB)
8.80
76.30
12.93
08.33
14.65
Key Bug Inclusion (KBI): Assesses the model's 's ability to recall critical issues that could lead to tangible losses. @ False Alarm Rate (FAR): Captures the proportion of irrel- evant or erroneous comments, with two variants AR, for all MRs and F AR for MRs where key bugs are recalled). @ Comprehensive Performance Index (CPI): Balances the completeness of key issue detection (KBI) and preci- also
Mismal-2407
26.89
75.44
(1258)
28.80
86.11
18.76
67.30
LLaMAS.1
W/b
31.11
89.41
15.80
73.10
28. 86
(405B)
20.00
77.96
20.97
67.59
24.73
L.M across all roles. This approach was chosen to isolate and clearly assess whether a single, powerful model could effectively address key challenges in code review. Recog- nizing the practical importance and potential benefits of diverse model deployments, we also conducted extended comparison experiments with heterogeneous LLM assign- ments for reviewer and validator roles. These experiments, detailed in Appendix O, show that strategic combinations, such as pairing a strong validator with a smaller reviewer, can achieve comparable or even superior performance while potentially optimizing resource usage.
sion (100- FAR), analogous to the F1-score. computed in two variants (CPI, and CP2)). @ Line Lo- calization Success Rate (LSR): Measures the accuracy of line-level references by checking whether comments point to the correct code lines.
4.4. Baselines and Experimental Setups
Since our framework focuses on C++, we selected state- of-the-art baselines that support this language: CodeRe- viewer (Li et al., 2022b): A T5 model pre-trained for code review tasks and then fine-tuned. CCT5 (Lin et al., 2023): A T5 model pre-trained on CodeChangeNet, then fine-tuned. LLaMA-Reviewer (Lu et al., 2023): A large LLM fine- tuned for code review tasks based on the LLaMA. DIS- COREV (Ben Sghaier& & Sahraoui, 2024): A T5 model enhanced via cross-task knowledge distillation for code re- view. The detailed experimental setups of our framework and baselines are presented in Section N.
For baselines, since they do not prioritize comments, we evaluated their comments based on whether they passed their respective "quality estimation" filters, which assess whether code snippet requires a comment. The results are in Table 2.
The results indicate that our framework significantly out- performs the baselines by factor of 10x across most key metrics, such as key bug inclusion (KBI) and comprehen- sive performance index (CPI). This marked improvement is likely due to our framework's 's end-to-end approach to code review automation, which addresses the key challenges of the task and introduces strategies specifically designed to tackle each challenge. We evaluated the performance of our framework the fault merge request dataset, comparing it with several baseline approaches. Our framework was tested with different large language model (LLM) engines. Our main experiments pri- marily utilized a homogeneous setup, employing the same
5. Evaluation Results
5.1. RQ1. Comparison with Baselines
Among the LLM engines tested in our primary setup, LLaMA3, demonstrated the best overall perfor- mance, which aligns with the general scaling laws of lan- guage models where capability often increases with param- eter count on complex tasks such as code review. However,
5

---

Towards Practical Defect-Focussee Automated Code Review
our evaluations (detailed in Table 2) also included more compact LLMs.. These results show that certain smaller models, particularly those with strong inherent reasoning ca- pabilities, can still achieve competitive performance within our framework. This finding is particularly relevant given the industry trend towards increasing "capacity density' in newer architectures, where smaller models are progressively narrowing the performance gap. While the largest models may provide peak effectivvness, these observations suggest that range of LLMs can be effectively utilized, allowing for a balance between performance and computational resource demands, a point further explored in our heterogeneous model assignments (Appendix O).
Full Flow, likely due the large language model's reduced capability when provided with longer contexts, which can cause distraction. This finding supports our assumption that providing targeted and relevant code context is critical for maximizing I.I.M performance in code review tasks, an ob- servation consistent with the principles underpinning RAG systems where curated information significantly enhances model outputs.
During our analysis of the recalled merge requests (MRs), we found another interesting pattern. Although some slicing algorithms perform worse overall, each algorithm uniquely succeeds in specific cases. This means that each slicing strat- egy provides valuable context in certain situations. Figure 4 presents a Venn diagram showing the union and differences among the key bugs recalled by each slicing algorithm un- der the "All" and *+Meta Reviewer" settings. Notably, Left Flow and Full Flow recall most, with significant overlap, but almost each method also uniquely recalls some.
Summary of RQ1. Our framework surpasses baseline ap- proaches significantly (up to 10x on KBI/CPI), thanks to its end-to-end design. LLaMA3.1-4053 stands out among tested engines, highlighting the role of model capability. Investigations into heterogeneous LLM combinations also suggest the potential for optimized deployments. (See Ap- pendix T.1 for the extended conclusion.)
This phenomenon mirrors how human reviewers oper- ate expandinn their focus to different levels of granularity, such as inspecting parent functions or understanding vari- able usage in different contexts. Some defects are easier to spot in one context, while others require different view. Therefore, combination of various slicing strategies might be promising direction.
5.2. RQ2. Effectiveness of Code Slicing
We tested the four code slicing algorithms described in Sec- tion 3.2: Original Diff, Parent Function, Left Flow, and Full Flow. It is important to clarify that while our framework does not employ an explicit Retrieval-Augmented Gener- ation (RAG) pipeline, our code slicing mechanism is de- signed with RAG-aligned objective. Specifically, it serves a similar purpose to RAG by strategically retrieving and providing the LLM with only the most relevant contextual code 'slices' from the broader codebase. This process aims to focus the model on pertinent information, thereby enhanc- ing its reasoning and effectiveness in the code review task. Our focus in this section is on K BI and CPI, as these metrics indicate how input content affects the maximum recall capability of LLMs for code review.
Summary of RQ2. Left Flow and Full Flow significantly improve key bug inclusion and overall performance com- pared to simpler slicing. Left Flow often outperforms Full Flow, possibly because shorter context helps maintain fo- cus. Notably, each slicing approach has exclusive successes, suggesting that combining them could further improve de- tection. (See Appendix T.2 for the extended conclusion.)
5.3. RQ3. Effectiveness of Multi-role System
To better understand the capabilities of our multi-role system, we conduct experiments on: O Leveraging the non-determinism of large language models; @ The self- correction capability (validator); @ The chain-of-thought (CoT) prompting strategy.
The experiments were structured to evaluate the comments generated by the large language models under different con- ditions, including all comments, comments after applying a coarse filter, and top-k ranked comments (based on scores from Q3). We also tested multi-reviewer settings, where the meta-reviewer merges the comments, and validator settings, where validators further refine the comments. The average results are shown in Table 3, based on the 1AMMM....-05B AnQ-Int4 LLM engine. To provide further insight into the variability of these results, the minimum and maximum values for each reported metric across the three runs are detailed in Appendix R.
5.3.1. NUMBER OF REVIEWERS
Previous research has shown that the non-determinism of large language models (LLMs) can impact results. Specifi- cally, with a best-of-N sampling approach, smaller LLMs can sometimes match or surpass larger models. Since our framework includes a multi- neulierrviewwrr scenario, where a merges comments from multiple reviewers, we conduct experiments to assess whether increasing the number of reviewers improves performance.
The results reveal that using only the diff or parent func- tion is less effective, while more detailed slicing (Left Flow and Full Flow) improves performance, especially in key bug inclusion. Surprisingly, Left Flow performs better than
The results in Table 4 show that increasing the number of reviewers from one to three improves KBI but also
6

---

Towards Practical Defect-Focusee Automated Code Review
Table Impact comparison of different code slicing algorithms on key bug inclusion (K BT) and the comprehensive performance index (CPI), based LLaMA3. 1.00A.....00BBB.. 1--05----AWW BEBB--AAQInnt Int4. Experiments for a single reviewer are conducted three times to compute the average. "All" represents comments generated by reviewer: "Coarse filter" refers filtering using Ql Q2 scores during generation: "Top-k" denotes truncated comments sorted by Q3 scores; M+Meta Reviewer" and Validator" settings are evaluated under Top-5 truncation.
Cade Slieing
Single Revigwer
Multi Reviewers
Algurithms
All
Course Filter
Top-10
Top-5
Top-3
Mcta Reviewer
Validator
GPT
CP11
GPT
CPT1 1
1
CPT, 1
CP 1
KBI f
CPT,
Original
12.70
5.21
11.11
10:46
Parent F Function
31.55
0.32
22.22
15.18
17.04
13.57
14.81
13. 40
8.15
9.53
20.00
11.01
11.11
10.81
Left Flow
37.04
9.77
33.33
12.80
32.59
13.94
25.93
14.26
17.78
12.77
31.11
17.51
20.00
22.07
Fult Flow
39.26
9.67
32.59
11.95
31.85
13.18
25.93
13.90
13.33
10.23
31.11
15.80
20.00
20.97
Original DifT
Parent Function
6
Left Flow
Full Flow
AlI
Meta Reviewer
Figure 4. Venn diagram of recalled key bugs identified by different code slicing algorithms. The "All" setting represents all comments. while the "+Meta Reviewer" setting denotes multi-reviewer comments merged by the meta-reviewer. analyze per-category performance, a breakdown across logic, security, and performance-related hugs is shown in Appendix P.
leads higher FAR, and FAR2, which negatively affect CPI, and CPI, in the "+Meta Reviewer" setting, How- ever, after introducing the validator, the performance for three reviewers significantly improves in terms of CPI, and CPI2. While more reviewers boost K KBI, they also in- crease false alarms, making the validator essential to overall performance.
Table 4. Impact of increasing the number of reviewers from one to three. The "+Meta Reviewer" setting represents mela-reviewer merging the reviewers' comments, while the "+Vaiiatto" setting denotes the validator refining the comments after the meva-reviewer. All settings 5op-5 truncation of reviewer comments.
Summary of RQ3.1. Increasing the number of reviewers lifts key bug inclusion but raises false alarms. A valida- tor mitigates these alarms, implying a trade-off between coverage and precision. (See Appendix T.3 for extended conclusions.)
Processing Stage
Reviewer Num
KBF
FAR
CPI,
FAR, CPI,T
Oviginal Day
Reviewer
8.89
13.50
+ Validator
1
4.44
76.39
7.47
73.33
7.62
11.11
90.11
10.46
71.00
16.07
Parvid Funetion
5.3.2. SELE-CORRECTION ABILITY LLMS
Meta Reviewer
15.58
80.37
17.36
73. 81
22.02
19.52
+ Validator
3
11.11
6.67
89. -8-
63.70
11.26
10.8L
65.33
56
11.50
16.83
In our framework, the validator refines and validates gen- erated comments correct hallucinations. Table 5 shows that the validator lowers F FAR and F FAR;, but also reduces
Left Flow
Meta Reviower
1
26.67
83.26
20.57
70.56
27.99
17.51
67.98
31.56
K BI, indicating trade-off between precision and recall.
Validator
20.00
11.11
75.37
22.07
43.55
29.54
19.41
Our analysis suggests such erroneous rejections of valid comments by validators primarily stem from factors includ- ing context propagation from earlier pipeline stages, minor inaceuracies in comment positioning, occasional model in- put token limits, and inherent scoring variances.
Full Flow
Meta Reviewer
3
31.11
22.22
78.04
89.41
22.09
15.80
73.10
71.17
28.86
25.10
20.60
77.96
20.97
67.59
24.73
Summary of RQ3.2. Self-correction (validator) reduces false alarms but can inadvertently discard critical bug-
7

---

Towards Practical Defect-Focusee Automated Code Review
Table The self-correction ability of L.LMs through the Validator role. "w/o" denotes without Validator. "w/" denotes with Validator.
Table 7. The BII FAR1, and CPT, results for different code slicing algorithms utilizzig our filtering mechanism. This table illustrates impact of sequential filter stages, including different Top-k truncation values (k=10, 5, 3) for single-reviewer paths. For the multi-reviewer path results shown here (+Meta Reviewer, + +Validator), Top-k is set to 5. A comprehensive discussion of
Validator Status
KBIt
EAR1L
C
FAR24
CPl
Original Daif
w/o
13.33
96.74
5.24
76.56
17.25
w/
11.11
90.11
10.46
71.00
16.07
Parent Famettin
who
20.00
92.41
11.01
73.15
22.92
Top-k sensitivity, covering both single-reviewer variations and multi-reviewer settings, is presented in Appendix S.
11.11
10.81
65.33
Flow
Roviewer
Filter/ Trune
KRIT
FAR
CPI,
KBIT
FAR
C
31.11
87.85
17.51
67.98
31.58
w/
20.00
75.37
22.07
43.52
20.54
Original Duf
Purent Fanction
Fall Flow
All
23.70
96.74
5.71
31.85
96.98
5.52
w/o 31.11
89.41
15.80
73.10
28.66
Crars Filter
17.04
88.72
12.70
22.22
87.17
16.18
w/
20.00
77.96
20.97
67.59
24.73
Single
Top-10
17.04
13.57
Top-5
14.81
68.11
12.48
14.81
87.67
13.40
Top-3
11.11
59.14
10.90
8.15
88.15
9.63
Meta
13.33
96.74
5.24
20.00
92.41
11.01
Table 6. Impact of Chain-of- Thouuhhtt (CoT) on the framework, pre- senting paired slicing algorithm comparisons. "SR" denotes Single Reviewer, "MR" denotes Multi Reviewers. All multi-reviewer settings use three reviewers and Top-5 truncation.
Multi
Validator
11.11
00.11
10:46
11.11
89. 48
10.81
Taft Flow
Full Flow
All
37. 04
94.36
9.77
39.26
94. 48
9.67
Counse Filter
33.33
02.03
12.80
32.59
92.65
11.95
Single
Top
32.59
91.13
13.94
31.85
91.68
13.18
Stage
CaT
KBIT & CPI,
KBIT
F
CPI,
Top
90.15
25.93
13.90
Top-3
17.78
60.00
12.77
13.33
91.60
10.23
Original Dij
Poresz Function
31.11
87.81
17.51
15.80
SR All
who
20.74 T-
97.08
5.12
31.11
96.91
5.60
Muli
Validator
20.00
75.37
22.07
20:00
77.96
20.97
96.74
5.71
31.85
SR Top.5 5
w/o
17.04
89.17 17
13.05
16.30
94.57
8.03
w/
14.81
89.11
12.48
14.81
87-67
13.40
MR Meta
wio
15.56
82.74
16.36
5.24
17.78
89.09
13.52
11.01
MR- Val
W/ 11.11
6.67
90.11
63.89
10.46
11.26
11.11
11.11
89 48
75.
10.81
15.35
ing (Left Flow, Full Flow), adding these filters sequentially decreases F FAR and improves CPI,, In simpler slicing (Original Diff, Parent Function), only the coarse filter proves particularly effective, likely due to limited context causing more hallucinations. A comprehensive sensitivity analy- sis of the Top-k truncation hyperparameter k detailing its impact single-reviewer paths with various k values (as presented in Table 7) and extended analysis within our multi-reviewer framework- provided in Appendix S.
Lagtt Fow
Full Mov
SK All
W/ 33.004
wik 34.81
94.36
95. 16
9.77
8.49
40.00
39.26
94.45
34.40
9.67
9.83
SR Top
w/o 20.74
92.37
11.10
20.74
91.94
11.50
25.93
13.90
MR. Meta
W/ 31
W/l 17.78
87.81
88.74
13.79
17.51
31.11
26.67
69 41
82.81
20.90
15.80
Val
whh 11.11
20.00
75.37
75.56
22.07
16.28
6.67
68.33
20.97
11.01
detecting comments. Balancing these factors is crucial. (See Appendix T.3 for extended conclusions.)
Summary of RQ4, Our comment filter significantly reduces false alarms and improves performance in more detailed slicing methods. In simpler slicing, the coarse filter stage is the most impactful step. (See Appendix T.4 for the extended conclusion.)
5.3.3. EFFECTIVENESS OF SHAIN-OF-THOUGHA
We compared our specified CoT approach with free-form reasoning. Table 6 shows that CoT prompts often excel in complex slicing tasks (Left Flow, Full Flow), but in simpler tasks (Original Diff, Parent Function), free-form can be just as good or better.
5.5. RQ5. Line Number Position
Line number localization is crucial for real-world appli- cations. We tested three formats: No: No line position information is provided; Relative: Code is provided with a separate list containing relative line positions; and Inline: Position information is integrated directly into the code using the format in Table 1.
Summary of RQ3.3. CoT prompting is especially benefi- cial in complex contexts. For simpler code slices, the model may perform well without explicit CoT guidance. As more powerful reasoning models, such as GPT-OI and DeepSeek- RI, emerge, the advantage of specified CoT over free-form reasoning may further diminish. (Appendix T.3)
Table 8 shows that providing line number information (es- pecially inline) significantly improves performance and lo- calization success rate (LSR).
5.4. RQ4. Effectiveness of Comment Filter Mechanism
Summary of ROS. Embedding line numbers inline yields the highest performance and LSR, likely because it helps the model anchor comments to specific lines accurately. (See Appendix T.5 for the extended conclusion.)
The comment filter mechanism includes Coarse reviewer filter, @ Top-k truncation, @ Meta-reviewer filter, and A Validator validation. Table shows that in flow-based slic-
8

---

Towards Practical Defect-Focused Automated Code Review
tive, this approach provides limited and idealized view of code review, often evaluated with text similarity metrics, such as BLEU or ROUGE, which do not fully capture the expectations of real-world developers for finding defects.
Table Impact of line number position information. "All" repre- sents the average of all comments generated by reviewers, while "+Meta Reviewer" denotes the neviiere workflow with three reviewers and Top-5 truncation. LSR (Line Success Rate) mea- sures whether LLMs provide valid lines, regardless of correctness.
In practice, code review is more complex, evaluated at the level of entire merge requests of repository codebases rather than individual code-comment pairs. The focus on text similarity fails to consider the broader context, including how comments address the full scope of changes in MR. Although contributing valuable insights, these studies fall short of replicating the holistic, real-worke workflow.
Position
KBIT
F
C
FAR24
CPL, LSK?
"ARl" Senting
No 30.37
95.66
7.58
93.12
11.17
90.54
Relative
42.96
94.60
9.58
92.69 92.75
12.32
Inlline
37.04
94.36
9.77
90.66
14.79
91.11
-a Mena Reviewer" Sentnng
No
17.78
900.7
72.71
Relative
17.78
93.32
9.50
78.04
20:41
Inline
31.11
87.81
17.51
67.98
31.06
7. Conclusion
6. Related Work
Motivated by the limitations of prior research that oversim- plified code review automation and fell short of practical applications, explored the complete automation pipeline within real-world company. We identified and addressed key challenges such as capturing relevant code context, im- proving key bug inclusion (KBI), reducing false alarm rates (FAR), and integrating human-centric workflows. Our ap- proach introduces four code slicing algorithms, a multi-role LLM framework, a comment filtering mechanism, and a prompt format with inline line number localization. Evalua- tions on real-world data demonstrated that we significantly outperforms existing methods, achieving up to a 10x im- provement in the comprehensive performance index (CPI) over previous baselines.
Code review comments play a crucial role in maintaining software quality, leading to significant research efforts in automating this process. Early studies, such as Gupta & Sundaresan (2018), employed retrieval-based methods, utilizing LSTM models to match new code snippets with historical changes to recommend comments. Siow et al. (2020) advanced this approach by incorporating attention mechanisms to capture semantic nuances more effectively.
With the advent of deep learning, the focus shifted towards automated comment generation. Pioneering efforts by Tu- fano al. (2021; 2022) introduced models trained on di- verse datasets, including technical texts and code snippets. Subsequent innovations included specialized models such as CodeReviewer (Li et al., 2022b), which leveraged pre- training on code review data, and AUGER (Li et al., 2022a), which used review tags to streamline the task. Another ap- proach, CommentFinder (Hong et al., 2022), presented an efficient retrieval-based model tailored to new code. More recently, Reviewrr (Lu et al., 2023) trained large language models specifically for code review tasks, and DISCOREV (Ben Sghaier& & Sahraoui, 2024) improved performance by applying cross-task knowledge distillation across successive tasks, and Yu et al. (2024b) focused on fine-tuning LLMs to improve both the accuracy and com- prehensibility of automated code reviews. Alongside these advancements in direct comment generation, recent stud- ies have also explored the application of LLMs to other related aspects of the software development lifecycle, such as enhancing code reviewer recommendation (Wang et al., 2024) and automating commit message generation (Tao et al., 2024), underscoring the expanding utility of large models in diverse software engineering contexts.
Key insights include: O Flow-based slicing (Left Flow and Full Flow) provided better context and outperformed sim- pler methods. @ Increasing the number of reviewers im- proved KBI but required validation to manage false alarms effectively. @ The validator role reduced hallucinations but slightly lowered KBI, highlighting a trade-off between precision and recall. Chain-of-thee -f-thoughtt guidance proved more valuable in complex slicing scenarios. O Inline line number localization enhanced both comment accuracy and localization success rates.
Looking ahead, four key areas for future research are: 1 Enhancing code slicing algorithms to capture more rele- vant context, potentially combining different slicing levels. @ Refining L.M interactions and enhancing engine L.M capability to improve key bug recall. O Further optimiz- ing the filtering mechanism, including the investigation of adaptive or learned thresholds, to reduce nitpicks and hallu- cinations more effectively. O Streamlining pipeline to make automation more accessible.
Despite these advances, previous works have oversimplified the code review process by treating it as a set of snippet- level code-comment pairs. These approaches typically split merge requests into independent snippets and framed the task as a one-to-one neural machine translation (NMT) prob- lem, converting code into natural language. While innova-
Limitation. We discuss limitations in Section V.
Data Availability. We publicly release our codes at https: / zezodoo org/ hwtgg//errr////e/ 479175. Details regard- ing their open-source source status can be found in Section U.
9

---

Towards Practical Defect-Focusee Automated Code Review
Acknowledgments
Facilitating multi-agent collaboration and exploring emer- gent behaviors. In ICIR, 2024a.
This work was supported by the National Key Research and Development Program of China (No. 2023YFB3307002) and the Alliance of International Science Organizations Collaborative Research Program (No. ---2-22-- 03).
Chen, Y., Hu, Z., Zhi, C., Han, J., Deng. S., and Yin, J. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pp. 572-576, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400706585. 3663529.366380) URL https://doi org/10. 000.335299.66881...
Impact Statement
doi: 10.1145/
This work advances the practical defect-focused applica- tions of automated code review. We redefine the code re- view task by shifting from snippet-level code-to-text for- mulation to end-to-end, rerge-request-level codebase analysis. Our approach fills previously overlooked sub-tasks in the automation pipeline and introduces more practical and objective metrics, better aligning the process with de- velopers' expectations for defect detection in real-world software development. These refinements not only estab- lish more comprehensive foundation for future research in automated code review but also offer insights applicable to other software engineering tasks. Furthermore, our generic framework, including the proposed context slicing algo- rithms, provides a versatile methodology that can inspire broader applications in code intelligence.
Cheng, X., Wang, H., Hua, J., Xu, G., and Sui, Y. Deep- wukong: Statically detecting software vulnerabilities us- ing deep graph neural network. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(3): 1-33, 2021.
Fagan, M. Design and code inspections reduce errors in program development. In Software pioneers, pp. 575-607. Springer, 2002.
Gauthier, Aider ai pair programming in your terminal, 2024. URL https: /ttps://github com/ paul-gauthier/aidee. Accessed: 2024-09-12.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In 33rd International Joint Conference on Artificial Intelli- gence (IJCAI 2024). IJCAI, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint 0rX1v:2333.08774. 2023.
Gupta, A. and Sundaresan, N. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' 18) Deep Learning Day, 2018.
Ackerman, A. F., Buchwald, L. S., and Lewski, F. H. Soft- ware inspections: an effective verification process. IEEE software, 6(3):31-36, 1989.
A1Omar, E. A., Venkatakrishnan, A., Mkaouer, M. W., New- man, C., and Ouni, A. How to refactor this code? an exploratory study on developer-chatgot refactoring con- versations. In Proceedings of the 21st International Con- ference Mining Software Repositories, pp. 202-206, 2024.
Hong, Y., Tantithamthavom. C., Thongtanunam, P., and Aleti, A. Commentlinder: simpler, faster, more accurate code review comments recommendation. In Proceedings of the 30th ACM Joinl European Software Engineering Conference and Symposium on the Foundations of Soft- ware Engineering, pp. 507-519, 2022.
Bacchelli, A. and Bird, C. Expectations, outcomes, and challenges of modern code review. In 2013 35th Interna- tional Conference on Software Engineering (ICSE), pp- 712-721. IEEE, 2013.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. Ruler: What's the real context size of your long-contex. language models? In First Conference on Language Modeling, 2024.
Ben Sghaier, O. and Sahraoui, H. Improving the learning of code review successive tasks with cross-task knowl- edge distillation. Proceedings of the ACM on Software Engineering, 1
Jiang, T., Wang, Z., Liang, J., Li, C., Wang, Y., and Wang, T. Robustkv: Defending large language mod- els against jailbreak attacks via ky eviction. In Proceedings of the 13th International Conference on Learning Representations (ICLR). OpenReview, April 2025. URL https: /ttpss////pp/eeeeeeee new. forum? i id=1.5godA0C2z. Poster presentation.
2024.
Chen, W. Su. Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., al. Agentverse:
10

---

Towards Practical Defect-Focuree Automated Code Review
Kononenko, O., Baysal, O., and Godfrey, M. W. Code re- view quality: How developers see it. In Proceedings of the 38th international conference software engineer- ing, Pp. 1028-10200 1038, 2016.
compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M.. Petroni, F,, and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E.,, Zhang. H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Lu, J., Yu, Li, X., Yang, L., and Zuo, C. Llama-reviewer: Advancing code review automation with large language models through parameter-effiiien fine-tuning. In 2023 IEEE 34th International Symposium on Software Relia- bility Engineering (ISSRE), pp. 647-658. IEEE, 2023.
Li, L., Yang, L.,, Jiang, H., Yan, J., Luo, T., Hua, Z., Liang, G., and Zuo, C. Auger: automatically generating review comments with pre-training models. In Proceedings of the 30th ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering, pp. 1009-1021, 2022a.
Lu, J., Li, Z., Shen, C., Yang, L., and Zuo, C. Exploring the impact of code review factors the code review comment generation. Automated Software Engineering. (1(2):71, 2024.
Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., and Zhong, Y. Vuldeepecker: A deep learning-based system for vulnerability detection. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018. The Internet Society, 2018. URL https: wwwww/www.........oo... hww.:w..dssssoo... symposium org/ wp- cop-content//oe http://lllaad//1llle// /22 20t./aaaa///222/0//////21111 ndss2018_ 03A-2 Li pappr.ppppp.... pdf.
Lu, J., Li, X., Hua, Z., Yu, L., Cheng, S., Yang, L., Zhang, F. and Zuo, C. Deeperceval: Revisiting the evaluation of code review comment generation. In International Conference on Fundamental Approaches to Software En- gineering, pp. 43-64. Springer, 2025.
Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B.. Liang, J., Zhou, M., and Xiao, Knowlog: Knowledge enhanced pre-trained language model for log understanding. In Pro- ceedings of the 46th IEEE/ACM International Conference Software Engineering, pp. 1-13, 2024a.
Li, Z., Zou, D., Xu, Jin, H., Zhu, Y., and Chen, Z. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing, 19(4):2244-223) 2021.
Ma, Z., Chen, A. R., Kim, D. J., Chen, T.-H., and Wang, S. LImparser: An exploratory study on using large language models for log parsing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. pp. 1-13, 2024b.
Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Ma- jumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. Automating code review activities by large-scale pre- training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1035-1047, 2022b.
Marjamäki, D. Cppcheck static analysis of c/c++ code, 2024. URL https: /// ////tttt/ fggituu........ com/danmes oppcheck. Accessed: 2024-09-12.
Liang, J., Wang, Y., Li, C., Zhu, R., Jiang, T., Gong, N., and Wang, T. Graphrag under fire. arXiv preprint arXiv:2501. 14050, 2025.
McAleese, N., Pokorny, R. M., Uribe, J. C., Nitishinskaya E., Trebacz, M., and Leike, J. Llm crities help catch Ilm bugs, 2024. URL https: htttps://ariiv httpiv.ivvvv///////////a orgg abs/
Lin, B., Wang. S., Liu, 7., Liu, Y., Xia, X., and Mao, X. Cct5; A code-change-oriente pre-trained model. In Pro- ceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, 2023.
Organization, P. Flask - the python micro framework for building web applications, 2024. URL https: //github. wtt.huu..cooopaaetss/laaaae Accessed: 2024- 09-12.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp- 74-81, 2004.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm

---

Towards Practical Defect-Focused Automated Code Review
Ray, P. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physicee Systems, 3:121-154, 2023.
Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., and Bavota, G. Towards automating code review activities. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 163-174. IEEE, 2021.
Rigby, P. C. and Bird, C. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th joint meeting on foundations of software engineering, pp. 202-212, 2013.
Tufano, R., Masiero, S., Mastropaolo, A., Pascarella, L.,, Poshyvanyk, D., and Bavota, G. Using pre-trained models to boost code review automation. In Proceedings of the 44th International Conference Software Engineering, pp. 2291 22022, 2022.
Rigby, P. C.., German, D. M., and Storey, M.-A. Open source software peer review practices: case study of the apache server. In Proceedings of the 30th international conference on Software engineering, pp. 541-550, 2008.
Tufano, R., Martin-Loppee Lopez, A., Tayeb, A., Dabić, O., Haiduc, S., and Bavota, G. Deep learning- based code reviews: paradigm shift or a double-edged sword? arXiv preprint arXiv:2411.1144 2024.
Rigby, P. C.,, German, D. M., Cowen, I.., and Storey, M.-A. Peer review on open-source software projects: Parame- ters, statistical models, and theory. ACM Transactions on Software Engineering and Methodology (TOSEM), 23(4): 1-33, 2014.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Sadowski, C., Söderberg, E., Church, L., Sipko, M.. and Bacchelli, A. Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, pp. 180-190, 2018.
Votta Jr, L. G. Does every inspection need meeting? In Proceedings of the Ist ACM SIGSOFT Symposium on Foundations of Software Engineering, pp. 107-114, 1993.
Wang, L.., Zhou, Y., Zhuang, H., Li, Q., Cui, D., Zhao, Y., and Wang, L. Unity is strength: Collaborative Ilm-based agents for code reviewer recommendation. In Proceed- ings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 2235-222522222 1233-223,,,22 2024.
Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software En- gineering, 2023.
Yang, X., Kula, R. G., Yoshida, N., and lida, H. Mining the modern code review repositories: A dataset of people, process and product. In Proceedings of the 13th Interna- tional Conference on Mining Software Repositories, pp. 460-463, 2016.
Shan, Q. Sukhdeo, D., Huang, Q. Rogers, S., Chen, L.., Paradis, E., Rigby, P. C., and Nagappan, N. Using nudges to accelerate code reviews at scale. In Proceedings of the 30th ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering, pp. 472-482, 2022.
Yu, L., Lu, J., Liu, X., Yang, L., Zhang, F., and Ma, J. Psevfinder: prompt-tuning based framework for smart contract vulnerability detection. In 2023 IEEE 34th Inter- national Symposium on Software Reliability Engineering (ISSRE), pp. 556-567. IEEE, 2023.
Siow, J. K., Gao, C., Fan, L., Chen, S., and Liu, Y. Core: Automating review recommendation for code changes. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). pp. 284- 295. IEEE, 2020.
Yu, L., Chen, S., Yuan, H., Wang, P., Huang. Z., Zhang, J., Shen, C., Zhang, F., Yang, I.,, and Ma, J. Smart- Ilama: Two-stage post-training of large language models for smart contract vulnerability detection and explanation. arXiv preprint arXiv:241 0et224111.622,,1 2024a.
Sommerville, I. Software engineering 9th edition. ISBN-10, 187035152:18, 2011.
Tao, W.. Zhou, Y., Wang, Y.. Zhang. H., Wang, H., and Zhang, W. Kadel: Knowledge-awand aware denoising learn- ing for commit message generation. ACM Transactions on Software Engineering and Methodology, 3--5)-1-32, 2024.
Yu, L., Huang, Z., Yuan, H., Cheng, S.. Yang, L., Zhang, F., Shen, C., Ma, J., Zhang, J., Lu, J., and Zuo, C. Smart- llama-dpo: Reinforced large language model for explain- able smart contract vulnerability detection. Proceedings of the ACM on Software Engineering, 2(ISTTA), July 2025. ISSN 2994-970X. doi: 10.1145/3728878. Licensed under Creative Commons Attribution 4.0 International License.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023.
12

---

Towards Practical Defect-Focused Automated Code Review
Yu, Y., Rong, G., Shen, H., Zhang, H., Shao, D., Wang, M., Wei, Z., Xu, Y., and Wang, J. Fine-tuning large language models improve accuracy and comprehensibility of automated code review. ACM transactions on software engineering and methodology, 34(1):1-26, 2024b.
Yuan, H., Yu, I.., Huang, Z., Zhang, J., Lu, J., Cheng, S., Yang, L., Zhang, F., Ma, J., and Zuo, C. Mos: To- wards effective smart contract vulnerability detection through mixture-of-experts tuning of large language mod- cls. arXiv preprint arXiv:2504. 12234, 2025.
Zhang. Y., Qiu, Z., Stol, K.-.,, Zhu, W., Zhu, J., Tian, Y., and Liu, H. Automatic commit message generation: A critical review and directions for future work. IEEE Transactions on Software Engineering, 2024.
13

---

Towards Practical Defect-Focused Automated Code Review
A. The Central Role of Defect Detection in Code Review
Identifying defects has consistently been recognized as the core and most fundamental goal of code review. This under- standing aligns with its historical origins, current industry expectations, state-of-the-art research directions. and pressing real-world needs:
Historical Foundations:
Defect detection was the original purpose of code reviews, tracing back to the concept of code inspection proposed by Michael Fagan at IBM in 1976 (Fagan, 2002). Fagan Inspections introduced structured process aimed at reducing long-term costs by detecting and fixing defects early. Subsequent decades of research continued to center on discovering and resolving faults (Ackerman et al., 1989; Sommerville, 2011; Votta Jr, 1993).
Contemporary Expectations: Empirical studies show that both developers and managers consider defect detection the primary expectation of code reviews (Bacchelli & Bird, 2013). For example, comprehensive survey involving 165 managers and 873 programmers at Microsoft revealed that while code review can serve multiple functions, identifying defects remains the foremost motivation for all stakeholders.
Current Research Directions: Recent state-of-the-art (SOTA) work in automated code wvview-patticularyy generation-based methods--cntinile to emphasize defect detection as the core research objective (Tufano et al., 2024). While these studies simulate code review by injecting known defects, our approach leverages actual, historically documented defects, providing a more realistic and robust evaluation scenario.
Real-World Industrial Needs: Industry practitioners, especially "super reviewers" overseeing thousands of developers and large codebases, highlight an urgent and practical need for more effective defect detection. These expectations were extracted from the Objectives and Key Results set by approximately 50 experienced super reviewers (the concept comes from Kononenko et al. (Kononeeeo et al., 2016)) within major production environment of 4000+ developers.
In conclusion, defect detection stands as the central and most essential aspect of code review. While other dimensions- as code improvement, comprehension, and communication do appear frequently, they often do not align with the urgent expectations of practitioners. This disconnect between what teams urgently need (effective defect detection) and what reviews often deliver (broader but less critical commentary) warrants a focused research effort (Bacchelli & Bird, 2013).
B. Detailed World Workflow Integration
As briefly mentioned in the Introduction and illustrated in Figure our automated code review framework is deeply integrated into the real-world Continuous Integration/Continuuus Deployment (CI/CD) pipeline and DevOps platform of large-scale online service company. This appendix provides more detailed description of this integration, designed to be seamless for developers and provide actionable feedback directly within their existing workflows.
The primary goal of this integration is to automate aspects of the code review process without disrupting established development practices, thereby enhancing both efficiency and code quality. The workflow is triggered upon the submission or update of Merge Request (MR) within the company's internal DevOps system and proceeds through several automated stages:
1. MR Trigger and Initial Verification: When developer submits an MR, a webhook notifies our automated review system. The system first performs essential verification checks. This includes confirming the submitting user's permissions and ensuring that the changed files fall within the scope of automated review (e.g., correct programming language, project-specific configurations for review). This step is crucial for security, access control, and efficient resource allocation.
2. Code Analysis and Comment Generation Launch: Once the MR is verified, the system retrieves the relevant code changes. The core analysis process is then launched:
Code Slicing: The modified code segments are processed by our code slicing algorithms (detailed in Section 3.2)
extract relevant contextual information necessary for effective review.
Multi-role Review: These code slices, along with their context, are distributed to our multi-role LLM framework (described in Section 3.3). Each role, potentially with specialized roles (c.g., Reviewer, Validator), analyzes the code identify potential issues and generate draft review comments.
14

---

Towards Practical Defect-Focused Automated Code Review
3. Comment Filtering and Refinement:
The raw comments generated by the LLM roles undergo a rigorous filtering process using our Redundancy Comment Filter Mechanism (explained in Section 3.4). This multi-stage process (involving Q1-Q3 scoring, coarse filtering, meta-reviewer processing, and validator re-scoring) aims to eliminate nitpicks, false positives, and less critical suggestions, ensuring that only high-quality, actionable comments proceed.
4. Seamless Injection into DevOps Platform and Developer Notification: This stage is critical for effective real-world integration:
DevOps System Integration: The filtered and validated comments are programmatically injected into the company's internal DevOps platform using its provided APIs. Each comment is associated with the specific MR and the relevant commit.
Line-Aware Comment Positioning: A key feature for developer adoption is the precise positioning of each comment directly at the relevant line number(s) in the diff view of the MR. This is achieved by accurately parsing diff hunks and mapping comment locations. The effectiveness and importance of this line-aware comment injection for providing clear, contextualized, and actionable feedback was specifically evaluated in Section 5.5 (RQ5) and found to be highly valued by developers.
Developer Notification: Developers (typically the MR author and assigned human reviewers) are notified of the automated review comments through the DevOps platform's 's standard notification mechanisms (e.g., email, internal messaging/chat system integration). The comments appear within the MR's 's discussion or review interface, similar to comments made by human colleagues.
This end-to-end automated workflow, from MR submission to the delivery of precisely positioned, filtered comments directly within the developers' familiar DevOps environment, constitutes a seamless integration into their daily activities. minimizes context switching, presents feedback in an actionable format, and leverages existing platform features for discussion and resolution of the identified issues. The positive developer feedback regarding the non-intrusiveness and utility of the system, particularly due reliable line-aware comment placement, substantiates our claim of successful real- world workflow integration. This integration was a prerequisite for addressing the key challenges outlined in Section C within a practical, industrial setting.
C. Four Challenges Identified in Code Review Process
Challenge 1: Capturing Proper Code Context. During the merge request process, code is integrated into the repository. Prior studies often split the input into method-level or diff hunk-level segments, which introduces two major problems: O the omission of critical context beyond the method, hunk, or even file level, such as variable declarations, definitions, and in languages like C++ misleads large language models (L.LMs) into generating false alarms or missing key bugs; and @ the truncation of long code snippets, leading to significant omissions. However, feeding excessively long inputs to LLMs also degrades performance due to the models' sensitivity to the relative position of the target sequence (Hsieh et al., 2024; Liu et al., 2024). Therefore, finding an optimal way to capture proper code context is essential.
Challenge 2: Improving Key Bug Inclusion (KBI). A primary goal of automated code review is identify key bugs introduced by new merge requests that could compromise system reliability and performance. In 2022, 30% of severe P1+ incidents (asset loss $350,000) and 24.04% of P4+ incidents in the company were attributed to preventable low-level faults, highlighting the need for robust code reviews and automated systems. Prior studies have validated models based on overall text similarity metrics, which are often misleading. Text similarity does not necessarily correlate with model's recall ability, as higher similarity scores may reflect linguistic style rather than the identification of critical bugs(Lu et al., 2025). OpenAl researchers have proposed tritique-Bug Inclusion (CBI) as a key metric for evaluating LLM performance (McAleese et al., 2024), which we adopt for our task. Increasing the KBI capability of the framework a key focus of our approach.
Challenge 3: Reducing False Alarm Rates (FAR). Generative models frequently produce redundant irrelevant comments, including nitpicks and hallucinations, even for single code snippet (McAleese et al., 2024). In real-world scenarios, merge requests often contain numerous code snippets, and managing redundant comments can overwhelm human reviewers. Most previous studies (Gupta & Sundaresan, 2018; Siow et al., 2020; Tufano et al., 2021; 2022; Li et al., 2022a; Hong et al., 2022) have not addressed this issue, simplifying the review task to merely generating natural language comments from problematic code. Some works attempted to introduce discriminators to assess comment quality (Lin et al., 2023; Li et al., 2022b), but
15

---

Towards Practical Defect-Focused Automated Code Review
these approaches have been shown be ineffective (McAleese et al., 2024). Thus, robust filtering mechanism is needed minimize redundant comments and reduce false alarms.
Challenge 4: Human-Centric Workflow Integration. Simplified code review tasks often overlook essential real-world workflow components, such as attaching comments to specific code lines. This step reduces the cognitive load on developers by making it easier to verify the validity of comments. Despite its importance, this aspect is frequently ignored in previous work (Gupta Sundaresan, 2018; Siow et al., 2020; Tufano et al., 2021; 2022; Li et al., 2022a; Hong et al., 2022; Lin et al., 2023; Li al., 2022b), which hinders real-world usability, Incorporating such functionality critical for improving user experience alongside KBI and FAR.
D. Demands and Expectations from Reviewers
We conducted qualitative studies, including surveys and in-depth interviews, to understand the expectations and demands for automated code review from a group of 50 experienced "super reviewers" within a large technology company's 's core infrastructure teams. From the reviewers' perspective, an effective automation solution must fulfill two primary demands regarding its operational behavior and meet three key expectations concerning the quality and utility of its feedback:
The two demands, which aim to simulate desirable aspects of human review, are:
D1:
Automation should operate at the merge request level, reviewing all changes holistically rather than focusing on partial code snippets or individual diff hunks.
Automation should incorporate global repository knowledge beyond the diff, enabling access broader information such as variable declarations or function definitions not included in the immediate changes.
D2:
The three key expectations for the generated feedback are:
El: Identify as many key issues as possible, thereby reducing the number of missed critical bugs. This directly informed our Key Bug Inclusion (KBI) metric, introduced in the Introduction. In our interviews, developers consistently emphasized that "ccaching critical bugs" is the absolute top priority for any automated review tool.
E2: Minimize nitpicking comments and hallucinations (i.e., non-existent issues), which corresponds to reducing the False Alarm Rate (FAR). This is crucial for lessening the burden on human reviewers when verifying automated feedback. Developers particularly stressed this point, noting that reducing irrelevant comments is vital for the adoption and continued trust in an automated system, with one sentiment being that "even one false positive can erode trust." " Initial feedback from the system's current deployment within an internal development team has further underscored this, revealing FAR to be an especially sensitive metric directly impacting developer perception and engagement.
E3 Support human-centric interaction by attaching comments precisely to the correct code lines. This aids reviewers or code owners efficiently verifying issues and prevents confusion or misleading information, factor captured by our Line Localization Success Rate (LSR).
The demand D2 aligns with Challenge 1 (Capturing Relevant Context) from the Introduction, while expectations E1, E2, and E3 correspond to Challenges 2 (Improving KBI), 3 (Reducing FAR). and 4 (Integrating Human Workflows) respectively. Previous work primarily focused on aspects related to E1, often using text similarity as an indirect proxy, but this does not guarantee precision or the detection of key bugs, rendering such approaches less suitable for demanding real-world applications. contrast, our approach, informed by these direct developer insights, addresses these challenges comprehensively. We introduce of merge- level evaluation metrics tailored to these real- world demands, including Key Bug Inclusion (KBI), False Alarm Rate (FAR), Comprehensive Performance Index (CPI), and Line Localization Success Rate (LSR).
While our metrics and system design are thus grounded in extensive interactions with professional developers and initial deployment feedback, we acknowledge that formal, systematic user studies evaluating perceived review quality and overall system utility have not yet been conducted. Such studies represent an important avenue for future work to further strengthen the validation of our approach.
16

---

Towards Practical Defect-Focusee
Automated Code Review
E. Background: Code Slicing
Providing the entire repository as input ensures comprehensive context, but large language models (LLMs) have token limitations. Increasing the imput size within these constraints can reduce model performance and lead delays in inference (Hsieh et al., 2024; Liu et al., 2024). Code slicing offers a potential solution by efficiently providing sufficient context while remaining concise. This technique uses static analysis to form code units, parsing code into an abstract syntax tree and slicing it based on node relationships. For large code snippets, code slicing divides them into small, independent segments. For small snippets, like diff hunks in code reviews, it enriches the context by retrieving statements and variable usages related to the changes. While code slicing has been effectively applied to tasks such as vulnerability detection (Li et al., 2018; 2021; Cheng et al., 2021; Yu et al., 2023), its explicit and systematic integration into M.M-aadedd automated code review
for enhancing context-aware defect localization and guiding comment generation generation-appears to be less explored in prior literature. While Lu al. (2024) made initial attempts at testing context components, the bulk of prior research in automated code review has remained largely centered on snippet-level analysis comment naturalness. Consequently, the richer repository-Iever contextual understanding, such that afforded by code slicing, has often been underutilized. To address real- world needs and explore this promising direction, we designed four code slicing algorithms that provide appropriate context for automated reviews.
F. Background: Large Language Models and Multi-role Systems for Software Engineering
Transformer-based large language models (LLLMs) al., 2017) have achieved notable success in natural language processing (asks(Ray, 2023). The ongoing evolution and refinement of these models involve diverse research efforts, including explorations into advanced knowledge integration techniques (Liang et al., 2025) and the development of robust safeguards against potential misuses (Jiang et al., 2025). Given that LLMs now often include code in their training corpora, they have developed strong abilities in code-related tasks alongside their general language capabilities. This blurs the line between code-specific and general models. For instance, GPT-4 excels in code generation, while models like ChatGPT (Achiam et al., 2023) and LLaMA (Touvron et al., 2023) have shown potential in generating commit messages (Zhang et al., 2024), tests (Chen et al., 2024b; Schäfer et al., 2023), method renaming (AlOmar al., 2024), log analysis (Ma et al., 2024a;b), and smart contract vulnerability detection Yu et al., 2024a; 2025; Yuan et al., 2025).
Complex tasks like automated code review, which demand deep-level reasoning. often exceed the capabilities of individual LLMs. Multi-role systems have emerged as an effective approach to decompose and tackle such tasks, assigning specialized roles to different LLM roles that collaborate to solve the overall problem (Guo et al., 2024; Chen et al., 2024a). In this work, we adopt a multi-role system utilizing mainstream open-source LLMs with large parameter counts to enhance Key Bug Inclusion (KBI) and filter redundant comments, thereby making the code review process more efficient and precise.
G. Code Slicing Algorithms
The code slicing process in this paper is composed of several interrelated steps designed to isolate, identify, and group code statements relevant to given set of changes. Specifically, the detailed algorithms of our approach are presented in Algorithms 1-9, each focusing on a different aspect or mode of slice generation. These algorithms work together as follows:
Algorithm (CodeSlicing) drives the entire process: It clones the target repository at the specified commit, initializes data structures, and orchestrates the slicing workflow for each file.
Algorithm 2 (ProcessAST) processes each file's Abstract Syntax Tree (AST) using a chosen slicing option, determining which statements intersect with the diff and storing those statements in cache.
Algorithm (GenerateNew.Slic)) takes contiguous segments of diff statements as seeds and expands them according to the chosen slicing strategy, removing statements from the cache once they are incorporated into slice.
Algorithm (GeeCattiguussssffeeeeeet)t serves as a helper function that extracts a cohesive set of adjacent statements from the cache, ensuring that logically connected changes are processed together.
Algorithm 5 (ApplySticingAAloorrtnn) acts as a dispatcher, selecting one of four specific slicing methods based on the chosen option:
- Algorithm 6 (OriginalDiff): Focuses on the original diff statements and their direct dependencies.
17

---

Towards Practical Defect-Focused Automated Code Review
Algorithm 7 (ParentFunction); Retrieves the smallest function containing all diff statements, providing function- level context for the slice.
-Algorithm (Lef:Flow): Performs a backward data-flow trace from the diff statements by analyzing L-values and their defining statements.
-Algorithm 9 (FullFlow): Extends LeftFlow by also including forward data-flow tracing for R-values and callees, capturing both backward and forward dependencies.
Through this sequence of algorithms, relevant code segments are iteratively located, expanded, and grouped, resulting in collection of slices tailored to the user's chosen level of context or detail.
Algorithm 1 CodeSlicing
1: main CodeSlicing
2: Input: repository, commit, slicingOption
3: Output: slices
4: clone repository and checkout to commit
5: ASTss + ApplyStaticAnalysiss Toooepssssssol
6: cache InitializeCache()
7: slices (Initialize slice list)
8: for each AST in ASTTs do
9: ProcessAST(AST) slicingOption, cache)
10: end for
11: while not cache.isf.mpt(() do
12: seed + GelComiiguuuDDiiiseeeeennaaattde
newSS + GenerateNew cache, slicingOption) slices (ipieeeeaapn
15:
end while end main
16:
Algorithm 2 ProcessAST
function ProcessAST
2: Input: AST, option, cache
3: Output: updated cache
4: slice +-
option)
5: for each statement in slice do
6: slatement intersects with diff then
7: cache.add statemmnn)))
8: end if
9: end for
10:
end function
H. Integration of Chain-of-Though (CoT) in the Review Process
We integrate Chaou-of-Thought (CoT) prompts to guide the roles in the review process. Below are the CoT prompts for each:
For the Reviewer:
1. System Introduction: Introduces the task and provides guidance related to the code repository and input format.
2. Understand:
Helps the model comprehend the purpose of the code changes.
3. Analyze: Instructs the model analyze the code for defects performance issues.
4. Re-evaluate: Guides the model to review its analysis, minimizing nitpicks and hallucinations. Three specific questions are posed to quantify nitpicks, ballucinations, and severity, inspired by (McAleese al., 2024).
18

---

end for Towards Practical Defect-Focused Automated Code Review
Algorithm 3 GenerateeNew.llie
function GenerateNewSSlce
2: Input: seed, cache, option
3: Output: newSlice
4: newSlice 01 (Start forming new slice from the seed]
5: Add seed statements to newSlice
6: Remove seed statements from cache
7: for each statement in newSlice do
8: Expand the slice (Expanding the slice)
9: expondedSlice ApplySlicing (statement, option)
for each expSlaalement in expondedSlice do if expStatement is in cache then
11:
12:
newSlice new.Slice.append((rrp) cuche.removeeerpSlaeeeeee end
13:
14:
15:
end for end function
Algorithm 4 GeCConigguosSDiiiSegemn
1: function DeCContiguoussDifffegemmn
2: Input: cache
Output: contiguousSegment
4: conliguousSegmenl Extract contiguous segment of cached diff statements
5: end function
Algorithm 5 ApplySSicingAllgrrith
function ApplySlicingAlgorrtti
2: Input: AST, option
3: Output: sliced statements
4: switch (option)
5: case "OriginalDiff":
6: OriginalDiff(AST)
7: case "ParentFunction":
8: Paren(Function(ASST)
9: case "LeftFlow":
10: LeftFlow(AST)
11: case "FullFlow":
12:
FullFlow(AST)
13: end switch
14:
end function
Algorithm 6 OriginalDiff
1: function OriginalDiff
2: Input: AST
3: Output: sliced set S
4: + diff statements in AST
5:
S 0
6: for each in D do
7: 5- SU [d]u dependencies of d
8:
end for
9: end function
19

---

end for Towards Practical Defect-Focused Automated Code Review
Algorithm 7 ParentFunction
1: function ParentFunction
2: Input: AST
3: Output: sliced set 5
4: f- diff statements in AST
5: - smallest function containing all
S all statements and declarations in F
7: end function
Algorithm LeftFlow
I: function LeftFlow
2: Input: AST
3: Output: sliced set S
4: D diff statements in AST
5:
S+0 0
6: for each d in D do
7:
L + all L-values affected by d
8:
for each in L do
9:
5 SU backward trace
10: end for
end function
Algorithm 9 FullFlow
1: function FullFlow
2: Input: AST
3: Output: sliced set S
4: diff statements in AST
5: S (ASTFloo(AST))
6: for each d in D do
7:
R <- all R-values and callees affected by d
8: for each in R do
9:
5 SU forward trace of r
10: end for
end for
end function
20

---

Towards Practical Defect-Focused Automated Code Review
5. Organize Your Thoughts: Directs the model to write detailed review comment specifying the issue, affected lines, root cause, recommended solution, and example code.
6. Final Comment:
Instructs the model to output the final comment in JSON format.
For the Meta-Reviewer:
1. System Introduction: Introduces the task of merging Reviewer comments and provides guidelines the required
format.
2. Analyze: Instructs the model to analyze Reviewer comments, focusing on patterns, discrepancies, and insights.
3. Organize and Sort Final Comments: Guides the model to format the refined comments prioritized JSON list, calculating the overall scores and sorting by criticality.
For the Validator:
1. System Introduction: Similar the Reviewer, but with a focus on accuracy and relevance.
2. Validate the Comment: Guides the model review and validate the existing comments, aiming to reduce false alarms.
3. Refine the Comment: Ensures the comment is refined for clarity and correctness.
4. Final Comment: Outputs the validated comment in JSON format suitable for the development environment.
For the Translator:
1. System Introduction: Introduces the translation task and explains the input format.
2. Translation and Formatting Requirements: Guides the model to translate items into the target language, ensuring proper formatting.
3. Translated Comments: Outputs the translated comments in ISON format for direct integration into the development environment.
I. Rationale for Offline Validation
The primary goal of our review system is to recall as many historical faults as possible while minimizing irrelevant comments that could burden developers. Each recalled fault suggests that our system could potentially prevent similar future issues.
To evaluate the system, we use several key performance metrics, including the key bug inclusion rate (KBI), false alarm rate (FAR), comprehensive performance index (CPI), and line localization success rate (LSRR,, which defined in Section 4.3. The CPI serves as overall measure, balancing the trade-off between effective bug detection (KBI) and minimizing false alarms (FAR).
We hypothesize that this validation approach, which goes heyond simple text text similarity metrics, provides a more accurate measure of the system's 's ability to handle real-world code review challenges. For instance, a change from "and" logic to "or" in code can fundamentally alter the program's behavior, but such subtle difference may missed by conventional text-based comparisons. Additionally, we observed that the same issue often described in multiple ways, complicating simple text-based comparisons and highlighting the need for more nuanced metrics.
J. Research Questions in Detail
RQI: How does the overall performance of our framework compare with previous works? This question evaluates our framework's 's overall performance against existing baselines, focusing on metrics such as KBI, FAR, and CPI (defined in Section 4.3). Since our framework decouples from base LLMs, we utilize various open-source large language model
21

---

Towards Practical Defect-Focused Automated Code Review
(LLM) engines provide comprehensive evaluation. Specifically, for RQ2-5, We conduct ablation studies using the LLaMA3.1-405B L.LM engine, which is one of the most representative models.
RQ2: How do code slicing algorithms impact the performance of the framework? This question investigates the effect of code slicing on the code review process. As the first to apply code slicing in this context, we compare the effectiveness of four slicing algorithms and study the results using a Venn diagram. This analysis focuses primarily on KBI and CPI, particularly the ability to recall key bugs.
RQ3: How do the different components of the multi-role system impact the performance of our framework? This question explores the influence of various components of our multi-role system, including the number of reviewers, the self-correction capability of LLMs (validator), and the impact of Chain-of-Thought (CoT) prompts. We measure KBI, FAR, and CPI to assess overall performance.
RQ4: How does the redundancy comment filter mechanism address nitpicks and hallucinations? This question evaluates the effectiveness of our redundancy comment filter mechanism in reducing nitpicks and hallucinations. We sequentially evaluate the contribution of each component of the filter mechanism, with a focus on KBI, FAR, and CPI.
RQ5: How does the representation of line number position information impact overall performance and line number localization success rate? This question assesses the impact of our code representation format overall performance and the line localization success rate (LSR). We compare our method with two other formats: no line position information and supplementary line position in a separate list. Here, LSR is considered alongside KBI, FAR, and CPI.
K. Dataset Statistics
key point that may lead to misunderstanding is that our evaluation cases operate the merge-request level, rather than focusing on isolated code snippets. Over the past three years (starting from 2021), we have collected all recorded faulty merge requests from four repositories maintained by over 4,000 developers. Fault selection follows practical criterion: each fault must have caused user-visible issue and been formally logged in the company's 's internal defect tracking system (Section 3.6). This result-oriented strategy emphasizes real impact, even if it does not fully cover all C++ error types. Each merge request often involves multiple, interdependent code changes, making the evaluation scenario more complex and realistic than snippet-level analyses.
Our focus is on C++ code, as it represents a critical portion of the company's 's software infrastructure. The dataset consists of 45 real-world fault reports, each corresponding to significant issue that caused financial losses, along with the associated merge request snapshots. Among these 45 cases, 12 are logic errors, 31 are code security errors, and 2 are performance- related errors. We have released a desensitized ISON folder of fault descriptions in our Zenodo repository. The dataset includes both edge and typical cases, e.g.:
Case 4694_23117: array out-of-bounds and null-pointer dereference.
Case 16231_13308: misuse of boost: :tw.ddddiieettaaiiiiiiiitte.
On average, each merge request includes 8.02 changed C++ files, with 416.8 newly added lines spread across 14.84 modified functions (a total of 1,403.36 lines affected). Unlike previous datasets that focus on individual code snippets, the faults in our dataset span multiple, interconnected code changes at the merge-request level.
The CodeReviewer dataset (Li et al., 2022b) is well-known benchmark for code review comment generation. However, we do not adopt it because (1) it is a snippet-level dataset lacking repository-level context, and (2) it does not focus on defect detection and lacks structured fault reports for comprehensive understanding (Iuu et al., 2025). These limitations reduce its applicability to real-worid, defect-focused code review automation.
To provide context, we compare our dataset's scale with that of CodeReviewer. Table 9 juxtaposes the size and complexity of the datasets, demonstrating that ours of comparable magnitude, but situated at a more realistic granularity (merge-request level) that better reflects practical development workflows.
22

---

Towards Practical Defect-Focused Automated Code Review
Table Dataset statisties. The statistics of the CoveReviewer dataset are estimated hased the language distribution (Li al., 2022h) and the distribution of comment types (Bacchelli & Bird, 2013).
Dataset
# of Code Snippets
LOC (Lines Code)
CodeReviewer Test (C++ Subset)
814
~44k
CodeReviewer Test (C++ & Defects Subset) Ours -Merge-Requess Level)
1 14
~ lk
668
633
L. Excluded Models and Justifications
Our framework is designed to be model-agnostic and independent of any specific base model. However, for the purposes of our study, we only present results on selected representative open-source models. Below, we outline the reasons for not including certain other models in our experiments:
Closed-Source Models:
This category includes proprietary models such the GPT series and the Claude series. These models remain inaccessible for self deployment due to their closed nature, and utilizing fault reports for evaluation poses potential security and data transmission risks.
Small and Weak Models: We also experimented with several smaller or less powerful models as the base model for our framework, including Gemma-2-27B, CodeGemma-7BB MiniCPM3-44B GLM-4-9B-Chat, CodeGeeX4-All-pp Da--Seek-V--Lit--CC
DeepSeek-Coded Coder- V2-Lite, CodeLlama- 70B, Phi-3-Medium- ---duMddmu---8K, Phi-3.5-MoE, and Aya-23- 35B. Unfortunately, these models frequently failed due to their limited capabilities, making it unfair to include them in comparisons under such conditions.
Excessively Large Models: Our experiments were conducted on an infrastructure consisting of eight A A100-40G GPUs. However, since FP8 quantization
supported on Ampere-architecture A100s, we were unable deploy (MoE) models with very high parameter count. This limitation prevented from testing models such as DeepSeek-V2-Chaa---0 V2-Chat-0628,, DeepSeek-Coder -ooeeee--Coor-V- V2, and Mixtral-8x222-Innttee
M. Metric Formulations
We intentionally avoid BLEU and ROUGE due to their limitations in evaluating the quality of code review comments:
1. Our task involves many-to-many mappings between code and reviews, violating BLEU's single-reference assumption.
2. Code review requires reasoning and domain expertise; recent studies show that BLEU and ROUGE fail to reflect quality in such tasks.
3. Real fault reports and -enerrtee comments differ significantly in style and expression, making surface-level textual similarity unreliable.
Regarding vagueness: rather than evaluating linguistic style, we focus nutcome-based metrics that directly reflect the effectiveness of review comments. Specifically, propose three core metrics- Key Bug Inclusion (KBI), False Alarm Rate (FAR), and the composite Comprehensive Performance Index (P1)~which are objective, interpretable, and domain-relevant. Further discussion on their behavior and limitations provided in Appendix Q.
M.1. Key Bug Inclusion (KBI)
By "key bugs," we refer to issues that can lead to tangible losses (e.g., performance degradation or potential future failures), even if the negative impact is not immediate. The P1-P4 incidents mentioned in our dataset (Section 4.2) all qualify as key bugs. This framing ensures our focus remains on high-impact defects rather than trivial concerns. KBI measures the model's ability to recall key issues that cause system faults. is calculated as the percentage of recalled key issues out of the total issue set:
Number of recalled key issues
KBI
100
(1)
Total number of key issues
23

---

Towards Practical Defect-Focusee Automated Code Review
M.2. False Alarm Rate (FAR)
FAR evaluates the extent to which the model generates irrelevant erroneous comments (false alarms). We consider all comments unrelated to key issues mentioned in the fault reports false alarms. FAR is calculated as the percentage of false alarm comments relative to total comments. Two types of FAR are defined:
1. F Calculates the False Alarm Rate for each individual MR and then averages these rates across all MRs to provide an overall measure of the model's ability to avoid false alarms.
N FAR =
1 N
Number of false alarm comments in MR, Total number of comments in MR, x 100
(2)
2. F FARy; Focuses on MRs where key bugs were successfully recalled, offering insight into precision when key issues are identified.
Number of false alarm comments in recalled MR, 100) M Σ
FAR2
Total number of comments in recalled MR,
Where N the total number of MRs, and M the number of MRs where key bugs were recalled.
M.3. Comprehensive Performance Index (CPT)
To balance KBI and FAR, we propose the Comprehensive Performance Index (CPI), which harmonizes KBI and FAR in a similar manner the F1-Score. CPI evaluates the completeness of key issue detection and the precision of the model's comments. Two versions of CPI:
1. CPI:: Based on F FAR,, considering all MRs.
KBI (100- FAR, KBI (100 FAR,)
CPI = 2
(4)
2. CPI: Based on FAR2, focusing MRs where key bugs were recalled.
KBI (100 FAR2) KBI (100 FAR₂)
CPL. = 2
(5)
M.4. Line Localization Success Rate (LSR)
LSR evaluates the model's ability to successfully associate comments with the valid code lines. A success is recorded if the correct line number provided and valid in the code. LSR is calculated as the percentage of correct line number cases:
N
Number of correct line number cases in MR;
LSR
Total number of comments in MR; x 100
(6)
N
N. Experimental Setups in Detail
The code slicing component of our framework is implemented using Cppcheck(Marjamäkki 2024), while the LLM engines are integrated through an API supported by the vLLM framework (Kwon al., 2023), and baselines are integrated via Flask(Organization,, 2024). All models and baselines eight Nvidia A100-40G GPUs. For large models such as LLaMA3.1 1-405B, we utilize an Int4 version quantized using AWQ (Lin et al., 2024). For other models, we use the original half- prefisrec floating-point format (FP16). For context on overall throughput for large-scale evaluations, processing our entire dataset, which includes many large merge requests (often over 20 modified files each), with models having 120B activated parameters took approximately 9 hours on our server setup.
hosted server equipped with an AMD EPYC 7702 CPU and
24

---

Towards Practical Defect-Focused Automated Code Review
100min
90min
80min
70min
Runtime (min)
60min
50min
40min
30min
20min
10min
4.8min
7.7min
6.2min
6.3min
Original Diff
Parent Function Slicing Algorithms
Left Flow
Full Flow
Figure Runtime per merge request under different slicing algorithms LLaMA3.1-405B base models. The most time- consumnngg algorithm is Function, due to its inclusion of the largest extra context. However, all runtimes are within an acceptable range based on our analysis.
25

---

Towards Practical Defect-Focused Automated Code Review
Regarding typical per-merge request runtime, report detailed timings using violin plots. Figure 5 illustrates these distributions.
The median runtime per MR is 6.2 minutes. The overall CI/CD pipeline (including compilation, analysis, and deployment checks) typically takes 15-30 minutes. Our module rums in parallel from the beginning and does not introduce blocking delays. Thus, we believe the overhead acceptable in practical scenarios.
O. Performance of Heterogeneous Model Combinations
While primary experiments (Section 5.1) employ homogeneous Large Language Model (LLM) setup across all roles to isolate the impact of single powerful model, exploring heterogeneous model combinations offers the potential for optimizing both performance and resource utilization. This section details supplementary experiments where LLM assignments for reviewer and validator roles were varied. The results, presented in Table 10, demonstrate that strategically pairing a strong validator model with computationally less intensive (i.e., smaller) reviewer model can yield performance comparable or, in certain metrics, even superior to configurations relying solely on a powerful model for both roles.
Table 10. Performance under different reviewer-validator model combinations and slicing algorithms. The results suggest that the validator plays critical role, is closer to the final decision output. Interestingly, combination of weaker reviewer and stronger validator can achieve comparable or even superior performance, indicating potential room for improvement through heterogeneous model pairing.
Slicing Algorithm
KBIt
FAR,
CPI1
FAR21
CPL,1
LLaMA3.1-4033 as reviewer, 1.aMMA.117000 as validator
Original Diff
2.22
42.22
4.28
0.00
4.35
Parent Function
6.67
20.00
12.31
0.00
12.50
Left Flow
2.22
39.26
4.29
66.67
4.17
Full Flow
0.00
35.56
LLaMA3.1-70B as reviewer, LLaMA3. as validator
Left Flow
17.78
55.74
25.37
51.04
26.08
Full Flow
13.33
66.11
19.14
45,83
21.40
L 1-05BB as both reviewer and validator
Original Diff
11.11
90.11
10.46
71.00
16.07
Parent Function
11.11
89.48
10.81
65.33
16.83
Left Flow
20.00
75.37
22.07
43.52
29.54
Full Flow
20.00
77.96
20.97
67.59
24.73
P. Performance by Error Category
To better understand the effectiveness of different slicing strategies, we analyze their performance across three fault categories:
logic, security, and performance-related bugs. As shown in Table 11, different slicing strategies exhibit distinct strengths across fault categories. Specifically, flow-based slicing (i.e., Left Flow and Full Flow) is particularly effective for identifying security issues. This is likely because these methods capture detailed jump. data, and control flow information, including the lifecycle of variables, which are often crucial for uncovering security vulnerabilities. In contrast, logic bugs appear to benefit more from the broader and more continuous context provided by parent function slicing, which can aid LLMs in understanding the overarching code logic and intent. Performance issues remain the most difficult to detect, as they tend to be subtle and delayed in manifestation.
Q. Discussion on High FAR
The initially reported False Alarm Rate (FAR) in Table appears relatively high. This is partly due to our strict definition: if the framework fails to detect the key bug in a merge request but still generates comments, we consider the FAR for that request to be 100%. To offer a more nuanced perspective, we introduce FARg, a metric that evaluates only cases where the key bug successfully recalled. Under the "LLaMA3.1 405B + Left Flow with Validator" configuration (Table 2), we
26

---

Towards Practical Defect-Focusee Automated Code Review
Table 11. Performance by category. Flow-based slicing excels detecting security-related bugs, while function-level context
enhances logic bug identification. Performance bugs remain hard to detect due to their implicit and non-local nature.
Category
KBIt
F
CPL1
FAR24
CPL2
Left Flow
Overall
20.00
75.37
22.07
43.52
29.54
Security
25.81
73.92
25.94
48.96
34.28
Logic
8.33
75.00
12.50
0.00
15.38
Performance
0.00
100.00
0.00
Full Flow
Overall
20.00
77.96
20.97
67 .7.
24.73
Security
25.81
72.31
26.71
67.71
28.69
Logic
8.33
97.22
4.17
66.67
13.33
Performance
0.00
50.00
0.00
Original Diff'
Overall
11.11
90.11
10.46
71.00
16.07
Security
16.13
88.87
13.17
71.00
20.73
Logic
0.00
91.67
0.00
Performance
0.00
100.00
0.00
Parent Function
Overall
11.11
89.48
10.81
65.33
16.83
Security
3.23
91.94
4.61
50.00
6.06
Logic
33.33
81.39
23.89
69.17
32.03
Performance
0.00
100.00
0.00
reduce FAR2 to below 50%.
In practical scenarios, verifying the correctness of comments with team of five experienced C++ developers- simulltinn a real-world inspection process- took approximately six minutes per new case. This time can be further reduced if the reviewers are already familiar with the codebase or if significant portion of the comments pertain minor suggestions (e.g., unnecessary try catch statements).
Although FAR is stringent m classifying all non-key-bug comments as false alarms it provides objective and quantifiable baseline. In contrast, prior work often relies subjective assessments such as "usefulness" or "goodness,"" which can vary significantly among reviewers. We view FAR foundational measure, with future research potentially refining it to better align with industry-specific tolerance thresholds.
R. Min/Max Performance Ranges of Slicing Algorithms
To provide more comprehensive understanding of the performance of our code slicing algorithms, beyond the average metrics presented in Section 5.2 (RQ2, Table 3), this appendix details the minimum and maximum values observed for key metrics across three experimental runs. This analysis addresses the need to understand result variability and offers deeper insights into the consistency and raw potential of each slicing strategy, particularly before extensive filtering is applied.
Table 12 presents these min-max performance ranges for the four slicing algorithms under various stages of the single- reviewer filtering pipeline. As emphasized in our response to reviewer feedback, the "Single Reviewer All" setting particularly insightful, as reflects the unfiltered, raw output from the Large Language Model when provided with context from different slicing methods. This setting best illustrates the inherent potential of each slicing algorithm to surface relevant information.
Examining the "Single Reviewer - All" setting in Table 12, we can observe the initial capabilities of each slicing algorithm. For instance, Flsw methods (Left Flow and Full Flow) demonstrate a high Key Bug Inclusion (KBI) potential, with Full Flow reaching up to 40.00% and Left Flow also achieving a maximum 40.00%. Parent Function also shows substantial KBI, peaking at 35.56%, while Original Diff ranges from 17.78% to 28. 8.% In terms of consistency this
27

---

Towards Practical Defect-Focused Automated Code Review
raw KBI output, Full Flow exhibits a relatively tight range (37.88%-40.00% spread of approx. 2.22%), suggesting more stable high performance across runs compared to Left Flow (spread of approx. 6.67%) Original Diff (spread of approx. 11.11%). As expected, the False Alarm Rate (FAR) for all methods in the "All" setting is uniformly high, typically in the 90s range, given the absence of any filtering.
As comments pass through subsequent filtering stages (Coarse Filter, Top-10, Top-5, Top-3), the min-max ranges for metrics like KBI and FAR evolve. For example, while the maximum KBI potential might be reduced by aggressive Top-k truncation (e.g., for Left Flow, KBI max drops from 40.00% at "All" or "Coarse Filter" to 22.22% at "Top-3"), the filtering stages generally aim to reduce FAR while preserving KBI, leading to improvements in Comprehensive Performance Index (CPI) metrics. The ranges also provide insights into the stability of these improvements. For instance, under "Single Reviewer- Top-5," Left Flow shows a KBI range of (20.00%. 31. 1%) and Full Flow (22.22%. 31,11%), indicating that even after significant filtering, these methods can still achieve high bug recall in some runs.
This min-max analysis complements the average performance data presented in RQ2. It highlights that while average performance provides general comparative measure, the variability across runs can differ between algorithms and filter settings. Understanding these ranges is valuable for assessing the robustness of each approach and identifying methods that not only perform well on average but also maintain consistency or offer high peak performance.
Table 12. Min-mat max ranges of key metries under different code slicing strategies for single reviewer settings.
Slicing Algorithm
KBI
FAR:
CPI,
FARg 1
CPI, f
Single Reviewer- - All
Original Diff
(17.78, 28.89)
(95.43, 98.07)
(3.48, 7.89)
(84.18, 89.16)
(13.47, 20.44)
Parent Function
(26.67, 35.56)
(96.73, 97.35)
(4.82, 5.96)
(90.06, 91.14)
(14.18, 15.16)
Left Flow
(33.33, 40.00)
(94.04, 94.73)
(9.31, 10.11)
(88.80, 92.39)
(12.79, 16.77)
Full Flow
(37.78, 40.00)
(93.81, 95.24)
(8.45, 10.72)
(90.07, 93.29)
(11.39, 15.91)
Single Reviewer- Coarse Filter
Original Diff
(15.56, 20.00)
(84.89, 92.40)
(10.21, 17.21)
(76.23, 80.01)
(17.60, 20.00)
Parent Function
(15.56, 26.67)
(87.36, 88.91)
(13.94, 16.33)
(75.92, 91.01)
(13.15, 22.27)
Left Flow
(24.44, 40.00)
(91.75, 92.20)
(11.83, 13.68)
(77.17, 90.38)
(15.15, 23.61)
Full Flow
31.11,333333
(91.73, 93.78)
(10.48, 13.07)
(87.69, 90.70)
(14.54, 17.64)
Single Reviewer- 10p-10
Original Diff
(13.33, 20.00)
(84.83, 91.81)
(10.26, 17.26)
(70.83, 79.69)
(18.30, 20.15)
Parent Function
(13.33,
(87.44, 89.61)
(11.68, 15.13)
(72.08, 88.39)
(15.25, 18.76)
Left Flow
(11111,33.33)
(90.58, 91.55)
(13.29, 14.68)
(78.42, 87.12)
(18.21, 26.20)
Full Flow
(28.89, 35.56)
(90.05, 92.68)
(11.85, 15.55)
(83.62, 88.77)
(16.17, 21.57)
Single Reviewer Top-5
Original Diff
(13.33, 17.78)
(84.44, 91.63)
(10.28, 16.59)
(67.78, 75.00)
(18.35, 20.78)
Parent Function
(11.11, 17.78)
(86.22, 89.22)
(10.94, 14.64)
(63.00, 80.00)
(17.09, 20.81)
Left Flow
(20.00, 31.11)
(89.19, 91.33)
(12.09, 16.05)
(67.78, 78.89)
(23.57, 29.26)
Full Flow
(22.22, 31.11)
(90.04, 90.67)
(13.51, 14.45)
(75.17, 80.00)
(22.00, 26.51)
Single Reviewer- Top-3
Original Diff
(8.89, 13.33)
(84.07, 92.22)
(8.30, 14.51)
(60.00, 63.89)
(14.37, 19.48)
Parent Function
(6.67, 8.89)
(86.67, 88.89)
(8.33, 10.67)
(50.00, 66.67)
(11.11, 15.09)
Left Flow
(13.33, 22.22)
(88.89, 91.48)
(10.40, 14.81)
(52.78, 66.67)
(20.80, 28.57)
Full Flow
(11.11, 15.56)
(90.74, 92.22)
(9.40, 10.93)
(63.89, 66.67)
(16.67, 21.67)
S. Sensitivity Analysis of Top-k Truncation in Multi-Reviewer Settings
This section details experiments evaluating the impact of different Top-k truncation values (k) on performance within our multi-reviewer framework, where all settings utilize three reviewers. The results, presented in Table 13, indicate that the optimal choice of i often contingent on the employed slicing strategy and the specific evaluation metrics prioritized.
28

---

Towards Practical Defect-Focused Automated Code Review
For slicing strategies that typically generate sparser of initial comments, such Original Diff and Parent Function:
With Original Diff, A 3 (Top-3) generally yields favorable CPI, results (9.30 before validation, 11.81 after) and achieves the highest KBI (15.56) before validation. After validation, while Top-5 or Top-10 showed slightly higher KBI (11.11 vs. 8.89 for Top-3), Top-3 maintained the best CPI1
- For Parent Function, the trends are more varied. Before validation, Top-5 led to the highest KBI (20.00), while Top-10 had the best CP1, (11.04). After validation, 3 achieved the best CPI, (13.62) with KBI comparable other & values.
For richer slicing strategies such as Left Flow and Full Flow, which capture more extensive context and often produce more candidate comments:
With Left Flow, larger k values (Top-10 10 or Top-5) consistently outperformed Top-3 in KBI both before and after validation. T generally provided the best CPI, (17.51 before validation, 22.07 after).
With Full Flow, Top- 10 initially showed the highest KBI (35.56) before validation. However, notable behavior was observed in this specific case with Top- 10 truncation, particularly after the Validator stage: there a significant drop in KBI (from 35.56 13.33) and CPI, (from 15.92 12.01). In contrast, Top-5 for Full Flow maintained a higher KBI (20.00) and CP1, (20.97) post-validation. We attribute this decline for Top-10 to the large volume of text processed when 10 for verbose slicing method like Full Flow. This may approach the context token limits of the LLaMA3.1 engine during the validation phase, potentially reducing the validator's 's effectiveness.
Overall, these findings demonstrate that while the optimal i can be tuned, the Top-k filter's behavior is generally stable across reasonable threshold ranges. Furthermore, it is adaptable to the context richness provided by different slicing algorithms, allowing for optimized configurations based on the desired balance between metrics like KBI and FAR.
T. Extended Conclusions
T.1. RQ1 Extended Conclusion
Conclusion
(Original)
Our framework surpasses baseline approaches significantly, achieving up to 10x better performance on key metrics like KBI and CPI. This success is attributed to our comprehensive approach to code review automation, which addresses the full pipeline and its associated challenges. Additionally, LLaMA3. emerged as the best- performing LLM engine, reinforcing the importance of model size and capability achieving optimal results. Further investigations into LLM configurations show that heterogeneous setups, such pairing strong validator with smaller reviewer, can yield comparable even improved performance (details in Appendix O).
T.2. RQ2 Extended Conclusion
Conclusion 2 (Original)
The results show that Left Flow and Full Flow significantly improve key bug inclusion (K BI) and overall perfor- mance (CPII) compared to simpler approaches like Original Diff and Parent Function. Among them, Left Flow performs better in more settings, likely due to its more concise context, which helps the large language model maintain focus without being overwhelmed. Each code slicing algorithm, however, has exclusive cases where performs well, suggesting that combining different strategies could further enhance key bug detection in future work.
29

---

Towards Practical Defect-Focused Automated Code Review
Table 13. The impact of Top-k truncation on comment quality under different slicing algorithms. All settings use three reviewers.
Top-k
KBIt FAR,
CPL1
Original Diff (Multi Reviewer + Meta Reviewer)
Top-10
13.33
94.90
7.37
Top-5
13.33
96.74
5.24
Top-3
15.56
93.37
9.30
Parent Function (Multi Reviewer + Meta Reviewer)
Top-10
15.56
91.44
11.04
Top-5
20.00
92.41
11.01
Top-3 11.11
94.19
7.63
Left Flow (Multi Reviewer + Meta Reviewer)
Top-10 31.11
88.93
16.33
Top-5
31.11
87.81
17.51
Top-3 17.78
91.96
11.07
Full Flow (Multi Reviewer Meta Reviewer)
Top-10 35.56
89.74
15.92
Top-5
31.11
89.41
15.80
Top-3
24.44
90.04
14.16
Original Diff (Multi Reviewer Meta Reviewer
Validator)
Top-10
11.11
89.07
11.02
Top-5
11.11
90.11
10.46
Top-3
8.89
82.41
11.81
Parent Function (Multi Reviewer + Meta Reviewer + Validator)
Top-10
11.11
85.11
12.73
Top-5
11.11
89.48
10.81
Top-3
11.11
82.41
13.62
Left Flow (Multi Reviewer Meta Reviewer
Validator)
Top-10 22.22
82.04
19.87
Top-5 20.00
75.37
22.07
Top-3 8.89
83.70
11.50
Full Flow (Multi Reviewer + Meta Reviewer + Validator)
Top-10 13.33
89.07
12.01
Top-5 20.00
77.96
20.97
Top-3
11.11
73.70
15.62
30

---

Towards Practical Defect-Focused Automated Code Review
T.3. RQ3 Extended Conclusion
Conclusion 3.1 (Original)
Increasing the number of reviewers improves key bug inclusion (K BT) but also increases false alarms (F and F' F.AR2). Validators are essential for maintaining comprehensive performance (CPI, and CP12) by reducing false alarms. While leveraging multiple reviewers is beneficial, the added computational cost and need for validation must be considered practical implementations.
Conclusion 3.2 (Original)
The self-correction ability of LLMs, implemented by the validator role, improves precision by reducing false alarms (F (FAR) and F FAR2) but may reduce key bug inclusion (IB)), This indicates a trade-off between precision and recall. The validation process valuable for reducing hallucinations, but care must be taken to ensure that important bug-detecting comments are not removed.
Conclusion 3.3 (Original)
The effectiveness of Chain-of- Thought (CoT) guidance varies based on the complexity of the code slicing algorithm. While LLMs perform better without CoT in simpler formats like Original Diff and Parent Function, CoT significantly improves results in more complex flow-hased slicing (Left Flow and Full Flow). This suggests that CoT guidance especially valuable when handling more intricate contexts. However, as more powerful reasoning models, such as GPT-O1 and DeepSeek-Ro emerge, the advantage of specified CoT over free-form reasoning may further diminish.
Chain-of-Thought
T.4. RQ4 Extended Conclusion
Conclusion 4 (Original)
The comment filter mechanism effectively reduces false alarms (F (FAR)) and improves overall performance (CPI) in flow-based slicing methods (Left Flow and Full Flow). For simpler slicing methods (Original Diff and Parent Function), the coarse filter is the most effective stage, as these methods lack sufficient code details to accurately filter nitpicks and hallucinations.
T.5. RQ5 Extended Conclusion
Conclusion 5 (Original)
Adding line number information improves both performance and localization success rate (LSR). The inline format outperforms the relative format, likely because embedding position data directly into the code allows for better association of comments with specific lines.
U. Status of Our Open-Source Code Artifacts
We have open-sourced all the core components of our framework and the experiments reported in the paper, except the certain non-essene menesssntial,, company-specife modules are not included. To achieve this, we carefully modularized the framework and separated any company-inteman interfaces, ensuring that external users can readily experiment, customize, and extend the system.
As for the omitted modules, these are excluded for three main reasons:
They are not generally useful beyond our specific internal environment, offering little to no benefit for external adaptation.
They depend on proprietary interfaces and data sources that are inherently inaccessible external researchers.
31

---

Towards Practical Defect-Focused Automated Code Review
Including them risks violating the double anonymity requirement, not due mere vocabulary text replacements. but because their logic and usage patterns could reveal organizational details. For code and comments that could be anonymized, we have already performed the necessary replacements.
V. Threats to Validity
recognized threat to the external validity our study that the presented evaluation and results are exclusively for C++ projects. This current language focus in our implementation is primarily due to the choice of Cppcheck for code slicing. tool specific C++. We prioritized C++ due to its significant presence in the core framework code of many companies. However, the underlying framework and its prompting strategies for L.MMs are are designed to be largely language-agnostin These components do not inherently rely on C++-specific features. Consequently, extending the framework to support other compiled languages is considered feasible, mainly requiring the integration of suitable language-specinf AST parsing tools or static analyzers.
principes--includingg the AST-based code slicing methodology and the
Another potential threat arises from how we calculate the False Alarm Rate (FAR). In our study, we classify all comments directly related the key bug as false alarms. However, some of these comments may still identify relevant issues, such as potential risks or code quality concerns, that do not immediately lead system failures but warrant attention. As result, the actual FAR may be lower than reported figures. Despite this, chose this conservative approach emphasize critical issues and minimize the burden on developers, making our assumption practical in the context of prioritizing key bugs.
32


Reviews:
[{"review_id": "RlNCNhAi3I", "reviewer": "Reviewer_jwAu", "summary": "In this paper, the authors proposed a framework for automated code review. More specifically, the authors first used code slicing to enable the Multi-Agent Code Review System to obtain sufficient context fragments of the code. Then, the Multi-Agent Code Review System conducted code reviews, filters, aggregates, and ranks the reviews. Finally, the proposed approach localized the issues mentioned in the reviews. The authors created a dataset at the merge request level using data from four repositories. The authors experimentally validated the performance  using Key Bug Inclusion, False Alarm Rate, Comprehensive Performance Index, and Line Localization Success Rate. The proposed approach performed better on both the Left Flow and Full Flow slicing algorithms, showing effective localization capability in the Inline representation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for the constructive comments and the recognition of our framework’s practical impact. We have addressed the concerns by enhancing dataset transparency, adding new evaluations (including error category analysis and heterogeneous LLM roles), clarifying metric rationale, and updating implementation files.**\n\n---\n\n### **1. Dataset Scope and Fault Selection**  \n\n**(Response to “Questions for Authors” and “Methods and Evaluation Criteria”)**\n\n> Can the authors provide more information about the dataset?...  \n> It is unclear how the key issues are defined... Is it feasible to report the performance across different error categories?\n\nWe appreciate your interest in the dataset. To improve transparency, we added a desensitized JSON folder of fault descriptions to our updated Zenodo repository and will revise **Appendix J** accordingly.\n\n**Fault selection** follows a practical criterion: all faults caused user-visible issues and were formally logged in the company’s internal defect tracking system (Section 3.6). This **result-oriented** strategy emphasizes **real impact**, even if it doesn’t fully cover all C++ error types.\n\nThe dataset includes both edge and typical cases, e.g.:\n\n- **Case 4694_23117**: array out-of-bounds and null-pointer dereference.  \n- **Case 16231_13308**: misuse of `boost::random::beta_distribution`.\n\nTo analyze per-category performance, we added a breakdown across **logic**, **security**, and **performance-related** bugs:\n\n🔗 https://anonymous.4open.science/r/16368/Performance%20by%20error%20category.pdf\n\nFindings show that flow-based slicing benefits security bugs, while broader context helps with logic bugs.\n\n---\n\n### **2. Evaluation Metrics and Comment Quality**  \n\n**(Response to “Claims and Evidence” and “Weakness 4”)**\n\n> The authors pointed out that current evaluations rely excessively on BLEU, ROUGE...  \n> The authors evaluate the comments based on certain issues... assess whether the answers are vague.\n\nWe intentionally did **not** use BLEU or ROUGE due to several limitations:\n\n1. Our task involves **many-to-many** mappings between code and reviews, violating BLEU’s single-reference assumption.  \n2. Code review requires reasoning and domain expertise, and recent studies from earlier this year show that BLEU and ROUGE fail to capture quality effectively in such tasks.\n3. Real fault reports and LLM outputs differ significantly in style and expression, making textual similarity unreliable.\n\nOn vagueness: rather than evaluating writing style, we focus on key outcome metrics like *key bug inclusion* (KBI) and *false alarm rate* (FAR), which directly reflect review effectiveness. These metrics are more objective and interpretable and will be further discussed in **Appendix N**.\n\n---\n\n### **3. LLM Combinations and Time Cost**  \n\n**(Response to “Experimental Designs or Analyses” and “Weakness 1, 2”)**\n\n> The proposed approach does not validate different LLM combinations...  \n> The authors do not discuss whether the time and resource costs are acceptable...\n\nOur main experiments use the same LLM across all agents to isolate whether a **strong model** alone can resolve key challenges in code review. However, we agree that heterogeneous combinations are worth exploring.\n\nWe added experiments varying reviewer/validator assignments, showing that a strong validator paired with a smaller reviewer often yields comparable or better results:\n\n🔗 https://anonymous.4open.science/r/16368/Performance%20of%20combinations.pdf\n\nRegarding runtime, we now report detailed timings using violin plots:\n\n🔗 https://anonymous.4open.science/r/16368/Runtime%20violin%20plot.pdf\n\n- **Median runtime per MR** is **6.2 minutes**.  \n- The overall CI/CD pipeline (including compilation, analysis, and deployment checks) typically takes 15–30 minutes.  \n- Our module runs **in parallel** from the beginning and does **not introduce blocking delays**.\n\nThus, we believe the overhead is acceptable in practical scenarios. Further analysis will be added to **Appendix M**.\n\n---\n\n### **4. Prompting Strategy and Retrieval-Augmented Generation**  \n\n**(Response to “Weakness 3”)**\n\n> In the LLM prompts, the authors can discuss the effectiveness of Retrieval-Augmented Generation...\n\nThank you for raising this point. Although we do not use explicit RAG pipelines, our **slicing mechanism** serves a similar purpose: retrieving and providing only **relevant context slices** to the model. This is evaluated in **Section 5.2 (RQ2)** and will be clarified as RAG-aligned in the revision.\n\n---\n\n### **5. Minor Corrections and Reference Additions**\n\n- The incorrect reference to Section 2.1 in **Appendix A** will be fixed.  \n- We will include the three suggested references and briefly discuss their relation to our work.\n\n---\n\n**We hope this response addresses your concerns and supports a more favorable assessment. Your feedback has helped us strengthen the clarity and rigor of our work.**", "claims_and_evidence": "The authors pointed out that current evaluations rely excessively on textual similarity metrics (e.g., BLEU, ROUGE), which fails to measure real-world effectiveness. Is there a connection between metrics like BLEU and the evaluation metrics used by the authors? The authors could compare the proposed approach and baselines with BLEU and ROUGE.", "methods_and_evaluation_criteria": "The authors have established criteria based on key issues; however, it is unclear how the key issues are defined. Is there a specific set of evaluation criteria, and are these standards reasonable? Regarding the Dataset, a C++ dataset was conducted by using requests. The authors categorized errors into logic errors, code security errors, and performance-related errors. Does the dataset comprehensively cover the majority of error types encountered in C++? Is it feasible to report the performance of the proposed method across these different error categories?", "theoretical_claims": "The paper does not contain proofs for theoretical claims.", "experimental_designs_or_analyses": "The authors evaluated the performance of different LLMs; however, further analysis was needed to examine the performance differences resulting from various combinations of LLMs. Additionally, did different tasks yield performance variations when different LLMs were employed? For instance, using LLaMA 3.1 (405B) as the Reviewer and LLaMA 3.1 (70B) as the Validator. For Time Cost in Appendix M, Experimental Setups in Detail, the authors mentioned that one round of automated code review took approximately 9 hours. In the context of code review, were such time and resource overheads acceptable?", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "There are some more recent related work. For example:\n\n[1] Wang L, Zhou Y, Zhuang H, et al. Unity Is Strength: Collaborative LLM-Based Agents for Code Reviewer Recommendation[C]. IEEE/ACM International Conference on Automated Software Engineering. 2024: 2235-2239.\n\n[2] Wei Tao, Yucheng Zhou, et al., 2024. KADEL: Knowledge-Aware Denoising Learning for Commit Message Generation. ACM Trans. Softw. Eng. Methodol. 33, 5, Article 133 (June 2024), 32 pages. \n\n[3] Yu Y, Rong G, Shen H, et al. Fine-tuning large language models to improve accuracy and comprehensibility of automated code review[J]. ACM transactions on software engineering and methodology, 2024, 34(1): 1-26.", "other_strengths_and_weaknesses": "Strengths: \n1. The proposed merge-request-based code review could be useful for real-world applications. \n2. The authors introduce code localization based on the review, which could help quickly determine whether the issues are genuine.\n\nWeakness: \n1. The proposed approach does not validate the combination of different LLMs. \n2. The authors do not discuss whether the time and resource costs of the multi-agent code review system are acceptable.\n3. In the LLM prompts, the authors can discuss the effectiveness of Retrieval-Augmented Generation in LLM-generated outputs. \n4. The authors evaluate the comments based on certain issues. However, the evaluation should be improved. For instance, the authors should assess whether the answers are vague.", "other_comments_or_suggestions": "Section 2.1 is mentioned in Appendix A, but where is it in the paper?", "questions_for_authors": "Can the authors provide more information about the dataset? Does the dataset comprehensively cover the majority of error types? How were the 45 fault reports selected? Are they representative of common code review scenarios (e.g., edge cases vs. typical defects)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vghbM4lwB", "reviewer": "Reviewer_6FuX", "summary": "This paper proposes an advanced method for automating code reviews, focused on defect detection and improving real-world code review workflows. To address the challenges, the authors introduce a multi-agent LLM framework that utilizes code slicing algorithms, a filtering mechanism to remove irrelevant comments, and a new prompt design for better integration into human workflows. Their system was validated using real-world industry data, achieving significant improvements over previous methods in detecting key bugs, reducing false alarms, and improving code review performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for the constructive and insightful feedback. We appreciate your recognition of our system’s architecture and the observed performance improvements on real-world data. Below, we address your concerns regarding generality, LLM dependency, filter sensitivity, and workflow integration.**\n\n---\n\n### **1. Generality Beyond C++**\n\n**(Response to “Weakness 1”)**\n\n> The framework is mainly tailored for C++, and its adaptability to other languages may need further research and development.\n\nWe agree clarification is needed. The current results are based on C++ and the 10× improvement reflects this. We will state this explicitly in the revised abstract. As noted in **Appendix Q**, we chose C++ because:\n\n- It is one of the most widely used industrial languages;\n- It is among the most complex in mainstream use.\n\nNonetheless, our framework is **inherently language-agnostic**, relying on universal structures like ASTs to extract elements such as control/data flow and function scopes. These structures exist across most compiled languages, and the framework’s design—including slicing and prompting—does not rely on C++-specific features. Extension to other languages is thus feasible and ongoing work.\n\n---\n\n### **2. LLM Dependency and Computational Cost**\n\n**(Response to “Weakness 2, 3”)**\n\n> The framework depends on large LLMs... Detailed slicing may be computationally expensive...\n\nWe acknowledge that performance depends on model capability. While we use large models (e.g., LLaMA 3.1) in our main experiments, we also evaluated smaller models in **Section 5.1 (RQ1)**. These results show that compact models with strong reasoning abilities can still perform competitively, especially there is a trend towards increasing capacity density.\n\nFor slicing cost: while fine-grained slicing (e.g., Full Flow) is more expensive, it remains within practical limits. We report detailed runtimes via violin plots:\n\n🔗 https://anonymous.4open.science/r/16368/Runtime%20violin%20plot.pdf\n\n- **Median runtime per MR** is **6.2 minutes**.\n- This fits comfortably within typical CI/CD pipelines (15–30 minutes), which include compilation, linting, static analysis, testing, and deployment checks.\n\nOur system runs **in parallel** with these processes and does not introduce blocking delays. We believe this makes the cost acceptable for real-world use. Further analysis is included in **Appendix M**.\n\nWe also emphasize that our focus is on solving core challenges like *defect inclusion* and *false alarm suppression*. Cost-efficiency optimization (e.g., model scaling, caching) is important but future work.\n\n---\n\n### **3. Sensitivity Analysis of Filter Thresholds**\n\n**(Response to “Experimental Designs”)**\n\n> The paper does not provide an in-depth justification for the chosen thresholds of the redundancy comment filter...\n\nThank you for this suggestion. Our current thresholds are based on heuristic rules aimed at **interpretability**. For Q1/Q2, a score above 4 on a 1–7 scale (inspired by prior work) indicates actionable, serious issues. This setting aligns with developer feedback during internal piloting.\n\nFor Q3, we conduct a sensitivity study using top-*k* truncation in **Section 5.4 (RQ4)**. We now extend this to multi-reviewer settings:\n\n🔗 https://anonymous.4open.science/r/16368/The%20impact%20of%20Top-k%20truncation.pdf\n\nKey observations:\n\n- **Smaller top-*k*** (e.g., 3) performs better for shorter slicing strategies (Original Diff, Parent Function), where fewer high-quality comments exist.\n- **Larger top-*k*** (5–10) works better for richer slicing (Left Flow, Full Flow), which yields more relevant outputs.\n- For Full Flow with Top-10, performance declines after validation—likely due to hitting token limits, reducing model focus.\n\nThese findings demonstrate that the filter behavior is stable across reasonable thresholds, and adaptable to context richness.\n\n---\n\n### **4. Real-World Workflow Integration**\n\n**(Response to “Claims And Evidence”)**\n\n> The claim “Real-world Workflow Integration” is limitedly supported.\n\nThank you for pointing this out. The workflow is illustrated in **Figure 1**, but we agree it merits a more detailed explanation. We will provide a full description in an additional appendix.\n\nBriefly, when a merge request is submitted:\n\n1. The system verifies user and file access;\n2. Slicing and multi-agent review are launched;\n3. Filtered comments are injected into the internal DevOps system;\n4. Comments are positioned at exact line numbers and pushed to developers via messaging.\n\nThis enables seamless integration into daily development workflows. In particular, **line-aware comment injection**, evaluated in **Section 5.5 (RQ5)**, was critical for developer adoption and feedback.\n\n---\n\n**We will incorporate these clarifications and additions in the revised version. We hope this addresses your concerns and supports a more favorable assessment. Your feedback has helped improve both clarity and rigor.**", "claims_and_evidence": "1. The evaluation primarily focuses on performance metrics rather than directly on qualitative assessments and user studies that demonstrate improved usability or reduced developer burden. Thus, the claim \"Real-world Workflow Integration\" is limitedly supported.\n2. The title didn't specifically mention C++ focus, but the \"10× improvement\" is primarily based on C++ code. This raises the concern about the generality of the framework, and the scope of the paper", "methods_and_evaluation_criteria": "Assuming the paper focuses on C++ related code review, the methods and evaluation make sense.", "theoretical_claims": "The paper primarily focuses on designing and empirically evaluating its defect-focused automated code review framework.", "experimental_designs_or_analyses": "The paper does not provide an in-depth justification for the chosen thresholds of the redundancy comment filter. A sensitivity analysis regarding these thresholds could strengthen confidence in the filtering mechanism's robustness.", "relation_to_broader_scientific_literature": "The paper extended existing automated code review methods from isolated snippet-level generation to holistic, context-rich analysis. And by proposing a multi-agent, collaborative framework with evaluation metrics that better capture the practical realities of defect detection. These innovations build directly on and address limitations identified in prior work.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n\n1. The use of multiple agents adds flexibility and scalability, ensuring that the review process is thorough and precise.\n\n2. The method achieves up to 10x improvement over previous baselines in detecting critical bugs and reducing irrelevant comments.\n\n\n### Weaknesses\n\n1. The framework is mainly tailored for C++, and its adaptability to other languages may need further research and development.\n\n2. The effectiveness of the framework strongly depends on the underlying LLM engine, with larger models like LLaMA3.1 performing better, which might be a limitation for certain applications where computational resources are constrained.\n\n3. The more detailed slicing methods (like Left Flow and Full Flow) showed better performance but could potentially be computationally expensive or difficult to implement in all environments.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WplsjkdtlB", "reviewer": "Reviewer_ysLJ", "summary": "The paper presents a language-model based system for automated code review. The methodology boils down to using static analysis tools to identify the most relevant parts of the code base, and then passing this through several LLM calls to generate the review, identify the key components, and filter out noise. The authors state that this approach is inspired by interviews with real-world code reviewers. Empirically, the authors find that on a dataset consisting of merge requests from an internal base, their method outperforms prior work. In particular, they evaluate their method in terms of whether the review correctly identifies the key bug introduced in the merge request, as well as how many \"false alarms\" the review raises and whether the faulty line(s) are correctly identified.\n\n## update after rebuttal\nFollowing the end of the rebuttal period I have decided to further raise my score to Accept (4). I like the motivations of the work and think it could have real-world impact, especially as C++ is at this point criminally understudied by academia in comparison to its popularity in the industry. The uncertainty quantification included by the authors in their final reply has convinced me that the findings are meaningful, and I appreciate that they have been responsive to changes requested in both the scope of the work as well as its exposition.\n\nI will note that I am not sure if ICML is the right venue for this work, as opposed to conferences such as ICSE or FSE, but given that none of the other reviewers seemed bothered by this I will chalk it up to being a symptom of AI/ML becoming increasingly relevant to many other areas of computer science and will not hold it against this paper in particular.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank the reviewer for the constructive and thoughtful feedback. We appreciate your recognition of the clarity of the paper, the strength of the experimental design, and the potential real-world impact of our work. Below, we respond to your questions and key suggestions.**\n\n---\n\n### **1. Motivation for Submitting to ICML**  \n\n**(Response to “Questions for Authors”)**\n\nWe submitted this work to ICML because we believe that integrating large language models (LLMs) into software engineering workflows represents a growing and impactful frontier at the intersection of machine learning and software development.\n\nWith the rapid advancement of LLMs, core tasks in software engineering—such as code review automation—are approaching an inflection point. Engaging the ML community is essential to foster deeper cross-domain innovation, especially as challenges like contextual reasoning, modular prompting, and learning from structured data remain open research problems.\n\nMoreover, our defect-focused formulation reflects a broader ML interest in real-world, system-level tasks beyond conventional generation. Practically, we also note the limited availability of top-tier SE venues with winter deadlines.\n\n---\n\n### **2. Novelty of Code Slicing in Code Review Context**  \n\n**(Response to “Questions for Authors” & “Essential References”)**\n\nTo the best of our knowledge, our work is the first to explicitly incorporate **static code slicing** into an **LLM-based automated code review pipeline** to guide review comment generation and filtering.\n\nWhile slicing is a well-established technique in software engineering, prior works on code review automation have largely focused on snippet-level generation or comment naturalness, rather than **context-aware defect localization**. Under such task formulations, slicing has typically been overlooked, as the repository-level context is not utilized.\n\nWe acknowledge this is a strong claim and will revise the relevant statement in **Appendix D** to present it more cautiously.\n\n---\n\n### **3. Use of the Term “Multi-Agent”**  \n\n**(Response to “Other Comments”)**\n\nWe appreciate this observation. Our usage of “multi-agent” was intended to convey modular collaboration among LLM roles (e.g., Reviewer, Validator, Aggregator), not reinforcement learning-style agents interacting with an environment.\n\nTo avoid confusion, we will revise the terminology in the revised version to use terms such as **“multi-role framework”** or **“role-based architecture.”**\n\n---\n\n### **4. Additional Suggestions and Minor Corrections**\n\n- **Figures and tables** will be updated with larger fonts, improved contrast, and clearer layout to ensure readability in both digital and printed formats.\n- **Citation formatting** in Section 6 will be corrected as suggested (e.g., “Gupta & Sundaresan (2018)”).\n\n---\n\n**We thank the reviewer again for the generous feedback and helpful suggestions. We hope these clarifications address your concerns and reinforce the value of our contribution.**", "claims_and_evidence": "Overall, the claims made appear to be well supported by the experiments. Of course it is difficult to judge the validity of evaluating on an internal dataset, but I do not see any immediate causes for concern. The one thing that stands out to me as perhaps not being completely justified by the experiments is whether the method reduces the False Alarm Rate (FAR); the definition of this metric appears to be quite strict, as acknowledged by the authors in Appendix N and Q. Furthermore, it seems to go against the training objectives of prior work like CodeReviewer, so the fact that this new method outperforms it on this metric seems like an obvious result. However, this limitation is acknowledged by the authors (albeit hidden in the appendix), and if prior work failed to take real-world concerns such as limiting the cognitive burden imposed on the user then that is a shortcoming of their approach, not of this paper.", "methods_and_evaluation_criteria": "Yes, the internal dataset used for the evaluation appears to be well suited for the study.  As mentioned in the preceding section, the FAR metric appears to be somewhat arbitrary. In particular, having a code review system point out minor issues and not just major faults seems like a feature, not a failure, to me. However, I appreciate that the authors are attempting to limit the amount of information that is presented to the end-user, since this is likely a make-or-break factor when applying the system in the real world.", "theoretical_claims": "N/A; no theoretical claims.", "experimental_designs_or_analyses": "I have checked the soundness of the experiments presented in the main text, and they appear valid and sound. The code slicing experiment in 5.2 seems a bit weak, however, since the absolute numbers of key bugs is relatively small, so it is hard to tell whether the different spreads in the Venn diagram are significant or just due to chance. Another, more important issue is that the text mentions that each experiment is repeated 3 times to account for the stochasticity of the system, but as far as I can see the results of each experiment is just reported as the mean of these 3 runs, rather than their complete spread. Just reporting the min and the max of each number would already be a significant improvement in terms of making it easier to tell if the results are significant or not.", "relation_to_broader_scientific_literature": "Automated code review is a problem with significant real-world impact, and has been studied in the software engineering community for many years. This paper appears to be a significant step forward in this domain, since it evaluates the performance of their system on real-world fault reports and merge requests, using metrics that (are intended to) mimic the desiderata of real software developers. The methodology is not itself particularly novel or interesting, but the experiments are extensive and may inspire future work in this direction. I am not sure how interesting this paper would be to an ICML audience, rather than a software engineering audience, though.", "essential_references_not_discussed": "None that I am aware of; this problem is well-studied in the software engineering literature, but the authors appear to have cited the most relevant works already (in particular those by Tufano et al.). I was surprised by the authors' claim in Appendix D that slicing (which is a very well established technique across software engineering) had not yet been applied to automated code review, but could not find any references myself for this, so it may be true.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well organized, with clear hypotheses and thorough discussion of the experiments\n- The authors release the source code of their system, which will help others seeking to replicate or build upon their work\n\nWeaknesses:\n- The figures and tables are, on the whole, completely illegible (at least when printed). The text is much too small (in particular in Figures 1,2, 3, 4) and the black-on-dark-green formatting of the tables is difficult to read.\n- While very much understandable, it is unfortunate that the dataset could not be released, as this would certainly have been a very useful resource for others working in this area.", "other_comments_or_suggestions": "Section 6 uses a very strange reference format, which should be simplified. For example, \"Gupta et al. (Gupta & Sundaresan, 2018)\" should just be \"Gupta & Sundaresan (2018)\".\n\nI really disagree with framing your system as being \"multi-agent\". There is no interaction with an environment, so there is in fact no agent involved at all. This is an abuse of the nomenclature.", "questions_for_authors": "1. What was your motivation for submitting this work to ICML instead of ICSE, FSE or ASE, where the majority of prior work in this area was published?\n2. How certain are you that there is no prior work applying slicing in the context of automated code review?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tN8FlU4JQh", "reviewer": "Reviewer_ngPf", "summary": "The paper introduces an end‐to‐end automated code review system designed specifically for defect detection in large-scale, industrial codebases. The authors identify four key challenges in automating code review: capturing the full, relevant code context; improving key bug inclusion to ensure that critical defects are detected; reducing the false alarm rate by filtering out redundant or irrelevant comments; and integrating the system into human workflows. \n\nTo overcome these challenges, the paper proposes an approach that includes:\n• A static analysis system using several code slicing algorithms\n• A multi-agent framework where different LLM roles (Reviewer, Meta-Reviewer, Validator, and Translator) collaborate\n• A robust filtering mechanism that systematically reduces false positives by eliminating nitpicks and hallucinations\n• A line-aware prompt design that ensures review comments are accurately attached to the relevant lines of code\n\nEmpirical evaluations demonstrate that this integrated approach can achieve good performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank the reviewer for the thoughtful and encouraging feedback. We greatly appreciate your recognition of the system design, the practical orientation of our evaluation, and the paper’s contributions toward large-scale industrial defect detection. Below, we address your questions in turn.**\n\n---\n\n### **1. Trade-offs with the Validator Role**  \n\n**(Response to “Question 1”)**\n\nThe Validator role involves important trade-offs:\n\n- It is highly effective at removing hallucinated comments, which was a core motivation for introducing it.\n- However, we observed some negatives cases, where valid comments were mistakenly rejected. These cases typically stem from:\n  - **Context loss** during the Meta-Reviewer phase (e.g., mismatched file names or snippet IDs), which prevented the Validator from tracing back to the correct source code.\n  - **Position mismatches**, where a comment was attached to a nearby but unrelated line.\n  - **Token limit issues**, where long inputs (context + chain-of-thought + comment pairs) exceeded the model’s context window.\n  - **Score variance**, where randomly low Q1–Q3 scores led to inadvertent filtering.\n\nWe believe that introducing more robust communication between roles and improving traceability can reduce such errors. This is a promising direction for future work.\n\n---\n\n### **2. Generalization to Other Languages or Domains**  \n\n**(Response to “Question 2”)**\n\nOur system is **inherently language-agnostic**. It relies on structural elements like **function boundaries, control/data flow, and AST-derived slicing**, which are common across many programming languages.\n\nWe chose C++ due to its industrial relevance and high complexity. However, both our slicing algorithms and prompting strategies are adaptable to other languages. Extending the framework to support multiple languages is part of our planned roadmap.\n\n---\n\n### **3. Sensitivity to Filtering Thresholds**  \n\n**(Response to “Question 3”)**\n\nThe thresholds for Q1 (nitpicks) and Q2 (hallucinations) were set heuristically for interpretability and practical deployment. As described in Section 3.4, we adopted a 1–7 scoring scale (inspired by McAleese et al., 2024), with a threshold of 4+ to denote serious, actionable issues.\n\nWe evaluated Q3 (redundancy) sensitivity via a top-*k* truncation strategy in **Section 5.4**, and extended it further:\n\n🔗 https://anonymous.4open.science/r/16368/The%20impact%20of%20Top-k%20truncation.pdf\n\nKey observations:\n\n- **Top-3** truncation works better for **shorter context slices** (e.g., Original Diff, Parent Function), which produce fewer relevant comments.\n- **Top-5 to Top-10** perform better with **richer slicing methods** (e.g., Left Flow, Full Flow), which generate more detailed reviews.\n- **Full Flow + Top-10** occasionally shows performance drop after validation, likely due to hitting the token limit of LLaMA3.1, reducing effectiveness.\n\nWe agree that adaptive or learned thresholds would be valuable. However, to avoid overfitting to a specific codebase, we chose to retain a generalizable, interpretable heuristic strategy in this version. We will reflect this design choice and its limitations more clearly.\n\n---\n\n### **4. Alignment of Metrics with Real-World Review Quality**  \n\n**(Response to “Question 4”)**\n\nWhile we have not yet conducted formal user studies, our metrics (KBI, FAR, LSR) are grounded in interviews with professional developers. Developers repeatedly emphasized two points:\n\n1. **Catching critical bugs** is a top priority.\n2. **Reducing irrelevant comments** is crucial for adoption—\"even one false positive can erode trust.\"\n\nThis informed our focus on *key defect inclusion* and *false alarm suppression*, which are not well captured by traditional text similarity metrics.\n\nThe system is currently deployed in an internal development team, though we do not require developers to reply to each comment. Based on initial feedback, FAR has emerged as a particularly sensitive metric, directly affecting developer perception. We agree that systematic user studies would further strengthen the validation and plan to incorporate them in future work.\n\n---\n\n**We thank the reviewer again for the constructive questions and positive evaluation. We hope these clarifications further highlight the robustness and extensibility of our work.**", "claims_and_evidence": "Most of the paper's claims are supported by empirical evaluations and ablation studies.\nThe evidence might be less convincing on the generalizability of the approach to other programming languages.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in the paper are well-aligned.\nOn the evaluation side, the authors introduce metrics such as Key Bug Inclusion (KBI), False Alarm Rate (FAR), Comprehensive Performance Index (CPI), and Line Localization Success Rate (LSR).", "theoretical_claims": "The paper does not include formal proofs or theoretical claims.", "experimental_designs_or_analyses": "One potential issue is that while these metrics and the accompanying analyses are well-structured, some evaluation criteria (like KBI and CPI) are specific to this work and might not fully capture all dimensions of code review quality as experienced by developers and can not be well compared with the literatures..", "relation_to_broader_scientific_literature": "The key contributions of the paper extend prior findings to an integrated, context-rich, and workflow-aware system that is validated on industrial-scale data.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper presents an innovative integration of multiple techniques—including diverse code slicing algorithms, a multi-agent framework with chain-of-thought reasoning, and a robust filtering mechanism—to tackle real-world challenges in automated code review.\nThe experimental evaluation is comprehensive and grounded in industrial-scale data, using metrics (KBI, FAR, CPI, LSR) that directly address practical defect detection concerns.\n\nWeaknesses:\n\nSome design choices, such as the thresholds in the filtering mechanism, appear heuristic, and it remains unclear how sensitive the results are to these parameters.\nThe evaluation is primarily confined to a specific industrial C++ codebase, leaving questions about the generalizability of the approach to other programming languages or domains.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "1. Could you elaborate on the trade-offs observed with the validator role? \n\n2. Could you discuss how your approach might generalize to other programming languages or domains? \n\n3. How sensitive is the system’s performance to the thresholds, and have you considered adaptive or data-driven methods for threshold selection? \n\n4. Have you conducted any user studies or qualitative evaluations with developers to validate that these metrics align well with real-world expectations of code review quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated that the work was tested in practically realistic setting and combines multiple techniques into a strong-performing tool. The paper is well-written and the authors made prompt and thorough improvements based on reviewer feedback. Given this, we recommend the paper for acceptance.
Decision: Accept (spotlight poster)