The Code Review Comprehension Assessment for Language Models
Authors: Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude
Accepted at: acl25


Reviewer paper summaries: This paper introduces CodeReviewQA, a novel benchmark for evaluating large language models' ability to comprehend code review comments and make appropriate code refinements. The authors decompose the code refinement task into three essential reasoning steps: change type recognition, change localization, and solution identification. Each step is formulated as a multiple-choice question with varying difficulty levels. The benchmark includes 900 manually curated examples across 9 programming languages, and evaluates 65 state-of-the-art models. Results show that CodeReviewQA effectively reveals model capability gaps in different reasoning tasks.

This paper proposes a new benchmark for code review. This benchmark is designed using a multi-choice question-answering format. Specifically, the authors describe how they split the code review into three steps, change type recognition, change localization, and solution identification, and construct corresponding questions for these three steps. After constructing the new benchmark, the authors evaluate amang LLMs on their benchmark to demonstrate the understanding capabilities of existing LLMs in code review tasks.

This paper introduces **CodeReviewQA**, a new benchmark to evaluate large language models (LLMs) on the complex task of automated code refinement in the context of code review. Unlike prior benchmarks that rely on end-to-end generation and surface-level metrics (e.g., BLEU, EM), this work decomposes the task into three intermediate reasoning steps: Change Type Recognition (CTR), Change Localisation (CL), and Solution Identification (SI), each reformulated as multiple-choice question answering (MCQA). The authors present a high-quality, manually curated dataset across 9 programming languages and evaluate 65 LLMs under invariant testing for robustness. The benchmark mitigates training data contamination through synthetic reformulation and distractor generation. This work is timely, well-scoped, and offers practical insights into LLM comprehension of real-world, conversational software engineering tasks.
Summary of strengths: - Comprehensive evaluation: Testing 65 models across multiple programming languages provides valuable insights into model performance.

- High-quality dataset: Manual curation ensures the examples represent meaningful real-world code review scenarios.

- Practical relevance: Addresses an important practical challenge in software engineering that affects developer productivity.

1. Previous code review benchmarks suffer from data contamination, and this newly proposed benchmark and its construction methods could avoid data contamination.
2. The construction process of the benchmark is quite rigorous, with many irrelevant confounding factors eliminated.
3. The authors evaluate 65 LLMs on their benchmark, which is quite comprehensive. I believe it could set a good standard for future works.

- **Novel Evaluation Framework**: Reformulating code refinement into MCQA probes (CTR, CL, SI) enables fine-grained analysis of LLM capabilities, beyond holistic metrics like BLEU or EM.

- **Focus on Real-World Developer Communication**: The dataset addresses an underrepresented but critical modality—code review comments—which are ambiguous, colloquial, and collaborative in nature.

- **Manual Data Curation**: The dataset is carefully filtered, manually verified, and includes 900 high-quality samples from 199 GitHub repositories, addressing prior noise issues in automatic mining.

- **Data Contamination Mitigation**: The benchmark explicitly tackles pretraining contamination risks by using novel formulations, distractors, and invariant prompting.

- **Comprehensive Model Evaluation**: The benchmark is applied to 65 LLMs across different parameter sizes, using rigorous permutation-based evaluation to ensure robustness and answer consistency.

- **Insightful Findings**: The results show discrepancies between end-to-end generation performance and individual reasoning capabilities, highlighting areas where model scaling does not guarantee comprehension gains.
Summary of weaknesses: - Limited scope of evaluation: The study focuses solely on understanding and implementing single code review comments, but real-world code review often involves multi-turn conversations and negotiation.

- Lack of baseline comparison with human performance: No human baseline is provided to contextualize model performance metrics.

- Missing error analysis: The paper lacks detailed error analysis of specific failure cases to better understand model limitations.

1. The authors state at the beginning of the paper that one of the challenges of code review is that review comments are often ambiguous. However, after reading this paper, I do not see any discussions about the ambiguity of review comments in the proposed benchmark, which makes me feel that the motivation and the proposed benchmark are not strongly related. Are the review comments in this benchmark also ambiguous? If not, I have to doubt whether this benchmark is suitable for the code review task.
2. The benchmark seems quite easy. For example, according to Table 2, Qwen2.5-72B can achieve an accuracy of 97.1% and 90.9% on easy and hard subsets of the solution identification part. And this is achieved without any advanced prompting techniques. It is widely believed that LLM's performance can be further enhanced with prompting techniques like CoT.
3. It is better to evaluate with some basic prompting methods like CoT or in-context learning. As we know, prompting techniques are strongly related to LLMs. Showing results with the most famous prompting techniques could be better for future studies.

1. **Limited Dataset Size**  
   While the data quality is commendable, the benchmark includes only 900 samples. This raises concerns about generalization and statistical power, especially for evaluating frontier LLMs.

2. **Hard-Coded Task Decomposition**  
   The CTR–CL–SI pipeline is human-designed and static. While effective, it might constrain generalization or overlook alternative reasoning pathways that models could learn implicitly.

3. **Line-Level Localization Granularity**  
   The localisation task is restricted to line-level precision. This misses finer-grained phenomena like token-level changes, which could be important in subtle refactoring or variable renaming tasks.

4. **Prompt Uniformity**  
   The same prompt templates and hyperparameters are applied across all models. While this ensures fairness, it may penalize models sensitive to prompt phrasing. Some adaptive prompting could yield stronger insights.

5. **Lack of Task Interaction Modeling**  
   The benchmark evaluates each subtask in isolation. While informative, it misses out on modeling or analyzing error propagation—i.e., how mistakes in CTR influence SI.


Questions: 

1. **Task Interaction Modeling**: Did you explore how failure in earlier tasks (e.g., CTR) propagates to downstream tasks (e.g., SI)? Could joint modeling improve outcomes?

2. **Prompt Design Sensitivity**: How sensitive are the models to prompt wording and format? Would retrieval-augmented or instruction-specific prompting improve invariant accuracy?

3. **Expansion to Token-Level Localisation**: Do you plan to extend CL beyond line granularity? Could this increase diagnostic depth, especially for code style and semantic edits?

4. **Generalization Beyond Code Reviews**: Could this approach (MCQA decomposition) be applied to other conversational software engineering tasks like bug triage or issue resolution?
Comments, suggestions and typos: see weakness

1. Question: In Line 341, how do you get the embedding of H_post from the surrogate model? The surrogate model (Codestral-22B) seems a generative model, not specifically designed for getting embeddings.
2. Comment: The authors may consider merging some steps to improve the difficulty of the benchmark. For example, we could merge change type recognition and change localization. For example, the choice may be: A. add in Line 6. B. modify in Line 7,8,9 C. delete Line 1, 2 ...

As above
Overall assessment scores: 3

3

2.5
Metareview score: 3
Metareview: This paper introduces CodeReviewQA, a novel benchmark designed to evaluate the fine-grained code comprehension abilities of large language models (LLMs) in the context of code review. The benchmark reformulates automated code refinement into three structured multiple-choice tasks: Change Type Recognition (CTR), Change Localisation (CL), and Solution Identification (SI). Across 900 curated examples in nine programming languages, 65 LLMs are evaluated under an invariant testing regime that mitigates surface-level data contamination and ensures answer robustness.

Reviewers broadly agree that the benchmark is timely, practically relevant, and methodologically sound, offering meaningful insights into model limitations not captured by generation-based metrics (e.g., BLEU or EM). The paper is particularly praised for its rigorous dataset construction, careful decomposition of subtasks, and its ability to reveal diagnostic differences among models with similar surface performance.

However, some reviewers voiced concerns about limited dataset size, lack of prompt adaptation, and the static nature of the task decomposition. Authors addressed these thoroughly in their rebuttal, clarifying that the benchmark prioritizes interpretability and modularity, while acknowledging future work may explore more adaptive prompting, token-level localization, or multi-turn review dynamics. Multiple reviewers flagged that several weaknesses merely repeated the authors' own stated limitations, and the authors rightly contested these as invalid criticisms per ARR guidelines.

Reviewers iTyP, wf3E, HAHR, ynaj, and bt8i ultimately converged on a positive consensus, suggesting acceptance at Findings, with recognition of the benchmark’s practical contribution and diagnostic utility.
Summary of reasons to publish: - Novel Diagnostic Benchmark: CodeReviewQA decomposes code refinement into interpretable subtasks (CTR, CL, SI), enabling fine-grained evaluation of LLMs beyond traditional holistic metrics.

- Rigorous Data Curation: The benchmark comprises 900 manually verified, real-world code review examples across 9 languages, with high inter-annotator agreement (κ = 0.89) and mitigation of contamination.

- Comprehensive Evaluation and Insightful Findings: Evaluation across 65 LLMs reveals meaningful performance disparities, demonstrating the benchmark’s ability to differentiate model capabilities and guide future model development.