ConGra: Benchmarking Automatic Conflict Resolution

## Abstract

Abstract Resolving conflicts from merging different software versions is a challenging task.
To reduce the overhead of manual merging, researchers develop various program analysis-based tools which only solve specific types of conflicts and have a limited scope of application.
With the development of language models, researchers treat conflict code as text, which theoretically allows for addressing almost all types of conflicts.
However, the absence of effective conflict difficulty grading methods hinders a comprehensive evaluation of large language models (LLMs), making it difficult to gain a deeper understanding of their limitations.
Furthermore, there is a notable lack of large-scale open benchmarks for evaluating the performance of LLMs in automatic conflict resolution.
To address these issues, we introduce ConGra , a CON flict- GRA ded benchmarking scheme designed to evaluate the performance of software merging tools under varying complexity conflict scenarios.
We propose a novel approach to classify conflicts based on code operations and use it to build a large-scale evaluation dataset based on 44,948 conflicts from 34 real-world projects.
We evaluate state-of-the-art LLMs on conflict resolution tasks using this dataset.
By employing the dataset, we assess the performance of multiple state-of-the-art LLMs and code LLMs, ultimately uncovering two counterintuitive yet insightful phenomena. ConGra will be released at https://github.com/HKU-System-Security-Lab/ConGra .

## 1 Introduction

Code merging has become a challenging task for developers during project development and maintenance.
Git, the most popular version control system (Spinellis ()), uses a text-based code-merging mechanism. Despite its efficiency, developers often struggle with manually merging different versions of code when Git fails to resolve conflicts automatically.
These conflicts come from text or even the syntax and the functionality of the code from different versions.

To address conflict resolution, researchers leverage program analysis to achieve syntax-error-free code merging (Zhu et al. (); Sousa et al. (); Larsén et al. (); Shen et al. (); Apel et al. (, )).
Based on abstract syntax trees (AST), these tools merge AST vertices and edges to ensure syntax correctness and better merge results.
Nevertheless, a conflict is still generated and need to be resolved by developers if the merge of AST nodes fails.
With the development of language models and even LLMs, they are now applied to conflict resolution (Dinella et al. (); Zhang et al. (); Svyatkovskiy et al. (); Dong et al. ()).
Trained on extensive datasets of prior and manually merged codes, these models predict suitable resolution for each conflicting code segment without manual efforts.

However, evaluating the performance of LLMs on conflict resolution tasks is challenging due to the wide variation in conflict difficulty and the lack of effective grading methods to reflect these differences.
For example, ConflictBench (Shen and Meng ()) classifies conflicts based on the source of conflict-resolved code, such as from either merging candidate versions or newly introduced by developers.
This classification does not accurately reflect the complexity of the conflicts.
Additionally, the community lacks comprehensive conflict resolution benchmarks (discussed in §2.4), especially for extreme cases involving long code contexts.

To this end, we introduce ConGra, which is designed to evaluate code merging tools across a diverse range of merging scenarios and assess their ability to resolve conflicts of varying complexities.
We propose a novel approach to construct graded conflict dataset according to conflict’s resolving complexity.
We evaluated six state-of-the-art LLMs (three general LLMs and three code LLMs) on ConGra to assess their abilities to resolve conflict in various merging scenarios.
The dataset is constructed using 44,948 conflict cases sourced from 34 large-scale open-source projects written in C, C++, Java, and Python.
The results show that LLMs with longer contexts support do not always yield better results compared to models with shorter contexts.
Additionally, general LLMs (e.g. LLama3-8B and DeepSeek-V2) outperform specialized code LLMs in automatically resolving conflict.
Besides, we will release our datasets and a benchmark at [https://github.com/HKU-System-Security-Lab/ConGra](https://github.com/HKU-System-Security-Lab/ConGra).

In summary, we made the following contributions:

- 1.
We introduce the first classification approach to generate a complexity-graded conflict dataset.
Using this appraoch, collected conflicts can be classified into seven categories.
- 2.
We release a large-scale graded dataset for conflict resolution benchmarking. which contains 44,948 conflict cases from popular projects written in C, C++, Java, and Python.
- 3.
We conduct the first comprehensive evaluation of LLM’s performance on conflict resolution task, and find two thought-provoking counter-intuitive phenomena.

## 2 Background

### 2.1 Task Definition of Automatic Merge Conflict Resolution

Git offers multiple strategies to merge code (Atlassian ()).
The most common and widely-used strategy is the three-way merging strategy (Vinkler ()).
The fundamental concept behind three-way merging is to locate the most recent common ancestor ($O$) through the historical commit graph, given two merging candidates ($A$ and $B$), and to generate the automerged code version ($M$) based on the difference $A-O$ and the difference $B-O$.
Before generating $M$, Git identifies multiple difference matching triplets $A_{i}-O_{i}-B_{i}$ by comparing $A-O$ and $B-O$.
For each difference matching triplet,
1) if $A_{i}-O_{i}$ equals $B_{i}-O_{i}$, it indicates that both $A$ and $B$ have made identical changes to the same code segment. In this case, the difference block will be applied to $M$;
2) if $A_{i}-O_{i}$ is empty, but $B_{i}-O_{i}$ exists, it suggests that only $B$ has modified this code segment. Consequently, the difference block from $B$ will be applied to $M$, and vice versa;
3) if $A_{i}-O_{i}$ and $B_{i}-O_{i}$ are not empty nor equal, it indicates that $A$ and $B$ have made different modifications to the same code segment, resulting in a conflict in $M$. In this paper, we assume that $M$ always contains at least one conflict.

Due to the presence of conflicts, Git-based automated code merging may encounter exceptions, prompting developers to manually address them. Following conflict resolution, developers will release the resolved code version ($R$).
In large projects, merging two versions results in numerous conflicts, involving substantial code modifications. This significantly raises developers’ project maintenance costs (Vale et al. ()).
To this end, various automatic conflict resolution (ACR) systems have been proposed to mitigate these challenges.

**Table 1: Evaluation features implemented in 1) program analysis-based ACR (tagged with †); 2) machine learning-based ACR (tagged with ‡); 3) conflict resolution benchmark (tagged with ✠). The symbol "-" means the feature is not applicable to the corresponding work’s evaluation.**
| Evaluation Features | $Feature_{1}$ | $Feature_{2}$ | $Feature_{3}$ | $Feature_{4}$ | $Feature_{5}$ | $Feature_{6}$ |
| --- | --- | --- | --- | --- | --- | --- |
| IntelliMerge† | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ |
| Spork† | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ |
| SafeMerge† | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| FSTMerge† | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Mastery† | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ |
| JDIME† | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| MergeBERT‡ | ✗ | ✗ | - | ✓ | ✓ | ✗ |
| GMerge‡ | ✗ | ✗ | - | ✓ | ✓ | ✗ |
| DeepMerge‡ | ✗ | ✗ | - | ✓ | ✓ | ✗ |
| MergeGen‡ | ✗ | ✗ | - | ✓ | ✓ | ✗ |
| ConflictBench✠ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ |
| ConGra ✠ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

### 2.2 Program Analysis-based ACR

Program analysis-based ACR (Zhu et al. (); Larsén et al. (); Sousa et al. (); Apel et al. (); Shen et al. (); Apel et al. ()) ensures the syntactic integrity of the merged code by merging on AST.
As they implement the three-way merging strategy, conflicts inevitably arise for unmergeable AST nodes, requiring manual intervention.

To assess the performance of program analysis-based ACR, researchers focus on metrics like the number of generated conflicts, the accuracy of resolutions, and resource consumption.
However, these tools often lack convincing evaluation datasets and performance comparisons.
As shown in Table 1, tools like Spork, SafeMerge, and Mastery (Larsén et al. (); Sousa et al. (); Zhu et al. ()) evaluate all merging scenarios without classification, obscuring performance differences across conflict complexities.
JDIME and FSTMerge (Apel et al. (, )) separately categorize structural and textual merges, but this is essentially an ablation study, leaving the dataset’s resolving complexity intertwined.
IntelliMerge (Shen et al. ()) classifies merge scenarios but fails to differentiate these categories in the final evaluation, preventing performance comparisons across scenarios.
Additionally, These tools evaluate the quality of generated resolutions by exact string matching with the manual resolution (ground truth).
However, this approach is inadequate for measuring code similarity, as it is influenced by programming style and code structure.

### 2.3 Machine learning-based ACR

Machine learning-based ACR (Zhang et al. (); Dong et al. (); Svyatkovskiy et al. (); Dinella et al. ()) leverages a vast number of code merge examples during pre-training and harnesses the language-aware capabilities of machine learning models to understand the intricacies of code merging.
Once trained, these models offer resolution suggestions for merge conflicts.
They does not rely on the three-way merge strategy, eliminating the need for manual conflict resolution.

Table 1 illustrates the features implemented in machine learning-based ACRs’ evaluation.
Machine learning-based ACRs must provide accurate resolutions for all conflicts, underscoring the importance of resolution correctness.
Prior research efforts have primarily concentrated on improving two key metrics: precision and accuracy. Accuracy measures the percentage of total conflicts for which these tools produce the correct resolution, while precision indicates the percentage of correct resolutions among all resolution suggestions provided by the tools.
By the same token as depicted in §2.2, the matching algorithm between generated resolution and ground truth are exact string matching, which cannot reflect the real code similarity.
Furthermore, existing language model-based ACR test datasets also lack classification of merge scenarios based on different complexities, hindering the assessment of model performance across various levels of code merging tasks’ difficulty.

### 2.4 Conflict Resolution Benchmarks

ConflictBench (Shen and Meng ()) is the only benchmark for software code merging evaluation to date.
It categorizes merging scenarios by resolution types and uses three metrics with an exact string matching algorithm to evaluate merging tools.
However, as shown in Table 1, its classification strategy doesn’t fully capture the complexity of merging scenarios.
For example, conflicts can be resolved by retaining all edits from $A$ or $B$, or by introducing new edits.
ConflictBench categorizes these into three groups, but the complexity boundaries between them are often blurred.
Additionally, ConflictBench’s datasets include only 180 merging cases, and further classification limits the data volume in each category, potentially leading to inaccurate evaluation results.

## 3 Benchmarking Pipeline

### 3.1 Overview

Figure: Figure 1: Benchmarking pipeline overview.
Refer to caption: extracted/5852181/images/pipeline_overview.png

Figure 1 provides an overview of ConGra’s benchmarking pipeline.
ConGra starts with collecting open-source projects’ historical merging scenarios with conflict from Github.
After the raw dataset is constructed, ConGra further colors the conflict snippets with code operations which are extracted via a lightweight but powerful syntax tree level analysis.
These conflicts then are classified into various categories based on the code operations taken in the conflict code block to construct ConGra dataset.
Then, we collect the conflict along with its context to construct the prompt for LLMs.
Finally, LLMs will respond to the query and provide conflict resolution suggestions.
The results will be assessed by ConGra’s evaluation metrics.

### 3.2 Coloring

Firstly, analyzing conflict code snippets is challenging because the snippets in $M$ generated by Git are often fragmented and can contain any part or format of code.
Existing program analysis tools (Lattner and Adve (); Wang and Shoshitaishvili (); Lam et al. ()) cannot handle these incomplete snippets syntactically or semantically.
To address this, ConGra applies $M$’s modifications from $A$ and $B$ separately to generate two merged files ($M_{a}$ and $M_{b}$).
This approach applies all the difference blocks from either $A$ or $B$, allowing for code analysis of the conflicting snippets by recording the text ranges and analyzing the whole code from $M_{a}$ and $M_{b}$.

Secondly, for the benchmarking pipeline, automating code complexity assessment must use the lightest code analysis.
Traditional program analysis requires a strict program context, including the project environment and necessary compilation settings, which are hard to preserve in datasets.
Additionally, program analysis tools perform data flow or control flow analysis (Chess and McGraw (); Nielson et al. ()), consuming significant resources, especially for large-scale datasets.
ConGra addresses this through code operation abstraction.
It extracts the operations of conflicting code fragments from $M_{a}$ and $M_{b}$ to reflect code complexity.
Since ConGra’s analysis involves only lexical and grammatical analysis without deeper compilation or context dependency, it achieves efficient benchmarking.

Given $M_{a}$, $M_{b}$ and respective conflict-related code ranges, ConGra extracts code operations from these conflict-related code blocks in the coloring phase.
ConGra first leverages tree_sitter (Brunsfeld ()), a lightweight multi-language code parsing framework, to convert plain text code into a syntax tree that is suitable for analysis.
We pre-define types of code operations based on the node of the syntax tree.
These code operations are arranged in ascending order of priority as shown below:

- 1.
Composite Type Definition (CTD): Definition of composite types, e.g. the definition of class and the definition of struct in C++ language.
- 2.
Function Body Definition (FBD): Definition of the function body content. Any modification to the function body is included in this operation.
- 3.
Function Prototype Definition (FPD): Definition of function prototype, i.e. function name, return value type and parameter list.
- 4.
Language-Specific Operations (LSO): Operations related to language features, such as macro definitions in C/C++ and the introduction of third-party libraries in Python.
- 5.
Commenting (CMT): Developers’ comments on the code.
- 6.
Variable Declaration (VD): Declaration of all variables. These variables refers to global variables, local variables, and member variables defined in composite types.

These code operations have different priorities, and operations with higher priorities can override operations with lower priorities.
For instance, the modification of a local variable name in a function body can belong to both function body definition and variable declaration.
To this end, ConGra will give priority to variable declaration as the result of code operation extraction.
After analyzing each syntax tree node, ConGra only retains code operations related to the conflicting code area, and generates code operation lists for the two versions of the code, namely $P_{a}$ and $P_{b}$.
These code operations serve as the basis for conflict classification in the subsequent stage.

### 3.3 Classifier

Given $P_{a}$ and $P_{b}$, ConGra traverses each pair of conflicting code blocks in $A$ and $B$ and classifies the conflicts according to the following conditions.
For each pair of $A$ ’s conflict and $B$ ’s conflict (i.e., $P_{ai}$ in $P_{a}$ and $P_{bi}$ in $P_{b}$):

- 1.
Text conflict: At least one of $P_{ai}$ and $P_{bi}$ contains CMT.
2(a) presents a text conflict example.
- 2.
Functional conflict: At least one of $P_{ai}$ and $P_{bi}$ contains FBD.
2(b) presents a functional conflict example.
- 3.
Syntax conflict: Respectively construct operations sets based on $P_{ai}$ and $P_{bi}$ (denoted as $S_{a}$ and $S_{b}$) with only remaining CTD, FPD, LSO, and VD.
Finally, the difference between $S_{a}$ and $S_{b}$ is not empty.
2(c) presents a syntax conflict example.

Figure: (a) Text conflict.
Refer to caption: extracted/5852181/images/text-conflict.png

Considering that the syntax among languages varies, the benchmarking of ConGra only focuses on syntax problems caused by missing declarations or definitions which will occur in all programming languages.
Since the classifier of ConGra adopts the whitelist mode, the correctness of the classification results can be guaranteed.
As one conflict may be classified into multiple categories, ConGra supports seven classifications, i.e., all permutations of text, syntax, and functional conflict.

### 3.4 Generate Resolutions

Figure: Figure 3: The process of generating resolutions by LLMs.
Refer to caption: extracted/5852181/images_source/llm_process.png

In this section, we present how to obtain merge conflict resolution solutions using LLMs, as depicted in  Figure 3. Here, we describe the specific steps as follows: (1) Step 1: Acquire the merge conflict and its context. Merge conflict refers to the current improvements and incoming changes. We select the previous and subsequent text at the conflict location as the context. (2) Step 2: Construct the prompt. We employ the thought-chain method to create the prompt, with the specific content provided in the  Appendix A. To avoid outputting redundant context, we offer an example for LLMs to reference.
(3) Step 3: Check the input. Since LLMs can only support a limited context length, we utilize the corresponding tokenizer to process the prompt. If the number of tokens surpasses the LLMs’ maximum input length, revert to Step 1 and reduce the context’s lines. (4) Step 4: Query LLMs and obtain the resolution. We input the prompt to LLMs to receive answers, and extract the relevant code block as the final generated resolution, ensuring a logically coherent and standardized format.

## 4 Graded Conflicts Dataset

We target on the well-known open-source projects in Github.
Our target selection criteria is that the project should contain 10K+ lines of code and historical commits on main branch.
This is because a larger code base and a more active development history can help generate conflicts with multiple complexities.
Upon generating the datasets, we first traverse the historical commits of each project and reproduce each merge process through Git merge.
When the merge fails, we record the conflicting file, namely $M$, and extract $A$, $B$, $O$ and $R$ from the commit graph.
Finally, we utilize the coloring and classifier introduced in §3 to classify all the collected cases.
In summary, we collected data from 34 open-source projects on GitHub: 14 written in C/C++, 11 in Java, and 9 in Python.
We gathered 23,334 conflict files encompassing 44,948 conflict scenarios classified in 7 complexity types as shown in  Figure 4.
The statistics on the length (in lines of code) of all conflicts and their corresponding manual resolutions are presented in  Figure 5.
More visualizations of ConGra are shown in  Appendix F.

Figure: Figure 4: Venn diagrams for conflict types.
Refer to caption: extracted/5852181/images/overlap.png

Figure: Figure 5: Average length of different types of conflicts. T for Text, S for Syntax, F for Functional.
Refer to caption: extracted/5852181/images/type_length_bar.png

## 5 Benchmarking for Auto-Conflict Resolutoin

### 5.1 Benchmakring Settings

Evaluation Metric.
We choose the accuracy and precision of generated resolutions as the key metrics in our evaluation.
Accuracy is the percentage of generated resolutions that match the ground truth to the total number of conflicts.
Precision refers to the percentage of generated resolutions that match ground truth to all generated resolutions.
We use the combination of normalized edit distance, winnowing, and cosine semantic similarity (Ristad and Yianilos (); Schleimer et al. (); Rahutomo et al. ()) as the code-matching standard (i.e., ES, WS, and SS).
For a generated resolution, ConGra regards the resolution matches the ground truth when at least one of the above values greater than 80%.

Benchmarked Models.
As depicted in  Figure 5, ConGra exhibits a substantial number of conflict instances, characterized by their contextual information. Consequently, we opt for LLMs that are capable of accommodating long contexts as our benchmarked models. These language models, supporting over 8K tokens, comprise three general language models (Llama3-8B (^1^11[https://llama.meta.com/llama3/](https://llama.meta.com/llama3/)), DeepSeek-V2(^2^22We are using their official API service, which supports a context of 32K tokens instead of 128K.)( DeepSeek-AI et al. ()), GLM-3-turbo (^3^33[https://open.bigmodel.cn/dev/api#glm-3-turbo](https://open.bigmodel.cn/dev/api#glm-3-turbo))) and three code language models (CodeLlama-7B( Roziere et al. ()), CodeLlama-34B( Roziere et al. ()), DeepSeek-Coder( Guo et al. ())).

Experiment Settings.
We set the temperature coefficient uniformly at 0.7 to fully ensure the creativity of LLMs in the experiments. For each conflict, we take at most the previous 100 lines and the following 100 lines of text as context, and decrement line by line until the number of tokens is lower than the maximum context length supported by LLMs. In addition, for each conflict that does not correctly obtain the model output, we will repeat the experiment up to 10 times, otherwise it will be regarded as an unprocessable case. For open-source models, we use vLLM( Kwon et al. () as the backend for model deployment and conduct experiments based on 6 $\times$ NVIDIA A800 80GB GPUs.

### 5.2 Benchmark Results and Analysis

**Table 2: Benchmark Result on Python and Java.**
|  |  | Python | Java |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Context<br>Length | Accuracy | Precision | ES | WS | SS | Accuracy | Precision | ES | WS | SS |
| LLama3-8B | 8K | 75.82 | 77.45 | 0.71 | 0.36 | 0.66 | 82.93 | 83.00 | 0.75 | 0.42 | 0.67 |
| DeepSeek-V2 | 32K | 75.07 | 75.53 | 0.67 | 0.53 | 0.83 | 84.38 | 84.40 | 0.74 | 0.61 | 0.84 |
| GLM-3-turbo | 128K | 57.05 | 57.31 | 0.53 | 0.33 | 0.70 | 62.52 | 64.75 | 0.58 | 0.38 | 0.74 |
| CodeLlama-7B | 16K | 50.68 | 59.92 | 0.55 | 0.41 | 0.76 | 73.61 | 73.66 | 0.67 | 0.51 | 0.79 |
| CodeLlama-34B | 16K | 61.47 | 62.34 | 0.61 | 0.30 | 0.63 | 70.82 | 71.03 | 0.66 | 0.40 | 0.70 |
| DeepSeek-Coder | 16K | 56.49 | 57.31 | 0.55 | 0.41 | 0.76 | 74.52 | 74.6 | 0.67 | 0.53 | 0.82 |

**Table 3: Benchmark Result on C and C++.**
| Model | Context<br>Length | C | C++ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy | Precision | ES | WS | SS | Accuracy | Precision | ES | WS | SS |  |  |
| LLama3-8B | 8K | 72.45 | 73.11 | 0.64 | 0.36 | 0.69 | 78.13 | 79.22 | 0.71 | 0.38 | 0.70 |
| DeepSeek-V2 | 32K | 54.42 | 71.06 | 0.61 | 0.45 | 0.79 | 70.86 | 77.31 | 0.69 | 0.53 | 0.83 |
| GLM-3-turbo | 128K | 58.86 | 60.32 | 0.52 | 0.30 | 0.70 | 64.10 | 64.13 | 0.57 | 0.32 | 0.70 |
| CodeLlama-7B | 16K | 59.09 | 60.55 | 0.52 | 0.35 | 0.73 | 64.28 | 64.9 | 0.58 | 0.38 | 0.75 |
| CodeLlama-34B | 16K | 67.75 | 68.23 | 0.61 | 0.26 | 0.63 | 69.83 | 70.39 | 0.66 | 0.29 | 0.63 |
| DeepSeek-Coder | 16K | 62.36 | 62.72 | 0.54 | 0.38 | 0.77 | 62.33 | 62.9 | 0.57 | 0.39 | 0.78 |

#### 5.2.1 Overall Performance

In this section, we present a comprehensive evaluation of advanced LLMs on our proposed ConGra dataset. The experimental results can be found in Tables  2 and  3. Upon careful comparison, we draw the following observations: (1) LLMs with longer context support do not always yield optimal results. GLM-3-Turbo, which supports a 128K context, is generally outperformed by LLama3-8B, which only supports an 8K context, with the exception of semantic similarity. For instance, in Python and Java, the Precision metric for GLM-3-Turbo is nearly 20% lower than that of LLama3-8B. We speculate that this may result from LLMs with ultra-long context support not being sufficiently trained on merge conflict datasets, hindering their ability to effectively extract valuable information from the context to suggest merge resolutions. (2) Code LLMs do not appear to demonstrate a distinct advantage. Notably, LLama3-8B and DeepSeek-V2 surpass Code LLMs in Precision across all four languages. We attribute this to two factors: (i) general models are in fact trained on extensive code repositories, so their code comprehension capabilities are not significantly inferior to Code LLMs. (ii) Elements within conflicts such as comments and variable names contribute a plethora of semantic information, extending beyond mere code understanding, which presents a challenge for Code LLMs. (3) Both LLama3-8B and DeepSeek-V2 prove to be well-suited for automatic conflict resolution.

Figure: Figure 6: Heatmap on precision of LLMs and types. T for Text, S for Syntax, F for Functional.
Refer to caption: x1.png

#### 5.2.2 The Impact of Conflict Type

Furthermore, we investigate the performance of LLMs under different conflict types. Based on Section  3.3, we consider seven conflict types, namely Text, Syntax, Functional conflicts, and their combinations. The results are visualized in Figure  6. In Figure  6, the closer the result is to blue, the better the model performs; the closer it is to green, the worse the model performs.

For LLMs, a simpler conflict does not necessarily mean it is easier to handle. This is actually a counterintuitive phenomenon. Specifically, for most models and languages (e.g., Python, C/C++), LLMs exhibit better performance on the most complex conflicts (i.e., F+S+T) and poorer performance on the simplest conflicts (i.e., F, S, and T). In the context of text conflicts, the performance of LLMs on different samples can be ranked as follows: T+F+S $>$ T+F $\approx$ T+S $>$ T. We believe that this occurs because, for LLMs, the simpler the samples within the conflict area (such as type T), the less effective guidance information the conflict can provide, causing LLMs to produce conservative answers. For example, when the conflict involves a comment, LLMs tend to preserve as much information as possible from both branches a and b. In contrast, the more complex the conflict area, the clearer the direction for conflict resolution. In summary, LLMs need to extract more valuable guidance information from the conflict area itself to improve their performance.
Notice that in Figure 6, the precision of F in Java is relatively higher.
This is due to Java’s encapsulation.
To hide "sensitive" member variables of a class from users, these variables are often declared as private, and the class must provide public get and set methods to access and update these values.
Consequently, these getter and setter methods are simple in terms of functionality.
Since most of the F conflicts in the Java dataset arise from the rewriting of these encapsulated methods, the LLMs can easily infer the resolutions, resulting in high precision.

### 5.3 The Role of Context

Figure: Figure 7: The impact of contextual information on Precesion.
Refer to caption: extracted/5852181/images_source/context_exp.png

In the vast majority of cases, LLMs without conflict context information significantly outperform those with context information. As observed in Figure  7, the red and gray solid circles almost entirely encompass their corresponding dashed circles, with only a few exceptions. We analyze this phenomenon from two perspectives: (1) the context understanding capability of large language models. Although existing LLMs have undergone extensive training on code data, this does not necessarily imply that they can effectively extract useful information from the rich context for automatic conflict resolution. (2) The choice of context. In our experiments, we provided the most basic context information. However, it is possible that the adjacent context does not supply crucial information for resolving conflicts and instead introduces a significant amount of noise. As a result, selecting the appropriate context for automatic conflict resolution may be a valuable research direction to explore.

## 6 Discussion

Limitation.
The conflict classification strategy in §3 ensures correct classification but may fail to identify the exact category of a conflict due to advanced language usage.
For example, the use of template class definitions in C++ (Vandevoorde and Josuttis ()) prevents tree_sitter (Brunsfeld ()) from capturing the type used as the template argument during instantiation.
Consequently, a conflict that should be classified as "syntax" may be regarded as "non-syntax."
We will continue to refine ConGra’s classification in future work.

Societal Impacts.
Our work alerts the software engineering and machine learning communities to the problem of code merging, without obvious negative societal impact.

## 7 Conclusion

We propose ConGra, a complexity-graded conflict benchmarking system.
ConGra implements a highly efficient and accurate conflict classification algorithm to construct a complexity-graded conflict dataset, which is used to evaluate the performance of merging tools under various conflict scenarios.
ConGra utilizes three code matching metrics of different granularities and combines them to calculate the accuracy and precision of auto-generated resolutions.
We evaluate six LLMs on ConGra, and the results show that LLMs with longer context support often perform worse than those with shorter context support, and general LLMs outperform specialized code LLMs in precision.

## Appendix A Prompt

Figure: Figure 8: Prompt
Refer to caption: x2.png

## Appendix B Demo Case

### B.1 Conflict

Figure: Listing 1: Demo 1: text conflict

Figure: Listing 2: Demo 2: text and functional conflict

Figure: Listing 3: Demo 3: syntax conflict

Figure: Listing 4: Demo 4: text, syntax, and functional conflict

### B.2 Ground Truth

Figure: Listing 5: Ground truth of demo 1

Figure: Listing 6: Ground truth of demo 2

Figure: Listing 7: Ground truth of demo 3

Figure: Listing 8: Ground truth of demo 4

### B.3 Resolution generated by LLama3-8B

Figure: Listing 9: Demo 1: LLama3-8B resolution

Figure: Listing 10: Demo 2: LLama3-8B resolution

Figure: Listing 11: Demo 3: LLama3-8B resolution

Figure: Listing 12: Demo 4: LLama3-8B resolution

### B.4 Resolution generated by Deepseek-Chat

Figure: Listing 13: Demo 1: Deepseek-Chat resolution

Figure: Listing 14: Demo 2: Deepseek-Chat resolution

Figure: Listing 15: Demo 3: Deepseek-Chat resolution

Figure: Listing 16: Demo 4: Deepseek-Chat resolution

### B.5 Resolution generated by Deepseek-Coder

Figure: Listing 17: Demo 1: Deepseek-Coder resolution

Figure: Listing 18: Demo 2: Deepseek-Coder resolution

Figure: Listing 19: Demo 3: Deepseek-Coder resolution

Figure: Listing 20: Demo 4: Deepseek-Coder resolution

## Appendix C Git Merge Conflict Example

Figure 9 shows a code merging scenario with conflict.
Both version A (9(a)) and version B ( 9(b)) implement quick sort, but version A selects the middle element of arr as the pivot, while version B selects the first element.
This discrepancy causes Git to encounter an impasse and report a conflict, as depicted in  9(c).

Figure: (a) Quick sort from version A.
Refer to caption: extracted/5852181/images/code_merge_conflict-a.png

## Appendix D License

We utilized the source code of 34 open-source projects in this research. These projects and their license information are as follows:
1) Linux Kernel, Android Kernel, Raspberry Pi Kernel, Git, MySQL, ReactOS, JDK, NewPipe, Ansible Cpython, are licensed under GNU General Public License (GPL);
2) Bitcoin, GCC, Jenkins, are licensed under the Massachusetts Institute of Technology (MIT);
3) LLVM, Swift, Tensorflow, AOSP, dbeaver, Ghidra, hadoop, Micronaut, Netty, Sprint-boot, Sprint-framework, Keras, Transformers, are under the Apache License;
4) Mongo is under Server Side Public License;
5) PHP is under PHP License;
6) V8, Django, Pandas, Pytorch, Scrapy, are under BSD-3-Clause License;
7) Eclipse is under Eclipse Public License;
8) Youtube-dl is unlicensed.
We acknowledge the contributions of the open-source community in developing and maintaining these projects.

## Appendix E Code Similarity Metrics

- •
Edit Similarity (ES). Edit distance is a metric used to measure the difference between two strings (Ristad and Yianilos []).
It is defined as the minimum number of edit operations required to transform one string into another.
In ConGra, we normalize the edit distance, treating the normalized result as the similarity measure in character level.
- •
Winnowing Similarity (WS). Winnowing is an algorithm used for text similarity detection and fingerprint extraction (Schleimer et al. []).
It has been implemented in the Moss code plagiarism detection system (Aiken []) to determine code similarity.
In ConGra, we normalize the winnowing result to assess the similarity of the whole conflict code snippets.
- •
Semantic Similarity (SS). We use the cosine similarity between the generated resolution and the actual resolution as the semantic similarity. In ConGra, in order to effectively model the semantic information of the resolution, we use BCEmbedding NetEase Youdao [] as the embedding model to obtain the representation of each resolution.

## Appendix F Visualization of the number of conflicts in ConGra

Here we visualize the distribution characteristics of the number of conflicts contained in each file in four languages in  Figure 10. Overall, most files contain only one or two conflicts, and a very small number of files contain more than 10 conflicts. We also visualize the number of different types of conflicts for each language in  Figure 11.

Figure: Figure 10: Violin plots of the number of conflicts per file.
Refer to caption: x3.png

Figure: (a) Python
Refer to caption: x4.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - From how many actual (merge) commits are the 44,948 conflict cases?
- What are the pros/cons of handling conflicts individually vs considering all connected to a sigle merge attempt?
- What is the impact of training data on the results?
- What would it take to apply the dataset to the existing automated conflict resolution tools?
soundness: 3
strengths: - Impressive merge conflict data collection established
- Paper establishes that LLMs with chain of thought can do a decent job in resolving merge conflicts
summary: This paper seeks to benchmark the applicability of large language models for the purpose of conflict resolution when merging git commits.
To that end, it proposes a pipeline consisting of:

1. The collection of merge commits with conflicts
2. 'coloring' of individual blocks containing conflicts into six categories (types, function body, function signature, variables, ...)
3. Classifying commits into 3 categories (text, functional, syntactic)
4. Applying six different large language models (each supporting context lenght of at least 8K) to let them merge the underlying conflicting commits
5. Metrics to assess the outcomes (normalized edit distance, winnowing, and cosine similarity)

This pipeline has been applied to 34 large open source projects, resulting in 23,334 conflict files and 44,948 'conflict scenarios' (which I assume are chunks in the code with a conflict).

Findings include that larger context support is not necessarily helpful, nor that code llms have advantages. Llama3-8B and DeepSeek-V2 appear to be the winners, with accuracy/precision hovering around 75% (depending on the programming language). Also, simpler conflict types are not necessarily easier to handle.
weaknesses: In section 3.1 I was confused about the way in which merges are detected. 
It turns out this is explained on page 6 in the first paragraph of section 4.
I think this belongs in section 3, as it is independent of the nine systems chosen.

In Section 5 I was disappointed by the fact that comparison with existing merging tools was considered infeasible. While I can follow the reasoning to some extent, this does undermine the whole purpose of having a benchmark. In particular, the abstract states:

> we introduce CONGRA, a CONflict-GRAded benchmarking scheme designed to evaluate the performance of software merging tools under varying complexity conflict scenarios

I find it odd that existing software merging tools then are not taken into account.

The implicit hypothesis under this paper is that the content of the conflicting commits is enough to make merge decisions. However, this can also be a project-level decision, where e.g., feature A or feature B is considered more important.

Another assumption is that conflicts can be handled individually. But I'd say that choosing between A and B should be consistent -- one cannot pick some changes from A and others from B. The approach proposed doesn't seem to take this into account.

The findings suggest the 'context type' doesn't make much of a difference -- this can also mean that it is pointless to collect this type of information (as it doesn't truly affect the merge decisions).

The paper doesn't contain an analysis of whether the systems studied are part of the training data of the language models used. If so, the language models "know" what the eventual code should look like, which should help them.

Detail: Table 1 would be easier to follow if more meaningful names than Feature_1..6 were used

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Given that the dataset is constructed from publicly available code repositories, how do you address potential bias in evaluation where LLMs may have encountered the "correct" resolution blocks during training?
2. As coding practices evolve, particularly with the growing influence of LLMs as coding assistants, how will CONGRA maintain its relevance as a benchmark since it is based on a static set of questions?
3. Sections 3.2 and 3.3 would benefit from clearer explanation of how the coloring process directly informs the classification methodology. Currently, these sections are not sufficiently clear for readers of all expertise. Could this workflow be elaborated?
4. What were the considerations in limiting the dataset to C/C++, Java, and Python? Is there potential for expansion to other languages such as JavaScript or Rust? Please consider adding this to Future Work.
5. Could the paper include visualizations depicting the distribution of conflicts across languages and conflict types to better illustrate the dataset composition?
6. The experimental setup using temperature=0.7 could be enhanced by either, a) Using near-deterministic settings (temperature~0) or b) Running multiple samples at higher temperatures. This would provide more statistically robust results.
soundness: 2
strengths: 1. The construction of a substantial dataset containing 44,948 conflict cases from 34 real-world projects across multiple programming languages (C, C++, Java, and Python) is a valuable contribution to the field. Using real-world conflict cases from open-source projects is appropriate and lends credibility to the evaluation.
2. It introduces a new method for classifying code merge conflicts based on code operations extracted from syntax trees, enabling a complexity-graded dataset.
3. It provides insights into how different conflict types and contextual information impact the performance of LLMs.
4. The use of traditional metrics like accuracy and precision makes evaluation reliable instead of using automated frameworks like LLM-as-a-Judge.
summary: The paper introduces **CONGRA**, a CONflict-GRAded benchmarking scheme designed to evaluate code merging capabilities of LLMs across varying conflict complexities. It proposes a novel method for classifying code conflicts into seven categories based on code operations, enhancing the understanding of conflict difficulty. Using this approach, the authors build a large-scale dataset of 44,948 conflict cases from 34 real-world projects written in C, C++, Java, and Python. They conduct the first comprehensive evaluation of LLMs on automatic conflict resolution using this dataset, revealing counterintuitive findings:
- LLMs with longer context lengths do not always perform better
- General purpose LLMs can outperform specialized code LLMs.
weaknesses: 1. The paper claims generalist LLMs outperform specialized code LLMs in automatic conflict resolution tasks. However, the code-focused models evaluated are relatively older and potentially less capable than the general-purpose LLMs used. Including recent code-specific LLMs would strengthen the authors' claims and provide a fairer basis for comparison.
2. The authors' explanations for their findings lack sufficient depth and appear speculative. For example, the claim that long-context models underperform because they are not adequately trained on merge conflict datasets is presented without substantial evidence or analysis (line 413). Incorporating more rigorous, possibly statistical, analyses would help substantiate these claims and make the conclusions more convincing.
3. For each task, different models are compared (eg. Table 2, Table 4, Table 6) which complicates direct comparisons of their performance. Populating these tables with the same set of models would help draw more insightful conclusions.
4. The authors use only the preceding and following code lines as context for resolving conflicts. Expanding the context to include cross-file information could potentially yield better results. Implementing methods such as tree-search combined with semantic similarity measures to identify and incorporate the most relevant context across files might improve the models' performance in conflict resolution tasks.
5. The authors propose a new dataset for evaluating automatic conflict resolution, however the dataset itself is not provided in the supplementary material.
---

If my concerns are addressed, I'm happy to revise my score.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What are your criteria for selecting projects? 

2. These machine learning-based ACRs also involved large-scale conflict datasets. Except for the complexity type information, are there any other differences between their dataset and the dataset created in this paper? 

3. In 5.1, the authors explained why they are not compared to the existing baselines. They used RPredictor and MESTRE to illustrate the reasons for not comparing to machine learning-based ACRs. While some other machine learning-based ACRs, such as DeepMerge and MergeGen, are still comparable, as explained in 2.3, these ML-based ACRs are trained to offer resolution suggestions for merge conflicts. Why do authors exclude DeepMerge and MergeGen?
soundness: 2
strengths: + A new dataset for evaluating ACR tools
+ Performance and limitations of existing LLMs on the new data were explored
summary: The work presented CONGRA, a dataset for evaluating automatic conflict resolution (ACR) tools. The authors aim to address limitations in existing benchmarks by introducing a complexity-graded dataset containing ~45k conflict cases from real-world projects. The built dataset includes conflicts categorized by type (text, syntax, and functional), which can be used for evaluating various conflict resolution models under differing complexities.  The authors evaluated several general-purpose and code-specific LLMs on this dataset and reported insights regarding the performance of specialized versus general LLMs.
weaknesses: - motivation of labeling conflict with the complexity scenario info is unclear
- dataset construction lacks transparency about the criteria used for project selection
- complexity type info (e.g., Text, Functional, Syntax) is too coarse-grained to be practical
- missing benchmarking existing machine learning-based ACR (e.g., DeepMerge, MergeGen)
- lack of deep analysis regarding the pros and cons of examined LLMs
- focused on limited syntax problems (i.e., declarations or definitions only)

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Classification of Merge Conflicts: The current classification of merge conflicts feels too restrictive and maybe disconnected from the merge task, appropriate for broader infilling tasks -- what portion of real-world merge conflicts would it cover? I believe the classification should reflect the difficulty of resolution. From this perspective, cases resolved using "take ours" or "take theirs" merge strategies are the simplest. Next are cases that can be resolved by rearranging physical lines of code across the base, incoming, and current versions, followed by those that allow rearrangement of token sequences within a line. Finally, there are cases where developers introduce new tokens not present in the conflicted chunks. This classification respects the unique challenges of program merging, while you could also impose your syntax-based classifications on top.

"Propagation of the large-scale datasets for LM's training and testing." -> Can you correlate it to major LLMs training data cutoffs like what swe bench does? Once the benchmark is established major LLMs will surely respect it and not include it in the training split.

"DeepMerge and MergeBERT are essentially classification models that select answers from candidates, while our paper only focuses on LMs" --> This is not entirely accurate and needs to be corrected. DeepMerge composes resolution based on the line interleaving of the lines present in the incoming, current and baseline code chunks. While MergeBERT is a classifier, it first performs a token-level merge then selects from a set of composable patterns. I think the major point you are making about these two models is that they do not allow new tokens in the resolutions which are not present in the inputs.

"these methods do not consider the contextual content and are divorced from the actual scenario" -> MergeBERT does include surrounding code context lines, up to the max limit of the model's context window, similarly to your setting. These models are not open source afaik which is a good enough reason to not evaluate them in the paper. But the statement about them being "divorced from the actual scenario" is kind of overly bold and also incorrect.

## Minor comments
abstract:
"With the development of language models, researchers treat conflict code as text," -> Even before LLMs, actually, the ubiquitous git merge algorithm handles program merging as a line-level unstructured merge process, treating code purely as text agnostic to the language syntax or grammar. While AST-aware structured program merging algorithms have been developed, I see your point that unstructured text-based merging is more feasible path forward with the rise of LLMs.

introduction
"Git fails to resolve conflicts automatically" -> git does not attempt to merge conflicts automatically, if a 3-way merge attempt fails, the conflict markers would be generated, prompting user to resolve it manually

Main text:
"For LLMs, a simpler conflict does not necessarily mean it is easier to handle" -- what do you mean by "simpler" here?
soundness: 3
strengths: 1) The paper tackles an important software engineering problem -- automated merge conflict resolution
2) The effort promotes open science, by promising to open source the benchmark which could help drive the research in automated program merge
summary: The paper presents CONGRA, a benchmark for program merge conflict resolution. The CONGRA dataset  includes 44,948 merge conflict cases and resolutions collected from 34 real-world projects spanning multiple programming languages. This dataset classifies conflicts into seven types, enabling more granular evaluations. 

The benchmark uses edit distance, winnowing, and cosine similarity to measure the accuracy and precision of conflict resolution results. Authors evaluate six LLMs on the CONGRA benchmark, finding that general-purpose LLMs frequently outperform code-specialized ones in resolving code conflicts and that the models with shorter context windows sometimes achieve better results than those with longer context windows.
weaknesses: The benchmark matching criteria are not optimal, and are not well aligned with the evaluation metrics used in the existing merge conflict research. It appears to be more optimized towards a typical code in-filling task. More specifically, CONGRA regards the resolution candidate matching the ground truth when similarity is greater than 80%. To my opinion, this is a major issue and I see two possible solutions:
1) require a stricter 100% syntactic match (modulo the white space and indentation). A variant of this could also exclude local identifiers defined and used only in the conflict region scope from the similarity calculation, given the freedom in choosing the particular identifier names in this case. Moreover, there is no clear reason to choose 80% similarity threshold -- the numbers in the table 2-3 are already quite high suggesting the benchmark maybe too easy and is at risk of becoming obsolete in the next year or so.
2) better option: verify functional correctness instead. When constructing the benchmark you could select only the conflicts that have unit test coverage for the lines in the conflicted region in the ground truth user-resolved version of the code. In this case, the functional correctness would be verified wrt to the unit tests

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see "weaknesses".
soundness: 2
strengths: +: This paper first investigates the performance of LLMs in resolving software version conflicts.

+: Proposed a conflict dataset in multiple programming languages.
summary: This paper proposes ConGra, a large-scale benchmark for evaluating LLMs on solving conflicts when merging different software versions. The authors built the ConGra dataset in 3 steps: collect Github commits, coloring by program analysis, and classify the different types of conflicts. Then, the authors propose a general pipeline for using LLMs to generate merged code. This paper evaluates 6 open-source LLMs on the ConGra dataset, and reaches several findings: first, LLMs with longer contexts do not always perform better than models with shorter contexts. Second, general-purpose LLMs outperform code LLMs in resolution precision and accuracy.
weaknesses: -: The authors only adopt a few models for evaluation, which makes their conclusion less convincing. For example, it is better to include results for more proprietary models, such as GPT-4o.

-: The authors stated that GPT-4o-mini is only evaluated on the Java subset due to its token limitation, but the context window of GPT-4o-mini is 128K, which is comparable to other models in the evaluation.

-: The finding "general-purpose LLMs outperform code LLMs" may be inaccurate. The code LLMs in this paper, CodeLlama and DeepSeek-Coder, are fine-tuned on older versions of base models than Llama-3 and DeepSeek-V2. To make a fair comparison, the authors should compare these code LLMs with their corresponding base models.