Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses merge-conflict resolution in source code repositories. Whereas prior neural work (DeepMerge) can only do line-level resolutions (i.e., predict a resolution that chooses some sequence of entire lines from the original file version or the two divergent branches), this work refines the task further towards token level resolutions, thereby covering more interesting cases. Whereas DeepMerge cast the task as a pointer-sequence prediction (predicted a sequence of lines from the input), MergeBERT casts the task as a classification problem: for each conflict instance (some token-sequence conflict), one of 9 recipes are chosen to merge (e.g., just the original tokens, just one of the branches, some concatenation of the three, and some substitution of tokens in the original by concatenations). Similarly to DeepMerge, the input is encoded as a combination of aligned token sequences of the original snapshots and the divergent branches, as well as an encoding of the token-wise edit "script" for the pairwise diffs (same, inserted, deleted, replaced). But instead of encoding this edit sequence explicitly, MergeBERT uses it as a token type embedding in a Transformer encoder, along with a positional embedding and the token embedding itself.

The Transformer encoder is pre-trained with the masked language model objective, before fine-tuning on merge-conflict resolutions. The resulting tool is shown to resolve more merge conflicts than prior work, and also to generalize to a language unseen during pre-training or fine-tuning.
main_review: This is an important problem, and I'm excited to see more work in finding a neural solution to it.

However, this submission seems more like a refinement of DeepMerge than a novel contribution on its own. The data representation for conflicts is a generalization from DeepMerge, and the model architecture is a straighforward form of a Transformer-based classifier. However, the limited novelty might be offset by the positive progress on this important problem, modulo a number of questions below.

I wonder if there's room to incorporate the strategies of syntactic or lexical merging employed by FSTMerge and its variants and other non-neural baselines into the actions predicted by something like MergeBERT. It seems excessive to throw out the engineering that went into such non-neural tools, if they can be used for something more sophisticated than o, a, b, oa, ob, etc. Q1: I'd be curious if you think the strategies in those earlier tools are incompatible with an encoder like your Transformer-based encoder, too coarse grained, or perhaps just too complicated.

That said, I like the data-driven design of the task (e.g., the merge resolution "recipes" described in section 4). I would have liked to see where the 9 edit patterns came from though (presumably some analysis of real conflict resolutions?), since otherwise they appear arbitrary. Q2: are your classification labels arbitrary? Did you choose them by analyzing actual merge-conflict resolutions?

## Non-conflicting token edits

My biggest concern with your work is that you seem to treat token conflicts as independent of each other, but of course they are most certainly not, in general. For example, on page 3, with respect to Figure 1, I'd be careful about calling a token with no direct conflict a "non-conflicting edit" (e.g., the statement on `var` versus `let`). Consider the following diff:
```
<<<<<<<< A
a = func2(y, 10)
|||||||| O
a = func2(y, 9)
========
a = func3(y, 10, 12)
>>>>>>>> B
```
where `func2` is a function with arity 2, and `func3` is a function with arity 3. Here, although the `func2` to `func3` change on `B` is "non-conflicting", it is not independent of the change from `9)` to `10)` or `10, 12)`, since one keeps the arity at 2 and the other changes it to 3. Perhaps a model can learn these interdependencies but your model does not. I'd caution you against calling independent token changes "non-conflicting". Maybe there's a better motivating example than this one, or again, perhaps there's some analysis of real conflict resolutions showing that real conflicts are indeed token-independent.

## Evaluation

I don't quite understand your definition of precision and recall. Is precision perfect accuracy modulo whitespace per conflict region? What's recall? Why is it not 100%? Do you have a threshold on classification probability you're using that's causing you to lose some examples? It seems you have one interpretation of recall for diff3 (it cannot resolve the conflict) and possibly another for MergeBERT? Q3: Please explain your metrics with specificity.

It's not obvious how BLEU-4 is a relevant metric. In cases where token-level conflict resolution kicks in, A or B or even O probably share many n-grams with the resolution. What kind of solution does BLEU-4 allow you to score positively that would be genuinely acceptable for this task? An example would be helpful here. Q4: Please explain why BLEU-4 is a good metric for this task, perhaps with an example.

The zero-shot generalization to Scala is puzzling. Presumably the value proposition here is to argue that you don't have to pre-train for every language, and a single strong (potentially multilingual) model can do the job. But then I would want to see the comparison of pre-training/fine-tuning on Scala versus the zero-shot version with neither pre-training nor fine-tuning. In the absence of that, it's not clear what the headroom is for this language. Is the result good? Is it bad? Maybe pre-training and fine-tuning on Scala gives super high F1. Q5: How does MergeBERT do if you pre-train on Scala and fine-tune on Scala, and how does that compare to your zero-shot results?

I don't understand the point of Section 8.2. You already do MLM pre-training on your unilingual and multilingual pre-training datasets. What does this section add to what you already do? Q6: Please explain what research question Section 8.2 is answering, and what the answer is.


summary_of_the_review: Important problem, but there are many unanswered questions making the submission feel incomplete.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents MergeBERT, a deep model for resolving program merge conflicts in software development. The authors introduce a new, hierarchical differencing and cast the problem as classifying over a fixed set of merge patterns, instead of generating . The model is pre-trained on a large corpus of GitHub code.
main_review: Pros:

+ Hierarchical two-level differencing appears to be an intuitive and good idea.
+ Casting the problem as classification over a set of merge patterns appears advantageous in both learning and computation, since most of the time the resolution comes from either of the two change versions.
+ Pre-training can be applied directly.

Cons:

- This idea is somewhat incremental, largely based on the work of Dinella et al., 2020.
- MergeBERT is not as good as *diff3* in terms of precision, which is to me a more important measure. 
- Some parts of the techniques and experiments are not clear, for example the class label and data split. I have more detailed questions below.


Questions:

1. It is somewhat unintuitive that `let` is not part of the token-level conflict. I understand that `var` is not changed in the version B, but it is important from both A and B perspectives that there is a change that needs an agreement in resolution. In that case, I don’t think merging `let` right away is an indisputable decision.

2. For this change from `--num_cores=2` to `--max_length=256 --num_cores=2`, what is the result of the two-level diff?

3. Is treating conflicting regions independently in Eq. (2) is oversimplified?

4. How do you split the train and test dataset (Table 4)? Is there a validation dataset?

5. How do you deal with conflicts from more than 2 versions?


summary_of_the_review: Overall, the paper presents several new technical contributions and insights, but some parts are not entirely convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is about using BERT for the automatic resolution of merge conflicts. The main idea is to cast the problem of automatic merge resolution as a classification problem with 9 classes (token-level merge patterns). The technique use token-level differencing to represent the input to the merge problem as four aligned token sequences, together with two edit-type embeddings. The input sequences are fed to BERT and the results are aggregated and used for classification. 
The technique is compared to state-of-the-art automatic-merge tools and is shown to outperform them in all benchmarks. 

main_review: Pros:
+ This is a compelling task, merge conflicts are an important problem
+ Reducing the problem to using a classifier is valuable practically 
+ The nine primitive merge resolution patterns covering 99% of merge cases is an interesting observation

Cons:
+ The encoding is token-based and rather standard
+ Missing ablation study
+ Experimental evaluation partial with respect to most important baseline 

* Tackling the merge conflicts problem is interesting. Using a learned model to decide how to resolve it is interesting as well (but not novel), since, it’s not always clear what is the best resolution (which is consistent with the context).

* It’s not clear how the model looks like, Do you train BERT only for word vectors? If so, what about exploiting the context? Is the encoder of the aligned sequences the pre-trained BERT? It looks like you are not finetuning it (based on the blue squares of Figure 2, showing that only the Edit Type Embeddings are finetuned), so how should it handle the new kind of inputs? (token embedding + type embedding + position embeddings)

* How important is the edit type embedding used in finetuning? The paper refers to Figure 5 in the appendix that shows how these are used, but not how important are they for the final result. In general, the paper would benefit from a more extensive ablation study. 

* The comparison with DeepMerge includes cases where DeepMerge cannot produce a prediction, as these cases are not line-level merges. This is not an apples-to-apples comparison. What would happen if we compare DeepMerge/MergeBert considering only line-level merges?

* Comparisons to JDime and jsFSTMerge are useful for showing the superiority of your neural approach, but I am more interested in understanding what makes your approach work well.

* Table 3 is pretty much due to properties of BERT and not of MergeBERT, right? Similar experience reported in CodeBERT?

* Primitive Merge Resolution Types: Appendix 10.1 seems to emphasize this, but I do have some comments:
"Our analysis shows that over 99% of all the merge conflicts can be represented using these labels"
But FIgure 3 (left) says that you failed to label 21.64% of the TypeScript examples. Can you please elaborate on this point?
It’s not clear how the colors in Figure 3 are related to your 9 classes. Does “Remove base” correspond to classes 6-9?, is so, what does pink (“Other”) refer to?
Please align the legends on both sides of Figure 3.
I would emphasize that there are almost 21.64% non-trivial examples (in TypeScript), which the token-level diff method can solve by using a classification approach.

* It would be helpful to add an encoder-decoder baseline - given (A, B, O), generate the resolution.

* Table 1- what is the meaning of the diff3 row? I guess it’s for the examples where the token granularity diff managed to solve conflicts the line-diff couldn’t. Please clarify that. 

* Table 1 - what about using BERTMerge with line diff and DeepMerge with token diff? It’s essential for a fair comparison.

* Section 8.2: The impact of pretraining is not disputed, but again it is not a contribution of the technique presented here. 

Minor: 

* Figure 2: should all edit operations in the blue squares be <->? 

* Figure 7(b) needs to be corrected: `data +=` is also a conflict, and there is a piece of code after the last conflict - ``});`.

summary_of_the_review: The problem of automatic merge resolution is compelling. The authors present a solution that is based on token-level diff (no ML contribution), identify 9 merge patterns (no ML contribution), and finally use BERT with minor adaptations to classify which merge pattern should be used. For this model, some details of what exactly is being fine-tuned are not clear. 

The suggested approach shows improvement compared to existing baselines. 

The experimental evaluation is hard to follow because neither of the existing baselines addresses the exact same kind of merges. The closest baseline in terms of technique is DeepMerge, and the comparison to that baseline is quite partial. 

Overall, this may be a great software-engineering contribution, but I'm afraid that there's not a lot here for ICLR.
review_title: null

Meta-review: This paper is a fair effort, making some headway on a problem of practical importance. 
There was some discussion of scoping and whether the contribution was Machine-Learning-y enough. 
I'm kind of ambivalent on that particular question: I think the general rule is that the further out-of-scope the paper seems, the better the results need to be for people to overlook it. 
I think in this case, unfortunately, even the two most positive reviewers did not evince enough excitement about this paper for it to get accepted in light of the scoping concerns. 
Given the various constraints involved, I don't think I can recommend acceptance.

In order to get it accepted into a future conference I would recommend either:
a) Submit to a more Software-Engineering focused venue
b) Really shore up the evaluation such that the reviewers sympathetic to this kind of paper will find it unimpeachable and score it more generously.
Decision: Reject