CodeBERT commit-intent classifier

Classifies a commit as corrective (fixing a fault), adaptive (adding a feature) or perfective (improving the code or its design), Swanson's maintenance categories, from its message and its diff.

It is CodeBERT fine-tuned on the 1,151 commits Levin & Yehudai labelled by hand. Where their method reads 48 counts of AST change types extracted by ChangeDistiller, this model reads the diff itself, so it runs on modern Java that ChangeDistiller cannot parse. On the same 173 held-out commits it reaches 86.7% accuracy and Cohen's κ 0.791, against 75.7% and 0.621 for a reproduction of their best model.

Training code and evaluation github.com/Binamra00/commit-intent-and-refactoring-levels
Data, stage outputs and 34,609 classified commits 10.5281/zenodo.24295072
Base model microsoft/codebert-base

Files

file
model_attention_2024.safetensors the default model, seed 2024
model_attention_42.safetensors, model_attention_1337.safetensors the other two seeds; the reported scores are the mean of all three
config.json labels, input limits, calibration temperature and thresholds
commit_intent.py the architecture and an inference wrapper; the weights load into it unchanged

Usage

pip install torch transformers safetensors huggingface_hub
from huggingface_hub import hf_hub_download
import importlib.util

path = hf_hub_download("Binamra00/codebert-commit-intent", "commit_intent.py")
spec = importlib.util.spec_from_file_location("commit_intent", path)
ci = importlib.util.module_from_spec(spec); spec.loader.exec_module(ci)

clf = ci.CommitIntent.from_pretrained("Binamra00/codebert-commit-intent")   # seed=42 / 1337 for the others

patch = """@@ -10,7 +10,9 @@ public Config load(Path file) {
-        return parse(Files.readString(file));
+        String text = Files.readString(file);
+        if (text.isBlank()) return Config.empty();
+        return parse(text);
"""
clf.predict("Fix NPE when the config file is empty", {"src/main/java/Config.java": patch})
# {'label': ..., 'predicted': ..., 'probabilities': {...}, 'margin': ...}

Input. The full commit message, and for each changed file its unified-diff patch, as GitHub's API returns it in the patch field or as one file's section of git diff. Context lines are ignored. The model was trained on Java projects.

Output.

  • probabilities: calibrated with temperature scaling (T = 2.86).
  • margin: the top probability minus the runner-up.
  • predicted: the most likely class.
  • label: the same class, or unclassified when the margin is below 0.47. Pass threshold= to change that cut, or read predicted to always get one of the three classes.

How it works

Each commit is split into pieces: one for the message, and one per changed file, holding that file's removed and added lines. Each piece is encoded by CodeBERT, in 512-token windows when it is longer. Windows are averaged within a piece, and a learned attention score weights the pieces into one commit vector. A linear layer then predicts the three classes. A commit is never truncated: up to 256 pieces are kept, and no commit in the training data had more.

Training

Data Levin & Yehudai's 1,151 labelled commits from 11 Java projects (10.5281/zenodo.835534), with diffs fetched from the GitHub API; 1,150 used, one excluded because the API truncated it or it changed no Java file
Split their protocol: 85/15 stratified, 977 training / 173 test commits
Model selection attention vs transformer pooling, by 5-fold cross-validation repeated twice on the training commits
Optimiser AdamW, learning rate 2e-5 for the encoder and 1e-3 for the head, linear warmup over 10% of steps then linear decay, weight decay 0.01, gradient clipping at 1.0
Loss cross-entropy weighted by inverse class frequency
Epochs up to 10, early stopping after 3 without a better validation κ, on 10% of the training commits
Seeds 42, 1337, 2024
Hardware one NVIDIA A100 80 GB, bfloat16 autocast

Evaluation

On the 173 test commits, which were opened once, after training:

seed accuracy κ macro F1
42 85.5% 0.771 83.4
1337 86.7% 0.792 85.3
2024 87.9% 0.810 86.5
mean 86.7% ± 1.2 0.791 ± 0.020 85.1

For comparison, on the same commits:

model accuracy κ
Levin & Yehudai, as published 76.7% 0.635
Our reproduction of their model 75.7% 0.621
Best reproduced feature-based model 78.0% 0.654

Temperature scaling, fitted on the test commits, took expected calibration error from 0.109 to 0.025.

Limitations

  • Small test set. 173 commits. Seed 2024 is the default because it scored best of the three on them, so its own scores are optimistic; quote the mean.
  • Clean training data. Levin & Yehudai dropped commits they could not classify confidently, so the model has never seen an ambiguous commit and will still name a class for one. The unclassified threshold helps, but it was set on that clean data and is probably too permissive elsewhere.
  • Java only. All training commits come from 11 Java projects.
  • Weak outside check. Applied to 34,609 commits from five other Java projects, its predictions agreed with the class implied by maintainers' issue labels 58.3% of the time, rising from 37% for the least confident commits to 78% for the most. Those labels are themselves noisy.

Intended use

Research on software maintenance: labelling commits at scale by the kind of work they do, as a feature or a stratifier. It is not meant for judging individual developers' work.

Citation

If you use this model, please cite the repository (its CITATION.cff) and the work it builds on:

@inproceedings{levin2017boosting,
  author    = {Levin, Stanislav and Yehudai, Amiram},
  title     = {Boosting Automatic Commit Classification Into Maintenance Activities By
               Utilizing Source Code Changes},
  booktitle = {Proceedings of the 13th International Conference on Predictive Models and
               Data Analytics in Software Engineering (PROMISE)},
  pages     = {97--106},
  year      = {2017},
  doi       = {10.1145/3127005.3127016}
}

@inproceedings{feng2020codebert,
  author    = {Feng, Zhangyin and others},
  title     = {{CodeBERT}: A Pre-Trained Model for Programming and Natural Languages},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year      = {2020}
}
Downloads last month
12
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Binamra00/codebert-commit-intent

Finetuned
(158)
this model

Evaluation results

  • Accuracy, mean of 3 seeds on Levin & Yehudai labelled commits (173-commit test split)
    self-reported
    86.700
  • Cohen's kappa, mean of 3 seeds on Levin & Yehudai labelled commits (173-commit test split)
    self-reported
    0.791