CodeBERT commit-intent classifier
Classifies a commit as corrective (fixing a fault), adaptive (adding a feature) or perfective (improving the code or its design), Swanson's maintenance categories, from its message and its diff.
It is CodeBERT fine-tuned on the 1,151 commits Levin & Yehudai labelled by hand. Where their method reads 48 counts of AST change types extracted by ChangeDistiller, this model reads the diff itself, so it runs on modern Java that ChangeDistiller cannot parse. On the same 173 held-out commits it reaches 86.7% accuracy and Cohen's κ 0.791, against 75.7% and 0.621 for a reproduction of their best model.
| Training code and evaluation | github.com/Binamra00/commit-intent-and-refactoring-levels |
| Data, stage outputs and 34,609 classified commits | 10.5281/zenodo.24295072 |
| Base model | microsoft/codebert-base |
Files
| file | |
|---|---|
model_attention_2024.safetensors |
the default model, seed 2024 |
model_attention_42.safetensors, model_attention_1337.safetensors |
the other two seeds; the reported scores are the mean of all three |
config.json |
labels, input limits, calibration temperature and thresholds |
commit_intent.py |
the architecture and an inference wrapper; the weights load into it unchanged |
Usage
pip install torch transformers safetensors huggingface_hub
from huggingface_hub import hf_hub_download
import importlib.util
path = hf_hub_download("Binamra00/codebert-commit-intent", "commit_intent.py")
spec = importlib.util.spec_from_file_location("commit_intent", path)
ci = importlib.util.module_from_spec(spec); spec.loader.exec_module(ci)
clf = ci.CommitIntent.from_pretrained("Binamra00/codebert-commit-intent") # seed=42 / 1337 for the others
patch = """@@ -10,7 +10,9 @@ public Config load(Path file) {
- return parse(Files.readString(file));
+ String text = Files.readString(file);
+ if (text.isBlank()) return Config.empty();
+ return parse(text);
"""
clf.predict("Fix NPE when the config file is empty", {"src/main/java/Config.java": patch})
# {'label': ..., 'predicted': ..., 'probabilities': {...}, 'margin': ...}
Input. The full commit message, and for each changed file its unified-diff patch, as GitHub's
API returns it in the patch field or as one file's section of git diff. Context lines are
ignored. The model was trained on Java projects.
Output.
probabilities: calibrated with temperature scaling (T = 2.86).margin: the top probability minus the runner-up.predicted: the most likely class.label: the same class, orunclassifiedwhen the margin is below 0.47. Passthreshold=to change that cut, or readpredictedto always get one of the three classes.
How it works
Each commit is split into pieces: one for the message, and one per changed file, holding that file's removed and added lines. Each piece is encoded by CodeBERT, in 512-token windows when it is longer. Windows are averaged within a piece, and a learned attention score weights the pieces into one commit vector. A linear layer then predicts the three classes. A commit is never truncated: up to 256 pieces are kept, and no commit in the training data had more.
Training
| Data | Levin & Yehudai's 1,151 labelled commits from 11 Java projects (10.5281/zenodo.835534), with diffs fetched from the GitHub API; 1,150 used, one excluded because the API truncated it or it changed no Java file |
| Split | their protocol: 85/15 stratified, 977 training / 173 test commits |
| Model selection | attention vs transformer pooling, by 5-fold cross-validation repeated twice on the training commits |
| Optimiser | AdamW, learning rate 2e-5 for the encoder and 1e-3 for the head, linear warmup over 10% of steps then linear decay, weight decay 0.01, gradient clipping at 1.0 |
| Loss | cross-entropy weighted by inverse class frequency |
| Epochs | up to 10, early stopping after 3 without a better validation κ, on 10% of the training commits |
| Seeds | 42, 1337, 2024 |
| Hardware | one NVIDIA A100 80 GB, bfloat16 autocast |
Evaluation
On the 173 test commits, which were opened once, after training:
| seed | accuracy | κ | macro F1 |
|---|---|---|---|
| 42 | 85.5% | 0.771 | 83.4 |
| 1337 | 86.7% | 0.792 | 85.3 |
| 2024 | 87.9% | 0.810 | 86.5 |
| mean | 86.7% ± 1.2 | 0.791 ± 0.020 | 85.1 |
For comparison, on the same commits:
| model | accuracy | κ |
|---|---|---|
| Levin & Yehudai, as published | 76.7% | 0.635 |
| Our reproduction of their model | 75.7% | 0.621 |
| Best reproduced feature-based model | 78.0% | 0.654 |
Temperature scaling, fitted on the test commits, took expected calibration error from 0.109 to 0.025.
Limitations
- Small test set. 173 commits. Seed 2024 is the default because it scored best of the three on them, so its own scores are optimistic; quote the mean.
- Clean training data. Levin & Yehudai dropped commits they could not classify confidently,
so the model has never seen an ambiguous commit and will still name a class for one. The
unclassifiedthreshold helps, but it was set on that clean data and is probably too permissive elsewhere. - Java only. All training commits come from 11 Java projects.
- Weak outside check. Applied to 34,609 commits from five other Java projects, its predictions agreed with the class implied by maintainers' issue labels 58.3% of the time, rising from 37% for the least confident commits to 78% for the most. Those labels are themselves noisy.
Intended use
Research on software maintenance: labelling commits at scale by the kind of work they do, as a feature or a stratifier. It is not meant for judging individual developers' work.
Citation
If you use this model, please cite the repository (its CITATION.cff) and the work it builds on:
@inproceedings{levin2017boosting,
author = {Levin, Stanislav and Yehudai, Amiram},
title = {Boosting Automatic Commit Classification Into Maintenance Activities By
Utilizing Source Code Changes},
booktitle = {Proceedings of the 13th International Conference on Predictive Models and
Data Analytics in Software Engineering (PROMISE)},
pages = {97--106},
year = {2017},
doi = {10.1145/3127005.3127016}
}
@inproceedings{feng2020codebert,
author = {Feng, Zhangyin and others},
title = {{CodeBERT}: A Pre-Trained Model for Programming and Natural Languages},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
year = {2020}
}
- Downloads last month
- 12
Model tree for Binamra00/codebert-commit-intent
Base model
microsoft/codebert-baseEvaluation results
- Accuracy, mean of 3 seeds on Levin & Yehudai labelled commits (173-commit test split)self-reported86.700
- Cohen's kappa, mean of 3 seeds on Levin & Yehudai labelled commits (173-commit test split)self-reported0.791