|
--- |
|
license: mit |
|
library_name: transformers |
|
tags: |
|
- code |
|
--- |
|
|
|
## JonBERTa-attn-ft-coco-2L |
|
|
|
Model for the paper [**"A Transformer-Based Approach for Smart Invocation of Automatic Code Completion"**](https://arxiv.org/abs/2405.14753). |
|
|
|
#### Description |
|
This model is fine-tuned on a code-completion dataset collected from the open-source [Code4Me](https://github.com/code4me-me/code4me) plugin. The training objective is to have a small, lightweight transformer model to filter out unnecessary and unhelpful code completions. To this end, we leverage the in-IDE telemetry data, and integrate it with the textual code data in the transformer's attention module. |
|
|
|
- **Developed by:** [AISE Lab](https://www.linkedin.com/company/aise-tudelft/) @ [SERG](https://se.ewi.tudelft.nl/), Delft University of Technology |
|
- **Model type:** [JonBERTa](https://github.com/Ar4l/curating-code-completions/blob/main/modeling_jonberta.py) |
|
- **Language:** Code |
|
- **Finetuned from model:** [`CodeBERTa-small-v1`](https://huggingface.co/huggingface/CodeBERTa-small-v1). |
|
|
|
Models are named as follows: |
|
|
|
- `CodeBERTa` → `CodeBERTa-ft-coco-[1,2,5]e-05lr` |
|
- e.g. `CodeBERTa-ft-coco-2e-05lr`, which was trained with learning rate of `2e-05`. |
|
- `JonBERTa-head` → `JonBERTa-head-ft-[dense,proj,reinit]` |
|
- e.g. `JonBERTa-head-ft-dense-proj`, where all have `2e-05` learning rate, but may differ in the head layer in which the telemetry features are introduced (either `head` or `proj`, with optional `reinit`ialisation of all its weights). |
|
- `JonBERTa-attn` → `JonBERTa-attn-ft-[0,1,2,3,4,5]L` |
|
- e.g. `JonBERTa-attn-ft-012L` , where all have `2e-05` learning rate, but may differ in the attention layer(s) in which the telemetry features are introduced (either `0`, `1`, `2`, `3`, `4`, or `5L`). |
|
|
|
Other hyperparameters may be found in the paper or the replication package (see below). |
|
|
|
#### Sources |
|
|
|
- **Replication Repository:** [`Ar4l/curating-code-completions`](https://github.com/Ar4l/curating-code-completions/tree/main) |
|
- **Paper:** [**"A Transformer-Based Approach for Smart Invocation of Automatic Code Completion"**](https://arxiv.org/abs/2405.14753) |
|
- **Contact:** https://huggingface.co/Ar4l |
|
|
|
To cite, please use |
|
|
|
```bibtex |
|
@misc{de_moor_smart_invocation_2024, |
|
title = {A {Transformer}-{Based} {Approach} for {Smart} {Invocation} of {Automatic} {Code} {Completion}}, |
|
url = {http://arxiv.org/abs/2405.14753}, |
|
doi = {10.1145/3664646.3664760}, |
|
author = {de Moor, Aral and van Deursen, Arie and Izadi, Maliheh}, |
|
month = may, |
|
year = {2024}, |
|
} |
|
``` |
|
|
|
#### Training Details |
|
This model was trained with the following hyperparameters, everything else being `TrainingArguments`' default. The dataset was prepared identically across all models as detailed in the paper. |
|
|
|
```python |
|
num_train_epochs : int = 3 |
|
learning_rate : float = 2e-5 |
|
batch_size : int = 16 |
|
``` |
|
|
|
#### Model Configuration |
|
|
|
```python |
|
num_telemetry_features :int = 26 |
|
|
|
add_feature_embeddings :bool = True |
|
feature_hidden_size :int = num_telemetry_features * 4 |
|
feature_dropout_prob :float = 0.1 |
|
add_feature_bias :bool = True |
|
|
|
add_self_attn :bool = True |
|
self_attn_layers :list[int] = search(sum( |
|
[[i,j,k] for i in range(6) for j in range(6) for k in range(6) if i < j < k], |
|
[[i,j] for j in range(6) for i in range(6) if i < j], |
|
[[i] for i in range(6)], |
|
[] |
|
)) |
|
``` |