File size: 3,928 Bytes
574c97f
 
8edc10e
 
ea71c24
8edc10e
ea71c24
 
 
 
 
 
 
 
 
8edc10e
 
574c97f
 
298c26a
574c97f
7fb4b37
 
298c26a
7fb4b37
298c26a
bff9db4
016356f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4141523
 
016356f
 
 
 
 
298c26a
574c97f
8edc10e
 
 
 
 
574c97f
 
7fb4b37
574c97f
a6f080d
 
 
 
 
 
2ba7b66
a6f080d
7fb4b37
a6f080d
574c97f
 
7fb4b37
574c97f
 
 
7fb4b37
574c97f
7fb4b37
574c97f
8edc10e
 
7fb4b37
 
2ba7b66
7fb4b37
2ba7b66
7fb4b37
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
---
license: apache-2.0
datasets:
- JetBrains/KExercises
base_model: meta-llama/CodeLlama-7b-hf
results:
- task:
    type: text-generation
  dataset:
    name: MultiPL-HumanEval (Kotlin)
    type: openai_humaneval
  metrics:
  - name: pass@1
    type: pass@1
    value: 42.24
tags:
- code
---

# Kexer models

Kexer models are a collection of open-source generative text models fine-tuned on the [Kotlin Exercices](https://huggingface.co/datasets/JetBrains/KExercises) dataset. 
This is a repository for the fine-tuned **CodeLlama-7b** model in the *Hugging Face Transformers* format.

# How to use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load pre-trained model and tokenizer
model_name = 'JetBrains/CodeLlama-7B-Kexer'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')

# Create and encode input
input_text = """\
This function takes an integer n and returns factorial of a number:
fun factorial(n: Int): Int {\
"""
input_ids = tokenizer.encode(
    input_text, return_tensors='pt'
).to('cuda')

# Generate
output = model.generate(
    input_ids, max_length=60, num_return_sequences=1, 
    early_stopping=True, pad_token_id=tokenizer.eos_token_id,
)

# Decode output
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
```

As with the base model, we can use FIM. To do this, the following format must be used: 
```
'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
```

# Training setup

The model was trained on one A100 GPU with the following hyperparameters:

|         **Hyperparameter**           |             **Value**              |
|:---------------------------:|:----------------------------------------:|
|           `warmup`            |           10%            |
|        `max_lr`        |          1e-4          |
|        `scheduler`        |          linear          |
|        `total_batch_size`        |          256 (~130K tokens per step)          |
|        `num_epochs`        |          4          |

More details about fine-tuning can be found in the technical report.

# Fine-tuning data

For tuning this model, we used 15K exmaples from the synthetically generated [Kotlin Exercices dataset](https://huggingface.co/datasets/JetBrains/KExercises). Every example follows the HumanEval format. In total, the dataset contains about 3.5M tokens. 

# Evaluation 

For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).

Here are the results of our evaluation:

|         **Model name**           |             **Kotlin HumanEval Pass Rate**              |
|:---------------------------:|:----------------------------------------:|
|           `CodeLlama-7B`            |           26.89            |
|        `CodeLlama-7B-Kexer`        |          **42.24**         |

# Ethical considerations and limitations

CodeLlama-7B-Kexer is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-Kexer's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-Kexer, developers should perform safety testing and tuning tailored to their specific applications of the model.