File size: 6,130 Bytes
fe0c877 11500b8 fe0c877 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
---
license: apache-2.0
datasets:
- JetBrains/KStack-clean
base_model: meta-llama/CodeLlama-7b-hf
results:
- task:
type: text-generation
dataset:
name: MultiPL-HumanEval (Kotlin)
type: openai_humaneval
metrics:
- name: pass@1
type: pass@1
value: 37.89
tags:
- code
---
**Exllamav2** quant (**exl2** / **4.0 bpw**) made with ExLlamaV2 v0.0.21
Other EXL2 quants:
| **Quant** | **Model Size** | **lm_head** |
| ----- | ---------- | ------- |
|<center>**[2.2](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-2_2bpw_exl2)**</center> | <center>2055 MB</center> | <center>6</center> |
|<center>**[2.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-2_5bpw_exl2)**</center> | <center>2279 MB</center> | <center>6</center> |
|<center>**[3.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_0bpw_exl2)**</center> | <center>2663 MB</center> | <center>6</center> |
|<center>**[3.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_5bpw_exl2)**</center> | <center>3047 MB</center> | <center>6</center> |
|<center>**[3.75](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-3_75bpw_exl2)**</center> | <center>3244 MB</center> | <center>6</center> |
|<center>**[4.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-4_0bpw_exl2)**</center> | <center>3437 MB</center> | <center>6</center> |
|<center>**[4.25](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-4_25bpw_exl2)**</center> | <center>3629 MB</center> | <center>6</center> |
|<center>**[5.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-5_0bpw_exl2)**</center> | <center>4209 MB</center> | <center>6</center> |
|<center>**[6.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-6_0bpw_exl2)**</center> | <center>5006 MB</center> | <center>8</center> |
|<center>**[6.5](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-6_5bpw_exl2)**</center> | <center>5383 MB</center> | <center>8</center> |
|<center>**[8.0](https://huggingface.co/Zoyd/JetBrains_CodeLlama-7B-KStack-clean-8_0bpw_exl2)**</center> | <center>6176 MB</center> | <center>8</center> |
# Model description
This is a repository for the **CodeLlama-7b** model fine-tuned on the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset with rule-based filtering, in the *Hugging Face Transformers* format. KStack-clean is a small subset of [KStack](https://huggingface.co/datasets/JetBrains/KStack), the largest collection of permissively licensed Kotlin code, automatically filtered to include files that have the highest "educational value for learning algorithms in Kotlin".
# How to use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load pre-trained model and tokenizer
model_name = 'JetBrains/CodeLlama-7B-KStack-clean'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
# Create and encode input
input_text = """\
This function takes an integer n and returns factorial of a number:
fun factorial(n: Int): Int {\
"""
input_ids = tokenizer.encode(
input_text, return_tensors='pt'
).to('cuda')
# Generate
output = model.generate(
input_ids, max_length=60, num_return_sequences=1,
pad_token_id=tokenizer.eos_token_id
)
# Decode output
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
```
As with the base model, we can use FIM. To do this, the following format must be used:
```
'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
```
# Training setup
The model was trained on one A100 GPU with following hyperparameters:
| **Hyperparameter** | **Value** |
|:---------------------------:|:----------------------------------------:|
| `warmup` | 100 steps |
| `max_lr` | 5e-5 |
| `scheduler` | linear |
| `total_batch_size` | 32 (~30K tokens per step) |
| `num_epochs` | 2 |
More details about fine-tuning can be found in the technical report (coming soon!).
# Fine-tuning data
For tuning the model, we used 25K exmaples from the [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset, selected from the larger [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset according to educational value for learning algorithms. In total, the dataset contains about 23M tokens.
# Evaluation
For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).
Here are the results of our evaluation:
| **Model name** | **Kotlin HumanEval Pass Rate** |
|:---------------------------:|:----------------------------------------:|
| `CodeLlama-7B` | 26.89 |
| `CodeLlama-7B-KStack-clean` | **37.89** |
# Ethical Considerations and Limitations
CodeLlama-7B-KStack-clean is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model. |