metadata

license: apache-2.0
datasets:
  - JetBrains/KStack-clean
base_model: meta-llama/CodeLlama-7b-hf
results:
  - task:
      type: text-generation
    dataset:
      name: MultiPL-HumanEval (Kotlin)
      type: openai_humaneval
    metrics:
      - name: pass@1
        type: pass@1
        value: 37.89
tags:
  - code

Model description

CodeLlama-7B-KStack-clean model is a fine-tuned open-source generative text model fine-tuned on JetBrains/KStack-clean dataset. This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.

Model use

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load pre-trained model and tokenizer
model_name = 'JetBrains/CodeLlama-7B-KStack-clean'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')

# Create and encode input
input_text = """\
This function takes an integer n and returns factorial of a number:
fun factorial(n: Int): Int {\
"""
input_ids = tokenizer.encode(
    input_text, return_tensors='pt'
).to('cuda')

# Generate
output = model.generate(
    input_ids, max_length=60, num_return_sequences=1, 
    pad_token_id=tokenizer.eos_token_id
)

# Decode output
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

As with the base model, we can use FIM. To do this, the following format must be used:

'<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'

Training setup

The model was trained on one A100 GPU with following hyperparameters:

Hyperparameter	Value
`warmup`	100 steps
`max_lr`	5e-5
`scheduler`	linear
`total_batch_size`	32 (~30K tokens per step)
`num_epochs`	2

More details about finetuning can be found in the technical report

Fine-tuning data

For this model we used 25K exmaples of KStack-clean selected according to educational value for learning algorithms. In total dataset contains about 23M tokens. For more information about the dataset follow the link.

Evaluation

To evaluate we used Kotlin Humaneval (more infromation here)

Fine-tuned model:

Model name	Kotlin HumanEval Pass Rate
`base model`	26.89
`fine-tuned model`	37.89

Ethical Considerations and Limitations

CodeLlama-7B-KStack-clean and its variants are a new technology that carries risks with use. The testing conducted to date could not cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.