Titovs mshny commited on
Commit
7700af1
1 Parent(s): c44895d

Update README.md (#2)

Browse files

- Update README.md (91c39340f3377857bbbee0e40edfd5c2150c7396)


Co-authored-by: Mikhail Arkhipov <mshny@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +92 -3
README.md CHANGED
@@ -1,3 +1,92 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - JetBrains/KStack-clean
5
+ base_model: meta-llama/CodeLlama-7b-hf
6
+ results:
7
+ - task:
8
+ type: text-generation
9
+ dataset:
10
+ name: MultiPL-HumanEval (Kotlin)
11
+ type: openai_humaneval
12
+ metrics:
13
+ - name: pass@1
14
+ type: pass@1
15
+ value: 37.89
16
+ tags:
17
+ - code
18
+ ---
19
+
20
+ # Model description
21
+
22
+ CodeLlama-7B-KStack-clean model is a fine-tuned open-source generative text model fine-tuned on [JetBrains/KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) dataset.
23
+ This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
24
+
25
+ # Model use
26
+
27
+ ```python
28
+ from transformers import AutoModelForCausalLM, AutoTokenizer
29
+
30
+ # Load pre-trained model and tokenizer
31
+ model_name = 'JetBrains/CodeLlama-7B-KStack-clean'
32
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
33
+ model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
34
+
35
+ # Create and encode input
36
+ input_text = """\
37
+ This function takes an integer n and returns factorial of a number:
38
+ fun factorial(n: Int): Int {\
39
+ """
40
+ input_ids = tokenizer.encode(
41
+ input_text, return_tensors='pt'
42
+ ).to('cuda')
43
+
44
+ # Generate
45
+ output = model.generate(
46
+ input_ids, max_length=60, num_return_sequences=1,
47
+ pad_token_id=tokenizer.eos_token_id
48
+ )
49
+
50
+ # Decode output
51
+ generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
52
+ print(generated_text)
53
+ ```
54
+
55
+ As with the base model, we can use FIM. To do this, the following format must be used:
56
+ ```
57
+ '<PRE> ' + prefix + ' <SUF> ' + suffix + ' <MID>'
58
+ ```
59
+
60
+ # Training setup
61
+
62
+ The model was trained on one A100 GPU with following hyperparameters:
63
+
64
+ | **Hyperparameter** | **Value** |
65
+ |:---------------------------:|:----------------------------------------:|
66
+ | `warmup` | 100 steps |
67
+ | `max_lr` | 5e-5 |
68
+ | `scheduler` | linear |
69
+ | `total_batch_size` | 32 (~30K tokens per step) |
70
+ | `num_epochs` | 2 |
71
+
72
+ More details about finetuning can be found in the technical report
73
+
74
+ # Fine-tuning data
75
+
76
+ For this model we used 25K exmaples of [KStack-clean](https://huggingface.co/datasets/JetBrains/KStack-clean) selected according to educational value for learning algorithms. In total dataset contains about 23M tokens.
77
+ For more information about the dataset follow the link.
78
+
79
+ # Evaluation
80
+
81
+ To evaluate we used Kotlin Humaneval ([more infromation here](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval))
82
+
83
+ Fine-tuned model:
84
+
85
+ | **Model name** | **Kotlin HumanEval Pass Rate** |
86
+ |:---------------------------:|:----------------------------------------:|
87
+ | `base model` | 26.89 |
88
+ | `fine-tuned model` | 37.89 |
89
+
90
+ # Ethical Considerations and Limitations
91
+
92
+ CodeLlama-7B-KStack-clean and its variants are a new technology that carries risks with use. The testing conducted to date could not cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-clean potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-clean, developers should perform safety testing and tuning tailored to their specific applications of the model.