Areyde commited on
Commit
545a540
1 Parent(s): 7a39883

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -17
README.md CHANGED
@@ -16,12 +16,11 @@ tags:
16
  - code
17
  ---
18
 
19
- # KStack-full models
20
 
21
- KStack-full models is a collection of fine-tuned open-source generative text models fine-tuned on KStack dataset with rule-based filtering.
22
- This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
23
 
24
- # Model use
25
 
26
  ```python
27
  from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -70,31 +69,31 @@ The model was trained on one A100 GPU with following hyperparameters:
70
  | `total_batch_size` | 128 (~65K tokens per step) |
71
  | `num_epochs` | 1 |
72
 
73
- More details about finetuning can be found in the technical report
74
 
75
- # Data filtering
76
 
77
- To increase the quality of the dataset and filter out statistical outliers such as homework assignments, we filter out the dataset entries according to the following rules:
78
- * We filter out files which belong to the low-popular repos (the sum of stars and forks is less than 6)
79
- * Next, we filter out files which belong to the repos with less than 5 Kotlin files
80
- * Finally, we remove files which have less than 20 SLOC
81
 
82
  We clean the content of the remaining dataset entries according to the following rules:
83
  * We remove all non-ASCII entries
84
- * We remove all package lines such as _package kotlinx.coroutines.channels_
85
- * We remove half of the import lines.
86
 
87
  # Evaluation
88
 
89
- To evaluate we used [Kotlin Humaneval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval)
90
 
91
- Fine-tuned model:
92
 
93
  | **Model name** | **Kotlin HumanEval Pass Rate** |
94
  |:---------------------------:|:----------------------------------------:|
95
- | `base model` | 26.09 |
96
- | `fine-tuned model` | **29.19** |
97
 
98
  # Ethical Considerations and Limitations
99
 
100
- CodeLlama-7B-KStack-full and its variants are a new technology that carries risks with use. The testing conducted to date could not cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack-full's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack-full, developers should perform safety testing and tuning tailored to their specific applications of the model.
 
16
  - code
17
  ---
18
 
19
+ # Model description
20
 
21
+ This is a repository for the **CodeLlama-7b** model fine-tuned on the [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset with rule-based filtering, in the *Hugging Face Transformers* format. KStack is the largest collection of permissively licensed Kotlin code, and so the model is fine-tuned to work better with Kotlin code.
 
22
 
23
+ # How to use
24
 
25
  ```python
26
  from transformers import AutoModelForCausalLM, AutoTokenizer
 
69
  | `total_batch_size` | 128 (~65K tokens per step) |
70
  | `num_epochs` | 1 |
71
 
72
+ More details about fine-tuning can be found in the technical report.
73
 
74
+ # Fine-tuning data
75
 
76
+ For tuning the model, we used the [KStack](https://huggingface.co/datasets/JetBrains/KStack) dataset, the largest collection of permissively licensed Kotlin code. To increase the quality of the dataset and filter out outliers, such as homework assignments, we filter out the dataset entries according to the following rules:
77
+ * We filter out files, which belong to low-popular repos (the sum of stars and forks is less than 6)
78
+ * Next, we filter out files, which belong to repos with less than 5 Kotlin files
79
+ * Finally, we remove files which have fewer than 20 SLOC
80
 
81
  We clean the content of the remaining dataset entries according to the following rules:
82
  * We remove all non-ASCII entries
83
+ * We remove all package lines, such as _package kotlinx.coroutines.channels_
84
+ * We remove half of the import lines
85
 
86
  # Evaluation
87
 
88
+ For evaluation, we used the [Kotlin HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval) dataset, which contains all 161 tasks from HumanEval translated into Kotlin by human experts. You can find more details about the pre-processing necessary to obtain our results, including the code for running, on the [datasets's page](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval).
89
 
90
+ Here are the results of our evaluation:
91
 
92
  | **Model name** | **Kotlin HumanEval Pass Rate** |
93
  |:---------------------------:|:----------------------------------------:|
94
+ | `CodeLlama-7B` | 26.09 |
95
+ | `CodeLlama-7B-KStack` | **29.19** |
96
 
97
  # Ethical Considerations and Limitations
98
 
99
+ CodeLlama-7B-KStack is a new technology that carries risks with use. The testing conducted to date has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, CodeLlama-7B-KStack's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate or objectionable responses to user prompts. The model was fine-tuned on a specific data format (Kotlin tasks), and deviation from this format can also lead to inaccurate or undesirable responses to user queries. Therefore, before deploying any applications of CodeLlama-7B-KStack, developers should perform safety testing and tuning tailored to their specific applications of the model.