File size: 2,181 Bytes
8576059 1fcdb2a 8576059 1fcdb2a 8576059 1fcdb2a 8576059 060754b 8576059 c01d315 8576059 fc02f17 8576059 1fcdb2a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
---
license: llama2
model-index:
- name: ETRI_CodeLLaMA_7B_CPP
results:
- task:
type: text-generation
dataset:
type: HumanEval-X
name: humanevalsynthesize-cpp
metrics:
- name: pass@1
type: pass@1
value: 34.3%
verified: false
---
## **ETRI_CodeLLaMA_7B_CPP**
We used LoRa to further pre-train Meta's CodeLLaMA-7B-hf model with high-quality C++ code tokens.
Furthermore, we was fine-tuned on CodeM's C++ instruction data.
## Model Details
This model was trained using LoRa and achieved a pass@1 of 34.3% on HumanEvalX-cpp.
ETRI_CodeLLaMA_7B_CPP is a C++ specialized model.
## Dataset Details
We pre-trained CodeLLaMA-7B further using 543 GB of C++ code collected online, and fine-tuned it using CodeM's C++ instruction data. We utilized 1 x A100-80GB GPU for the training.
## Requirements
```
peft==0.3.0.dev0
tokenizers==0.13.3
transformers==4.33.0
bitsandbytes==0.41.1
```
## How to reproduce HumanEval-X results
We use Bigcode-evaluation-harness repo for evaluating our trained model.
bigcode-evaluation-harness
```
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
```
Then, run main.py as follows.
```
accelerate launch bigcode-evaluation-harness/main.py \
--model DDIDU/ETRI_CodeLLaMA_7B_CPP \
--max_length_generation 512 \
--prompt continue \
--tasks humanevalsynthesize-cpp \
--temperature 0.2 \
--n_samples 100 \
--precision bf16 \
--do_sample True \
--batch_size 10 \
--allow_code_execution \
--save_generations \
```
## Model use
```
from transformers import AutoTokenizer
import transformers
import torch
model = "DDIDU/ETRI_CodeLLaMA_7B_CPP"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.bfloat16,
device_map="auto",
)
sequences = pipeline(
'import socket\n\ndef ping_exponential_backoff(host: str):',
do_sample=True,
top_k=10,
temperature=0.1,
top_p=0.95,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
max_length=200,
)
for seq in sequences:
print(f"Result: {seq['generated_text']}")
``` |