Hugging Face's logo

MBZUAI
/
bactrian-x-llama-7b-lora

Model card Files Community
haonan-li commited on
Commit
c4c541b
•
1 Parent(s): cbe5dbd

update model

Browse files
Files changed (3) hide show
  1. README.md +5 -3
  2. adapter_config.json +2 -2
  3. adapter_model.bin +1 -1
README.md CHANGED
@@ -2,6 +2,9 @@
2
  license: mit
3
  ---
4
 
 
 
 
5
  This repo contains a low-rank adapter (LoRA) for LLaMA-7b
6
  fit on the [Stanford-Alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca)
7
  and [databricks-dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data) data in 52 languages.
@@ -29,7 +32,6 @@ This version of the weights was trained with the following hyperparameters:
29
  - Lora _r_: 64
30
  - Lora target modules: q_proj, k_proj, v_proj, o_proj
31
 
32
- #### Current Training Steps: 21000
33
 
34
  That is:
35
 
@@ -40,7 +42,7 @@ python finetune.py \
40
  --batch_size=128 \
41
  --cutoff_len=512 \
42
  --group_by_length \
43
- --output_dir='./bactrian-x-7b-lora' \
44
  --lora_target_modules='q_proj,k_proj,v_proj,o_proj' \
45
  --lora_r=64 \
46
  --micro_batch_size=32
@@ -57,7 +59,7 @@ Instructions for running it can be found at https://github.com/MBZUAI-nlp/Bactri
57
 
58
  ```
59
  @misc{bactrian,
60
- author = {Haonan Li and Fajri Koto and Timothy Baldwin},
61
  title = {Bactrian-X: A Multilingual Replicable Instruction-Following Model},
62
  year = {2023},
63
  publisher = {GitHub},
 
2
  license: mit
3
  ---
4
 
5
+ #### Current Training Steps: 40000
6
+
7
+
8
  This repo contains a low-rank adapter (LoRA) for LLaMA-7b
9
  fit on the [Stanford-Alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca)
10
  and [databricks-dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data) data in 52 languages.
 
32
  - Lora _r_: 64
33
  - Lora target modules: q_proj, k_proj, v_proj, o_proj
34
 
 
35
 
36
  That is:
37
 
 
42
  --batch_size=128 \
43
  --cutoff_len=512 \
44
  --group_by_length \
45
+ --output_dir='./bactrian-x-llama-7b-lora' \
46
  --lora_target_modules='q_proj,k_proj,v_proj,o_proj' \
47
  --lora_r=64 \
48
  --micro_batch_size=32
 
59
 
60
  ```
61
  @misc{bactrian,
62
+ author = {Haonan Li and Fajri Koto and Minghao Wu and Alham Fikri Aji and Timothy Baldwin},
63
  title = {Bactrian-X: A Multilingual Replicable Instruction-Following Model},
64
  year = {2023},
65
  publisher = {GitHub},
adapter_config.json CHANGED
@@ -10,10 +10,10 @@
10
  "peft_type": "LORA",
11
  "r": 64,
12
  "target_modules": [
13
- "[q_proj",
14
  "k_proj",
15
  "v_proj",
16
- "o_proj]"
17
  ],
18
  "task_type": "CAUSAL_LM"
19
  }
 
10
  "peft_type": "LORA",
11
  "r": 64,
12
  "target_modules": [
13
+ "q_proj",
14
  "k_proj",
15
  "v_proj",
16
+ "o_proj"
17
  ],
18
  "task_type": "CAUSAL_LM"
19
  }
adapter_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a79d74d6cfed583c0a176438158a983133a2016ed923d25347e4335d95c7aab8
3
  size 268527949
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dabb634e58bfa65d99b894b6f5a390af86b12bf57246518b9a4eddaffdf471fc
3
  size 268527949