ahxt commited on
Commit
ad3b4ce
1 Parent(s): dc660c4

first commit

Browse files
.gitattributes copy ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,93 @@
1
  ---
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - llama2
6
+ - llama-2
7
+ - llama
8
+ - llama2 architecture
9
+ - litellama
10
+ datasets:
11
+ - Redpajama
12
+ metrics:
13
+ - MMLU
14
  license: mit
15
  ---
16
+
17
+ # LiteLLama: Reduced-Scale, Experimental Versions of Llama
18
+
19
+ In this series of repos, we present an open-source reproduction of Meta AI's [LLaMA](https://ai.meta.com/blog/large-language-model-llama-meta-ai/) and [LLaMa 2](https://ai.meta.com/llama/) large language models. However, with significantly reduced model sizes, the experimental version of [llama1_s](https://huggingface.co/ahxt/llama1_s_1.8B_experimental) has 1.8B parameters, and the experimental version of [llama2_xs](https://huggingface.co/ahxt/llama2_xs_460M_experimental) has 460M parameters. ('s' stands for small, while 'xs' denotes extra small).
20
+
21
+
22
+ ## Dataset and Tokenization
23
+ We train our models on part of [RedPajama](https://www.together.xyz/blog/redpajama) dataset. We use the [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/gpt2#transformers.GPT2Tokenizer) to tokenize the text.
24
+
25
+ ## Training Details
26
+
27
+ The model was trained with ~1T tokens (0.98T). num of tokens = steps*length*batch_size=499679*1024*192=98240888832≈0.98T.
28
+
29
+ The training curve is at https://wandb.ai/ahxt/llama2_xs_460M_training_loss/reports/reduced_train_loss-23-09-05-20-25-43---Vmlldzo1MzIwNDUx?accessToken=x2ch3n30jo77p1x8y7q9js4h4d8zpjtz1tzot4xxullyefixp4jwt7au2q37k2q6
30
+
31
+ ### Using with HuggingFace Transformers
32
+ The experimental checkpoints can be directly loaded by [Transformers](https://huggingface.co/transformers/) library. The following code snippet shows how to load the our experimental model and generate text with it.
33
+
34
+
35
+ ```python
36
+ import torch
37
+ from transformers import AutoTokenizer, AutoModelForCausalLM
38
+
39
+ model_path = 'ahxt/LiteLlama-460M-1T'
40
+
41
+ model = AutoModelForCausalLM.from_pretrained(model_path)
42
+ tokenizer = AutoTokenizer.from_pretrained(model_path)
43
+ model.eval()
44
+
45
+ prompt = 'Q: What is the largest bird?\nA:'
46
+ input_ids = tokenizer(prompt, return_tensors="pt").input_ids
47
+ tokens = model.generate(input_ids, max_length=20)
48
+ print( tokenizer.decode(tokens[0].tolist(), skip_special_tokens=True) )
49
+ # Q: What is the largest bird?\nA: The largest bird is the bald eagle.
50
+ ```
51
+
52
+ ## Evaluation
53
+
54
+ # We evaluate our models on the MMLU task.
55
+
56
+ | Models | #parameters |zero-shot | 5-shot |
57
+ | --- | --- | --- | --- |
58
+ | llama | 7B | 28.46 | 35.05 |
59
+ | openllama | 3B | 24.90 | 26.71 |
60
+ |TinyLlama-1.1B-step-50K-105b | 1.1B | 19.00 | 26.53 |
61
+ | LiteLlama-460M-1T | 0.46B | 21.13 | 26.39 |
62
+
63
+
64
+ # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
65
+ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ahxt__llama2_xs_460M_experimental)
66
+
67
+ | Metric | Value |
68
+ |-----------------------|---------------------------|
69
+ | Avg. | 26.65 |
70
+ | ARC (25-shot) | 24.91 |
71
+ | HellaSwag (10-shot) | 38.47 |
72
+ | MMLU (5-shot) | 26.17 |
73
+ | TruthfulQA (0-shot) | 41.59 |
74
+ | Winogrande (5-shot) | 49.88 |
75
+ | GSM8K (5-shot) | 0.0 |
76
+ | DROP (3-shot) | 5.51 |
77
+
78
+
79
+
80
+
81
+ ## Contact
82
+ This experimental version is developed by:
83
+ [Xiaotian Han](https://ahxt.github.io/) from Texas A&M University. The model is released
84
+
85
+
86
+
87
+
88
+
89
+
90
+
91
+
92
+
93
+
config copy.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 1024,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 4096,
11
+ "layernorm_epsilon": 1e-05,
12
+ "max_position_embeddings": 1024,
13
+ "model_type": "llama",
14
+ "num_attention_heads": 16,
15
+ "num_hidden_layers": 24,
16
+ "num_key_value_heads": 2,
17
+ "pad_token_id": 0,
18
+ "pretraining_tp": 1,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_scaling": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.31.0",
24
+ "use_cache": true,
25
+ "vocab_size": 50304
26
+ }
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 1024,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 4096,
11
+ "layernorm_epsilon": 1e-05,
12
+ "max_position_embeddings": 1024,
13
+ "model_type": "llama",
14
+ "num_attention_heads": 16,
15
+ "num_hidden_layers": 24,
16
+ "num_key_value_heads": 2,
17
+ "pad_token_id": 0,
18
+ "pretraining_tp": 1,
19
+ "rms_norm_eps": 1e-06,
20
+ "rope_scaling": null,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.31.0",
24
+ "use_cache": true,
25
+ "vocab_size": 50304
26
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.31.0"
7
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c5e3a81023fd29302653e5d337bb49c806a6ed7e82dea41b83947cf0944cc543
3
+ size 923454681
special_tokens_map.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "eos_token": "#",
3
+ "pad_token": "\"",
4
+ "unk_token": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": true,
8
+ "rstrip": false,
9
+ "single_word": false
10
+ }
11
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_prefix_space": false,
4
+ "bos_token": {
5
+ "__type": "AddedToken",
6
+ "content": "<|endoftext|>",
7
+ "lstrip": false,
8
+ "normalized": true,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "clean_up_tokenization_spaces": true,
13
+ "eos_token": {
14
+ "__type": "AddedToken",
15
+ "content": "<|endoftext|>",
16
+ "lstrip": false,
17
+ "normalized": true,
18
+ "rstrip": false,
19
+ "single_word": false
20
+ },
21
+ "errors": "replace",
22
+ "model_max_length": 1024,
23
+ "pad_token": null,
24
+ "tokenizer_class": "GPT2Tokenizer",
25
+ "unk_token": {
26
+ "__type": "AddedToken",
27
+ "content": "<|endoftext|>",
28
+ "lstrip": false,
29
+ "normalized": true,
30
+ "rstrip": false,
31
+ "single_word": false
32
+ }
33
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff