blockblockblock commited on
Commit
efde344
1 Parent(s): 0108855

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. README.md +43 -5
  2. config.json +3 -3
  3. output.safetensors +2 -2
README.md CHANGED
@@ -10,13 +10,20 @@ license: llama3
10
  <a href="https://www.gradient.ai" target="_blank"><img src="https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/TSa3V8YpoVagnTYgxiLaO.png" width="200"/></a>
11
 
12
  # Llama-3 8B Gradient Instruct 1048k
 
 
 
13
  Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. If you're looking to build custom AI models or agents, email us a message contact@gradient.ai.
14
 
15
  For more info see our [End-to-end development service for custom LLMs and AI systems](https://gradient.ai/development-lab)
16
 
 
 
17
  This model extends LLama-3 8B's context length from 8k to > 1040K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3's original pre-training data.
18
 
19
- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/6MKLoX2ruLIaREiyb6coO.png)
 
 
20
 
21
  **Approach:**
22
 
@@ -32,7 +39,7 @@ Notably, we layered parallelism on top of Ring Attention with a custom network t
32
 
33
  **Data:**
34
 
35
- For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B).
36
 
37
  **Progressive Training Details:**
38
 
@@ -41,7 +48,7 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
41
  | Initialize From | LLaMA-3 8B| 65K | 262K | 524k |
42
  | Sequence Length 2^N | 16 | 18 | 19 | 20 |
43
  | RoPE theta | 15.3 M | 207.1 M | 1.06B | 2.80B |
44
- | Batch Size | 1 | 1 | 16 | 16 |
45
  | Gradient Accumulation Steps | 32 | 16 | 1 | 1 |
46
  | Steps | 30 | 24 | 50 | 50 |
47
  | Total Tokens | 62914560 | 100663296 | 419430400 | 838860800 |
@@ -50,9 +57,37 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
50
  | GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
51
  | Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
52
 
53
- **Quants**:
54
- - [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  - [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
 
 
 
56
 
57
  ## The Gradient AI Team
58
 
@@ -72,6 +107,9 @@ Drop an email to [contact@gradient.ai](mailto:contact@gradient.ai)
72
 
73
  [3] https://github.com/jzhang38/EasyContext
74
 
 
 
 
75
 
76
  ----
77
 
 
10
  <a href="https://www.gradient.ai" target="_blank"><img src="https://cdn-uploads.huggingface.co/production/uploads/655bb613e8a8971e89944f3e/TSa3V8YpoVagnTYgxiLaO.png" width="200"/></a>
11
 
12
  # Llama-3 8B Gradient Instruct 1048k
13
+
14
+ Join our custom agent and long context (262k-1M+) waitlist: https://forms.gle/L6TDY7dozx8TuoUv7
15
+
16
  Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. If you're looking to build custom AI models or agents, email us a message contact@gradient.ai.
17
 
18
  For more info see our [End-to-end development service for custom LLMs and AI systems](https://gradient.ai/development-lab)
19
 
20
+ [Join our Discord](https://discord.com/invite/2QVy2qt2mf)
21
+
22
  This model extends LLama-3 8B's context length from 8k to > 1040K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is < 0.01% of Llama-3's original pre-training data.
23
 
24
+ **Update (5/3): We further fine-tuned our model to strengthen its assistant-like chat ability as well. The NIAH result is updated.**
25
+
26
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/-qaI__83ksClzoJzlqZjq.png)
27
 
28
  **Approach:**
29
 
 
39
 
40
  **Data:**
41
 
42
+ For training data, we generate long contexts by augmenting [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B). We also fine-tune on a chat dataset based on UltraChat [4], following a similar recipe for data augmentation to [2].
43
 
44
  **Progressive Training Details:**
45
 
 
48
  | Initialize From | LLaMA-3 8B| 65K | 262K | 524k |
49
  | Sequence Length 2^N | 16 | 18 | 19 | 20 |
50
  | RoPE theta | 15.3 M | 207.1 M | 1.06B | 2.80B |
51
+ | Batch Size | 1 | 1 | 16 | 8 |
52
  | Gradient Accumulation Steps | 32 | 16 | 1 | 1 |
53
  | Steps | 30 | 24 | 50 | 50 |
54
  | Total Tokens | 62914560 | 100663296 | 419430400 | 838860800 |
 
57
  | GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
58
  | Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
59
 
60
+
61
+ **Evaluation:**
62
+
63
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585dc9be92bc5f258156bd6/mWxIGZNi3ejlmeIDWafKu.png)
64
+
65
+ ```
66
+ EVAL_MAX_CONTEXT_LENGTH=1040200
67
+ EVAL_MIN_CONTEXT_LENGTH=100
68
+ EVAL_CONTEXT_INTERVAL=86675
69
+ EVAL_DEPTH_INTERVAL=0.2
70
+ EVAL_RND_NUMBER_DIGITS=8
71
+
72
+ HAYSTACK1:
73
+ EVAL_GENERATOR_TOKENS=25
74
+
75
+ HAYSTACK2:
76
+ EVAL_CONTEXT_INTERVAL=173350
77
+ EVAL_GENERATOR_TOKENS=150000
78
+
79
+ HAYSTACK3:
80
+ EVAL_GENERATOR_TOKENS=925000
81
+ ```
82
+
83
+ All boxes not pictured for Haystack 1 and 3 are 100% accurate. Haystacks 1,2 and 3 are further detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
84
+
85
+ **Quants:**
86
+ - [GGUF by Crusoe](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF). Note that you need to add 128009 as [special token with llama.cpp](https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k/discussions/13).
87
  - [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
88
+ - [Ollama](https://ollama.com/library/llama3-gradient)
89
+ - vLLM docker image, recommended to load via `--max-model-len 32768`
90
+ - If you are interested in a hosted version, drop us a mail below.
91
 
92
  ## The Gradient AI Team
93
 
 
107
 
108
  [3] https://github.com/jzhang38/EasyContext
109
 
110
+ [4] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan
111
+ Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling
112
+ high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
113
 
114
  ----
115
 
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "gradientai/llama3-run1-stage524k-fm-v3",
3
  "architectures": [
4
  "LlamaForCausalLM"
5
  ],
@@ -19,10 +19,10 @@
19
  "pretraining_tp": 1,
20
  "rms_norm_eps": 1e-05,
21
  "rope_scaling": null,
22
- "rope_theta": 2804339835.0,
23
  "tie_word_embeddings": false,
24
  "torch_dtype": "bfloat16",
25
- "transformers_version": "4.39.1",
26
  "use_cache": true,
27
  "vocab_size": 128256,
28
  "quantization_config": {
 
1
  {
2
+ "_name_or_path": "gradientai/llama3-8b-stage262k-chat",
3
  "architectures": [
4
  "LlamaForCausalLM"
5
  ],
 
19
  "pretraining_tp": 1,
20
  "rms_norm_eps": 1e-05,
21
  "rope_scaling": null,
22
+ "rope_theta": 3580165449.0,
23
  "tie_word_embeddings": false,
24
  "torch_dtype": "bfloat16",
25
+ "transformers_version": "4.41.0.dev0",
26
  "use_cache": true,
27
  "vocab_size": 128256,
28
  "quantization_config": {
output.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:25f12a2be40675da06de6244cf30eb3df44ddd2e417b8c4b595ccc99759e1b09
3
- size 4083984616
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bbd713c239143c6cb6363c5743187601228d26347e59bc439959ccca9e0cbdb9
3
+ size 4081601956