Hazzzardous SmerkyG commited on
Commit
500e167
0 Parent(s):

Duplicate from SmerkyG/rwkv-6-world-all-pth

Browse files

Co-authored-by: Dan Goldstein <SmerkyG@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - zh
5
+ - fr
6
+ - es
7
+ - de
8
+ - pt
9
+ - ru
10
+ - it
11
+ - ja
12
+ - ko
13
+ - vi
14
+ - ar
15
+ tags:
16
+ - pytorch
17
+ - text-generation
18
+ - causal-lm
19
+ - rwkv
20
+ license: apache-2.0
21
+ datasets:
22
+ - cerebras/SlimPajama-627B
23
+ - EleutherAI/pile
24
+ - bigcode/starcoderdata
25
+ - oscar-corpus/OSCAR-2301
26
+ ---
27
+
28
+ # RWKV-6 World
29
+
30
+ Use rwkv pip package 0.8.24+ for RWKV-6 inference: https://pypi.org/project/rwkv/ (pipeline = PIPELINE(model, "rwkv_vocab_v20230424") for rwkv-world models)
31
+
32
+ Online Demo 1: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2
33
+
34
+ Online Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1
35
+
36
+ GUI: https://github.com/josStorer/RWKV-Runner (see Releases)
37
+
38
+ For developer: https://github.com/BlinkDL/ChatRWKV/blob/main/API_DEMO_CHAT.py
39
+
40
+ https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_v6_demo.py
41
+
42
+ How it works: https://twitter.com/BlinkDL_AI/status/1685230712247795713
43
+
44
+ https://www.rwkv.com/
45
+
46
+ ## Model Description
47
+
48
+ RWKV-6 trained on 100+ world languages (70% English, 15% multilang, 15% code).
49
+
50
+ World = Some_Pile + Some_SlimPajama + Some_StarCoder + Some_OSCAR + All_Wikipedia + All_ChatGPT_Data_I_can_find
51
+
52
+ World v1 = 0.59T tokens
53
+
54
+ World v2 = 1.12T tokens
55
+
56
+ World v2.1 = 1.42T tokens
57
+
58
+ Recommended fine-tuning format (use \n for newlines):
59
+ ```
60
+ User: xxxxxxxxxxxxxxx
61
+
62
+ Assistant: xxxxxxxxxxxxxxx
63
+ xxxxxxxxxxxxxxx
64
+ xxxxxxxxxxxxxxx
65
+
66
+ User: xxxxxxxxxxxxxxx
67
+ xxxxxxxxxxxxxxx
68
+
69
+ Assistant: xxxxxxxxxxxxxxx
70
+ xxxxxxxxxxxxxxx
71
+ xxxxxxxxxxxxxxx
72
+ xxxxxxxxxxxxxxx
73
+ ```
74
+
75
+ A good chat prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
76
+ ```
77
+ User: hi
78
+
79
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
80
+
81
+ User: xxx
82
+
83
+ Assistant:
84
+ ```
85
+ QA prompt (better replace \n\n in xxx to \n, such that there will be no newlines in xxx):
86
+ ```
87
+ Question: xxx
88
+
89
+ Answer:
90
+ ```
91
+ and
92
+ ```
93
+ Instruction: xxx
94
+
95
+ Input: xxx
96
+
97
+ Response:
98
+ ```
99
+
100
+ !!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
101
+
102
+ !!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
103
+
104
+ !!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
RWKV-x060-World-1B6-v2-20240208-ctx4096.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5c9c877fb60a65cab269af175328b6aaf16d02b8b09738923254f9986e5dc440
3
+ size 3199845663
RWKV-x060-World-1B6-v2.1-20240328-ctx4096.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cda1f0ebe802e2859bfa129546372fd1eac60319111f215f65ced67dd334db36
3
+ size 3199845663
RWKV-x060-World-3B-v2-20240228-ctx4096.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1235079a07084472de86996846a6533e41e3964b153cdc1e8462cc138d8521d
3
+ size 6199859158