LoneStriker commited on
Commit
9d93337
1 Parent(s): 4de2cb9

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ datasets:
6
+ - nampdn-ai/tiny-textbooks
7
+ ---
8
+
9
+ # Nuclues 1B Alpha1
10
+
11
+ <p align="center">
12
+ <img src="https://github.com/prp-e/nucleus/raw/main/nucleus-logo.png" width=256 height=256>
13
+ </p>
14
+
15
+ ## What is Nucleus?
16
+
17
+ Nucleus is a small language model based on Mistral (actually, the trimmed untrained version you can find [here](https://huggingface.co/lmlab/lmlab-mistral-1b-untrained)) and trained in different steps. First, we've pretrained it on TinyStories dataset, then [TinyTextBooks](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks) to make it a more specific model. This model is just a _proof of concept_ at this point, but showed good promises in early tests. So with proper training, can be a good product over time!
18
+
19
+ ## Inference
20
+
21
+ [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/prp-e/nucleus/blob/main/nucleus_1b_inference.ipynb)
22
+
23
+ First you need to install `transformers` and `accelerate` libraries in order to run this model. Then, you basically have to run the following code:
24
+
25
+ ```python
26
+
27
+ from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
28
+ import torch
29
+
30
+ model_name_or_id = "NucleusOrg/Nucleus-1B-alpha-1"
31
+
32
+ model = AutoModelForCausalLM.from_pretrained(model_name_or_id, torch_dtype=torch.float16, device_map="cuda")
33
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_id)
34
+
35
+ prompt = "### Lesson: Python Programming 101\n### Introduction\n"
36
+
37
+ inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
38
+
39
+ generation_config = GenerationConfig(
40
+ do_sample=True,
41
+ top_k=1,
42
+ temperature=0.9,
43
+ max_new_tokens=500,
44
+ repetition_penalty=1.5,
45
+ pad_token_id=tokenizer.eos_token_id
46
+ )
47
+
48
+ outputs = model.generate(**inputs, generation_config=generation_config)
49
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
50
+
51
+ ```
52
+
53
+ __Prompt Format__: This model does not have a specific prompt format, but the best results could be achieved with a _textbook_ type of format like:
54
+
55
+ ```
56
+ ### Chapter 1: Elon Musk and Iron Man
57
+ Elon met Tony at a Cafe in Monaco, then they had a conversation about
58
+ ```
59
+
60
+ You also can try something like this:
61
+
62
+ ```
63
+ Question: Who are you?
64
+ Answer:
65
+ ```
66
+
67
+ But since the model isn't made for chat/question answering, the result won't be good enough.
68
+
69
+ __Repetition Penalty__: Since most of these models like to repeat themselves, just keep that number there. You can increase or decrease it based on your liking,but keep in mind that a number lower than 1 makes the model _super repetitive_.
70
+
71
+ ## Known Issues
72
+
73
+ * Since we only had 420k rows of data, a lot of information are missing on this model. Since mentioned earlier in this very model card, it's a _proof of concept_ model.
74
+ * You probably may test it with coding. Let's say that the model is terrible at coding. We may release a coding optimized model as soon as possible.
75
+
76
+ ## Our Team
77
+
78
+ * Muhammadreza Haghiri ([X (formerly Twitter)](https://twitter.com/haghiri_ai) - [Website](https://haghiri75.com/en) - [Github](https://github.com/prp-e) - [LinkedIn](https://www.linkedin.com/in/muhammadreza-haghiri-1761325b))
79
+ * Mahi Mohrechi ([Website](https://mohrechi-portfolio.vercel.app/) - [Github](https://github.com/f-mohrechi) - [LinkedIn](https://www.linkedin.com/in/faeze-mohrechi/))
80
+
81
+ ## Special Thanks
82
+
83
+ * LMLabs for providing 1B untrained model.
84
+ * Mistral Team for providing the best open source base model ever.
85
+ * _Sina Rashidi_, who translated Alpaca dataset to Persian.
86
+ * [Jupyto](https://jupyto.com) team for providing our infrastructure.
config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/jupyter/Nucleus-1B-alpha-1",
3
+ "architectures": [
4
+ "MistralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 14336,
13
+ "max_position_embeddings": 32768,
14
+ "model_type": "mistral",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 4,
17
+ "num_key_value_heads": 8,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_theta": 10000.0,
20
+ "sliding_window": 4096,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "float16",
23
+ "transformers_version": "4.36.2",
24
+ "use_cache": true,
25
+ "vocab_size": 32000
26
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.36.2"
6
+ }
model.safetensors.index.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 2269192192
4
+ },
5
+ "weight_map": {
6
+ "lm_head.weight": "model-00003-of-00003.safetensors",
7
+ "model.embed_tokens.weight": "model-00001-of-00003.safetensors",
8
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
9
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
10
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
11
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
12
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
13
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
14
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
15
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
16
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
17
+ "model.layers.1.input_layernorm.weight": "model-00002-of-00003.safetensors",
18
+ "model.layers.1.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
19
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
20
+ "model.layers.1.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
21
+ "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
22
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
23
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
24
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
25
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
26
+ "model.layers.2.input_layernorm.weight": "model-00002-of-00003.safetensors",
27
+ "model.layers.2.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
28
+ "model.layers.2.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
29
+ "model.layers.2.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
30
+ "model.layers.2.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
31
+ "model.layers.2.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
32
+ "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
33
+ "model.layers.2.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
34
+ "model.layers.2.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
35
+ "model.layers.3.input_layernorm.weight": "model-00003-of-00003.safetensors",
36
+ "model.layers.3.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
37
+ "model.layers.3.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
38
+ "model.layers.3.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
39
+ "model.layers.3.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
40
+ "model.layers.3.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
41
+ "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
42
+ "model.layers.3.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
43
+ "model.layers.3.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
44
+ "model.norm.weight": "model-00003-of-00003.safetensors"
45
+ }
46
+ }
output.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e69b92e45971ccb385c8565ccaa0900697d48105c0bf1ec6dead1103db6bf8cf
3
+ size 1244598360
special_tokens_map.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "unk_token": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ }
23
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "added_tokens_decoder": {
5
+ "0": {
6
+ "content": "<unk>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "1": {
14
+ "content": "<s>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "2": {
22
+ "content": "</s>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ }
29
+ },
30
+ "additional_special_tokens": [],
31
+ "bos_token": "<s>",
32
+ "clean_up_tokenization_spaces": false,
33
+ "eos_token": "</s>",
34
+ "legacy": true,
35
+ "model_max_length": 1000000000000000019884624838656,
36
+ "pad_token": null,
37
+ "sp_model_kwargs": {},
38
+ "spaces_between_special_tokens": false,
39
+ "tokenizer_class": "LlamaTokenizer",
40
+ "unk_token": "<unk>",
41
+ "use_default_system_prompt": true
42
+ }