PocketDoc commited on
Commit
930c1b8
1 Parent(s): fd05f5d

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +68 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ ---
5
+
6
+ ### Description:
7
+ This is a multipurpose chat / chat instruct hybrid model in the same vein as the Pygmalion team's Metharme. It uses a curated pile of training data that has been normalized into a consistent training format. It has been trained on a wide array of one shot instructions, multi round instructions, and role playing scenarios.
8
+
9
+ The training parameters were suboptimal for the most recent run and I decided to stop after 2 epochs as 3 likely would have overtrained it. I plan on iterating the model and improving it further when I have access to more funds to do so.
10
+
11
+ ### Prompt format:
12
+ Metharme
13
+
14
+ The prompt should start with the cursor on the same line directly after "<|model|>" with no space. The following are all valid formats and can be extended to as many rounds as desired.
15
+ ```
16
+ <|system|>system message here<|user|>user message here<|model|>
17
+ ```
18
+ ```
19
+ <|system|>system message here<|user|>user message here<|model|>model message<|user|>user message here<|model|>
20
+ ```
21
+ ```
22
+ <|system|>system message here<|model|>
23
+ ```
24
+ ```
25
+ <|system|>system message here<|model|>model message<|user|>user message here<|model|>
26
+ ```
27
+
28
+ Some example prompts:
29
+ ```
30
+ <|system|>The following is a transcript between a helpful assistant and a user.<|user|>Why is the sky blue?<|model|>
31
+ ```
32
+ ```
33
+ <|system|>You are a Virtual Story Generator. You take the user's input and create an excellent and captivating story that goes in that direction. Use an abundance of sensory descriptions and eloquent prose.<|user|>Alpha Centauri has fallen, to the bears. This is a point of view tale about a soldier on the ground.<|model|>
34
+ ```
35
+ ```
36
+ <|system|>You are a professional editor with decades of experience, help the user with any task they have for you.<|user|>Can you rewrite this to flow better? "I knew I probably shouldnt have done that but oh well"<|model|>
37
+ ```
38
+ More will be added at a later date.
39
+
40
+ ### Perplexity Benchmarks:
41
+ - TBA
42
+
43
+ ### Training information:
44
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="150" height="24"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
45
+ - GPTQ 4 bit LoRA
46
+ - 2 Epochs
47
+ - 64 / 32 R / A
48
+ - 2048 Cutoff
49
+ - 42 hours on 1x RTX 4090
50
+
51
+ ### Data used in training:
52
+ - TBA
53
+
54
+ ### Models used:
55
+ For training:
56
+ https://huggingface.co/PocketDoc/llama-30b-gptq-4bit-128g
57
+
58
+ For merging:
59
+
60
+ https://huggingface.co/PocketDoc/Dans-PersonalityEngine-30b-LoRA
61
+
62
+ and
63
+
64
+ https://huggingface.co/huggyllama/llama-30b
65
+
66
+
67
+ ### Disclaimer:
68
+ It has not been aligned and no warranty is given for the quality or safety of its outputs.