teknium commited on
Commit
c952ecb
1 Parent(s): 487f046

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -0
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: gpl
3
+ language:
4
+ - en
5
+ tags:
6
+ - starcoder
7
+ - wizardcoder
8
+ - code
9
+ - self-instruct
10
+ - distillation
11
+ ---
12
+
13
+ # Model Card: Redmond-Hermes-Coder 15B
14
+
15
+ ## Model Description
16
+
17
+ Redmond-Hermes-Coder 15B is a state-of-the-art language model fine-tuned on over 300,000 instructions. This model was fine-tuned by Nous Research, with Teknium and Karan4D leading the fine tuning process and dataset curation, Redmond AI sponsoring the compute, and several other contributors.
18
+
19
+ This model was trained with a WizardCoder base, which itself uses a StarCoder base model.
20
+
21
+ The model is truly great at code, but, it does come with a tradeoff though. While far better at code than the original Nous-Hermes built on Llama, it is worse than WizardCoder at pure code benchmarks, like HumanEval.
22
+
23
+ It comes in at 39% on HumanEval, with WizardCoder at 57%. This is somewhat dissapointing to us, and we are exploring why now.
24
+
25
+ However, it does seem better at non-code than WizardCoder on a variety of things, including writing tasks.
26
+
27
+ ## Model Training
28
+
29
+ The model was trained almost entirely on synthetic GPT-4 outputs. This includes data from diverse sources such as GPTeacher, the general, roleplay v1&2, code instruct datasets, Nous Instruct & PDACTL (unpublished), CodeAlpaca, Evol_Instruct Uncensored, GPT4-LLM, and Unnatural Instructions.
30
+
31
+ Additional data inputs came from Camel-AI's Biology/Physics/Chemistry and Math Datasets, Airoboros' (v1) GPT-4 Dataset, and more from CodeAlpaca. The total volume of data encompassed over 300,000 instructions.
32
+
33
+ ## Collaborators
34
+ The model fine-tuning and the datasets were a collaboration of efforts and resources from members of Nous Research, includingTeknium, Karan4D, Huemin Art, and Redmond AI's generous compute grants.
35
+
36
+ Huge shoutout and acknowledgement is deserved for all the dataset creators who generously share their datasets openly.
37
+
38
+ Among the contributors of datasets, GPTeacher was made available by Teknium, Wizard LM by nlpxucan, and the Nous Research Instruct Dataset was provided by Karan4D and HueminArt.
39
+ The GPT4-LLM and Unnatural Instructions were provided by Microsoft, Airoboros dataset by jondurbin, Camel-AI datasets are from Camel-AI, and CodeAlpaca dataset by Sahil 2801.
40
+ If anyone was left out, please open a thread in the community tab.
41
+
42
+ ## Prompt Format
43
+
44
+ The model follows the Alpaca prompt format:
45
+ ```
46
+ ### Instruction:
47
+
48
+ ### Response:
49
+ ```
50
+
51
+ or
52
+
53
+ ```
54
+ ### Instruction:
55
+
56
+ ### Input:
57
+
58
+ ### Response:
59
+ ```
60
+
61
+ ## Resources for Applied Use Cases:
62
+ For an example of a back and forth chatbot using huggingface transformers and discord, check out: https://github.com/teknium1/alpaca-discord
63
+ For an example of a roleplaying discord bot, check out this: https://github.com/teknium1/alpaca-roleplay-discordbot
64
+
65
+ ## Future Plans
66
+ The model is currently being uploaded in FP16 format, and there are plans to convert the model to GGML and GPTQ 4bit quantizations. The team is also working on a full benchmark, similar to what was done for GPT4-x-Vicuna. We will try to get in discussions to get the model included in the GPT4All.
67
+
68
+ ## Benchmark Results
69
+ ```
70
+ HumanEval: 39%
71
+ ```
72
+
73
+ ## Model Usage
74
+ The model is available for download on Hugging Face. It is suitable for a wide range of language tasks, from generating creative text to understanding and following complex instructions.
75
+
76
+ Compute provided by our project sponsor Redmond AI, thank you!!