Julius ter Pelkwijk
commited on
Commit
•
07804a9
1
Parent(s):
10f06ce
Initial Commit
Browse files
README.md
CHANGED
@@ -1,3 +1,25 @@
|
|
1 |
---
|
|
|
2 |
license: mit
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
language: en
|
3 |
license: mit
|
4 |
---
|
5 |
+
# Fairseq-dense 13B - Nerys
|
6 |
+
## Model Description
|
7 |
+
Fairseq-dense 13B-Nerys is a finetune created using Fairseq's MoE dense model.
|
8 |
+
## Training data
|
9 |
+
The training data contains around 2500 ebooks in various genres (the "Pike" dataset), a CYOA dataset called "CYS" and 50 Asian "Light Novels" (the "Manga-v1" dataset).
|
10 |
+
Most parts of the dataset have been prepended using the following text: `[Genre: <genre1>, <genre2>]`
|
11 |
+
### How to use
|
12 |
+
You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
|
13 |
+
```py
|
14 |
+
>>> from transformers import pipeline
|
15 |
+
>>> generator = pipeline('text-generation', model='KoboldAI/fairseq-dense-13B-Nerys')
|
16 |
+
>>> generator("Welcome Captain Janeway, I apologize for the delay.", do_sample=True, min_length=50)
|
17 |
+
[{'generated_text': 'Welcome Captain Janeway, I apologize for the delay."\nIt's all right," Janeway said. "I'm certain that you're doing your best to keep me informed of what\'s going on."'}]
|
18 |
+
```
|
19 |
+
### Limitations and Biases
|
20 |
+
Based on known problems with NLP technology, potential relevant factors include bias (gender, profession, race and religion).
|
21 |
+
|
22 |
+
### BibTeX entry and citation info
|
23 |
+
```
|
24 |
+
Artetxe et al. (2021): Efficient Large Scale Language Modeling with Mixtures of Experts
|
25 |
+
```
|