RichardErkhov
commited on
Commit
โข
44b2a79
1
Parent(s):
b0a4c2b
uploaded readme
Browse files
README.md
ADDED
@@ -0,0 +1,145 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
Quantization made by Richard Erkhov.
|
2 |
+
|
3 |
+
[Github](https://github.com/RichardErkhov)
|
4 |
+
|
5 |
+
[Discord](https://discord.gg/pvy7H8DZMG)
|
6 |
+
|
7 |
+
[Request more models](https://github.com/RichardErkhov/quant_request)
|
8 |
+
|
9 |
+
|
10 |
+
hebrew-bad_wiki-gpt_neo-tiny - bnb 4bits
|
11 |
+
- Model creator: https://huggingface.co/Norod78/
|
12 |
+
- Original model: https://huggingface.co/Norod78/hebrew-bad_wiki-gpt_neo-tiny/
|
13 |
+
|
14 |
+
|
15 |
+
|
16 |
+
|
17 |
+
Original model description:
|
18 |
+
---
|
19 |
+
language: he
|
20 |
+
|
21 |
+
thumbnail: https://avatars1.githubusercontent.com/u/3617152?norod.jpg
|
22 |
+
widget:
|
23 |
+
- text: "ืืชืืืืงื:"
|
24 |
+
- text: "ืขืืืืช ืืืืื ืืช"
|
25 |
+
- text: "ืืืงืืคืืื ืืขืืจืืช"
|
26 |
+
- text: "ืืืืจืืืืืืื ืืื"
|
27 |
+
- text: "ืืื ืื-ืืืจืืื ืืื"
|
28 |
+
|
29 |
+
license: mit
|
30 |
+
---
|
31 |
+
|
32 |
+
# hebrew-bad_wiki-gpt_neo-tiny
|
33 |
+
|
34 |
+
## Table of Contents
|
35 |
+
- [Model Details](#model-details)
|
36 |
+
- [Uses](#uses)
|
37 |
+
- [Risks, Limitations and Biases](#risks-limitations-and-biases)
|
38 |
+
- [Training](#training)
|
39 |
+
- [Evaluation](#evaluation)
|
40 |
+
- [Environmental Impact](#environmental-impact)
|
41 |
+
- [How to Get Started With the Model](#how-to-get-started-with-the-model)
|
42 |
+
|
43 |
+
## Model Details
|
44 |
+
**Model Description:**
|
45 |
+
|
46 |
+
The model developer notes that the model is
|
47 |
+
> Hebrew nonsense generation model which produces really bad wiki-abstract text.
|
48 |
+
|
49 |
+
|
50 |
+
- **Developed by:** [Doron Adler](https://github.com/Norod)
|
51 |
+
- **Model Type:** Text Generation
|
52 |
+
- **Language(s):** Hebrew
|
53 |
+
- **License:** MIT
|
54 |
+
- **Resources for more information:**
|
55 |
+
- [GitHub Repo](https://github.com/Norod/hebrew-gpt_neo)
|
56 |
+
- [HuggingFace Space](https://huggingface.co/spaces/Norod78/Hebrew-GPT-Neo-Small)
|
57 |
+
|
58 |
+
|
59 |
+
## Uses
|
60 |
+
|
61 |
+
#### Direct Use
|
62 |
+
|
63 |
+
This model can be used for text generation.
|
64 |
+
|
65 |
+
#### Misuse and Out-of-scope Use
|
66 |
+
|
67 |
+
|
68 |
+
## Risks, Limitations and Biases
|
69 |
+
**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
|
70 |
+
|
71 |
+
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
|
72 |
+
|
73 |
+
|
74 |
+
|
75 |
+
|
76 |
+
## Training
|
77 |
+
|
78 |
+
|
79 |
+
#### Training Data
|
80 |
+
[Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020
|
81 |
+
|
82 |
+
|
83 |
+
|
84 |
+
|
85 |
+
#### Training Procedure
|
86 |
+
|
87 |
+
|
88 |
+
This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo).
|
89 |
+
|
90 |
+
Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
|
91 |
+
|
92 |
+
|
93 |
+
|
94 |
+
## Evaluation
|
95 |
+
|
96 |
+
|
97 |
+
#### Configs
|
98 |
+
|
99 |
+
Model configs for the hebrew-gpt_neo-tiny is available on the [hebrew-gpt_neo model github](https://github.com/Norod/hebrew-gpt_neo/tree/main/hebrew-gpt_neo-tiny/configs)
|
100 |
+
|
101 |
+
* **Activation Function:** gelu
|
102 |
+
* **Number_Head:** 12
|
103 |
+
* **Number_Vocab:** 50257
|
104 |
+
* **Train batch size:** 250
|
105 |
+
* **Eval batch size:** 64
|
106 |
+
* **Predict batch size:** 1
|
107 |
+
|
108 |
+
|
109 |
+
|
110 |
+
|
111 |
+
## Environmental Impact
|
112 |
+
|
113 |
+
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
|
114 |
+
|
115 |
+
|
116 |
+
- **Hardware Type:** [More information needed]
|
117 |
+
|
118 |
+
- **Hours used:** Unknown
|
119 |
+
|
120 |
+
- **Cloud Provider:** GCP tpu-v8s
|
121 |
+
|
122 |
+
- **Compute Region:** europe-west4
|
123 |
+
|
124 |
+
- **Carbon Emitted:** [More information needed]
|
125 |
+
|
126 |
+
|
127 |
+
## How to Get Started With the Model
|
128 |
+
|
129 |
+
A Google Colab Notebook is also available [here](https://colab.research.google.com/github/Norod/hebrew-gpt_neo/blob/main/hebrew-gpt_neo-tiny/Norod78_hebrew_gpt_neo_tiny_Colab.ipynb)
|
130 |
+
|
131 |
+
|
132 |
+
โโ
|
133 |
+
```
|
134 |
+
from transformers import AutoTokenizer, AutoModelForCausalLM
|
135 |
+
|
136 |
+
tokenizer = AutoTokenizer.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
|
137 |
+
|
138 |
+
model = AutoModelForCausalLM.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
|
139 |
+
|
140 |
+
|
141 |
+
```
|
142 |
+
|
143 |
+
|
144 |
+
|
145 |
+
|