|
--- |
|
language: he |
|
|
|
thumbnail: https://avatars1.githubusercontent.com/u/3617152?norod.jpg |
|
widget: |
|
- text: "מתמטיקה:" |
|
- text: "עליית המכונות" |
|
- text: "ויקיפדיה העברית" |
|
- text: "האירוויזיון הוא" |
|
- text: "דוד בן-גוריון היה" |
|
|
|
license: mit |
|
--- |
|
|
|
# hebrew-bad_wiki-gpt_neo-tiny |
|
|
|
## Table of Contents |
|
- [Model Details](#model-details) |
|
- [Uses](#uses) |
|
- [Risks, Limitations and Biases](#risks-limitations-and-biases) |
|
- [Training](#training) |
|
- [Evaluation](#evaluation) |
|
- [Environmental Impact](#environmental-impact) |
|
- [How to Get Started With the Model](#how-to-get-started-with-the-model) |
|
|
|
## Model Details |
|
**Model Description:** |
|
|
|
The model developer notes that the model is |
|
> Hebrew nonsense generation model which produces really bad wiki-abstract text. |
|
|
|
|
|
- **Developed by:** [Doron Adler](https://github.com/Norod) |
|
- **Model Type:** Text Generation |
|
- **Language(s):** Hebrew |
|
- **License:** MIT |
|
- **Resources for more information:** |
|
- [GitHub Repo](https://github.com/Norod/hebrew-gpt_neo) |
|
- [HuggingFace Space](https://huggingface.co/spaces/Norod78/Hebrew-GPT-Neo-Small) |
|
|
|
|
|
## Uses |
|
|
|
#### Direct Use |
|
|
|
This model can be used for text generation. |
|
|
|
#### Misuse and Out-of-scope Use |
|
|
|
|
|
## Risks, Limitations and Biases |
|
**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.** |
|
|
|
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). |
|
|
|
|
|
|
|
|
|
## Training |
|
|
|
|
|
#### Training Data |
|
[Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020 |
|
|
|
|
|
|
|
|
|
#### Training Procedure |
|
|
|
|
|
This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo). |
|
|
|
Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen). |
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
#### Configs |
|
|
|
Model configs for the hebrew-gpt_neo-tiny is available on the [hebrew-gpt_neo model github](https://github.com/Norod/hebrew-gpt_neo/tree/main/hebrew-gpt_neo-tiny/configs) |
|
|
|
* **Activation Function:** gelu |
|
* **Number_Head:** 12 |
|
* **Number_Vocab:** 50257 |
|
* **Train batch size:** 250 |
|
* **Eval batch size:** 64 |
|
* **Predict batch size:** 1 |
|
|
|
|
|
|
|
|
|
## Environmental Impact |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf). |
|
|
|
|
|
- **Hardware Type:** [More information needed] |
|
|
|
- **Hours used:** Unknown |
|
|
|
- **Cloud Provider:** GCP tpu-v8s |
|
|
|
- **Compute Region:** europe-west4 |
|
|
|
- **Carbon Emitted:** [More information needed] |
|
|
|
|
|
## How to Get Started With the Model |
|
|
|
A Google Colab Notebook is also available [here](https://colab.research.google.com/github/Norod/hebrew-gpt_neo/blob/main/hebrew-gpt_neo-tiny/Norod78_hebrew_gpt_neo_tiny_Colab.ipynb) |
|
|
|
|
|
|
|
``` |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny") |
|
|
|
model = AutoModelForCausalLM.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny") |
|
|
|
|
|
``` |
|
|
|
|
|
|
