Model Card

README.md

@@ -14,9 +14,113 @@ license: mit
 
 # hebrew-bad_wiki-gpt_neo-tiny
 
-Hebrew nonsense generation model which produces really bad wiki-abstract text. This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo). Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
+## Table of Contents
+- [Model Details](#model-details)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Environmental Impact](#environmental-impact)
+- [How to Get Started With the Model](#how-to-get-started-with-the-model)
 
-## Datasets
+## Model Details
+**Model Description:**
 
+The model developer notes that the model is 
+> Hebrew nonsense generation model which produces really bad wiki-abstract text. 
+
+
+- **Developed by:** [Doron Adler](https://github.com/Norod)
+- **Model Type:** Text Generation
+- **Language(s):** Hebrew
+- **License:** MIT
+- **Resources for more information:**
+- [GitHub Repo](https://github.com/Norod/hebrew-gpt_neo)
+- [HuggingFace Space](https://huggingface.co/spaces/Norod78/Hebrew-GPT-Neo-Small)
+
+
+## Uses
+
+#### Direct Use
+
+This model can be used for text generation.
+
+#### Misuse and Out-of-scope Use
+
+
+## Risks, Limitations and Biases
+**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
+
+
+
+
+## Training
+
+
+#### Training Data
  [Hebrew Wikipedia Dump](https://dumps.wikimedia.org/hewiki/latest/) (hewiki abstract) from May 2020
 
+
+
+
+#### Training Procedure
+
+
+This model was fined tuned upon [hebrew-gpt_neo-tiny](https://huggingface.co/Norod78/hebrew-gpt_neo-tiny) which was previously trained using [EleutherAI's gpt-neo](https://github.com/EleutherAI/gpt-neo). 
+
+Fine-tuning on the wiki-absract text was done using [@minimaxir](https://twitter.com/minimaxir)'s [aitextgen](https://github.com/minimaxir/aitextgen).
+
+
+
+## Evaluation
+
+
+#### Configs
+
+Model configs for the hebrew-gpt_neo-tiny is available on the [hebrew-gpt_neo model github](https://github.com/Norod/hebrew-gpt_neo/tree/main/hebrew-gpt_neo-tiny/configs) 
+
+* **Activation Function:** gelu
+* **Number_Head:** 12
+* **Number_Vocab:** 50257
+* **Train batch size:** 250
+* **Eval batch size:** 64
+* **Predict batch size:** 1
+
+
+
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
+
+
+- **Hardware Type:** [More information needed]
+
+- **Hours used:** Unknown
+
+- **Cloud Provider:** GCP  tpu-v8s
+
+- **Compute Region:** europe-west4
+
+- **Carbon Emitted:** [More information needed]
+
+
+## How to Get Started With the Model
+
+A Google Colab Notebook is also available [here](https://colab.research.google.com/github/Norod/hebrew-gpt_neo/blob/main/hebrew-gpt_neo-tiny/Norod78_hebrew_gpt_neo_tiny_Colab.ipynb)
+
+
+​​
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
+
+model = AutoModelForCausalLM.from_pretrained("Norod78/hebrew-bad_wiki-gpt_neo-tiny")
+
+
+```
+
+