Update README.md

README.md

@@ -1,15 +1,23 @@
 ---
 language:
 - en
+datasets:
+- English
 tags:
 - text generation
 - pytorch
 - causal-lm
-license: cc-by-4.0
+- Writer-data
+- GPT
+- NeMO
 pipeline_tag: text-generation
 library_name: transformers
 ---
-# Writer-5B
+
+license: cc-by-4.0
+
+
+# Palmyra-small
 
 <style>
 img {
@@ -17,115 +25,82 @@ img {
 }
 </style>
 
-|[![Model architecture](https://img.shields.io/badge/Model%20Arch-Transformer%20Decoder-green)](#model-architecture)|[![Model size](https://img.shields.io/badge/Params-5B-green)](#model-architecture)|[![Language](https://img.shields.io/badge/Language-en--US-lightgrey#model-badge)](#datasets)
+|[![Model architecture](https://img.shields.io/badge/Model%20Arch-Transformer%20Decoder-green)](#model-architecture)|[![Model size](https://img.shields.io/badge/Params-126M-green)](#model-architecture)|[![Language](https://img.shields.io/badge/Language-en--US-lightgrey#model-badge)](#datasets)
 
 
 ## Model Description
 
-Model description
-Writer LLM base was primarily pretrained with English text, there is still a trace amount of non-English data present within the training corpus that was accessed through CommonCrawl. A causal language modeling (CLM) objective was utilized during the process of the model's pretraining. Similar to GPT-3, Writer LLM base is a member of the same family of models that only contain a decoder. As a result, it was pretrained utilizing the objective of self-supervised causal language modeling.
-Writer LLM base uses the prompts and general experimental setup from GPT-3 in order to conduct its evaluation in accordance with GPT-3. Read the official paper if you want more information about this.
+Palmyra was primarily pretrained with English text, there is still a trace amount of non-English data present within the training corpus that was accessed through CommonCrawl. A causal language modeling (CLM) objective was utilized during the process of the model's pretraining. Similar to GPT-3, Palmyra is a member of the same family of models that only contain a decoder. As a result, it was pretrained utilizing the objective of self-supervised causal language modeling. Palmyra uses the prompts and general experimental setup from GPT-3 in order to conduct its evaluation in accordance with GPT-3. Read the official paper if you want more information about this.
 
-## Getting started
 
-### Step 1: Install NeMo and dependencies
+## Training data
 
-You will need to install NVIDIA Apex and NeMo. 
+Palmyra-base 5b was trained on Writer custom dataset
 
-```
-git clone https://github.com/ericharper/apex.git
-cd apex
-git checkout nm_v1.11.0
-pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
-```
+## Intended Use and Limitations
 
-```
-pip install nemo_toolkit['nlp']==1.11.0
-``` 
+Palmyra-base learns an inner representation of the English language that can be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating text from a prompt.
 
-### Step 2: Launch eval server 
+### How to use
 
-**Note.** The example below launches a model variant with Tensor Parallelism (TP) of 4 and Pipeline Parallelism (PP) of 1 on two GPUs.
+This model can be easily loaded using the `AutoModelForCausalLM` functionality:
 
+```python
 
-```
-git clone https://github.com/NVIDIA/NeMo.git 
-cd NeMo/examples/nlp/language_modeling
-git checkout v1.11.0
-python megatron_gpt_eval.py gpt_model_file=palmyara_gpt_5b.nemo server=True tensor_model_parallel_size=4 trainer.devices=4
-```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
 
-### Step 3: Send prompts to your model!
-```python
-import json
-import requests
-
-port_num = 5555
-headers = {"Content-Type": "application/json"}
-
-def request_data(data):
-    resp = requests.put('http://localhost:{}/generate'.format(port_num),
-                        data=json.dumps(data),
-                        headers=headers)
-    sentences = resp.json()['sentences']
-    return sentences
-
-
-data = {
-    "sentences": ["Tell me an interesting fact about space travel."]*1,
-    "tokens_to_generate": 50,
-    "temperature": 1.0,
-    "add_BOS": True,
-    "top_k": 0,
-    "top_p": 0.9,
-    "greedy": False,
-    "all_probs": False,
-    "repetition_penalty": 1.2,
-    "min_tokens_to_generate": 2,
-}
+model = AutoModelForCausalLM.from_pretrained("Writer/palmyra-base", torch_dtype=torch.float16).cuda()
 
-sentences = request_data(data)
-print(sentences)
-```
+# the fast tokenizer currently does not work correctly
+tokenizer = AutoTokenizer.from_pretrained("Writer/palmyra-base", use_fast=False)
 
+prompt = "What is the color of a carrot?\nA:"
 
-## Training Data
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
 
-|    part        | MassiveText (sampling) | tokens (B) | sampling ratio |
-|:---------------|-----------------------:|:----------:| --------------:|
-| mc4 filtered   |   MassiveWeb (48%)     |   1331     |   58%          |
-| TrustedWeb     |   -                    |    -       |   -            |
-| realnews       |   News (10%)           |    21      |   10%          |
-| c4             |   c4  (10%)            |    -       |   -            |
-| wikipedia-40B  |   wikipedia  (2%)      |    2       |   5%           |
-| github         |   github  (3%)         |    -       |   -            |
-| books          |   books (27%)          |    24      |   27%          |
-| youtube        |   -                    |    -       |   -            |
+generated_ids = model.generate(input_ids)
 
+tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
 
 
-## Evaluation results
+```
+
+### Limitations and Biases
 
-*Zero-shot performance.* Evaluated using [LM Evaluation Test Suite from AI21](https://github.com/AI21Labs/lm-evaluation)
+Palmyra's core functionality is to take a string of text and predict the next token. While language models are widely used for other tasks, there are many unknowns in this work. When prompting Palmyra, keep in mind that the statistically most likely next token is not always the token that produces the most "accurate" text. Never rely on Palmyra to produce factually correct results.
 
-| ARC-Challenge	| ARC-Easy | RACE-middle | RACE-high | Winogrande | RTE | BoolQA | HellaSwag | PiQA |
-| ------------- | -------- | ----------- | --------- | ---------- | --- | ------ | --------- | ---- |
-| 0.3976        | 0.5566  | 0.5007       | 0.4171    | 0.6133     | 0.5812 | 0.6356 | 0.6298 | 0.7492 | 
+Palmyra was trained on Writer custom data. As with all language models, it is difficult to predict how Palmyra will respond to specific prompts, and offensive content may appear unexpectedly. We recommend that the outputs be curated or filtered by humans before they are released, both to censor undesirable content and to improve the quality of the results.
 
-## Limitations
+## Evaluation results
 
-The model was trained on the data originally crawled from the Internet. This data contains toxic language and societal biases. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts.   
+Evaluation of Palmyra-base model on the SuperGLUE benchmark
 
-## References
 
-[1] [Improving Language Understanding by Generative Pre-Training](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
+|   Task     | Metric | Value |
+|------------|--------|-------|
+|   boolq    |  acc   | 64.43 |
+|   cb       |  acc   | 10.71 |
+|            |  f1    | 08.32 |
+|   copa     |  acc   | 76.00 |
+|   multirc  |  acc   | 01.26 |
+|   record   |  f1    | 84.02 |
+|            |  em    | 83.29 |
+|   wic      |  acc   | 50.00 |
+|   wsc      |  acc   | 36.54 |
 
-[2] [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/pdf/1909.08053.pdf)
 
-[3] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
+## Citation and Related Information
 
-[4] [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://arxiv.org/abs/2101.00027)
 
-## Licence
+To cite this model:
+```bibtex
+@misc{Palmyra,
+  author = {Kiran and Komatsuzaki, Aran},
+  title = {{GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model}},
+  howpublished = {\url{https://github.com/kingoflolz/mesh-transformer-jax}},
+  year = 2021,
+  month = May
+}
+```
 
-License to use this model is covered by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). By downloading the public and release version of the model, you accept the terms and conditions of the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) license.