Update README.md

README.md

@@ -27,24 +27,7 @@ img {
 
 ## Model Description
 
-Palmyra-small 128M is a transformer-based language model. GPT refers to a class of transformer decoder-only models similar to GPT-2 and 3 while. It has Tensor Parallelism (TP) of 1, Pipeline Parallelism (PP) of 1 and should fit on a single NVIDIA GPU.
-
-<figure>
-
-| Hyperparameter       | Value      |
-|----------------------|------------|
-| \\(n_{parameters}\\) | 6053381344 |
-| \\(n_{layers}\\)     | 28&ast;    |
-| \\(d_{model}\\)      | 4096       |
-| \\(d_{ff}\\)         | 16384      |
-| \\(n_{heads}\\)      | 16         |
-| \\(d_{head}\\)       | 256        |
-| \\(n_{ctx}\\)        | 2048       |
-| \\(n_{vocab}\\)      | 50257/50400&dagger; (same tokenizer as GPT-2/3)  |
-| Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864) |
-| RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
-<figcaption><p><strong>&ast;</strong> Each layer consists of one feedforward block and one self attention block.</p>
-<p><strong>&dagger;</strong> Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer.</p></figcaption></figure>
+Palmyra was primarily pretrained with English text, there is still a trace amount of non-English data present within the training corpus that was accessed through CommonCrawl. A causal language modeling (CLM) objective was utilized during the process of the model's pretraining. Similar to GPT-3, Palmyra is a member of the same family of models that only contain a decoder. As a result, it was pretrained utilizing the objective of self-supervised causal language modeling. Palmyra uses the prompts and general experimental setup from GPT-3 in order to conduct its evaluation in accordance with GPT-3. Read the official paper if you want more information about this.
 
 The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model
 dimension is split into 16 heads, each with a dimension of 256. Rotary Position Embedding (RoPE) is applied to 64
@@ -143,5 +126,3 @@ To cite the codebase that trained this model:
   month = May
 }
 ```
-
-If you use this model, we would love to hear about it! Reach out on [GitHub](https://github.com/kingoflolz/mesh-transformer-jax), Discord, or shoot Ben an email.