---
license: apache-2.0
datasets:
- wikitext
- ptb_text_only
language:
- en
metrics:
- perplexity
pipeline_tag: text-generation
model-index:
- name: distilgpt2
  results:
  - task:
      type: text-generation
    dataset:
      name: penn_treebank
      type: ptb_text_only
    metrics:
    - name: perlexity@distilgpt2:BASELINE
      type: dmx-perlexity
      value: 63.45857238769531
    - name: perlexity@distilgpt2:BASIC
      type: dmx-perlexity
      value: 64.36720275878906
  - task:
      type: text-generation
    dataset:
      name: wikitext2
      type: wikitext-2-raw-v1
    metrics:
    - name: perlexity@distilgpt2:BASELINE
      type: dmx-perlexity
      value: 46.05925369262695
    - name: perlexity@distilgpt2:BASIC
      type: dmx-perlexity
      value: 46.570838928222656
---
This is a d-Matrix functional reference of the GPT2 model family, with the following *revisions*: 
- [`distilgpt2`](https://huggingface.co/distilbert/distilgpt2)
- [`gpt2`](https://huggingface.co/openai-community/gpt2)
- [`gpt2-medium`](https://huggingface.co/openai-community/gpt2-medium) 
- [`gpt2-large`](https://huggingface.co/openai-community/gpt2-large) 
- [`gpt2-xl`](https://huggingface.co/openai-community/gpt2-xl)

The reference provides the following functional *configurations*:
  Configuration | Explanation
  :-- | :-- 
  **`BASELINE`** | a reference functionally equivalent to the original model
  **`BASIC`** | all linear algebraic operands quantized to `BFP16-64`, and all other operations transformed to approximated kernel simulations


### Usage

Install d-Matrix [ML Tools](https://github.com/d-matrix-ai/dmx-mltools) first.

```sh
pip install dmx-mltools
```

The following is an example model and its evaluation.  

```python
from mltools.dmx import pipeline

pipe = pipeline(
    task="text-generation",
    model="d-matrix/gpt2",
    revision="gpt2-xl",  # see above for other variants
    dmx_config="BASELINE",  # see above for other variants
)

results = pipe.evaluate(
    metric="d-matrix/dmx_perplexity",
    dataset="wikitext",
    dataset_version="wikitext-2-raw-v1",
)
```

### Evaluation results

- `perplexity` on `penn_treebank`
  Revision \ Configuration | **`BASELINE`** | **`BASIC`** 
  :-- | --: | --: 
  `distilgpt2` | 63.46 | 64.13
  `gpt2` | 35.77 | 35.93
  `gpt2-medium` | 27.06 | 27.10
  `gpt2-large` | 23.03 | 23.04
  `gpt2-xl` | 21.01 | 21.02

- `perplexity` on `wikitext2`
  Revision \ Configuration | **`BASELINE`** | **`BASIC`** 
  :-- | --: | --:
  `distilgpt2` | 46.06 | 46.44
  `gpt2` | 29.94 | 30.08
  `gpt2-medium` | 21.71 | 21.73
  `gpt2-large` | 19.42| 19.43
  `gpt2-xl` | 17.40| 17.40

- `perplexity` on `wikitext103`
  Revision \ Configuration | **`BASELINE`** | **`BASIC`** 
  :-- | --: | --:
  `distilgpt2` | 46.06 | 46.44
  `gpt2` | 29.94 |30.08
  `gpt2-medium` | 21.71 | 21.73
  `gpt2-large` | 19.43 | 19.43
  `gpt2-xl` | 17.40 | 17.40