---
language:
  - en
library_name: pytorch
tags:
  - language-model
  - gpt2
  - transformer
  - wikitext-103

model-index:

- name: gpt2_wt103-40m_12-layer
  results:
  - task:
      type: language-modeling
    dataset:
      type: wikitext
      name: Wikitext-103
    metrics:
      - type: perplexity
        value: 40.3 
    
---

# Model description

paper: [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://arxiv.org/abs/2210.13569)

This is a gpt2-small-like decoder-only transformer model trained on a 40M token subset of the [wikitext-103 dataset](https://paperswithcode.com/dataset/wikitext-103).

# Usage

You can download and load the model as follows:

```python
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("Kristijan/gpt2_wt103-40m_12-layer")

```

Alternatively, if you've downloaded the checkpoint files in this repository, you could also do:

```python
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained(path_to_folder_with_checkpoint_files)

```

To tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103-tokenizer)

# Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.