|
--- |
|
language: |
|
- en |
|
library_name: pytorch |
|
tags: |
|
- language-model |
|
- gpt2 |
|
- transformer |
|
- wikitext-103 |
|
|
|
model-index: |
|
|
|
- name: gpt2_wt103-40m_12-layer |
|
results: |
|
- task: |
|
type: language-modeling |
|
dataset: |
|
type: wikitext |
|
name: Wikitext-103 |
|
metrics: |
|
- type: perplexity |
|
value: 40.3 |
|
|
|
--- |
|
|
|
# Model description |
|
|
|
paper: [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://arxiv.org/abs/2210.13569) |
|
|
|
This is a gpt2-small-like decoder-only transformer model trained on a 40M token subset of the [wikitext-103 dataset](https://paperswithcode.com/dataset/wikitext-103). |
|
|
|
# Usage |
|
|
|
You can download and load the model as follows: |
|
|
|
```python |
|
from transformers import GPT2LMHeadModel |
|
|
|
model = GPT2LMHeadModel.from_pretrained("Kristijan/gpt2_wt103-40m_12-layer") |
|
|
|
``` |
|
|
|
Alternatively, if you've downloaded the checkpoint files in this repository, you could also do: |
|
|
|
```python |
|
from transformers import GPT2LMHeadModel |
|
|
|
model = GPT2LMHeadModel.from_pretrained(path_to_folder_with_checkpoint_files) |
|
|
|
``` |
|
|
|
To tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103-tokenizer) |
|
|
|
# Intended uses |
|
|
|
This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets. |
|
|