Trouble fine tuning Huggingface GPT-2 on Colab — Assertion error
Author: texturejc
Pull request: no

[Cross posted from SO]

I wish to fine tune Huggingface's GPT-2 transformer model on my own text data. I want to do this on a Google Colab notebook. However, it doesn't seem to work.

I install the various bits and pieces via the Colab:

```
!git clone https://github.com/huggingface/transformers
%cd transformers
!pip install .
!pip install -r ./examples/requirements.txt
```

Following the example, I upload the suggested WikiText sample data to the for training and run the suggested CLI commands in the notebook.

```
!export TRAIN_FILE=wiki.train.raw
!export TEST_FILE=wiki.test.raw

!python run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```

This chugs along for a bit, but then I get an assertion error:

```
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 790, in <module>
    main()
  File "run_lm_finetuning.py", line 735, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 149, in load_and_cache_examples
    return TextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "run_lm_finetuning.py", line 88, in __init__
    assert os.path.isfile(file_path)
AssertionError
```
When I run this script via CLI on my own machine it works fine, with the problem that it takes forever to do anything. Why does Colab present this specific problem? Thanks!

Comments:
The error raised means that it cannot find the files you gave it. Do you manage to load the files using the 
`with open` syntax without using the script?

>  Do you manage to load the files using the `with open` syntax without using the script?

Thanks for the reply. I guess the answer is 'no', as I'm not sure what you mean. The files are in the same directory as the script. Should I interpret what you say to mean that I open the files as text and then upload the result? I.e.

```
open("wiki.train.raw", "rb") as file:
    data = file.read()
with open("wiki_train.txt") as f:
    f.write(data)
```
Then upload wiki_train.txt to the Colab and use the CLI to access that when fine tuning?


LysandreJik wants to know if you can open and read the files with 'regular' python on Colab (i.e. just read the file and print some lines in another cell). The error message tells us that the script can't find the train dataset (i.e. wiki.train.raw) and this suggests somekind of a path issue. Maybe you can resolve this with by using the absolute path to the file. 

>LysandreJik wants to know if you can open and read the files with 'regular' python on Colab (i.e. just read the file and print some lines in another cell). 

Ah, sorry. But yes, I can do this no problem:

```
with open("/content/wiki.test.raw") as file:
    data = file.read()

data[:100]

' \n = Robert Boulter = \n \n Robert Boulter is an English film , television and theatre actor . He had '
```

I then attempted to use absolute filepaths, just to be sure, but no joy:

```
!export TRAIN_FILE=/content/wiki.train.raw
!export TEST_FILE=/content/wiki.test.raw

!python /content/transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```
```
Traceback (most recent call last):
  File "/content/transformers/examples/run_lm_finetuning.py", line 790, in <module>
    main()
  File "/content/transformers/examples/run_lm_finetuning.py", line 735, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/content/transformers/examples/run_lm_finetuning.py", line 149, in load_and_cache_examples
    return TextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "/content/transformers/examples/run_lm_finetuning.py", line 88, in __init__
    assert os.path.isfile(file_path)
AssertionError
```
I am entirely stumped by this. Any further ideas what might be happening?

Is there a way for you to share your colab notebook so that I can take a look?

> Is there a way for you to share your colab notebook so that I can take a look?

Absolutely; please use the link below. This has all the steps outlined in my previous replies. I'll keep the Colab active for as long as I can.

https://colab.research.google.com/drive/1qx2t0KleLyY_EncLyM1leSRFz7VooP0e

Environment variables exported with !export are not registered by google colab in them same shell (they are registered in a sub-shell). Just set them with %env like

```
%env TRAIN_FILE=/content/wiki.train.raw
%env TEST_FILE=/content/wiki.test.raw
```
or avoid them by setting them directly:
```
!python /content/transformers/examples/run_lm_finetuning.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=/content/wiki.train.raw \
    --do_eval \
    --eval_data_file=/content/wiki.test.raw
```

>Environment variables exported with !export are not registered by google colab in them same shell (they are registered in a sub-shell). Just set them with %env 

This is fantastic, thanks; it has completely resolved the issue with respect to the initial error. However, I'm now having a different set of issues. When I run the fine tuning script, training doesn't seem to occur; it stops before the first epoch and iteration is complete:

```
01/30/2020 18:25:44 - INFO - __main__ -   ***** Running training *****
01/30/2020 18:25:44 - INFO - __main__ -     Num examples = 244
01/30/2020 18:25:44 - INFO - __main__ -     Num Epochs = 1
01/30/2020 18:25:44 - INFO - __main__ -     Instantaneous batch size per GPU = 4
01/30/2020 18:25:44 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
01/30/2020 18:25:44 - INFO - __main__ -     Gradient Accumulation steps = 1
01/30/2020 18:25:44 - INFO - __main__ -     Total optimization steps = 61
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/61 [00:00<?, ?it/s]^C
```
But there is nevertheless a model saved: `content/gpt2_cached_lm_1024_GE_train.txt`

However, when I run the text generation script, 
```
!python /content/transformers/examples/run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=content/gpt2_cached_lm_1024_GE_train.txt
```
I get the following error:
```
Traceback (most recent call last):
  File "/content/transformers/examples/run_generation.py", line 237, in <module>
    main()
  File "/content/transformers/examples/run_generation.py", line 200, in main
    tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 309, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 410, in _from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name 'content/gpt2_cached_lm_1024_GE_train.txt' was not found in tokenizers model name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). We assumed 'content/gpt2_cached_lm_1024_GE_train.txt' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```
Now, I don't know if this is because the training hasn't occurred successfully, or if I'm loading the model incorrectly. Sorry for being such a pain on this, and thanks for the help so far!

> content/gpt2_cached_lm_1024_GE_train.txt

That is not the model, but the features generated from your training data. I assume that GE_train.txt is a file which contains your training data (environment variable $TRAIN_FILE)?

Does the script work for you with the WikiText-2 dataset? It works for me on colab with a reduced batch size (i.e. --per_gpu_train_batch_size=2).




>Does the script work for you with the WikiText-2 dataset? It works for me on colab with a reduced batch size (i.e. --per_gpu_train_batch_size=2).

Yes, reducing the batch size works nicely, thanks. 

Two final questions and I'll close this, if that's OK.

1. So when I fine tune in the colab,  I don't need to load the fine tuned model separately? As in, my local GPT-2 model _is_ the fine tuned model, and I can call it in the usual way?

2. Is there any way of increasing the length of both the prompt text and the generated text from `run_generation.py`? As of now, the prompt text just gets reproduced if it's too long and the generated text is usually just one sentence length piece. I assume that this is the line of code responsible for this,
`text = text[: text.find(args.stop_token) if args.stop_token else None]`
but i wonder is there any parameter I can adjust in the CLI without digging into the bowels of the script itself?

Glad you could get the script to work! Concerning your questions:

1. You can load the fine-tuned model as you would any model, just point the `model_name_or_path` from `run_generation` to the directory containing your finetuned model.

2. You can increase the length by specifying the `--length` argument to `run_generation`. Up until this morning there was an issue with the script where it wouldn't sample from the generation, instead always taking the argmax of all tokens generated. This generally results in some repetition, which might be what you were facing as you say the `prompt text just gets reproduced`.

I would try to pull the repository once again, making sure you have the last version so that it samples correctly according to the `--k` and `--p` arguments, which you can modify to generate different completions.

If you don't specify a stop token, it should not stop at the end of a sentence. For example, with the following arguments for `run_generation`:
```
--model_type=gpt2 
--model_name_or_path=gpt2 
--k=50 
--p=0.9 
--length=200
```

With the following sample text: `The horse is`, I get the following completion:

```
The horse is just barely alive as he's been playing with the dog. The dog is trying to get its life back. She's been waiting for him for a long time. It's been long and hard. I have to move her to the backseat of my car.

When we start talking, he looks over at us like it's about to move off the road. He tells me he's been trying to get a good driver since he moved in. I don't know what to think. The only way he can drive me like that is to ride his horse to get his life back.

My sister has been following my life. I am her daughter now. She has always been my mother and always has been.

My dad has been the only one in my family. He never told me. He was a nice guy, but he was very controlling. He didn't tell anyone. He always would take a picture of me with his horse.
!

```

>Glad you could get the script to work! Concerning your questions:

Thanks so much for all the help! This is a really comprehensive answer and therefore very useful––and not just for me either, I'll wager. Great, I'll get cracking with my project so with all these provisos duly noted!

Happy to help!