run_lm_finetuning.py regenerates examples cache when restored from a checkpoint, is this intended?
Author: mike-athene
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello,

I am finetuning a gpt2-medium model on a large (800mb+) input via run_lm_finetuning.py on a windown/conda env with recent git checkout of transformers and apex installed (--per_gpu_train_batch_size=1 --fp16 --fp16_opt_level O2 --gradient_accumulation_steps=10 --block_size=512).
On initiall run a cache file is created next to the input file using the model type as filename prefix, i.e. gpt2-medium_cached_lm_512_dataset.txt

This WAI and if I restart the finetuning process the cache is picked up, but when I try to resume by restoring a checkpoint (--model_name_or_path=FULL_PATH\output\checkpoint-200\) the cache is regenerated in the /output folder under a name of checkpoint-200_cached_lm_512_dataset.txt.
(also providing a relative path to model_name_or_path does not seem to work[1])

- Tracing the cache generation code it looks like it depends on the tokenizer which is restored from the checkpoint, but I am not sure if any state from the checkpoint is actually affecting the cache generation. Both files have the same checksum, so it does seem like unnecessary work?
- If not, Is it possible to store current version of the cache alongside other checkpoint data for faster resume?
- Am I restoring the model incorrectly? Even after cache regeneration the models OOMs, but starting from scratch I can finetune way past the checkpoint state. [2]

Thank you!

[1] 
```
Saving features into cached file C:\data\.\output\checkpoint-200\_cached_lm_512_dataset.txt
Traceback (most recent call last):
  File ".\transformers\examples\run_lm_finetuning.py", line 721, in <module>
    main()
  File ".\transformers\examples\run_lm_finetuning.py", line 666, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File ".\transformers\examples\run_lm_finetuning.py", line 130, in load_and_cache_examples
    block_size=args.block_size,
  File ".\transformers\examples\run_lm_finetuning.py", line 115, in __init__
    with open(cached_features_file, "wb") as handle:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\data\\.\\output\\checkpoint-200\\_cached_lm_512_dataset.txt'
```

[2]

```
  File ".\transformers\examples\run_lm_finetuning.py", line 721, in <module>
    main()
  File ".\transformers\examples\run_lm_finetuning.py", line 671, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File ".\transformers\examples\run_lm_finetuning.py", line 316, in train
    with amp.scale_loss(loss, optimizer) as scaled_loss:
  File "...\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "...\apex\amp\handle.py", line 111, in scale_loss
    optimizer._prepare_amp_backward()
  File "...\apex\amp\_process_optimizer.py", line 137, in prepare_backward_with_master_weights
    self._amp_lazy_init()
  File "...\apex\amp\_process_optimizer.py", line 309, in _amp_lazy_init
    self._lazy_init_maybe_master_weights()
  File "...\apex\amp\_process_optimizer.py", line 90, in lazy_init_with_master_weights
    self.load_state_dict(self.state_dict())
  File "...\torch\optim\optimizer.py", line 147, in load_state_dict
    state[param] = cast(param, v)
  File "...\torch\optim\optimizer.py", line 134, in cast
    return {k: cast(param, v) for k, v in value.items()}
  File "...\torch\optim\optimizer.py", line 134, in <dictcomp>
    return {k: cast(param, v) for k, v in value.items()}
  File "...\torch\optim\optimizer.py", line 130, in cast
    value = value.to(param.dtype)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.00 GiB total capacity; 8.61 GiB already allocated; 10.74 MiB free; 8.69 GiB reserved in total by PyTorch)
```

Comments:
I also use  --save_total_limit=10 which triggered another exception in checkpoint deletion code as it tried to delete the cache file like it was a folder

```
Deleting older checkpoint [./output\checkpoint-200_cached_lm_512_dataset.txt] due to args.save_total_limit
Traceback (most recent call last):
  File ".\transformers\examples\run_lm_finetuning.py", line 721, in <module>
    main()
  File ".\transformers\examples\run_lm_finetuning.py", line 671, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File ".\transformers\examples\run_lm_finetuning.py", line 360, in train
    _rotate_checkpoints(args, checkpoint_prefix)
  File ".\transformers\examples\run_lm_finetuning.py", line 169, in _rotate_checkpoints
    shutil.rmtree(checkpoint)
  File "...\lib\shutil.py", line 500, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "...\lib\shutil.py", line 382, in _rmtree_unsafe
    onerror(os.listdir, path, sys.exc_info())
  File "...\lib\shutil.py", line 380, in _rmtree_unsafe
    names = os.listdir(path)
NotADirectoryError: [WinError 267] The directory name is invalid: './output\\checkpoint-200_cached_lm_512_dataset.txt'
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.