run_language_modeling.py line 251: checking if it is a directory
Author: jihwangk
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi, I'm curious why you changed the code to check if model_args.model_name_or_path is a directory. I am trying to train OpenAI GPT in Google Colab, and using arguments as following:

!python /content/transformers/examples/run_language_modeling.py
--output_dir="/content/gdrive/My Drive/output"
--overwrite_output_dir
--model_type=openai-gpt
--model_name_or_path=openai-gpt
--do_train
--train_data_file="/content/gdrive/My Drive/train.txt"
--do_eval
--eval_data_file="/content/gdrive/My Drive/dev.txt"
--num_train_epochs 5
--learning_rate 1.5e-4
--save_steps 1
--save_total_limit 5

but this line causes following error:

Traceback (most recent call last):
File "/content/transformers/examples/run_language_modeling.py", line 283, in
main()
File "/content/transformers/examples/run_language_modeling.py", line 257, in main
trainer.train(model_path=model_path)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 363, in train
self._rotate_checkpoints()
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 458, in _rotate_checkpoints
checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 443, in _sorted_checkpoints
regex_match = re.match(".*{}-([0-9]+)".format(checkpoint_prefix), path)
File "/usr/lib/python3.6/re.py", line 172, in match
return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

It seems that openai-gpt is not a directory, thus making model_path variable equal to None. Is this due to my implementation or can this part be fixed?


<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
I also faced the similler issue while the arguments 
--save_steps 200
--save_total_limit 3 
Removing this arguments worked. But need to know what could be the issue

Did removing them worked through the end completely? Because there is a default value for save_steps, and I just set it to 1 to see if the bug still appears after some manual fixes... Looking at the function names like rotate_checkpoints(), I feel like save_total_limit might be the key, since its default is set to None, but again not sure....

Yes Removing them worked till the end for me. Total Optimization steps for me was around 1600. so it did save 3 times the checkpoints and correctly loaded them as well. 

Ah okay removing the --save_total_limit make it work for me, but then it would start saving gigabytes of models without top limit. I think the directory checking might come from removing the should_continue argument, but will require us to constantly update where we load the latest model from...?

I can confirm this is a bug. I'll push a fix on master shortly.