text: 
  
  TITLE
  Problem while pretraining MLM from scratch using Transformers
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
       
  - `transformers` version: 3.1.0
  - Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core
  - Python version: 3.6.9
  - PyTorch version (GPU?): 1.6.0 (True)
  - Tensorflow version (GPU?): 2.3.0 (True)
  - Using GPU in script?: Yes(Two GPUs, Nvidia P100)
  - Using distributed or parallel set-up in script?: No
  
  ### Who can help
  <!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
   If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
   Please tag fewer than 3 people.
   
  
   Trainer: **@sgugger** 
   
   -->
   Trainer: @sgugger
  ## Information
  
  Model I am using (RoBERTa): 
  
  The problem arises when using:
  * [* ] my own modified scripts: (give details below)
  '''Python
  config=RobertaConfig(vocab_size=30_000, max_position_embeddings=inputlen, num_attention_heads=12,
                       num_hidden_layers=6,hidden_dropout_prob=0.1,attention_probs_dropout_prob=0.1,
                       initializer_range=0.2, intermediate_size=3072, type_vocab_size=1)
  
  training_args=TrainingArguments(output_dir="/LAB_SHARED/Projects/011-Language_Model/Data/LanguageModel/BadgerBERT",overwrite_output_dir=True,
                                  num_train_epochs=10, do_train=True, do_eval=True, evaluate_during_training=True,
                                  per_gpu_train_batch_size=64, learning_rate=0.0004,
                                  gradient_accumulation_steps=32,
                                  logging_steps=128,
                                  warmup_steps=30000,
                                  weight_decay=0.01,
                                  eval_steps=128,
                                  save_steps=128,
                                  save_total_limit=2, prediction_loss_only=True)
  
  trainer=Trainer(model=model,args=training_args,data_collator=data_collator,train_dataset=dataset,eval_dataset=dataset1,prediction_loss_only=True)
  
  '''
  
  The tasks I am working on is:
  * [ *] my own task or dataset: (give details below)
  Training masked language model
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Please use the following configuration
  2. I used my own training dataset, but I guess there should be no difference. My input text file size is around 40GB
  
  
  <!-- If you have code snippets, error messages, stack traces please provide them here as well.
       Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
       Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
  
  Traceback (most recent call last):█████████████████████████████████████████████████████████████████▉| 279691/279692 [5:22:01<00:00, 15.48it/s]
    File "CreateLanguageModel.py", line 84, in <module>
      trainer.train()
    File "**/lib64/python3.6/site-packages/transformers/trainer.py", line 785, in train
      torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
    File "**/lib64/python3.6/site-packages/torch/serialization.py", line 361, in save
      with _open_file_like(f, 'wb') as opened_file:
    File "**/lib64/python3.6/site-packages/torch/serialization.py", line 229, in _open_file_like
      return _open_file(name_or_buffer, mode)
    File "**/lib64/python3.6/site-packages/torch/serialization.py", line 210, in __init__
      super(_open_file, self).__init__(open(name, mode))
  FileNotFoundError: [Errno 2] No such file or directory: '**/checkpoint-128/optimizer.pt'
  
  ## Expected behavior
  Instead of continue training, it stops. It seems that the problem is related to the check-points. I don't know why it tries to open (save) check-point-128 while it already saved 512!
  Besides that it already saved check-point-160 while I did not set any parameter as 160, so I don't know where the 160 came from.
  I expected the code to just report the evaluation loss periodically, save checkpoints and finish the training.
  
  <!-- A clear and concise description of what you would expect to happen. -->
  