Error pushing GPT2 flax training model to hub
Author: BirgerMoell
Pull request: no

While training a GPT-2 model using the following scripts the model crashes while pushing to hub. I made the saving step 10 since I suspected was related so saving.
```
#!/usr/bin/env bash
python3 swedish-gpt2-oscar/run_stream_trainer.py \
    --output_dir="${MODEL_DIR}" \
    --model_type="gpt2" \
    --config_name="${MODEL_DIR}" \
    --tokenizer_name="${MODEL_DIR}" \
    --dataset_name="mc4" \
    --dataset_config_name="sv" \
    --do_train --do_eval \
    --block_size="512" \
    --per_device_train_batch_size="64" \
    --per_device_eval_batch_size="64" \
    --learning_rate="5e-3" --warmup_steps="1000" \
    --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01" \
    --overwrite_output_dir \
    --max_steps="100000" \
    --decay_steps="100000" \
    --logging_steps="500" \
    --save_steps="10" \
    --eval_steps="2500" \
    --push_to_hub
```

The files that would be commited.
https://huggingface.co/birgermoell/ckpt-10/commit/d256a3e1fc7dd9da4833c98a21ea689d3caede18

Stacktrace
```
Model weights saved in /home/bmoell/swedish-gpt2-oscar/ckpt-10/flax_model.msgpack
07/11/2021 20:56:44 - INFO - huggingface_hub.repository - Uploading LFS objects: 100% (1/1), 498 MB | 33 MB/s, done.

Model pushed to the hub in this commit: https://huggingface.co/birgermoell/ckpt-10/commit/d256a3e1fc7dd9da4833c98a21ea689d3caede18
07/11/2021 20:56:45 - INFO - __main__ - checkpoint saved
07/11/2021 20:56:45 - INFO - absl - Saving checkpoint at step: 10
tcmalloc: large alloc 1373577216 bytes == 0x241732000 @  0x7ff956432680 0x7ff956452bdd 0x7ff67595f20d 0x7ff67596d340 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff675968bd3 0x7ff6759691fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x5f5b33 0x56aadf
tcmalloc: large alloc 2986590208 bytes == 0x293524000 @  0x7ff956432680 0x7ff956452bdd 0x7ff67595f20d 0x7ff67596d340 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff67596ce87 0x7ff675968bd3 0x7ff6759691fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x5f5b33 0x56aadf 0x568d9a 0x68cdc7 0x67e161
tcmalloc: large alloc 1493295104 bytes == 0x1eff42000 @  0x7ff956432680 0x7ff956453824 0x5f7b11 0x7ff675968c6f 0x7ff6759691fe 0x504d56 0x56acb6 0x568d9a 0x5f5b33 0x56bc9b 0x5f5956 0x56aadf 0x5f5956 0x56fb87 0x568d9a 0x5f5b33 0x56bc9b 0x568d9a 0x5f5b33 0x56aadf 0x568d9a 0x68cdc7 0x67e161 0x67e1df 0x67e281 0x67e627 0x6b6e62 0x6b71ed 0x7ff9562490b3 0x5f96de
07/11/2021 20:56:53 - INFO - absl - Saved checkpoint at swedish-gpt2-oscar/checkpoint_10
Traceback (most recent call last):
  File "swedish-gpt2-oscar/run_stream_trainer.py", line 818, in <module>
    main()
  File "swedish-gpt2-oscar/run_stream_trainer.py", line 805, in main
    save_checkpoint(training_args.output_dir, jax_utils.unreplicate(state), cur_step, keep=training_args.save_total_limit, overwrite=False)
  File "/home/bmoell/gpt2/lib/python3.8/site-packages/flax/training/checkpoints.py", line 139, in save_checkpoint
    if len(checkpoint_files) > keep:
TypeError: '>' not supported between instances of 'int' and 'NoneType'
https://symbolize.stripped_domain/r/?trace=7ff9562123f4,7ff95626820f,7f&map= 
*** SIGTERM received by PID 61911 (TID 61911) on cpu 29 from PID 60845; stack trace: ***
PC: @     0x7ff9562123f4  (unknown)  do_futex_wait.constprop.0
    @     0x7ff94d15e800        976  (unknown)
    @     0x7ff956268210  348884112  (unknown)
    @               0x80  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7ff9562123f4,7ff94d15e7ff,7ff95626820f,7f&map=2a762cd764e70bc90ae4c7f9747c08d7:7ff94021c000-7ff94d49d280 
E0711 20:56:53.386296   61911 coredump_hook.cc:250] RAW: Remote crash gathering disabled for SIGTERM.
E0711 20:56:54.372857   61911 process_state.cc:771] RAW: Raising signal 15 with default behavior
  0%|                                               | 11/100000 [01:37<245:42:13,  8.85s/it]
```

Comments:
The training script `run_stream_trainer.py` is not an official training script no? Where can I find `run_stream_trainer.py` ? The error also does not seem to be related to pushing to the hub but rather with the line `keep=training_args.save_total_limit`.

I noticed some strange git behaviour including the creation of another repo where the files were uploaded.
https://huggingface.co/birgermoell/ckpt-10/tree/main

This is likely related to git and not related to transformers so I'm closing the issue and I'm hoping to resolve it.

Thank you so much for the help debugging.


> The training script `run_stream_trainer.py` is not an official training script no? Where can I find `run_stream_trainer.py` ? The error also does not seem to be related to pushing to the hub but rather with the line `keep=training_args.save_total_limit`.

It's true that it's not an official script. Uploading the script manually now.