push_to_hub fails with "cannot lock ref" and "failed to push some refs"
Author: XekriRedmane
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Put together the script from [Automatic Speech Recognition](https://huggingface.co/docs/transformers/tasks/asr) into a file `main.py`, up to but not including Inference.

Run under Windows.

Training succeeds, but then:

```
Pushing to hub...
Several commits (2) will be pushed upstream.
The progress bars may be unreliable.
Upload file pytorch_model.bin: 366MB [02:42, 3.62MB/s]                                                                                                   remote: error: cannot lock ref 'refs/heads/main': is at aa0f87dd56de1a36e17cffb07b4a50a0d0f530f4 but expected 52d558ffa06199a1340c979ed1fbbc0e98c862c8     

To https://huggingface.co/xekri/my_awesome_asr_model
 ! [remote rejected] main -> main (failed to update ref)
error: failed to push some refs to 'https://huggingface.co/xekri/my_awesome_asr_model'

Upload file pytorch_model.bin: 100%|██████████████████████████████████████████████████████████████████████████████████| 360M/360M [02:43<00:00, 2.32MB/s]
Traceback (most recent call last):
  File "f:\eo-reco\.env\Lib\site-packages\huggingface_hub\repository.py", line 1099, in git_push
    raise subprocess.CalledProcessError(return_code, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'push', '--set-upstream', 'origin', 'main']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "f:\eo-reco\main.py", line 147, in <module>
    trainer.push_to_hub()
  File "f:\eo-reco\.env\Lib\site-packages\transformers\trainer.py", line 3661, in push_to_hub
    git_head_commit_url = self.repo.push_to_hub(
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "f:\eo-reco\.env\Lib\site-packages\huggingface_hub\repository.py", line 1307, in push_to_hub
    return self.git_push(
           ^^^^^^^^^^^^^^
  File "f:\eo-reco\.env\Lib\site-packages\huggingface_hub\repository.py", line 1102, in git_push
    raise EnvironmentError(exc.stderr)
OSError: remote: error: cannot lock ref 'refs/heads/main': is at aa0f87dd56de1a36e17cffb07b4a50a0d0f530f4 but expected 52d558ffa06199a1340c979ed1fbbc0e98c862c8
To https://huggingface.co/xekri/my_awesome_asr_model
 ! [remote rejected] main -> main (failed to update ref)
error: failed to push some refs to 'https://huggingface.co/xekri/my_awesome_asr_model'

The push command with PID 7788 failed.
To https://huggingface.co/xekri/my_awesome_asr_model
   52d558f..381666e  main -> main
```

Checking the repo on the hub shows that all files were seemingly committed.

The four commits to the hub were:
* `aa0f87dd56de1a36e17cffb07b4a50a0d0f530f4 `: "End of training"
* `381666e4197a0e5ce2b4d8a9b0c3f426cd2b2348`: "Training in progress, step 200"
* `52d558ffa06199a1340c979ed1fbbc0e98c862c8`: "Training in progress, step 100"
* `09c2ba5a5066b5b24e8fd2ddf333eda61f6c85da`: "initial commit"

In the attached file, the changes from the documented script are:

1. Loading dataset `mozilla-foundation/common_voice_13_0` (since dropbox is rejecting requests to download `PolyAI/minds14`, see [discussion](https://huggingface.co/datasets/PolyAI/minds14/discussions/6))
2. Modifications for columns present in that dataset.

[main.py.zip](https://github.com/huggingface/transformers/files/11413763/main.py.zip)


### Expected behavior

No scary git errors


Comments:
cc @Wauplin 

Hi @RobertBaruch, I'm sorry you're facing this issue. I'm not sure what happened here. Can be either a wrong git setup or a temporary server issue on git operations. Or maybe concurrent push to the Hub which made one fail while having the others correctly uploading. Just to be sure, is the final state of repo as you want it or are you missing something? If something is still missing, I would advice you to save the data locally (with `.save_pretrained`) and then upload the folder using [`huggingface_hub.upload_folder`](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-a-folder).

@sgugger I hope this is the type of unclear error that we could rid off when switching to a http-based approach (once https://github.com/huggingface/huggingface_hub/pull/1458 is merged) :)

This happens every time -- even with the [example for speech recognition](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#single-gpu-ctc). I'm pretty sure this has something to do with Windows.

The final state of the repo appears to be correct. However, the problem is that an error is raised, which means that anything in the program after pushing to the repo will not be executed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hi @Wauplin, I faced the same issue several times. Can you clarify how to replicate what the --push_to_hub does from the command line? Is there a command to create the model card and upload only the files that --push_to_hub (without all checkpoints)? 

how do I know what caused the issue of not uploading to the hub? Can it be that I don't have space left in my account? 

>  Can you clarify how to replicate what the --push_to_hub does from the command line?

Don't know about a command line equivalent but 
@sgugger mentioned  in https://github.com/huggingface/huggingface_hub/issues/1560#issuecomment-1634053167 that you can use `trainer.create_model_card()` to create a model card from your trainer. 

>  Is there a command (...) upload only the files that --push_to_hub (without all checkpoints)?

Once you have files saved locally, uploading them to the Hub can be quickly done using `huggingface_hub`. Here is a guide on [how to upload files to the Hub.](https://huggingface.co/docs/huggingface_hub/guides/upload#upload-files-to-the-hub). It is not a command line tool but rather a few lines of scripts to write. But that's only once you have files saved locally and know which ones you want to upload.

> how do I know what caused the issue of not uploading to the hub? Can it be that I don't have space left in my account?

If the upload fails, it is probably due to some network issues (see https://github.com/huggingface/huggingface_hub/issues/1560#issuecomment-1635878401). In any case, it is not a problem of not have space left on your Hugging Face account since it's unlimited.