push_to_hub returns "OSError: error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408"
Author: MicPie
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-96-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Following the docs at https://huggingface.co/course/chapter5/5#uploading-the-dataset-to-the-hugging-face-hub.
In my case I try to upload 413299 smaller jsonl files that are separate tasks that get joined depending on the data subset in the dataset setup later (which is why I would like to keep them separated).

This happens after some time when I run `repo.push_to_hub()` but nothing shows up on the dataset site on the HF hub (currently private until everything is finished):
```
Several commits (5) will be pushed upstream.
The progress bars may be unreliable.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File ~/miniconda3/envs/lmproj2/lib/python3.8/site-packages/huggingface_hub/repository.py:1201, in Repository.git_push(self, upstream, blocking, auto_lfs_prune)
   1200             if return_code:
-> 1201                 raise subprocess.CalledProcessError(
   1202                     return_code, process.args, output=stdout, stderr=stderr
   1203                 )
   1205 except subprocess.CalledProcessError as exc:

CalledProcessError: Command '['git', 'push', '--set-upstream', 'origin', 'main']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
Input In [100], in <cell line: 1>()
----> 1 repo.push_to_hub()

File ~/miniconda3/envs/lmproj2/lib/python3.8/site-packages/huggingface_hub/repository.py:1475, in Repository.push_to_hub(self, commit_message, blocking, clean_ok, auto_lfs_prune)
   1473 self.git_add(auto_lfs_track=True)
   1474 self.git_commit(commit_message)
-> 1475 return self.git_push(
   1476     upstream=f"origin {self.current_branch}",
   1477     blocking=blocking,
   1478     auto_lfs_prune=auto_lfs_prune,
   1479 )

File ~/miniconda3/envs/lmproj2/lib/python3.8/site-packages/huggingface_hub/repository.py:1206, in Repository.git_push(self, upstream, blocking, auto_lfs_prune)
   1201                 raise subprocess.CalledProcessError(
   1202                     return_code, process.args, output=stdout, stderr=stderr
   1203                 )
   1205 except subprocess.CalledProcessError as exc:
-> 1206     raise EnvironmentError(exc.stderr)
   1208 if not blocking:
   1210     def status_method():

OSError: error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date
```

### Expected behavior

```shell
Data is completely uploaded and shows up on the dataset site on the HF hub.
```


Comments:
Maybe there is also a git command line workaround?

Yes, can you try with a `git push` command line to see if it works that way?

Hi @julien-c,
thank you for the fast feedback, I just tried it two times, at the beginning it looks like it works but then gets stuck again:
```
> git push
Enumerating objects: 413419, done.
Counting objects: 100% (413419/413419), done.
Delta compression using up to 6 threads
Compressing objects: 100% (410375/410375), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: the remote end hung up unexpectedly
Writing objects: 100% (413417/413417), 561.79 MiB | 1.75 MiB/s, done.
Total 413417 (delta 3098), reused 413302 (delta 3042)
fatal: the remote end hung up unexpectedly
Everything up-to-date

> git push
Enumerating objects: 413419, done.
Counting objects: 100% (413419/413419), done.
Delta compression using up to 6 threads
Compressing objects: 100% (410375/410375), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
fatal: the remote end hung up unexpectedly3 MiB | 239.00 KiB/s
Writing objects: 100% (413417/413417), 561.79 MiB | 1.81 MiB/s, done.
Total 413417 (delta 3098), reused 413302 (delta 3042)
fatal: the remote end hung up unexpectedly
Everything up-to-date
```
The total size of the data directory is around 5GB.

@julien-c It seems that I run all the time into the issue from above. I guess the best workaround would be to go for a single jsonl files setup, or what do you think?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, it worked fine with me when i connected to internet via my mobile hotspot .
i don't know what was the reason but this was the solution

I also had the same problem, switching to a Wi-Fi network worked for me.

Switching to a Wi-Fi network also worked for me.

I also had the same problem, switching to a Wi-Fi network worked for me.
but i don't know y if any one know the answer please tell me

I also had the same problem, switching to a Wi-Fi network also worked for me.
If anybody know why please post it.

error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
I tried all possible method but it still fails. 


It didn't worked for me

switching wifi does not work for me.

So for those of you still suffering from this issue and for those in the future, here is a detailed log of what I have tried:
entering 
git config http.postBuffer <number of bytes, default = 1 MiB> 
into the terminal can work, but its a one-time fix and should be avoided. This is a good option if you've included large files, like zip files, into your commit history and dont want to fix that issue. Remember to set it back later. In my case, it just lead to another issue.

It could also be the time out feature when pushing/pulling. This could be the case if you are working with large files. Here you'd want to edit the http.lowSpeedLimit, http.lowSpeedTime config values. Documentation for this and http.postBuffer can be found here: https://git-scm.com/docs/git-config

Changing wiFi can work for some people, but depending on the root cause of the issue this wont work. If you sometimes get this error and sometimes don't regardless of what you are pushing/pulling, then this is most likely the solution for you.

I was finally able to fix my issue. First, I changed http.postBuffer to be 50MiB. This then showed me that lfs files were still being pushed despite being tracked by lfs. I had to do git lfs migrate import --everything --verbose --include="<file extension>" to fix it for me. I hope this helps!



> I also had the same problem, switching to a Wi-Fi network also worked for me. If anybody know why please post it.

Yes,for me also it had the same issue.
Thanks.

Increase the buffer size by typing 

git config http.postBuffer 9999999999999999

and use the SSH link instead of HTTP as SSH is more stable

git remote set-url origin **git@github.com:username/repository.git**

> Increase the buffer size by typing
> 
> git config http.postBuffer 9999999999999999
> 
> and use the SSH link instead of HTTP as SSH is more stable
> 
> git remote set-url origin **[git@github.com](mailto:git@github.com):username/repository.git**

Thank you. It worked for me.
9999999999999999 was too big that it caused an error in my case.

The solution was for me to type :
**git config http.postBuffer 99999999**

-> set to show hidden files in your pc and get access of .git folder
-> delete .git folder
-> init again, make sure you have README.md file
-> now commit and push

this steps will definitely work 

switching from wi-fi to cable wroked for me.

thanks. the issue was wii-fi 

> So for those of you still suffering from this issue and for those in the future, here is a detailed log of what I have tried: entering git config http.postBuffer <number of bytes, default = 1 MiB> into the terminal can work, but its a one-time fix and should be avoided. This is a good option if you've included large files, like zip files, into your commit history and dont want to fix that issue. Remember to set it back later. In my case, it just lead to another issue.
> 
> It could also be the time out feature when pushing/pulling. This could be the case if you are working with large files. Here you'd want to edit the http.lowSpeedLimit, http.lowSpeedTime config values. Documentation for this and http.postBuffer can be found here: https://git-scm.com/docs/git-config
> 
> Changing wiFi can work for some people, but depending on the root cause of the issue this wont work. If you sometimes get this error and sometimes don't regardless of what you are pushing/pulling, then this is most likely the solution for you.
> 
> I was finally able to fix my issue. First, I changed http.postBuffer to be 50MiB. This then showed me that lfs files were still being pushed despite being tracked by lfs. I had to do git lfs migrate import --everything --verbose --include="" to fix it for me. I hope this helps!

do git lfs migrate import work for me 

Hi, I had the same problem, and switching from wifi to mobile internet works for me

when i push my next js project in github i got this error whats the error how can i fix it ? . . here is error   objects: 274, done.
Counting objects: 100% (274/274), done.
Delta compression using up to 4 threads
Compressing objects: 100% (259/259), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (274/274), 156.53 MiB | 798.00 KiB/s, done.
Total 274 (delta 3), reused 0 (delta 0), pack-reused 0 (from 0)
fatal: the remote end hung up unexpectedly
Everything up-to-date