transformers-cli: LFS multipart uploads (> 5GB)
Author: julien-c
Pull request: yes

### Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs. This lets users upload large files >5GB 🔥.

Spec for LFS custom transfer agent is: https://github.com/git-lfs/git-lfs/blob/master/docs/custom-transfers.md

The PR introduces two commands to the CLI:
```
transformers-cli lfs-enable-largefiles ./path/to/repo
```
^ Do this once per model repo where you want to push >5GB files. It's documented in the error message you get if you just try to `git push` a 5GB file without having enabled it before.

```
transformers-cli lfs-multipart-upload
```

^ is the custom transfer agent itself. This is not meant to be called by the user, but by lfs directly.

### Things to experiment with:
- [ ] upload speed. Is it sufficient? Please comment with your upload speeds e.g. for https://huggingface.co/t5-3b

Experiment:
```bash
time git clone https://huggingface.co/t5-3b
cd t5-3b
git remote set-url origin https://huggingface.co/$USER/t5-3b-clone
# ^ After having created this model repo in your account
transformers-cli lfs-enable-largefiles .

git reset 5e0a32db352b33091ea9fb2f8d8782d47a505986 # go back to initial commit for lfs to reupload files
git add pytorch_model.bin
git commit -m "ok"
time git push
```

Comments:
:+1:  nice job!

Hi, this command may need a args?
```
transformers-cli lfs-enable-largefiles repo_path
```


> Hi, this command may need a args?

Yes, correct


Thanks a lot for the PR, and thanks for letting me know.

I will give my feedback after testing it with the 11B model soon. 

Hi, thanks a lot for the PR.
I try to reinstall transformers with this new branch and follow with this comment:
```
git clone https://huggingface.co/mymusise/CPM-Third-Party
cd CPM-Third-Party
git lfs install 
transformers-cli lfs-enable-largefiles .

cp ../models/tf_model.h5 ./
git add . && git commit -m 'add model'
git push
```
Then, after half an hour later, it raised an error:
```
$ git push
Git LFS: (0 of 1 files) 0 B / 9.68 GB
Git LFS: (0 of 1 files)  4.66 GB / 9.68 GB
EOFoading LFS objects:   0% (0/1), 9.31 B / 9.68 GB
error: failed to push some refs to 'https://huggingface.co/mymusise/CPM-Third-Party'
```

Did I do something wrong?

@mymusise might have been an intermittent server error. Can you try again?

> @mymusise might have been an intermittent server error. Can you try again?

Yes, I try again and again. But it always raises this error at `9.31 GB / 9.68 GB`.

:eyes:  But, now `git push` will return a ` 502 ` error :
```
fatal: unable to access 'https://huggingface.co/mymusise/CPM-Third-Party/': The requested URL returned error: 502
```

@mymusise Yes, looks like this is crashing/locking the server 😱

Do you mind trying again on Monday? As we'll have more bandwidth to fix then. Sorry about that :/

It's ok, guy, haha. Waiting for your good news. 
Happy weekend!

>Yes, I try again and again. But it always raises this error at 9.31 GB / 9.68 GB.

Hi, I try again today and push the big model file without any exception! :tada: 
Thank guys!

Hi, here I push another model file(4.9GB) again, but this time it gives me a **504** Gateway Time-out  error :sweat:

```
root@iZt4n9z4x3ph9oc3hhrdneZ:~/CPM-FP16-Third-Party# time git push 
Username for 'https://huggingface.co': mymusise
Password for 'https://mymusise@huggingface.co': 
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects:  66% (2/3), 2.52 GiB | 3.62 MiB/s      
Writing objects: 100% (3/3), 4.46 GiB | 3.77 MiB/s, done.
Total 3 (delta 0), reused 1 (delta 0)
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

real	22m8.653s
user	0m29.758s
sys	0m17.136s
```

Seems the big file is uploaded completely, it looks like there is some problem with the server configuration about the timeout. @julien-c 

Tried three times with the same result.

@Pierrci - I ran some tests on ~10 GB files (`t5-3b`) and didn't encounter any problems! However when doing it for ~45GB files (`t5-11b`), I encounter some problems.

```
cd t5-11b-repo
transformers-cli lfs-enable-largefiles /path/to/repo
git add .
git commit -m "add"
git push # <= this command failse
```

In case it is useful, here is a link to the `git trace` error message: https://github.com/patrickvonplaten/files_to_link_to/blob/master/output.txt

I can always go back to the way of manually uploading to the git-lfs hash path. Maybe the error message is helpful though :-) 

Thanks @patrickvonplaten, I see where it might be coming from, gonna look at it now!

I deployed a fix that should address your problem, can you try again @patrickvonplaten?

@mymusise Is it possible for you to try again with `GIT_CURL_VERBOSE=1 git push` so that we can try to get more information? From what you shared so far, your error seems to be different from Patrick's one.

Thanks, @Pierrci. Yes, I think my error is different from Patrick's one. Here is the [information](https://gist.github.com/mymusise/bfa331e3effe0876efbc0011334ead96) with `GIT_CURL_VERBOSE=1 git push`. 
Hope it help.


> I deployed a fix that should address your problem, can you try again @patrickvonplaten?
> 
> @mymusise Is it possible for you to try again with `GIT_CURL_VERBOSE=1 git push` so that we can try to get more information? From what you shared so far, your error seems to be different from Patrick's one.

Awesome it works now @Pierrci - thanks! :-) 

> Thanks, @Pierrci. Yes, I think my error is different from Patrick's one. Here is the [information](https://gist.github.com/mymusise/bfa331e3effe0876efbc0011334ead96) with `GIT_CURL_VERBOSE=1 git push`.
> Hope it help.

@mymusise Are you sure LFS is properly installed and configured for the repo? From your logs it seems your `git push` command isn't doing any LFS work (like running a `pre-push` hook or calling our LFS endpoint), trying instead to push all the files through the classic git endpoint, which can't work.