[models] why aren't .bin files compressed for faster download?
Author: stas00
Pull request: no
Labels: name: WIP

Why are not the pretrained model files compressed? 

It took ~25minutes yesterday to download 45GB t5-11b on a slow connection. 

I did a quick test on a random `pytorch_model.bin` with default gzip options and it's 1/3rd less in size. And surely there must be a better compressor that can be used - but will require to be on the client's side so gzip might be good enough. This is not much of a difference for under 1GB files, but for large models this starts to add up.

It's not like you can diff a .bin file, so there is little value in having it stored as is from the RCS point of view. But perhaps I'm missing other aspects. 

Perhaps 2 versions can be stored and the retriever could favor the compressed version for large files?

Cost-wise this change would introduce some 60% increase in storage if both versions are to be stored, but will have a huge saving in downloads.

The compression process can be a simple cronjob, so that it won't need to make users do anything special.

@julien-c, @LysandreJik, @patrickvonplaten, @sgugger 

Comments:
I'm in favor of this! It's also a bit annoying to me that downloading takes that much time and I think for people that try out a bunch of different checkpoitns to decide on which model they want to use, a 60% speed-up in downloading would be very nice.

A related issue is that t5-11b is actually too large to be served by Cloudfront (they have a limit of 20GB for a single file) so we have to fallback to serving using S3, which is way, way slower. (up to a few MB/s depending on where you are, vs. basically saturating your downlink when from Cloudfront)

If large models are here to stay, then we probably need to support the **splitting of models** in `save_pretrained`/`from_pretrained`

also cc @mfuntowicz @n1t0 

> If large models are here to stay, then we probably need to support the splitting of models in save_pretrained/from_pretrained

If we go with compression why not do the normal volumes of whatever common compressor tool we choose, as in:

```
pytorch.bin.rar00
pytorch.bin.rar01
```

So you kill 2 birds at the same time, get the compression and the splitting.

Again, the user doesn't need to do anything. the compression and splitting can be triggered upon the upload.

e.g. with tar.gz:

on upload:
```
tar cvzf - pytorch_model.bin | split -b 10G - pytorch_model.bin.tar.gz.
```
which should give:
```
pytorch_model.bin.tar.gz.aa
pytorch_model.bin.tar.gz.ab
pytorch_model.bin.tar.gz.ac
```
on download:
```
cat pytorch_model.bin.tar.gz.a* | tar xzvf -
```

well, we don't need tar here - it's just one file. so just gzip would be enough.

Just need to choose which compression is good enough, and doesn't take too long to decompress - e.d. don't use the highest compression possible and 100% available on all clients - so gzip and uncompress for sure, 7zip/rar can't be trusted to have, but if there is a python client that can handle it, it may work anywhere?

Operationally I'm wondering if instead of doing it at rest in an async way (which might prove difficult to easily scale to a much larger number of models), we should probably handle this in the `save_pretrained` (which means users will upload their models already in the supported format)

Is the intention to upload both compressed and uncompressed or just the former?

I propose to manage compression/decompression transparently on the server side and leave everything as is on the client side (other than download of the compressed version). 

Here are some quick pros/cons for 3 different scenarios I see.

### 1. Having only the compressed version on the client side:

Cons:
1. Will create a constant overheard of compression on `save_pretrained` and check pointing
2. Will create a constant wasteful overhead of decompression during `from_pretrained`
3. Should the max split size change - how do you tell the users that they all need to update their repo?

Pros:
1. Will make the upload faster


### 2. Having only the decompressed version on the client side:

Cons:
1. More expensive upload

Pros:
1. Everything else is simple



### 3. Having both versions on the client side:

This one is like case 2, but with an one additional change in each up/down direction:

Pros: same as in case 2

Cons:
1. Could be confusing to the user during upload if they need to upload only the compressed files
2. More to upload 

Extra notes:
1. Need to make sure that the decompression will happen once upon download and not on every `from_pretrained()` call. 


Please feel free to edit this post directly as I'm sure I've missed some aspects.

I agree that 2) is simpler, and is in line with the goal of keeping things simple for the user. As a PyTorch user I would prefer seeing my files in the native PyTorch format rather than a compressed format I don't know about, on which I'll need to apply additional pre-processing before using it in a custom model. Especially since we've seen users use `torch.load` instead of `from_pretrained` in the past.

Should we keep this one alive? Is this on someone's TODO list?

i think checkpoint-splitting (#13548) is going to be a better/more future-proof solution than compression (on top of already compressed binary files) where the size delta is going to be rather minimal

So I'd vote to close this issue and focus on #13548

Sounds good, Julien. Let's close this one.