[testing] when to @slow and when not to? (huge models download)
Author: stas00
Pull request: no

Looking at the [CI logs](https://app.circleci.com/pipelines/github/huggingface/transformers/12476/workflows/b4ca9141-e992-4c00-95c6-a6ae7fff63b3/jobs/88844) we do have huge models downloaded (i.e. not @slow):
```
Downloading: 100% 1.16G/1.16G [00:52<00:00, 22.3MB/s]
Downloading: 100% 433M/433M [00:08<00:00, 48.4MB/s]s]
Downloading:  43% 369M/863M [00:08<00:10, 45.4MB/s]
```
so it's very inconsistent. Why not have a whole bunch more of tests not be `@slow` then if we are downloading huge files anyway? A lot of those tests are very fast, other than the download overhead. Or, perhaps, those currently doing huge downloads should be `@slow` in first place? 

I'm asking since I was told not to run any fsmt tests with the full model unless it's `@slow` (size ~1.1GB). So it's unclear when it's OK to include huge models in the non-slow test suite and when not to.

Also, here is an alternative approach to think about - why not download large weights while other tests not needing them are running? i.e. fork a process early on on CI after pip installs are done and let it cache the models - then they will be ready to be used by the time the tests that need them get to run. This is an unpolished idea, since one needs to figure out how to re-sort the tests so that these large-model tests aren't run first...
 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I guess I haven't tagged that one so nobody looked at it - @LysandreJik, would you please have a look?

Hi, sorry for getting back to you so late. I believe this was due to the pipeline tests, but that should not be the case anymore since the refactor of the pipeline tests by Thom.

If some tests still download large files, then that's an error which we should resolve.

I had a quick look - we no longer have the job that runs all tests, so had to look in several places. I haven't check them all - here are some samples:

* This one downloads a handful of 250-440MB files, if you look at the top of the output for the test suite run:
https://app.circleci.com/pipelines/github/huggingface/transformers/15941/workflows/1f3fc641-82f3-4d34-a45d-30ebc2a04292/jobs/121749

* this one downloads a 500MB file
https://app.circleci.com/pipelines/github/huggingface/transformers/15941/workflows/1f3fc641-82f3-4d34-a45d-30ebc2a04292/jobs/121754


It'd be useful to have documented a MB threshold agreed on at which point the developer knows to add `@slow` on the test.

Okay, we should take a look and see if we can't replace some of these by smaller checkpoints, and if we can't just set these tests as slow.

Yes, that would be very useful. What do you think of a 100MB limit?  (maybe even smaller - we can set it to 100MB now and come back later if it's too high)

It all depends on how much it'll add up to - if now we are going to hypothetically enable 100 tests each downloading 100MB - that's 10TB which is probably too much overall.

They key is to keep the total run-time of the test suite to a reasonable length of time, taking into an account its constant growth.

In general we have mostly 2 types of tests (network io-wise):

1) dumb tests that only check the mechanics and require either downloaded tiny models or created on the fly, with tiny datasets - all <10MB.
2) quality-checking tests that typically require a download of full models, which are typically 400MB+

I don't think 100MB threshold fits into either group - it's too big for dumb tests and too small for most quality-checking tests. So it's more like 20MB and anything bigger is already type 2, which are currently mostly '@slow`.

The following is a bit of a brain dump on how to keep the total CI job runtime to the minimum in the context of network overhead impacting the total duration of the CI job run:

* In general to make an intelligent decision about how much download we should accommodate if we want CI jobs to be faster, we need to know an average download speed - which we can then translate to the time overhead added by the test being blocked while waiting for the download. For example, knowing this we perhaps could measure download demands of each test and then sort by the slowest and set them to slow if it's not an essential test or rewrite the slow test to be faster. Similar to duration, except here we are going more finegrained and look at the download aspect that impacts the total duration.

* A possible bruteforce solution, is to find a magical way to fork off a process as soon as feasible (probably as soon as pip installed `transformers`) that will start downloading all the largish models used by fast tests (and it'd do those in parallel if it's faster than doing sequential downloads - needs to check which side of the download is the bottleneck) so that when the test comes around the model is already there.

   Note that `pip install` takes about 45-60 secs to run and it's not doing any downloads (local cache) - we have a whooping minute to pre-download a lot of models. So I'd install some bare version of `transformers` that all it knows is to run `from_pretrained` and the caching/downloading code and fork it off to already start downloading models.

   But perhaps while some of the tests are blocking on IO, there are more computer resources for other tests to complete faster. For this we need to watch the CPU utilization/ load / swapping and see whether the system struggles or stays cool. For example, through measuring with `time(1)` I found that `pytest -n3` leads to the fastest completion of the test suite on my machine, and higher `-n` leads to a slower total walltime (the bottleneck is gpu in my case). I have 12 CPU cores so `make test` starts -n12, which is a really bad idea. It'd require at least 6GPUs to handle 12 pytest workers.


* Also I wonder whether the caching software could somehow be made even more advanced so that it can be made aware that another program is already in the process of downloading this model and not start a new one. That way in the worst case scenario, the test will block as before if the model download hasn't started, in the better case it'll just have to wait for a fraction of time for the model to complete to download when it started the download by another process and in the best case it'd be already downloaded. ( `pytest --dist=loadfile` at the moment mostly prevents from multiple attempts of downloading the same model by different pytest workers)

* I checked that circle CI has a huge pip cache - only a few newer pip packages get downloaded during the `pip install` step, most come from a local cache.

   Would it be possible to request to create a local cache for essential  `transformers` models? Then this whole issue of `@slow` because download is too big will be moot. Then `@slow` will be only bound by execution time.

   I think it'll have to be a local cache dir and not a cache to download from since even on a fast network it'll still add up to a large overheard.

I hope my communication was clear. 




Thank you for your comment, I think you make some very good points. The last one is especially important, as caching large model files would be very helpful. We did consider this at one point, but refrained from it as it could have added another layer for the detection of errors.

I think we should revisit this option, for two reasons:
- It would speed up the CI, as you've mentioned here.
- We have a lot of connection issues since using the new git-based system because the CI is bursting the website (cc @julien-c).

We can simply cache the `.cache/huggingface/transformers` folder for this.

---

Regarding your other points, indeed, it would be interesting to explore all of these options. Downloading the models while we are doing the docker instantiation/pip install/setup would be nice, but a download run in parallel to several commands is out of my knowledge. It may be a bit overkill, too, and require too much development time.

---

All in all, the simplest answer would be to simply put the limit to a threshold, like you've mentioned earlier. I chose 100MB specifically because it's not in either of the two categories. However, thinking about it, maybe some of the smaller distilled models are 80-90MBs large, so an integration test using those would still be under the limit - which ignores the purpose of the 100MB limit.

Thank you for reading my ideas and following up, @LysandreJik.

I made a tentative 50MB suggestion in https://github.com/huggingface/transformers/pull/8824

We can tweak it if it's not right down the road.