Flaky tests sometimes caused by S3 failures
Author: patrickvonplaten
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
The test: `tests/test_pipelines.py::MonoColumnInputTestCase::test_torch_fill_mask` is flaky on circle ci: https://app.circleci.com/pipelines/github/huggingface/transformers/7734/workflows/7c51cfcf-5425-4172-aa15-ed677e37f7fc/jobs/49891/steps

## To reproduce

Not sure. It pops up from time to time on circle ci.
Need to investigate more

## Expected behavior

The test should not be flaky

## Environment info
On circle ci

Comments:
I can't replicate (I understand that's the point :)) 

The traceback from your link is:

```python
OSError: Can't load weights for 'sshleifer/tiny-distilroberta-base'. Make sure that:
E               
E               - 'sshleifer/tiny-distilroberta-base' is a correct model identifier listed on 'https://huggingface.co/models'
E               
E               - or 'sshleifer/tiny-distilroberta-base' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
```

Contents are fine, it seems:

![image](https://user-images.githubusercontent.com/6045025/85339595-7ba7e300-b4b2-11ea-998e-64169948ead5.png)


@julien-c have you seen similar intermittent S3 failures?

From what I experience, this happens not just for the same model, but for different ones. Very hard to traceback what is going on there IMO.

Yes, I had a similar circleci failure trying to get the bert tokenizer yesterday.


I have found a clue. I hypothesize that some models have user posted metadata that somehow ducks them up. 

I have never seen flaky bart-large failure, and there is no metadata:

```bash

aws s3api head-object --bucket models.huggingface.co --key bert/facebook/bart-large/config.json

=> bytes	1264	application/json	"faf4c3c00764dc47a3a10a63a004dcc3"	Fri, 24 Apr 2020 15:58:48 GMT	JhIFsOvvLtrLn0vJjGNN6ZhJGUlbXBEP

```

whereas there are `Helsinki-NLP/opus-mt-en-ROMANCE` failures, and metadata:

```bash
aws s3api head-object --bucket models.huggingface.co --key bert/Helsinki-NLP/opus-mt-en-ROMANCE/config.json

=> bytes	1113	text/plain	"13ca8d49ee7f02a26f935cb4a60e6557"	Tue, 12 May 2020 22:39:10 GMT	jAI94kJ_exk0tG6z0Yr.Rea4_j0g02Ih
METADATA	atime:1589321950/ctime:1589321949/gid:1007/gname:shleifer/md5:13ca8d49ee7f02a26f935cb4a60e6557/mode:33188/mtime:1589321949/uid:1006/uname:shleifer
```



Now I need to figure out how to get rid of the metadata to test my hypothesis, if anyone knows how.

**Update:**
```bash
bytes	474	application/json	"622babbd58848ec69a2433ba0c6edab3"	Tue, 12 May 2020 01:26:15 GMT	UKLWouonOr8duRlrHqH4.iNZEF5bmOQY
METADATA	atime:1589246769/ctime:1589246726/gid:20/gname:staff/md5:622babbd58848ec69a2433ba0c6edab3/mode:33188/mtime:1589246726/uid:501/uname:shleifer
```
also shows metadata, and there are flaky failures with that model.


Should be easy, I can guide you through the S3 doc if needed.

I doubt that's the reason though. What about intermittent connectivity issues? They seem more likely.

It could certainly be connectivity. I manually deleted the metadata for all contents of `Helsinki-NLP/opus-mt-en-ROMANCE/` and `sshleifer/tiny-distilroberta-base`, so if there are more failures there we will know that my theory is wrong. I think the metadata is caused by using `s3cmd` instead of `awscli`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.