[wip] [deps] data_collator fails with older numpy, update numpy>=1.20.0
Author: stas00
Pull request: yes

Please ignore for now as this looks a pytorch 1.9.0-rc problem, filed an issue 
https://github.com/pytorch/pytorch/issues/59533

--------------------

with pytorch-1.9.0/nighty 1.9.0a0+git2a178d3

data collator via Trainer fails with numpy==1.19.5 with:
```
RuntimeError: Could not infer dtype of numpy.float32
```

Additionally getting a warning:

```
../../../../../home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/package/_mock_zipreader.py:17
  /home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/package/_mock_zipreader.py:17: UserWarning: Failed to initialize NumPy: module compiled against API version 0xe but this version of numpy is 0xd (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:67.)
    _dtype_to_storage = {data_type(0).dtype: data_type for data_type in _storages}
```

Full trace:
```
$ pip install numpy==1.19.5
$ pytest tests/test_trainer.py::TrainerIntegrationTest::test_fp16_full_eval
====================================================================== test session starts ======================================================================
platform linux -- Python 3.8.8, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface, configfile: pytest.ini
plugins: monitor-1.6.0, flakefinder-1.0.0, forked-1.3.0, instafail-0.4.2, xdist-2.2.1
collected 1 item                                                                                                                                                

tests/test_trainer.py F                                                                                                                                   [100%]

=========================================================================== FAILURES ============================================================================
__________________________________________________________ TrainerIntegrationTest.test_fp16_full_eval ___________________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_fp16_full_eval>

    def setUp(self):
        super().setUp()
        args = TrainingArguments(".")
        self.n_epochs = args.num_train_epochs
        self.batch_size = args.train_batch_size
        trainer = get_regression_trainer(learning_rate=0.1)
>       trainer.train()

tests/test_trainer.py:333: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/trainer.py:1237: in train
    for step, inputs in enumerate(epoch_iterator):
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/utils/data/dataloader.py:521: in __next__
    data = self._next_data()
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/utils/data/dataloader.py:561: in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
/home/stas/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py:47: in fetch
    return self.collate_fn(data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

features = [{'input_x': -0.54438275, 'labels': 1.8582585}, {'input_x': 0.64768857, 'labels': 4.288176}, {'input_x': 1.5792128, 'l...abels': 0.12356561}, {'input_x': -0.46947438, 'labels': 2.0574687}, {'input_x': -0.46572974, 'labels': 2.1507308}, ...]

    def default_data_collator(features: List[InputDataClass]) -> Dict[str, torch.Tensor]:
        """
        Very simple data collator that simply collates batches of dict-like objects and performs special handling for
        potential keys named:
    
            - ``label``: handles a single value (int or float) per object
            - ``label_ids``: handles a list of values per object
    
        Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs
        to the model. See glue and ner for example of how it's useful.
        """
    
        # In this function we'll make the assumption that all `features` in the batch
        # have the same attributes.
        # So we will look at the first element as a proxy for what attributes exist
        # on the whole batch.
        if not isinstance(features[0], (dict, BatchEncoding)):
            features = [vars(f) for f in features]
    
        first = features[0]
        batch = {}
    
        # Special handling for labels.
        # Ensure that tensor is created with the correct type
        # (it should be automatically the case, but let's make sure of it.)
        if "label" in first and first["label"] is not None:
            label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
            dtype = torch.long if isinstance(label, int) else torch.float
            batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
        elif "label_ids" in first and first["label_ids"] is not None:
            if isinstance(first["label_ids"], torch.Tensor):
                batch["labels"] = torch.stack([f["label_ids"] for f in features])
            else:
                dtype = torch.long if type(first["label_ids"][0]) is int else torch.float
                batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype)
    
        # Handling of all other possible keys.
        # Again, we will use the first element to figure out which key/values are not None for this model.
        for k, v in first.items():
            if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
                if isinstance(v, torch.Tensor):
                    batch[k] = torch.stack([f[k] for f in features])
                else:
>                   batch[k] = torch.tensor([f[k] for f in features])
E                   RuntimeError: Could not infer dtype of numpy.float32

src/transformers/data/data_collator.py:80: RuntimeError
```

The error goes away after installing the next release `numpy==1.20.0`.

Perhaps it can be fixed in the collator to support older numpy.

This PR is one way to approach it. Not sure if we have other dependencies that perhaps require numpy<=1.20.


Comments:
OK, so as I was concerned some dependency fixes numpy at 1.19.5 so things fail:
```
ERROR: Could not find a version that satisfies the requirement numpy>=1.20.0 (from transformers[all,quality]) (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0rc1, 1.13.0rc2, 1.13.0, 1.13.1, 1.13.3, 1.14.0rc1, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0rc1, 1.15.0rc2, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0rc1, 1.16.0rc2, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0rc1, 1.17.0rc2, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0rc1, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0rc1, 1.19.0rc2, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5)
ERROR: No matching distribution found for numpy>=1.20.0

Exited with code exit status 1
``` 

This is so not user-friendly as it could have said which dependency causes this conflict.

`pip check` comes to help:

```
$ pip check
WARNING: Ignoring invalid distribution -orch (/mnt/nvme1/anaconda3/envs/py38-pt18/lib/python3.8/site-packages)
tensorflow 2.5.0 has requirement numpy~=1.19.2, but you have numpy 1.20.0.
```

so `tensorflow 2.5.0` is the limiting culprit.

OK, that was a faulty pytorch build. Seems to work fine with the latest nightly or 1.9.0-rc.