[testing] test_trainer.py is failing
Author: stas00
Pull request: no

```
pytest tests/test_trainer.py
```
```
platform linux -- Python 3.7.9, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: xdist-2.1.0, forked-1.3.0
collected 11 items                                                                                                                                   

tests/test_trainer.py F.FF.FF...F                                                                                                              [100%]

====================================================================== FAILURES ======================================================================
____________________________________________________ TrainerIntegrationTest.test_custom_optimizer ____________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_custom_optimizer>

    def test_custom_optimizer(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression")
        model = RegressionModel()
        optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
        lr_scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda x: 1.0)
        trainer = Trainer(model, args, train_dataset=train_dataset, optimizers=(optimizer, lr_scheduler))
        trainer.train()
    
>       self.assertTrue(torch.abs(trainer.model.a - 1.8950) < 1e-4)
E       AssertionError: tensor(False, device='cuda:0') is not true

tests/test_trainer.py:240: AssertionError
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00,  4.15it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 584.41it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 570.73it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00,  3.06it/s]
_______________________________________________________ TrainerIntegrationTest.test_model_init _______________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_model_init>

    def test_model_init(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression", learning_rate=0.1)
        trainer = Trainer(args=args, train_dataset=train_dataset, model_init=lambda: RegressionModel())
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:249: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 540.05it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 510.99it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 553.37it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 130.01it/s]
______________________________________________ TrainerIntegrationTest.test_number_of_steps_in_training _______________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_number_of_steps_in_training>

    def test_number_of_steps_in_training(self):
        # Regular training has n_epochs * len(train_dl) steps
        trainer = get_regression_trainer(learning_rate=0.1)
        train_output = trainer.train()
>       self.assertEqual(train_output.global_step, self.n_epochs * 64 / self.batch_size)
E       AssertionError: 12 != 24.0

tests/test_trainer.py:129: AssertionError
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 547.43it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 573.03it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 557.12it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 136.18it/s]
_________________________________________________ TrainerIntegrationTest.test_reproducible_training __________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_reproducible_training>

    def test_reproducible_training(self):
        # Checks that training worked, model trained and seed made a reproducible training.
        trainer = get_regression_trainer(learning_rate=0.1)
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:118: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 388.21it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 556.31it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 544.31it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 117.50it/s]
_______________________________________________ TrainerIntegrationTest.test_train_and_eval_dataloaders _______________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_train_and_eval_dataloaders>

    def test_train_and_eval_dataloaders(self):
        trainer = get_regression_trainer(learning_rate=0.1, per_device_train_batch_size=16)
>       self.assertEqual(trainer.get_train_dataloader().batch_size, 16)
E       AssertionError: 32 != 16

tests/test_trainer.py:143: AssertionError
____________________________________________________ TrainerIntegrationTest.test_trainer_with_nlp ____________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_trainer_with_nlp>

    def test_trainer_with_nlp(self):
        np.random.seed(42)
        x = np.random.normal(size=(64,)).astype(np.float32)
        y = 2.0 * x + 3.0 + np.random.normal(scale=0.1, size=(64,))
        train_dataset = nlp.Dataset.from_dict({"input_x": x, "label": y})
    
        # Base training. Should have the same results as test_reproducible_training
        model = RegressionModel()
        args = TrainingArguments("./regression", learning_rate=0.1)
>       trainer = Trainer(model, args, train_dataset=train_dataset)

tests/test_trainer.py:212: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/trainer.py:285: in __init__
    self._remove_unused_columns(self.train_dataset, description="training")
src/transformers/trainer.py:311: in _remove_unused_columns
    dataset.set_format(type=dataset.format["type"], columns=columns)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Dataset(features: {'input_x': Value(dtype='float32', id=None), 'label': Value(dtype='float64', id=None)}, num_rows: 64), type = 'python'
columns = ['input_x', 'label'], output_all_columns = False, format_kwargs = {}

    def set_format(
        self,
        type: Optional[str] = None,
        columns: Optional[List] = None,
        output_all_columns: bool = False,
        **format_kwargs,
    ):
        """ Set __getitem__ return format (type and columns)
    
            Args:
                type (Optional ``str``): output type selected in [None, 'numpy', 'torch', 'tensorflow', 'pandas']
                    None means __getitem__ returns python objects (default)
                columns (Optional ``List[str]``): columns to format in the output
                    None means __getitem__ returns all columns (default)
                output_all_columns (``bool`` default to False): keep un-formated columns as well in the output (as python objects)
                format_kwargs: keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
        """
        # Check return type
        if type == "torch":
            try:
                import torch  # noqa: F401
            except ImportError:
                logger.error("PyTorch needs to be installed to be able to return PyTorch tensors.")
        elif type == "tensorflow":
            try:
                import tensorflow  # noqa: F401
            except ImportError:
                logger.error("Tensorflow needs to be installed to be able to return Tensorflow tensors.")
        else:
            assert not (
                type == "pandas" and (output_all_columns or format_kwargs)
            ), "Format type 'pandas' doesn't allow the use of `output_all_columns` or `**format_kwargs`."
            assert (
                type is None or type == "numpy" or type == "pandas"
>           ), "Return type should be None or selected in ['numpy', 'torch', 'tensorflow', 'pandas']."
E           AssertionError: Return type should be None or selected in ['numpy', 'torch', 'tensorflow', 'pandas'].

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/nlp/arrow_dataset.py:542: AssertionError
================================================================== warnings summary ==================================================================
/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:546
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:546: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    class IteratorBase(collections.Iterator, trackable.Trackable,

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:106
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:106: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    class DatasetV2(collections.Iterable, tracking_base.Trackable,

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

src/transformers/modeling_tf_utils.py:718
  /mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_tf_utils.py:718: DeprecationWarning: invalid escape sequence \s
    """

src/transformers/modeling_funnel.py:130
  /mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_funnel.py:130: DeprecationWarning: invalid escape sequence \d
    layer_index = int(re.search("layer_(\d+)", m_name).groups()[0])

tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer
tests/test_trainer.py::TrainerIntegrationTest::test_evaluate
tests/test_trainer.py::TrainerIntegrationTest::test_model_init
tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training
tests/test_trainer.py::TrainerIntegrationTest::test_predict
tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
    warnings.warn('Was asked to gather along dimension 0, but all '

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================== short test summary info ===============================================================
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_model_init - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - AssertionError: 12 != 24.0
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - AssertionError: 32 != 16
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_nlp - AssertionError: Return type should be None or selected in ['numpy', '...
===================================================== 6 failed, 5 passed, 11 warnings in 10.28s ==================================================
```
Env:
```
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
```

Thanks.

 @sgugger?


Comments:
So with more look, the issue has to do with multiple GPUs. The problem goes away if I do:

`CUDA_VISIBLE_DEVICES="0" pytest tests/test_trainer.py`

> Looks like those tests need a decorator to run on the CPU only.

They work with 1 gpu too, but we don't have such a setup to force 1 gpu?

Though isn't this exposing a problem and we will just hide it by setting CPU-only?

Yes, 2 GPUs end up doing half the updates, hence the error when counting the number of steps or the wrong value for a/b. This not a problem per se: it's logical to do half the steps since the batch size is multiplied by 2 and it's also logical that the seeded training end up to a different value with a different batch size.

But I wasn't expecting those to run on 2 GPUs, so we should add something to not use the 2 GPUs, which I don't think is possible since Trainer is there to automatically use all of them...

The best I can think of is to skip when detecting multiple GPUs. At least the tests that test the seeded training. For the number of steps, we can adapt to use the proper batch size.

ok, let me try - will post PR when working.

Odd, now it's failing with 1 or 0 gpus too, but only 4 tests. 
```
collecting ... 
 tests/test_trainer.py ✓                                                                                                                             9% ▉         

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TrainerIntegrationTest.test_custom_optimizer ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_custom_optimizer>

    @require_non_multigpu
    def test_custom_optimizer(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression")
        model = RegressionModel()
        optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
        lr_scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda x: 1.0)
        trainer = Trainer(model, args, train_dataset=train_dataset, optimizers=(optimizer, lr_scheduler))
        trainer.train()
    
>       self.assertTrue(torch.abs(trainer.model.a - 1.8950) < 1e-4)
E       AssertionError: tensor(False, device='cuda:0') is not true

tests/test_trainer.py:245: AssertionError
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1064.58it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1292.20it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1446.00it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 152.11it/s]

 tests/test_trainer.py ⨯                                                                                                                            18% █▊        

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TrainerIntegrationTest.test_model_init ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_model_init>

    @require_non_multigpu
    def test_model_init(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression", learning_rate=0.1)
        trainer = Trainer(args=args, train_dataset=train_dataset, model_init=lambda: RegressionModel())
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1122.63it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1000.25it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1124.70it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 131.09it/s]

 tests/test_trainer.py ⨯                                                                                                                            27% ██▊       

――――――――――――――――――――――――――――――――――――――――――――――――――――――― TrainerIntegrationTest.test_trainer_with_datasets ――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_trainer_with_datasets>

    @require_non_multigpu
    def test_trainer_with_datasets(self):
        np.random.seed(42)
        x = np.random.normal(size=(64,)).astype(np.float32)
        y = 2.0 * x + 3.0 + np.random.normal(scale=0.1, size=(64,))
        train_dataset = datasets.Dataset.from_dict({"input_x": x, "label": y})
    
        # Base training. Should have the same results as test_reproducible_training
        model = RegressionModel()
        args = TrainingArguments("./regression", learning_rate=0.1)
        trainer = Trainer(model, args, train_dataset=train_dataset)
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:218: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------
Set __getitem__(key) output type to python objects for ['input_x', 'label'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Iteration: 100%|██████████| 8/8 [00:00<00:00, 941.46it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1036.33it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1004.92it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 121.20it/s]

 tests/test_trainer.py ⨯                                                                                                                            36% ███▋      

――――――――――――――――――――――――――――――――――――――――――――――――――――――― TrainerIntegrationTest.test_reproducible_training ――――――――――――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_reproducible_training>

    @require_non_multigpu
    def test_reproducible_training(self):
        # Checks that training worked, model trained and seed made a reproducible training.
        trainer = get_regression_trainer(learning_rate=0.1)
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:119:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1168.66it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1169.27it/s]
Iteration: 100%|██████████| 8/8 [00:00<00:00, 1208.34it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 143.97it/s]

 tests/test_trainer.py ⨯✓✓✓✓✓✓                                                                                                                     100% ██████████
======================================================================== warnings summary ========================================================================
/home/stas/anaconda3/envs/main/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15
  /home/stas/anaconda3/envs/main/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/stable/warnings.html
==================================================================== short test summary info =====================================================================
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_model_init - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_datasets - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - AssertionError: tensor(False, device='cuda:0') is not true

Results (9.28s):
       7 passed
       4 failed
         - tests/test_trainer.py:235 TrainerIntegrationTest.test_custom_optimizer
         - tests/test_trainer.py:249 TrainerIntegrationTest.test_model_init
         - tests/test_trainer.py:206 TrainerIntegrationTest.test_trainer_with_datasets
         - tests/test_trainer.py:114 TrainerIntegrationTest.test_reproducible_training
```

OK, so I found at least one difference - fails on py38 env, but works on py37 - I will check if it's python or some libs that are different and report back.

**edit**: I rebuilt a fresh py38 env and the problem is gone, so it must be some other packages. I will post more if I find the culprit.

Hmm, I was trying to match the environments and accidentally updated the environment the test breaks in and it no longer breaks. So it must have been some package. The main suspect was pytorch as I had a nightly version, but I reverted back to that old version and it still works. 

So this issue is resolved once this is merged https://github.com/huggingface/transformers/pull/7059