[testing] test_trainer_callback.py cannot collect test class 'TestTrainerCallback' 
Author: stas00
Pull request: no

something is wrong here:
```
$ pytest tests/test_trainer_callback.py  
tests/test_trainer_callback.py:24
  /mnt/nvme1/code/transformers-comet/tests/test_trainer_callback.py:24: PytestCollectionWarning: cannot collect test class 'TestTrainerCallback' because it has a __init__ constructor (from: tests/test_trainer_callback.py)
    class TestTrainerCallback(TrainerCallback):
```

another one in the same test:
```
  /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
    warnings.warn('Was asked to gather along dimension 0, but all '

tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow
```

hint: these are in the warnings section of the test.

@sgugger 

Comments:
It's just pytest thinking that class is a test because it begins by Test. Will fix, but this is very low priority for me as it runs as intended since pytest does not collect it. No idea about the other warning, but again, not important since the goal of that test is not to test gathered tensors, just the event flow.

Yes, of course, there is no rush. I can take care of some of them.

In general - the problem with stray warnings is that if there is a problem one has to read through warnings to see if something related is related and if there is a lot of noise in there this wastes a huge amount of time, sifting grain from chaff. Warnings really should be treated as errors, IMHO, or be removed if they aren't important.

I'm guessing you don't have TensorFlow installed then :-)

I do:

```
- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-118-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201014 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

And you don't get a thousand warnings from it?

You must be alluding to tf generating a gazillion of useless logs and therefore what's another warning, correct?

I'd say if we had resources to do so there should be no warnings emitted in our test suite. So that when one appears we know that something is not right and fix it. It's like a house with a broken window that doesn't get mended. Others start breaking more windows because there is one broken already.

FWIW, most of the time, because our test suite is so noisy, I run pytest via `pytest --disable-warnings`, unless I'm trying to debug a failing test and I want to see if any warnings might be relevant to the issue at hand.

Moreover, I have been contributing for about 3 months now and I'm yet to be able to complete the test suite 100% error-free. There is always a handful of failing tests, which seem to be just fine on CIs. I'd like to work on that `USE_CUDA` elimination task, so I was hoping I could get `make test` to pass so that I'd have a working base line, but it's not quite there yet.

The problem is that I don't know how to catch TF useless (or them not having fixed deprecation warnings for other python libs so not so useless) warnings and keep others.
As for running the whole test suite, I've never tried on non-CPU (like what the CI does) so I don't know what's failing there or if it's important ;-)

> The problem is that I don't know how to catch TF useless warnings

It'd be quite easy to do if we are talking about tf loggers. I wrote a function that you can do this selectively for any of the offenders of choice (or all of them). https://github.com/huggingface/transformers/issues/3050#issuecomment-682167272

But really we are going back to the PR that wasn't received well, where I proposed to optionally silence the loggers for external modules for the test suite - https://github.com/huggingface/transformers/pull/6816

> As for running the whole test suite, I've never tried on non-CPU (like what the CI does) so I don't know what's failing there or if it's important ;-)

It's probably important ;) Most applications of transformers are probably run on `gpus` - I could be wrong.

And it'd help to have a normal CI that would run on GPU.

Now that we are going to eliminate USE_CUDA, so that gpu will be used by default, it'll improve things a lot, as more developers will run into such problems and hopefully fix them.

I think the slow tests run the CI on GPU (even GPUs) so the common tests are all passing on that setup. You may have other issues linked to your env.

Only on scheduled CI which runs only a limited range of options. e.g. not testing newer pytorch releases.