[testing] trainer tests fail - 2 issues
Author: stas00
Pull request: no

trainer tests seem to have several issues:

```
USE_CUDA=1 pytest tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer
```

1.

```
        if self.api_key is None:
>           raise ValueError(
                "Comet.ml requires an API key. Please provide as the "
                "first argument to Experiment(api_key) or as an environment"
                " variable named COMET_API_KEY "
            )
```

the test should skip and not fail if the user doesn't have Comet.ml

2. wandb seems to still be problematic when there is an error in the test.
```
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 180, in console_main
    code = main()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 157, in main
    ret = config.hook.pytest_cmdline_main(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 289, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/main.py", line 284, in wrap_session
    config._ensure_unconfigure()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 920, in _ensure_unconfigure
    fin()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 632, in stop_global_capturing
    self._global_capturing.pop_outerr_to_orig()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 522, in pop_outerr_to_orig
    out, err = self.readouterr()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 563, in readouterr
    out = self.out.snap()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/_pytest/capture.py", line 481, in snap
    self.tmpfile.seek(0)
OSError: [Errno 29] Illegal seek
```
I have to add `WANDB_DISABLED=true` to overcome those.
There was an issue about it earlier, but I am not able to find it.

Full report:
```
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_can_resume_training - ValueError: Comet.ml requires an API key. Please provide as the first argument...
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_can_resume_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_evaluate - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_evaluate - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_flos_extraction - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_flos_extraction - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_load_best_model_at_end - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_load_best_model_at_end - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_model_init - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_model_init - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_num_train_epochs_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_num_train_epochs_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_predict - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_predict - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_save_checkpoints - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_save_checkpoints - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_lm - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_lm - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_iterable_dataset - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_iterable_dataset - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_datasets - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_datasets - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_training_arguments_are_left_untouched - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_training_arguments_are_left_untouched - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_add_remove_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_add_remove_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_init_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_callback.py::TrainerCallbackTest::test_init_callback - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_utils.py::TrainerUtilsTest::test_distributed_tensor_gatherer - OSError: [Errno 29] Illegal seek
ERROR tests/test_trainer_utils.py::TrainerUtilsTest::test_distributed_tensor_gatherer - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_find_code_in_transformers - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_find_code_in_transformers - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_is_copy_consistent - OSError: [Errno 29] Illegal seek
ERROR tests/test_utils_check_copies.py::CopyCheckTester::test_is_copy_consistent - OSError: [Errno 29] Illegal seek
```

Thank you!

## Environment info

```
- `transformers` version: master
- Platform: Linux-4.15.0-118-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.0.dev20201014 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

@sgugger


Comments:
All tests are passing on my side and the CI so it's something in your environment that triggers the failures. Could you add what you have inside it?

This?
> the test should skip and not fail if the user doesn't have `Comet.ml`


I don't have `comet.ml` installed in my dev env. I think the problem might be having it installed without the key properly set.

Right, so this comes from `transformers`:

```
src/transformers/integrations.py:                experiment = comet_ml.Experiment(**args)
src/transformers/trainer_tf.py:            experiment = comet_ml.Experiment(**args)
```
and it fails in:
```
        if self.api_key is None:
>           raise ValueError(
                "Comet.ml requires an API key. Please provide as the "
                "first argument to Experiment(api_key) or as an environment"
                " variable named COMET_API_KEY "
            )
```
and it has a check:
```
if is_comet_available():
    import comet_ml
```
so the tests need to do something about it.

I have never explicitly installed `comet_ml`, but some other package installed it as a dependency. Yet, I don't have this service configured. It's like `wandb` - it needs to gracefully skip tests if `comet_ml` is installed but not configured. To configure you have to go to that service website, sign up, create API key, etc. etc.

I suppose there needs to be a skip decorator that does something like:
```
from .integrations import is_comet_available
def working_comet_ml(test_case):
    # handle the case where comel_ml is installed but not configured
    try:
        if is_comet_available:
            import comet_ml
            comet_ml.Experiment()
    except:
        return unittest.skip("test is slow")(test_case)
    return test_case
```
this is untested. 

Most likely you should be able to reproduce the problem by just installing `comet_ml`

Or perhaps those 2 places where it's used in `transformers` it should issue a warning, rather then raise an error. Which is probably a better way to handle that.

**edit**: thinking more, it is the library that shouldn't crash, and not the tests that should skip. It's an optional feature not a requirement, so it should behave gracefully.

Seems to come from: https://github.com/huggingface/transformers/commit/b923871bb78f538e3c2e4bf36776986c800da1ae
So I guess we need to ask @dsblank, who made the initial PR and I see you were in the reviewers too.

Yes I think raising a warning should be better than a hard error since it does not mean the training has to stop, and it blocks the tests. If you can suggest a PR with those changes, I'd happily review it.

Found a better solution https://github.com/huggingface/transformers/pull/7830

`comet_ml` needs to have `comet_ml.ensure_configured()` ala `wandb.ensure_configured()` - I am not sure whether ` @dsblank could add/ask for it. Thank you!

Until then it'll be emulated by:
```
try:
    # Comet needs to be imported before any ML frameworks
    import comet_ml  # noqa: F401

    # XXX: there should be comet_ml.ensure_configured(), like `wandb`, for now emulate it
    comet_ml.Experiment(project_name="ensure_configured")
    _has_comet = True
except (ImportError, ValueError):
    _has_comet = False
```


and wandb needed the latest version - which fixed the `OSError: [Errno 29] Illegal seek` issue