solve Illegal seek in wandb teardown (test suite)
Author: stas00
Pull request: yes

somehow wandb teardown fails in `tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc`:

```
pytest -n 1 tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc
====================================================================== test session starts =======================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-tests-1
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
gw0 [1]
FE                                                                                                                                                         [100%]
wandb: Waiting for W&B process to finish, PID 19525

============================================================================= ERRORS =============================================================================
_______________________________________________ ERROR at teardown of TrainerIntegrationTest.test_trainer_eval_mrpc _______________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <contextlib._GeneratorContextManager object at 0x7efc47610690>, type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               OSError: [Errno 29] Illegal seek

/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py:119: OSError
============================================================================ FAILURES ============================================================================
_________________________________________________________ TrainerIntegrationTest.test_trainer_eval_mrpc __________________________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <contextlib._GeneratorContextManager object at 0x7efc476bbd50>, type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               OSError: [Errno 29] Illegal seek

/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py:119: OSError
==================================================================== short test summary info =====================================================================
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
```

This is a possible solution, based on this solution 
https://github.com/wandb/client/issues/1138#issuecomment-654943065

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=h1) Report
> Merging [#5835](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0735def8e1200ed45a2c33a075bc1595b12ef56a&el=desc) will **decrease** coverage by `1.65%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5835/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5835      +/-   ##
==========================================
- Coverage   80.08%   78.43%   -1.66%     
==========================================
  Files         153      146       -7     
  Lines       27984    26012    -1972     
==========================================
- Hits        22412    20402    -2010     
- Misses       5572     5610      +38     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `43.98% <0.00%> (-46.86%)` | :arrow_down: |
| [src/transformers/tokenization\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcmVmb3JtZXIucHk=) | `41.66% <0.00%> (-40.00%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-32.51%)` | :arrow_down: |
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `76.47% <0.00%> (-11.93%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `80.98% <0.00%> (-10.18%)` | :arrow_down: |
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `87.70% <0.00%> (-7.98%)` | :arrow_down: |
| [src/transformers/benchmark/benchmark.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9iZW5jaG1hcmsvYmVuY2htYXJrLnB5) | `74.01% <0.00%> (-7.88%)` | :arrow_down: |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `93.80% <0.00%> (-5.37%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `74.41% <0.00%> (-4.20%)` | :arrow_down: |
| ... and [81 more](https://codecov.io/gh/huggingface/transformers/pull/5835/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=footer). Last update [0735def...9d566b1](https://codecov.io/gh/huggingface/transformers/pull/5835?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


CI doesn't have wandb installed.

I get an identical issue with wandb shutdown in other places as well, e.g. when running:

```pytest examples/test_examples.py::ExamplesTests::test_run_glue```

So far the solution seems to be to disable wandb. This works:

```
WANDB_DISABLED=true pytest examples/test_examples.py::ExamplesTests::test_run_glue
```

Unless wandb integration is being tested perhaps disabling it globally will solve this everywhere?


pinging @borisdayma in case



The wandb cli is being updated and will have a noop mode that we can use for easier testing.
For now `WANDB_DISABLED=true` should work.

> The wandb cli is being updated and will have a noop mode that we can use for easier testing.

would it be possible to ping this issue when it happened, so that we could resolve this? Thank you very much, @borisdayma 

Sounds good, I'll keep an eye on it!

Seems to have been resolved upstream (wandb) - I can no longer reproduce this problem, closing