testing utils: capturing std streams context manager
Author: stas00
Pull request: yes

A while ago I developed this set of classes `CaptureStd`, `CaptureStdout`,  `CaptureStderr`, which are very useful in testing outputs. 

Usage examples:

```
    with CaptureStdout() as cs:
        print("Secret message")
    print(f"captured: {cs.out}")

    import sys
    with CaptureStderr() as cs:
        print("Warning: ", file=sys.stderr)
    print(f"captured: {cs.err}")

    # to capture just one of the streams, but not the other
    with CaptureStd(err=False) as cs:
        print("Secret message")
    print(f"captured: {cs.out}")
```

it doesn't require passing the `capsys` arg to the test. It can be used anywhere.

Bonus: it properly handles outputs that rewrite themselves with \r, which normally lead to an inconsistent captured text, depending on whether `-s` option is used or not. See more details [here](https://github.com/fastai/fastai/blob/master/tests/utils/text.py#L6)

The full code is here: https://github.com/fastai/fastai/blob/master/tests/utils/text.py#L23

I think those would be handy in transformers and it looks that `src/transformers/testing_utils.py` is the right place for it, but perhaps it could have its own file as in the original. 

I have an immediate use for it in the following PR for implementing https://github.com/huggingface/transformers/issues/6070 I'm now working on an examples test that asserts on the contents of output of `--help` before `sys.exit`, so the output doesn't show up in the assert message. I could `capsys` at the test level, but that contains a lot of irrelevant crud. This nifty wrapper will allow us to capture just the stream we want in tests that need it.

p.s. not sure how one goes about importing code from another project, that uses Apache License 2.0. I need to add a disclaimer somewhere.

@sgugger 

edit: this is what we get with normal `capsys`:
```
E       assert 'lr_scheduler=cosine' in '\rValidation sanity check: 0it [00:00, ?it/s]\rValidation sanity check: 100%|██████████| 1/1.0 [00:00<00:00,  3.64it/...                              \x1b[A\rEpoch 1: 100%|██████████| 2/2 [00:00<00:00,  4.95it/s, loss=227.691, v_num=14]\n'
E        +  where '\rValidation sanity check: 0it [00:00, ?it/s]\rValidation sanity check: 100%|██████████| 1/1.0 [00:00<00:00,  3.64it/...                              \x1b[A\rEpoch 1: 100%|██████████| 2/2 [00:00<00:00,  4.95it/s, loss=227.691, v_num=14]\n' = CaptureResult(out='\rValidation sanity check: 0it [00:00, ?it/s]\rValidation sanity check: 100%|██████████| 1/1.0 [00:...                     \x1b[A\rEpoch 1: 100%|██████████| 2/2 [00:00<00:00,  4.95it/s, loss=227.691, v_num=14]\n', err='').out
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=h1) Report
> Merging [#6231](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/57eb1cb68d1c567b25ac256444e5c1a77b8817a7&el=desc) will **decrease** coverage by `0.50%`.
> The diff coverage is `28.57%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6231/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6231      +/-   ##
==========================================
- Coverage   79.29%   78.78%   -0.51%     
==========================================
  Files         146      146              
  Lines       26597    26646      +49     
==========================================
- Hits        21089    20994      -95     
- Misses       5508     5652     +144     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/testing\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90ZXN0aW5nX3V0aWxzLnB5) | `51.92% <28.57%> (-20.81%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `26.02% <0.00%> (-69.52%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `44.16% <0.00%> (-49.17%)` | :arrow_down: |
| [src/transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fY3RybC5weQ==) | `78.64% <0.00%> (-17.48%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.99% <0.00%> (-0.98%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `80.30% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `95.31% <0.00%> (+23.43%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6231/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `98.78% <0.00%> (+34.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=footer). Last update [57eb1cb...ab365a2](https://codecov.io/gh/huggingface/transformers/pull/6231?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> This looks useful to me for the tests. As for the disclaimer, I think just a mention of where the original code came from is enough.

done.


I'm not a 100% sure I understand exactly what this does. Do you mind showing a side-by-side comparison of a test output with and without it?

What I understand is that it captures one of the streams, and hides the other one (e.g. only captures stderr and throws away stdout). This seems a bit intimidating to me, as I can imagine having nightmarish debugging sessions trying to understand why random print statements are not output. If I understood this correctly, would it be possible to add a message saying that one of the streams has been hidden?

It does exactly the same thing as `capsys`, except:

- it gives you a much finer scope for when std streams (one or both) are overriden and captured. (the main feature)
- it doesn't get influenced by pytest cl args
- it cleans up all that `\r` output rewrites only returning the clean final output

So it actually gives you a much finer control over what and when gets captured. 

when you have a relatively complex test that uses `capsys`, you can't do debug printing and unless you are careful to dump the captured streams they get "eaten". With `CaptureStd` you just capture the slice of the stream exactly where you need it.

I wanted to use it here: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples.py#L332

So instead of the current:

```
def test_finetune_lr_shedulers(capsys):
    [...]
    with pytest.raises(SystemExit) as excinfo:
        args = parser.parse_args(args)
        assert False, "--help is expected to sys.exit"
    assert excinfo.type == SystemExit
    captured = capsys.readouterr()
    expected = lightning_base.arg_to_scheduler_metavar
    assert expected in captured.out, "--help is expected to list the supported schedulers"
```

it'd be:

```
def test_finetune_lr_shedulers(): # no capsys
    [...]
    with pytest.raises(SystemExit) as excinfo:
        with CaptureStdout() as cs:
            args = parser.parse_args(args)
        assert False, "--help is expected to sys.exit"
    assert excinfo.type == SystemExit
    expected = lightning_base.arg_to_scheduler_metavar
    assert expected in cs.out, "--help is expected to list the supported schedulers"
```
as you can see the stdout stream gets capture only in a scope of one line. and the rest of the test is not impacted at all - you can do debug, normal messages, etc.

Please let me know whether this helped to clarify why this would be a useful addition to the testing tools.

> What I understand is that it captures one of the streams, and hides the other one (e.g. only captures stderr and throws away stdout). This seems a bit intimidating to me, as I can imagine having nightmarish debugging sessions trying to understand why random print statements are not output. If I understood this correctly, would it be possible to add a message saying that one of the streams has been hidden?

No, these are just for convenience so a minimal overriding is done, you can capture either or both and it has aliases for even less typing: `CaptureStdout() ` is the same as `CaptureStd(err=False)`. That's the beauty of it, it leaves everything else intact.

And yes, `capsys`, can lead a nightmarish experience, that's why this helper came about.

Thanks @stas00 for the in-depth explanation. Indeed this seems useful, thanks a lot for your contribution!