Seq2SeqDataset uses linecache to save memory by @Pradhy729 (#5792) 
Author: sshleifer
Pull request: yes


Comments:
The isort check doesn't fail when I run it on my end. Other than that - I would say this is complete.


Awesome, I'm testing it now on WMT english-romanian translation.

OK, I have it running with many modifications and it works much better. Very little cpu ram wasted!

Is it OK if I make a new PR or do you prefer to add me as a contributor to your fork and so I can push to this branch? either way you will get all the credit in the release notes/pr summary :) 



Thanks - I'll add you as a contributor in mine. :)

> Thanks - I'll add you as a contributor in mine. :)

You can just click on "Allow edits from maintainers" on your PR. (in case you didn't know this feature)

Any updates here? Is it good to go?


I'm still working on cleaning up my code. Sorry for the delay. 

Biggest change:
- For MBart Tokenizer, we can't use the `encode_line` approach because there are special tokens all over the place, so I made a separate dataset. 

Stylistic:
- Address the `linecache` off by 1 error inside of `__getitem__` instead other places.
- `_get_examples` -> `get_char_lens`.
- `MbartTokenizer` cleanup.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=h1) Report
> Merging [#5792](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eae6d8d14f1d25d62c3fe9e7e410607bbaf69787&el=desc) will **increase** coverage by `0.91%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5792/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5792      +/-   ##
==========================================
+ Coverage   77.54%   78.46%   +0.91%     
==========================================
  Files         146      146              
  Lines       26200    26200              
==========================================
+ Hits        20318    20559     +241     
+ Misses       5882     5641     -241     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/5792/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmFydC5weQ==) | `95.45% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5792/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `26.02% <0.00%> (-69.52%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5792/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5792/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=footer). Last update [eae6d8d...79d73ee](https://codecov.io/gh/huggingface/transformers/pull/5792?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).