read each lines, require less memory
Author: knok
Pull request: yes

The original code reads whole data at once, so it requires so much memory to handle huge corpus.
This change is:
* read corpus by each lines
* flatten 2-dimension array by itertools.chain, it requies less memory and fast

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=h1) Report
> Merging [#2339](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/537a1de53d824b5851bce32cb5eafaef3f9ce5ef?src=pr&el=desc) will **increase** coverage by `1.11%`.
> The diff coverage is `75.31%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2339/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #2339      +/-   ##
=========================================
+ Coverage   73.49%   74.6%   +1.11%     
=========================================
  Files          87      87              
  Lines       14793   14802       +9     
=========================================
+ Hits        10872   11043     +171     
+ Misses       3921    3759     -162
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jYW1lbWJlcnQucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [src/transformers/tokenization\_bert\_japanese.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydF9qYXBhbmVzZS5weQ==) | `67.46% <ø> (ø)` | :arrow_up: |
| [src/transformers/configuration\_mmbt.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX21tYnQucHk=) | `55.55% <ø> (ø)` | :arrow_up: |
| [src/transformers/commands/serving.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9zZXJ2aW5nLnB5) | `0% <ø> (ø)` | :arrow_up: |
| [src/transformers/commands/convert.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9jb252ZXJ0LnB5) | `0% <ø> (ø)` | :arrow_up: |
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `36.76% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_mmbt.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tbWJ0LnB5) | `17.6% <0%> (ø)` | :arrow_up: |
| [src/transformers/tokenization\_camembert.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fY2FtZW1iZXJ0LnB5) | `35.71% <0%> (ø)` | :arrow_up: |
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `27.86% <0%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `25.3% <0%> (ø)` | :arrow_up: |
| ... and [72 more](https://codecov.io/gh/huggingface/transformers/pull/2339/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=footer). Last update [537a1de...9166b24](https://codecov.io/gh/huggingface/transformers/pull/2339?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


We recently merged a `LineByLineTextDataset` that should be equivalent:
https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L124

Feedback welcome.