Lazy text dataset loading for language modelling with PyTorch
Author: GCHQResearcher92457
Pull request: yes

#3083 

Added a lazy text dataset using linecache to run_language_modeling.py. Slightly refactored collate_fn construction to accommodate the different collate functions needed for a lazy dataset vs a in-memory dataset. 

Comments:
Can anyone advise on the failed tests? Seems to failing in parts of the code-base I haven't touched.

@GCHQResearcher92457 Yes, the failures are unrelated.

Most recent failures are unrelated.

Quick question in passing because I am working on something close, did you run some benchmark to see how this behaves speedwise?

> Quick question in passing because I am working on something close, did you run some benchmark to see how this behaves speedwise?

I've found in practice so far that training iterations are the same speed using this method as the previous methods, i.e. the bottleneck seems to be later on. Just did some crude tests on a file of 100 lines to test only the data loading performance.

Instantiating lazy dataset: 39.1 µs ± 408 ns
Instantiating cached dataset: 17.4 ms ± 41.6 µs 
Random access to single item (lazy): 3.34 µs ± 171 
Random access to single item (cached): 3.18 µs ± 86.3 ns
Creating a single tokenized batch (lazy): 1.33 ms ± 3.5 µs
Creating a single tokenized batch (cached): 81.5 µs ± 5.15 µs

I tested out the LazyLineByLineTextDataset and quickly ran out of memory.

It looks like linecache isn't capable of efficiently indexing into large files. My ~6GB training data causes linecache to stall & use up 7+ GB of RAM. 

Saw a similar issue [here](https://stackoverflow.com/questions/620367/how-to-jump-to-a-particular-line-in-a-huge-text-file/2727585). Might be better to use a system similar to the second answer in that post where you create a map of line breaks in the file and seek to them. 

```
class LineSeekableFile:
    def __init__(self, seekable):
        self.fin = seekable
        self.line_map = list() # Map from line index -> file position.
        self.line_map.append(0)
        while seekable.readline():
            self.line_map.append(seekable.tell())

    def __getitem__(self, index):
        # NOTE: This assumes that you're not reading the file sequentially.  
        # For that, just use 'for line in file'.
        self.fin.seek(self.line_map[Index])
        return self.fin.readline()
```

> I tested out the LazyLineByLineTextDataset and quickly ran out of memory.
> 
> It looks like linecache isn't capable of efficiently indexing into large files. My ~6GB training data causes linecache to stall & use up 7+ GB of RAM.
> 
> Saw a similar issue [here](https://stackoverflow.com/questions/620367/how-to-jump-to-a-particular-line-in-a-huge-text-file/2727585). Might be better to use a system similar to the second answer in that post where you create a map of line breaks in the file and seek to them.
> 
> ```
> class LineSeekableFile:
>     def __init__(self, seekable):
>         self.fin = seekable
>         self.line_map = list() # Map from line index -> file position.
>         self.line_map.append(0)
>         while seekable.readline():
>             self.line_map.append(seekable.tell())
> 
>     def __getitem__(self, index):
>         # NOTE: This assumes that you're not reading the file sequentially.  
>         # For that, just use 'for line in file'.
>         self.fin.seek(self.line_map[Index])
>         return self.fin.readline()
> ```

Did you run into out-of-memory issues, or did the process simply use a lot of memory? It is likely to be the latter, and that is exactly what line_cache_ is supposed to do: it reads as much of the file into memory as it can for quick access as much as possible (considering the available memory), and then does its work.

LineSeekableFile can be an alternative but definitely not a good replacement imo (it'll be slower, and expects a file handle to always be open which you often would not want).

> Did you run into out-of-memory issues, or did the process simply use a lot of memory? It is likely to be the latter, and that is exactly what line_cache_ is supposed to do: it reads as much of the file into memory as it can for quick access as much as possible (considering the available memory), and then does its work.
> 
> LineSeekableFile can be an alternative but definitely not a good replacement imo (it'll be slower, and expects a file handle to always be open which you often would not want).

I ran the code on a GCP VM instance with 13 GB of RAM. My RAM quickly went to 0 and I was kicked out of SSH. I was forced to restart the instance in order to regain access. 

From what I'm seeing, it seems like linecache is primarily designed to be used on Python source files, not large text files. From what I can tell, the [source code](https://github.com/python/cpython/blob/10dabbf8d2c1c929f6ac395e19c64b361bd58fdd/Lib/linecache.py#L82) reads all the lines in the file into memory, without any consideration for available memory. 

@ceremonious I tested this locally with a 50+GB file on my 32GB RAM system and it works as expected. Memory usage goes up to around 95% and stays there. Reproducible code:

```python
import linecache
import random


def get_n_lines(fin, size=65536):
    # borrowed from https://stackoverflow.com/a/9631635/1150683
    def blocks(files):
        while True:
            b = files.read(size)
            if not b:
                break
            yield b

    with open(fin, encoding="utf-8") as fhin:
        n_lines = sum(bl.count("\n") for bl in blocks(fhin))
    return n_lines

def main(fin):
    n_lines = get_n_lines(fin)
    while True:
        idx = random.randint(1, n_lines+1)
        line = linecache.getline(fin, idx)
        print(line)


if __name__ == '__main__':
    f = r'path/to/huge/file.txt'
    main(f)
```

I haven't dug into the source code (though I do see a MemoryError check in it), but I have used this for many projects on our own servers and I can tell you that it works (it will utilise as much RAM as it can but won't throw OOM errors). It is good to know that this won't work well with GCP, though! A note should be included in the class's docstring.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=h1) Report
> Merging [#3388](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d22894dfd40d5c858e8398e2783545103d191b47&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3388/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3388   +/-   ##
=======================================
  Coverage   78.26%   78.26%           
=======================================
  Files         106      106           
  Lines       17964    17964           
=======================================
  Hits        14060    14060           
  Misses       3904     3904           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=footer). Last update [d22894d...1ead846](https://codecov.io/gh/huggingface/transformers/pull/3388?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


The merge conflicts are a bit of a mess because of datasets and collation being moved outside the main script in `master`. I've opened a new PR where the code used here has been slotted-in to the the more modular format of this script. Please see PR #4009.

Any updates on this PR? Lazy loading sounds like an important functionality for massive datasets.

@misrasaurabh1 This PR is closed. See https://github.com/huggingface/transformers/pull/4009 for the continuation.