Memory error : load 200GB file in run_language_model.py
Author: omerarshad
Pull request: no
Labels: name: Discussion

name: wontfix

name: High-Level feature

name: Core: Tokenization


line 107 in run_language_model.py

with open(file_path, encoding="utf-8") as f:
                text = f.read()

any idea how to use generators to load large files?

Comments:
I agree that an on-the-fly tokenisation would be neat as an alternative to pre-processing the whole input file and saving the tensors in memory. 

Hi @BramVanroy , this is mentioned in the blog post bout training models from scratch, as something that could be done (https://huggingface.co/blog/how-to-train). Is it possible?
Thanks!

I am fairly new to how contributing to HuggingFace works but gave this a little thought today.
At first I thought we could maybe solve it like this:

If we consider this code:
`tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))`

My first intuition was that the space needed to save the result of this code is significantly less than the space needed to store f.read(). 
So if we could get the result of this code by just reading a line (or reading parts of the texts with overlaps to the previous part as big as max_len of tokens), we might solve it...
However I ran a litte experiment and it turns out that `tokenized_text` would still take up around 133GB with an input txt file of 200GB.
So not a solution.
Do you guys have any idea how to approach this differently? Because of the RandomSampler we also can't store parts of the file trivially.

The solution would be to work with a dataset that on every call, fetches the lines from the file that are required, rather than reading the whole file in memory. This is bound to be slower, but it is very lenient on memory. A good candidate is linecache, which does smart caching in the process.

```python
import linecache
from pathlib import Path

from torch.utils.data import Dataset


class LazyTextDataset(Dataset):
    def __init__(self, fin):
        self.fin = fin
        self.num_entries = self._get_n_lines(self.fin)

    @staticmethod
    def _get_n_lines(fin):
        with Path(fin).resolve().open(encoding='utf-8') as fhin:
            for line_idx, _ in enumerate(fhin, 1):
                pass

        return line_idx

    def __getitem__(self, idx):
        # linecache starts counting from one, not zero, +1 the given index
        idx += 1
        return linecache.getline(self.fin, idx)

    def __len__(self):
        return self.num_entries
```

Then you can add a custom `collate_fn` to your data loader that will automatically tokenize the whole batch.

I thought about this but figured it would definitely be too slow for such big files. I didn't know about linecache though, cool!

I've been using a similar solution to @BramVanroy for a couple of weeks, though I too was not aware of `linecache`, so assume that my solution can be improved by using that tool. 

I implemented this because the up-front loading was taking hours and hours. I did some rough comparisons on smaller data files and found that I was getting the same iters/second  using this method as the existing methods.


```python
class LazyUnSupervisedTextExamples:
    """
    Deals with file i/o for lazy retrieval of specific lines of text in file.

    """
    def __init__(self, path):
        """
        :args:
            path: str
                : The path to the data file to be loaded.
        """
        self.data_path = path
        self.data_stream = open(self.data_path, 'r')
        self.offsets = [0]
        for line in self.data_stream:
            self.offsets.append(self.offsets[-1] + len(line.encode('utf-8')))
        self.offsets = self.offsets[1:-1]
        self.data_stream.seek(0)
        self.current_offset = 0

    def __len__(self):
        return len(self.offsets)

    def __getitem__(self, _id):
        """
        :returns:
            str; the line of text given by _id if no errors.
            None if errors occur.

            PEP8 note: we really do want a bare exception here because an uncaught exception in here has the potential
            to bring down a large training run with an error in a single line of the data file.
        """

        offset = self.offsets[_id]
        try:
            self.data_stream.seek(offset)
            line = self.data_stream.readline()
            example = line.strip()
            self.data_stream.seek(self.current_offset)
        except:
            example = None
        return example

    def __next__(self):
        line = self.data_stream.readline()
        self.current_offset = self.data_stream.tell()
        return line.strip()

    def close(self):
        self.data_stream.close()


class LazyUnSupervisedTextDataset(Dataset):
    """
    Works with datasets of simple lines of text. Lines are loaded and tokenized
    lazily rather than being pulled into memory up-front. This reduces the memory
    footprint when using large datasets, and also remedies a problem seen when using
    the other Datasets (above) whereby they take too long to load all
    of the data and tokenize it before doing any training.

    The file i/o work is handled within self.examples. This class just indexes
    into that object and applies the tokenization.
    """
    def __init__(self,  tokenizer, file_path, block_size=512):
        """
        :args:
            tokenizer: tokenizer.implementations.BaseTokenizer object (instantiated)
                     : This tokenizer will be directly applied to the text data
                       to prepare the data for passing through the model.
            file_path: str
                     : Path to the data file to be used.
            block_size: int
                      : The maximum length of a sequence (truancated beyond this length).
        :returns: None.
        """
        self.examples = LazyUnSupervisedTextExamples(file_path)
        self.tokenizer = tokenizer
        self.max_len = block_size

    def __len__(self):
        return len(self.examples)

    def _text_to_tensor(self, item):
        """
        Defines the logic for transforming a single raw text item to a tokenized
        tensor ready to be passed into a model.

        :args:
            item: str
                : The text item as a string to be passed to the tokenizer.
        """
        return torch.tensor(self.tokenizer.encode(item, max_length=self.max_len))

    def _text_to_item(self, text):
        """
        Convenience functino to encapsulate re-used logic for converting raw
        text to the output of __getitem__ of __next__.

        :returns:
            torch.Tensor of tokenized text if no errors.
            None if any errors encountered.
        """
        try:
            if (text is not None):
                return self._text_to_tensor(text)
            else:
                return None
        except:
            return None

    def __getitem__(self, _id):
        """
        :returns:
            torch.Tensor of tokenized text if no errors.
            None if any errors encountered.
        """
        text = self.examples[_id]
        return self._text_to_item(text)

    def __next__(self):
        text = next(self.examples)
        return self._text_to_item(text)

    def close(self):
        """
        Since the LazyUnSupervisedTextExamples object self.examples contains a
        file handle, this method provides access to its close function to safely
        close the open data file when finished. This should be run when the
        dataset object is finished with.
        """
        self.examples.close()
```

The only change I found necessary to make to the `collate_fn` was a line to filter out lines that failed to load. I'm currently tokenising one item at a time, but prefer @BramVanroy's suggestion of batch tokenisation in the `collate_fn`. 

```python
def collate(examples: List[torch.Tensor]):
    examples = list(filter(lambda ex: ex is not None, examples))
    if tokenizer._pad_token is None:
        return pad_sequence(examples, batch_first=True)
    return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
```


Happy to make above-mentioned sensible changes and contribute.

Does anyone have any advice about more sophisticated performance testing to shore-up my above claim that lazy loading isn't any slower per iteration?

I would recommend to, indeed, run the tokenisation in collate_fn. You can use `batch_encode_plus` there. Concerning your collate function: the filter function can be simplified to `filter(None, examples)` but in fact I'd go with a list comprehension right away: `[ex for ex in examples if ex is not None]`. 

For timing you can use the timeit module.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> I've been using a similar solution to @BramVanroy for a couple of weeks, though I too was not aware of `linecache`, so assume that my solution can be improved by using that tool.
> 
> I implemented this because the up-front loading was taking hours and hours. I did some rough comparisons on smaller data files and found that I was getting the same iters/second using this method as the existing methods.
> 
> ```python
> class LazyUnSupervisedTextExamples:
>     """
>     Deals with file i/o for lazy retrieval of specific lines of text in file.
> 
>     """
>     def __init__(self, path):
>         """
>         :args:
>             path: str
>                 : The path to the data file to be loaded.
>         """
>         self.data_path = path
>         self.data_stream = open(self.data_path, 'r')
>         self.offsets = [0]
>         for line in self.data_stream:
>             self.offsets.append(self.offsets[-1] + len(line.encode('utf-8')))
>         self.offsets = self.offsets[1:-1]
>         self.data_stream.seek(0)
>         self.current_offset = 0
> 
>     def __len__(self):
>         return len(self.offsets)
> 
>     def __getitem__(self, _id):
>         """
>         :returns:
>             str; the line of text given by _id if no errors.
>             None if errors occur.
> 
>             PEP8 note: we really do want a bare exception here because an uncaught exception in here has the potential
>             to bring down a large training run with an error in a single line of the data file.
>         """
> 
>         offset = self.offsets[_id]
>         try:
>             self.data_stream.seek(offset)
>             line = self.data_stream.readline()
>             example = line.strip()
>             self.data_stream.seek(self.current_offset)
>         except:
>             example = None
>         return example
> 
>     def __next__(self):
>         line = self.data_stream.readline()
>         self.current_offset = self.data_stream.tell()
>         return line.strip()
> 
>     def close(self):
>         self.data_stream.close()
> 
> 
> class LazyUnSupervisedTextDataset(Dataset):
>     """
>     Works with datasets of simple lines of text. Lines are loaded and tokenized
>     lazily rather than being pulled into memory up-front. This reduces the memory
>     footprint when using large datasets, and also remedies a problem seen when using
>     the other Datasets (above) whereby they take too long to load all
>     of the data and tokenize it before doing any training.
> 
>     The file i/o work is handled within self.examples. This class just indexes
>     into that object and applies the tokenization.
>     """
>     def __init__(self,  tokenizer, file_path, block_size=512):
>         """
>         :args:
>             tokenizer: tokenizer.implementations.BaseTokenizer object (instantiated)
>                      : This tokenizer will be directly applied to the text data
>                        to prepare the data for passing through the model.
>             file_path: str
>                      : Path to the data file to be used.
>             block_size: int
>                       : The maximum length of a sequence (truancated beyond this length).
>         :returns: None.
>         """
>         self.examples = LazyUnSupervisedTextExamples(file_path)
>         self.tokenizer = tokenizer
>         self.max_len = block_size
> 
>     def __len__(self):
>         return len(self.examples)
> 
>     def _text_to_tensor(self, item):
>         """
>         Defines the logic for transforming a single raw text item to a tokenized
>         tensor ready to be passed into a model.
> 
>         :args:
>             item: str
>                 : The text item as a string to be passed to the tokenizer.
>         """
>         return torch.tensor(self.tokenizer.encode(item, max_length=self.max_len))
> 
>     def _text_to_item(self, text):
>         """
>         Convenience functino to encapsulate re-used logic for converting raw
>         text to the output of __getitem__ of __next__.
> 
>         :returns:
>             torch.Tensor of tokenized text if no errors.
>             None if any errors encountered.
>         """
>         try:
>             if (text is not None):
>                 return self._text_to_tensor(text)
>             else:
>                 return None
>         except:
>             return None
> 
>     def __getitem__(self, _id):
>         """
>         :returns:
>             torch.Tensor of tokenized text if no errors.
>             None if any errors encountered.
>         """
>         text = self.examples[_id]
>         return self._text_to_item(text)
> 
>     def __next__(self):
>         text = next(self.examples)
>         return self._text_to_item(text)
> 
>     def close(self):
>         """
>         Since the LazyUnSupervisedTextExamples object self.examples contains a
>         file handle, this method provides access to its close function to safely
>         close the open data file when finished. This should be run when the
>         dataset object is finished with.
>         """
>         self.examples.close()
> ```
> 
> The only change I found necessary to make to the `collate_fn` was a line to filter out lines that failed to load. I'm currently tokenising one item at a time, but prefer @BramVanroy's suggestion of batch tokenisation in the `collate_fn`.
> 
> ```python
> def collate(examples: List[torch.Tensor]):
>     examples = list(filter(lambda ex: ex is not None, examples))
>     if tokenizer._pad_token is None:
>         return pad_sequence(examples, batch_first=True)
>     return pad_sequence(examples, batch_first=True, padding_value=tokenizer.pad_token_id)
> ```
> 
> Happy to make above-mentioned sensible changes and contribute.
> 
> Does anyone have any advice about more sophisticated performance testing to shore-up my above claim that lazy loading isn't any slower per iteration?



> Does anyone have any advice about more sophisticated performance testing to shore-up my above claim that lazy loading isn't any slower per iteration?

When reading the file line in seek, numworkers in dataloader >1, afraid  about that the reading will be wrong. Not sure how you do it