extremely slow processing when using a custom dataset 
Opened by StephennFernandes (closed)

## processing a custom dataset loaded as .txt file  is extremely slow,  compared to a dataset of similar volume from the hub

I have a large .txt file of 22 GB which i load into HF dataset 

`lang_dataset = datasets.load_dataset("text", data_files="hi.txt")`

further i use a pre-processing function to clean the dataset 

 `lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)`

the following processing takes astronomical time to process, while hoging all the ram. 

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data. 
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi",  use_auth_token=True)`

the hours predicted to preprocess are as follows:

huggingface hub dataset:  6.5 hrs 
custom loaded dataset: 7000 hrs

note: both the datasets are almost actually same, just provided by different sources with has +/- some samples, only one is hosted on the HF hub and the other is downloaded in a text format. 

## Steps to reproduce the bug
```
import datasets
import psutil
import sys 
import glob 
from fastcore.utils import listify
import re 
import gc 

def remove_non_indic_sentences(example): 
    tmp_ls = []
    eng_regex = r'[. a-zA-Z0-9ÖÄÅöäå _.,!"\'\/$]*'
    for e in listify(example['text']):
        matches = re.findall(eng_regex, e)
        for match in (str(match).strip() for match in matches if match not in [""," ", "  ", ",", " ,", ", ", " , "]):
            if len(list(match.split(" "))) > 2:
                e = re.sub(match," ",e,count=1)
        tmp_ls.append(e)
        gc.collect()
    example['clean_text'] = tmp_ls
    return example

lang_dataset = datasets.load_dataset("text", data_files="hi.txt")

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)


## same thing work much faster when loading similar dataset from hub
 
lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", split="train", use_auth_token=True)

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

```

## Actual results

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data. 
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", use_auth_token=True)


**the hours predicted to preprocess are as follows:**
huggingface hub dataset:  6.5 hrs 
custom loaded dataset: 7000 hrs

**i even tried the following:**

-  sharding the large 22gb text files into smaller files and loading
- saving the file to disk and then loading 
- using lesser num_proc 
- using smaller batch size 
- processing without batches ie :  without `batched=True`


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2.dev0
- Platform: Ubuntu 20.04 LTS 
- Python version: 3.9.7 
- PyArrow version:8.0.0 



Comments:
Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.

Closing due to inactivity