Saving processed dataset running infinitely
State: open | Author: ayubSubhaniya (NONE)

I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.

Comments: @thomwolf @lhoestq can you guys please take a look and recommend some solution.

am suspicious of this thing? what's the purpose of this? pickling and unplickling
`self = pickle.loads(pickle.dumps(self))`

```
    def save_to_disk(self, dataset_path: str, fs=None):
        """
        Saves a dataset to a dataset directory, or in a filesystem using either :class:`datasets.filesystem.S3FileSystem` or any implementation of ``fsspec.spec.AbstractFileSystem``.

        Args:
            dataset_path (``str``): path (e.g. ``dataset/train``) or remote uri (e.g. ``s3://my-bucket/dataset/train``) of the dataset directory where the dataset will be saved to
            fs (Optional[:class:`datasets.filesystem.S3FileSystem`,``fsspec.spec.AbstractFileSystem``],  `optional`, defaults ``None``): instance of :class:`datasets.filesystem.S3FileSystem` or ``fsspec.spec.AbstractFileSystem`` used to download the files from remote filesystem.
        """
        assert (
            not self.list_indexes()
        ), "please remove all the indexes using `dataset.drop_index` before saving a dataset"
        self = pickle.loads(pickle.dumps(self))
  ```

It's been 24 hours and sadly it's still running. With not a single byte written

Tried finding the root cause but was unsuccessful.
I am using lazy tokenization with `dataset.set_transform()`, it works like a charm with almost same performance as pre-compute.

Hi ! This very probably comes from the hack you used.

The pickling line was added an a sanity check because save_to_disk uses the same assumptions as pickling for a dataset object. The main assumption is that memory mapped pyarrow tables must be reloadable from the disk. In your case it's not possible since you altered the pyarrow table.
I would suggest you to rebuild a valid Dataset object from your new pyarrow table. To do so you must first save your new table to a file, and then make a new Dataset object from that arrow file.

You can save the raw arrow table (without all the `datasets.Datasets` metadata) by calling `map` with `cache_file_name="path/to/outut.arrow"` and `function=None`. Having `function=None` makes the `map` write your dataset on disk with no data transformation.

Once you have your new arrow file, load it with `datasets.Dataset.from_file` to have a brand new Dataset object :)

In the future we'll have a better support for the fast filtering method from pyarrow so you don't have to do this very unpractical workaround. Since it breaks somes assumptions regarding the core behavior of Dataset objects, this is very discouraged.

Thanks, @lhoestq for your response. Will try your solution and let you know.