Cached dataset not loaded
Author: borisdayma
State: closed
Labels: name: bug

## Describe the bug
I have a large dataset (common_voice, english) where I use several map and filter functions.
Sometimes my cached datasets after specific functions are not loaded.
I always use the same arguments, same functions, no seed…

## Steps to reproduce the bug
```python
def filter_by_duration(batch):
    return (
        batch["duration"] <= 10
        and batch["duration"] >= 1
        and len(batch["target_text"]) > 5
    )

def prepare_dataset(batch):
    batch["input_values"] = processor(
        batch["speech"], sampling_rate=batch["sampling_rate"][0]
    ).input_values
    with processor.as_target_processor():
        batch["labels"] = processor(batch["target_text"]).input_ids
    return batch

train_dataset = train_dataset.filter(
    filter_by_duration,
    remove_columns=["duration"],
    num_proc=data_args.preprocessing_num_workers,
)

# PROBLEM HERE -> below function is reexecuted and cache is not loaded
train_dataset = train_dataset.map(
    prepare_dataset,
    remove_columns=train_dataset.column_names,
    batch_size=training_args.per_device_train_batch_size,
    batched=True,
    num_proc=data_args.preprocessing_num_workers,
)

# Later in script
set_caching_enabled(False)
# apply map on trained model to eval/test sets

```

## Expected results
The cached dataset should always be reloaded.

## Actual results
The function is reexecuted.

I have access to cached files `cache-xxxxx.arrow`.
Is there a way I can somehow load manually 2 versions and see how the hash was created for debug purposes (to know if it's an issue with dataset or function)?

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Linux-5.8.0-45-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Comments: It actually seems to happen all the time in above configuration:
* the function `filter_by_duration` correctly loads cached processed dataset
* the function `prepare_dataset` is always reexecuted

I end up solving the issue by saving to disk my dataset at the end but I'm still wondering if it's a bug or limitation here.

Hi ! The hash used for caching `map` results is the fingerprint of the resulting dataset. It is computed using three things:
- the old fingerprint of the dataset
- the hash of the function
- the hash of the other parameters passed to `map`

You can compute the hash of your function (or any python object) with
```python
from datasets.fingerprint import Hasher

my_func = lambda x: x + 1
print(Hasher.hash(my_func))
```

If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.

> If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.

Yes I think that was the issue.

For the hash of the function:
* does it consider just the name or the actual code of the function
* does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)

> does it consider just the name or the actual code of the function

It looks at the name and the actual code and all variables such as recursively. It uses `dill` to do so, which is based on `pickle`.
Basically the hash is computed using the pickle bytes of your function (computed using `dill` to support most python objects).

> does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)

Yes it does thanks to recursive pickling.

Thanks for these explanations. I'm closing the issue.