Repository: huggingface/datasets
Pull request: Add "ipykernel" to list of `co_filename`s to remove
Should resolve #5157 

Problem statement:
Consistent caching between python and jupyter
### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.


### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen

Hints from issue discussion:
Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`
Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!

Commit message: add ipykernel to co_filename to ignore

Patch to review:
diff --git a/src/datasets/utils/py_utils.py b/src/datasets/utils/py_utils.py
index c4b7152307e..ca86ab4b117 100644
--- a/src/datasets/utils/py_utils.py
+++ b/src/datasets/utils/py_utils.py
@@ -657,7 +657,9 @@ def _save_code(pickler, obj):
     #
     # Only those two lines are different from the original implementation:
     co_filename = (
-        "" if obj.co_filename.startswith("<") or obj.co_name == "<lambda>" else os.path.basename(obj.co_filename)
+        ""
+        if obj.co_filename.startswith("<") or "ipykernel" in obj.co_filename or obj.co_name == "<lambda>"
+        else os.path.basename(obj.co_filename)
     )
     co_firstlineno = 1
     # The rest is the same as in the original dill implementation


Code review comments:
diff_hunk: @@ -657,7 +657,9 @@ def _save_code(pickler, obj):
     #
     # Only those two lines are different from the original implementation:
     co_filename = (
-        "" if obj.co_filename.startswith("<") or obj.co_name == "<lambda>" else os.path.basename(obj.co_filename)
+        ""
+        if obj.co_filename.startswith("<") or "ipykernel" in obj.co_filename or obj.co_name == "<lambda>"
path: src/datasets/utils/py_utils.py
text: @user1:
Nice ! Can you also update the comments to explain how ipykernel functions are named ?

Also do you have a more robust way of detecting ipykernel functions ? If it comes from a filename equals to "path/to/ipykernel_folder/packaged/main.py" then it would trigger this "if" statement