nlp downloads to its module path
Author: danieldk
State: closed

I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`?

Comments: Indeed this is a known issue arising from the fact that we try to be compatible with cloupickle.

Does this also happen if you are installing in a virtual environment?

> Indeed this is a know issue with the fact that we try to be compatible with cloupickle.
> 
> Does this also happen if you are installing in a virtual environment?

Then it would work, because the package is in a writable path.

If it's fine for you then this is the recommended way to solve this issue.

> If it's fine for you then this is the recommended way to solve this issue.

I don't want to use a virtual environment, because Nix is fully reproducible, and virtual environments are not. And I am the maintainer of the `transformers` in nixpkgs, so sooner or later I will have to package `nlp`, since it is becoming a dependency of `transformers` ;).

Ok interesting. We could have another check to see if it's possible to download and import the datasets script at another location than the module path. I think this would probably involve tweaking the python system path dynamically.

I don't know anything about Nix so if you want to give this a try your self we can guide you or you can give us more information on your general project and how this works.

Regarding `nlp` and `transformers`, we are not sure `nlp` will become a required dependency for `transformers`.  It will probably be used a lot in the examples but I think it probably won't be a required dependency for the main package since we try to keep it as light as possible in terms of deps.

Happy to help you make all these things work better for your use-case 

@danieldk modules are now installed in a different location (by default in the cache directory of the lib, in `~/.cache/huggingface/modules`). You can also change that using the environment variable `HF_MODULES_PATH`

Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
We plan to do a release in the next coming days

Awesome! I’ll hopefully have some time in the coming days to try this.

> Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
> We plan to do a release in the next coming days

Thanks for making this change! I just packaged the latest commit on master and it works like a charm now! :partying_face: