SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Cannot pickle error in Dataset.from_generator()

Comment:
> Hi! It should work if you put `model = torch.compile(model)` inside the `generate_data` function. If a referenced object is outside, it needs to be pickable, and that's not the case for the compiled models (or functions).

Hi! Thank you for your reply! Everything works perfectly with your suggestion!

Closing the issue.
Issue: Ability to update a dataset

Comment:
In this case, I think we can avoid the `PermissionError` by unpacking the underlying `ConcatenationTable` and saving only the newly added data blocks (in new files).
Issue: Ability to update a dataset

Comment:
Thanks @stevhliu and @mariosasko , so saving to individual files then loading them later, concatenating again and saving again is the recommended way. Good to know.

Question that I hope doesn't sound rude: is this sort of thing (processing a dataset that doesn't fit in memory) outside of `datasets`'s core area of focus? Are there other tools you would recommend to do this sort of thing that play nice with `datasets`? Or is it just that I've found myself in a niche situation that hasn't specifically been catered for?
Issue: Setting `num_proc` errors when `.map` returns additional items.

Comment:
Hi ! Unfortunately I couldn't reproduce on my side locally and with datasets 2.11 and python 3.10.11 on colab.
What version of `multiprocess` are you using ?
Issue: Setting `num_proc` errors when `.map` returns additional items.

Comment:
I've got `multiprocess` version `0.70.14`.

I've done some more testing and the error only occurs in PyCharm's Python Console. It seems to be [this PyCharm bug](https://youtrack.jetbrains.com/issue/PY-51922/Multiprocessing-bug.-Can-only-run-in-debugger.), I'll close this.
Issue: Setting `num_proc` errors when `.map` returns additional items.

Comment:
For other users facing this, my workaround is to conditionally set `num_proc` so I can work interactively in the PyCharm Python Console while developing, then when I'm ready to run on the whole dataset, run it as a script and use multiprocessing.

```py
mapped_ds = ds.map(
    my_map_function,
    batched=True,
    remove_columns=ds.column_names,
    num_proc=1 if "PYCHARM_HOSTED" in os.environ else 8,
)
```
Issue: Easy way to create a Kaggle dataset from a Huggingface dataset?

Comment:
Hi @hrbigelow , I'm no expert for such a question so I'll ping @lhoestq from the `datasets` library (also this issue could be moved there if someone with permission can do it :) )
Issue: Easy way to create a Kaggle dataset from a Huggingface dataset?

Comment:
Hi ! Many datasets are made of several files, and how they are parsed often requires a python script. Because of that, datasets like wmt14 are not available as a single file on HF. Though you can create this file using `datasets`:

```python
from datasets import load_dataset

ds = load_dataset("wmt14", "de-en", split="train")

ds.to_json("wmt14-train.json")
# OR to parquet, which is compressed:
# ds.to_parquet("wmt14-train.parquet")
```

We are also working on providing parquet exports for all datasets, but wmt14 is not supported yet (we're rolling it out for datasets <1GB first). They're usually available in the `refs/convert/parquet` branch (empty for wmt14):

<img width="267" alt="image" src="https://user-images.githubusercontent.com/42851186/235878909-7339f5a4-be19-4ada-85d8-8a50d23acf35.png">
Issue: Easy way to create a Kaggle dataset from a Huggingface dataset?

Comment:
I've requested support for creating a Kaggle dataset from an imported HF dataset repo on their "forum" here: https://www.kaggle.com/discussions/product-feedback/427142 (upvotes appreciated 🙂)
Issue: load_dataset: TypeError: 'NoneType' object is not callable, on local dataset filename changes

Comment:
This error means a `DatasetBuilder` subclass that generates the dataset could not be found inside the script, so make sure `dushowxa-characters/dushowxa-characters.py `is a valid dataset script (assuming `path_or_dataset` is `dushowxa-characters`)

Also, we should improve the error to make it more obvious what the problem is.
Issue: wiki_dpr details for Open Domain Question Answering tasks

Comment:
Hi ! I don't remember exactly how it was done, but maybe you have to embed `f"{title}<sep>{text}"` ?

Using a HF tokenizer it corresponds to doing
```python
tokenized = tokenizer(titles, texts)
```
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
Implementing this makes sense (e.g., `tensorflow_datasets`' imagefolder returns image filenames). Also, in Datasets 3.0, we plan only to store the bytes of an image/audio, not its path, so this feature would be useful when the path info is still needed.
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
Hey @mariosasko, Can I work on this issue, this one seems interesting to implement. I have contributed to jupyterlab recently, and would love to contribute here as well.
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
@tsabbir96 if you are planning to start working on this, you can take on this issue by writing a comment with only the keyword: #self-assign
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
@albertvillanova thank you for letting me contribute here. 
@albertvillanova @mariosasko As I am totally new to this repo, could you tell me something more about this issue or perhaps give me some idea on how I can proceed with it? Thanks!
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
Hello there, is this issue resolved? @tsabbir96 are you still working on it? Otherwise I would love to give it a try
Issue: Return the name of the currently loaded file in the load_dataset function.

Comment:
Hey @mariosasko, I've taken the time to take a look at how we load the datasets usually. My main question now is about the final solution.

So the idea is that whenever we load the datasets we also add a new column in the _generate_tables() method from the builders called filename (or file_name) that should be related files contained in each split, right?

Do you have any suggestions of where I could add that?
Issue: Improve `Create a dataset` tutorial 

Comment:
I can work on this. The link to the tutorial seems to be broken though @polinaeterna.
Issue: Improve `Create a dataset` tutorial 

Comment:
@isunitha98selvan would be great, thank you! which link are you talking about? I think it should work: https://huggingface.co/docs/datasets/create_dataset
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Hi ! We're using process pools for parallelism right now. I was wondering if there's a package that implements the same API as a process pool but runs with Spark under the hood ? That or something similar would be cool because users could use whatever distributed framework they want this way.

Feel free to ping us when you'd like to open PRs for this kind of things, so that we can discuss this before you start working on it ^^
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Hi, thanks for taking a look and providing your input! I don't know of such packages, and even it exists, I don't think with the process pool API it's possible to run Spark as backend properly; otherwise I understand a unified API would be preferable.

The process pool API requires splitting the workload to a fixed number parts for multiprocessing; meanwhile distributed framework such as Spark has sophisticated scheduler to distribute the workload to the processes on multiple machines in a cluster, so the way of splitting things for `multiprocessing.pool` would not suit / be as flexible as directly calling the `sparkContext.parallelize` API.

I think this could be a good addition to scale the `datasets` implementation to distributed workers, and from my benchmark results so far it looks promising compared with multiprocessing.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
I see ! I think we only need an equivalent of `pool.map`. We use it to run download and conversion of data files on disk. That would require less changes in the internal code - and therefore less tests to write ;)

We also use `pool.apply_async` in some places with a `Queue` to get progress updates of the running jobs. I'm mentioning this in case there's a way to get a python generator from a running spark job ? This is less important though
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
For Spark, `rdd.map` (where `rdd` can be created by `sparkContext.parallelize`) is the most similar as `pool.map`, but it requires creating a Spark RDD first that is used for distributing the `iterable` and the actual parallelization is managed by the Spark framework; `pool.map` takes the splits of `iterable` that are split into `num_proc` parts by the Python code. You can also check my PR #5807 in the `src/datasets/utils/py_utils.py` file to compare the differences of the APIs, it might make more sense than the the above description.

Given the different inputs and mechanisms of calling the `map` functions, this is why I think it's not that feasible to reuse most of the `multiprocessing` code.

Progress bar updating might be challenging with Spark, I'll consider it as a followup work.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Indeed I think the current use of multiprocessing.Pool in `map_nested` can be rewritten to work like `sparkContext.parallelize` - without splitting the iterable.

Maybe from the user's perspective it's ok to let multiprocessing.Pool or spark distribute the load on their own, as long as it takes a list and runs jobs in parallel in the end :)
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
From your feedback, seems to me there are two paths to consider now for supporting spark's `map` function in `map_nested` now:
1. Keep the current `pool.map` implementation, and add an if statement for the spark's `map` code (which is what I did in my current PR) -- the code change is just a few lines in the `map_nested` function, and it has been tested by unit tests + manual testing on real Spark clusters; if you have other concerns I'd also be happy to address them.
2. Rewrite the current `pool.map` implementation to remove splitting the iterable, and we will still need to add an if statement to use either
```python
with Pool(...) as pool:
    mapped = pool.map(_single_map_nested, iterable)
```
or
```python
rdd = spark.sparkContext.parallelize(iterable)
mapped = rdd.map(lambda obj: _single_map_nested((function, obj, types, None, True, None))).collect()
```
because there is no unified API that supports both `pool.map` and `rdd.map`. This can be more unified and flexible in the long run, but might require more work, and it will change the existing multiprocessing behavior, which is why I'm not leaning towards this option.

Am I understanding correctly?
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Yup correct ! I think it's a nice path because it would be possible for users to define whatever parallel processing backend they want. I think we still need to discuss how that would look like in the `datasets` API : how to specify it has to use the "spark" parallel backend ? And how to specify the spark session parameters (number of executors etc.) ? Maybe there is something more practical than `use_spark=True`

I'll check with the team internally if they have some ideas, but feel free to share your thoughts here !
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Sure, please let me know if you have more updates regarding the API and implementation from the team.

For parameters we don't need to worry about setting them for Spark, because Spark will figure out the environment / number of worker nodes by itself, so it's preferable to just provide some parameter such as `use_spark` to use the RDD `map` function.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Hi! I wanted to check in to see if there is any update from the team.

A potential change of API I can think of is change the argument to `distributed_backend=...`, which accepts `str`, such as `load_dataset(..., distributed_backend="spark")`.

Implementation wise, we can add a class / function to abstract away the details of using multiprocessing vs. spark vs. other parallel processing frameworks in `map_nested` and `_prepare_split`.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
I found this quite interesting: https://github.com/joblib/joblib-spark with this syntax:

```python
with parallel_backend('spark', n_jobs=3):
    ...
```

cc @lu-wang-dl who might know better
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Joblib spark is providing Spark backend for joblib. We can implement a general parallel backend like
```
with parallel_backend("<parallel-backedn>", n_jobs=..):
```

It can support multiprocessing , spark, ray, and etc. https://joblib.readthedocs.io/en/latest/parallel.html#joblib.parallel_backend
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Thank you @lhoestq for finding this repo. I validated that it can distribute downloading jobs with Spark to arbitrary cluster worker nodes evenly with `n_jobs=-1`.

For the API, I think it makes sense to define it as
```python
load_dataset(..., parallel_backend=<str>)
```
where `parallel_backend` can be `spark`, `multiprocessing`,  and potentially other supported joblib backends including `ray` and `dask`.

Implementation-wise, do you think it is better to just use `joblib` for `spark` backend in `map_nested`, or also migrate the `multiprocessing.Pool` code to use `joblib`?
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Hello @lhoestq, I wanted to follow up on my previous comment with some prototyping code that demonstrates how `map_nested` would be like if we unify `multiprocessing` and `spark` with `joblib`. The snippet hasn't hashed out the details such as dealing with `tqdm` yet.

In terms of API, the way of using multiprocessing is still the same; for Spark, the user sets `parallel_backend='spark'`  can reuse the `num_proc` argument to pass in the number of executors, or preferably, just set `num_proc=-1` and joblib is able to decide it (I've validated it by running it on a Spark cluster).

```python
def map_nested(
  # ... same args
  parallel_backend: Optional[str] = None, # proposed new argument
):

  # ... same code

  # allow user to specify num_proc=-1, so that joblib will optimize it
  if (num_proc <= 1 and num_proc != -1) or len(iterable) < parallel_min_length:
    # same code
    mapped = [
      _single_map_nested((function, obj, types, None, True, None))
      for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    ]
  else:
    if not parallel_backend:
      parallel_backend = 'loky' # 'loky' is joblib's own implementation of robust multiprocessing
    
    n_jobs = min(num_proc, len(iterable))

    if parallel_backend == 'spark':
      n_jobs = -1 # 'loky' is joblib's own implementation of robust multiprocessing
      from joblibspark import register_spark
      register_spark()

    # parallelized with the same API
    with joblib.parallel_backend(parallel_backend, n_jobs=n_jobs):
      mapped = joblib.Parallel()(
        joblib.delayed(
          _single_map_nested((function, obj, types, None, True, None))
        )(obj) for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
      )
  
  # ... same code
```
We can always `joblib` for Spark and other distributed backends such as Ray if people want to support them later. It's worth noting that some distributed backends do not currently have `joblib` implementations.

I would appreciate your thoughts on this proposed new API. We can also discuss the pros and cons of migrating the `multiprocessing` code to `joblib` later.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Nice ! It should be quite easy to make the change then :)

I think adding spark support can actually be less than 20 lines of code and would roughly require one line of code to change in map_nested:

Maybe we can define a new `datasets.parallel` submodule that has the `parallel_backend()` context manager and a `parallel_map()` function that uses `Pool.map` by default and `joblib` otherwise.

`joblib` would be an optional dependency, and `joblib-spark` as well.

Then whenever someone wants to use Spark, they can do something like this (similar to scikit-learn parallel_backend):

```python
from datasets.parallel import parallel_backend

with parallel_backend("spark"):
    ds = load_dataset(...)
```

What do you think ?
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Although until we've switched to all the steps in `load_dataset` to use `datasets.parallel`, I would require the user to explicitly say which step should use Spark. Maybe something like this, but I'm not sure yet:

```python
from datasets.parallel import parallel_backend

with parallel_backend("spark", steps=["download"]):
    ds = load_dataset(...)
```
for now some steps can be NotImplemented:
```python
from datasets.parallel import parallel_backend

with parallel_backend("spark", steps=["download", "prepare"]):
# NotImplementedError: the "prepare" step that converts the raw data files to Arrow is not compatible with the "spark" backend yet
```

This way we can progressively roll out Spark support for the other data loading/processing steps without breaking changes between `datasets` versions
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Sounds good! I like the partial rollout idea.
So for example `map_nested` would call `parallel_map` under the hood if `num_proc != 1` or `parallel_backend` is specified right?
I would be happy to start a PR next week to explore this path.
Issue: Support parallelized downloading and processing in load_dataset with Spark

Comment:
Awesome ! I think map_nested can call `parallel_map()` if num_proc > 1, and `parallel_map` can be responsible to use Pool.map by default or joblib.
Issue: load_dataset is case sentitive?

Comment:
Hi @haonan-li , thank you for the report! It seems to be a bug on the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) site, there is even no such dataset as `mbzuai/bactrian-x` on the Hub.  I opened and [issue](https://github.com/huggingface/huggingface_hub/issues/1453) there.
Issue: load_dataset is case sentitive?

Comment:
I think `load_dataset("mbzuai/bactrian-x")` shouldn't be loaded at all and raise an error but because of [this fallback](https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L1194) to packaged loaders when no other options are applicable, it loads the dataset with standard `json` loader instead of the custom loading script.
Issue: TIFF/TIF support 

Comment:
The issue with multichannel TIFF images has already been reported in Pillow (https://github.com/python-pillow/Pillow/issues/1888). We can't do much about it on our side.

Still, to avoid the error, you can bypass the default Pillow decoding and define a custom one as follows:
```python
import tifffile # pip install tifffile

dset = dset.cast_column("image", datasets.Image(decode=False))

def decode_mutlichannel_tiff(batch):
    batch["image"] = [tifffile.imread(image["path"]) for image in batch["image"]]
    return batch

dset.set_transform(decode_mutlichannel_tiff)
```

Regarding the annotations, in which format are they? In the COCO format? I think this is a bit too specific to have a built-in loader for it.
Issue: Multiprocessing in a `filter` or `map` function with a Pytorch model

Comment:
Hi ! PyTorch may hang when calling `load_state_dict()` in a subprocess. To fix that, set the multiprocessing start method to "spawn". Since `datasets` uses `multiprocess`, you should do:

```python
# Required to avoid issues with pytorch (otherwise hangs during load_state_dict in multiprocessing)
import multiprocess.context as ctx
ctx._force_start_method('spawn')
```

Also make sure to run your main code in `if __name__ == "__main__":` to avoid issues with python multiprocesing
Issue: Multiprocessing in a `filter` or `map` function with a Pytorch model

Comment:
@lhoestq Hello, I also encountered this problem but maybe with another reason. Here is my code:
```python
tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir, model_max_length=training_args.model_max_length)
data = load_dataset("json", data_files=data_args.train_file, cache_dir=data_args.data_cache_dir)
def func(samples):
    # main operation
    for sentence_value in samples:
        sentence_ids = tokenizer.encode(sentence_value, add_special_tokens=False, max_length=tokenizer.model_max_length, truncation=True)
    ... ...
train_data = data["train"].shuffle().map(func, num_proc=os.cpu_count())
```
It hangs after the progress reaches 100%. Could you help me point out the reason?
Issue: Multiprocessing in a `filter` or `map` function with a Pytorch model

Comment:
@SkyAndCloud your issue doesn't seem related to the original post - could you open a new issue and provide more details ? (size of the dataset, number of cpus, how much time it took to run, `datasets` version)
Issue: Multiprocessing in a `filter` or `map` function with a Pytorch model

Comment:
@lhoestq Hi, I just solved this problem. Because the input is extremely long and the tokenizer requests a large amount of memory, which leads to a OOM error and may eventually causes the hang. I didn't filter those too-long sentences because I thought `tokenizer` would stop once the length exceeds the `max_length`. However, it actually firstly complete the tokenization of entire sentence and then truncate it.
Issue: Offset overflow while doing regex on a text column

Comment:
Hi! This looks like an Arrow bug, but it can be avoided by reducing the `writer_batch_size`.

(`ds = ds.map(get_text_caption, writer_batch_size=100)` in Colab runs without issues)
Issue: Support for various audio-loading backends instead of always relying on SoundFile

Comment:
Hi! 

You can use `set_transform`/`with_transform` to define a custom decoding for audio formats not supported by `soundfile`:
```python
audio_dataset_amr = Dataset.from_dict({"audio": ["audio_samples/audio.amr"]})

def decode_audio(batch):
    batch["audio"] = [read_ffmpeg(audio_path) for audio_path in batch["audio"]]
    return batch

audio_dataset_amr.set_transform(decode_amr) 
```

Supporting multiple backends is more work to maintain, but we could consider this if we get more requests such as this one.
Issue: Support for various audio-loading backends instead of always relying on SoundFile

Comment:
Considering the number of times a custom decoding transform has been suggested as a solution, an example in the [docs](https://huggingface.co/docs/datasets/process#format-transform) would be nice.

cc @stevhliu
Issue: Error using `load_datasets`

Comment:
It looks like an issue with your installation of scipy, can you try reinstalling it ?
Issue: Schrödinger's dataset_dict

Comment:
Hi ! Passing `data_files="path/test.json"` is equivalent to `data_files={"train": ["path/test.json"]}`, that's why you end up with a train split. If you don't pass `data_files=`, then split names are inferred from the data files names
Issue: datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory

Comment:
Note:
I listed the datasets and grepped around to find what appears to be an alternative source for this:

raw_datasets = load_dataset("espejelomar/code_search_net_python_10000_examples", "python")