noamaanMulla-03/datasets-issues huggingface dataset:
Passing `sample_by` to `load_dataset` when loading text data does not work

### Describe the bug

I have a dataset that consists of a bunch of text files, each representing an example. There is an undocumented `sample_by` argument for the `TextConfig` class that is used by `Text` to decide whether to split files into lines, paragraphs or take them whole. Passing `sample_by=“document”` to `load_dataset` results in files getting split into lines regardless. I have edited `src/datasets/packaged_modules/text/text.py` for myself to switch the default and it works fine.

As a side note, the `if-else` for `sample_by` will silently load an empty dataset if someone makes a typo in the argument, which is not ideal.

### Steps to reproduce the bug

1. Prepare data as a bunch of files in a directory.
2. Load that data via `load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)`.
3. Inspect the resultant dataset — every item should have the form of `{“text”: <a line from a file>}`.

### Expected behavior

`load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)` should result in a dataset with items of the form `{“text”: <one document>}`.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0


Thanks for reporting! We are working on a fix.
Fix CI test_load_dataset_distributed_with_script

Fix #6796.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Finally:
- the initial issue seems it was temporary
- there is a different issue now 

```
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport_errors_on_wrong_sha - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_with_script - AssertionError: assert 'dataset_with_script' == 'parquet'
  
  - parquet
  + dataset_with_script
```
Generating split is very slow when Image format is PNG

### Describe the bug

When I create a dataset, it gets stuck while generating cached data. 
The image format is PNG, and it will not get stuck when the image format is jpeg.

![image](https://github.com/huggingface/datasets/assets/22740819/3b888fd8-e6d6-488f-b828-95a8f206a152)

After debugging, I know that it is because of the `pa.array` operation in [arrow_writer](https://github.com/huggingface/datasets/blob/2.13.0/src/datasets/arrow_writer.py#L553), but i don't why.

### Steps to reproduce the bug

```
from datasets import Dataset

def generator(lines):
    for line in lines:
        img = Image.open(open(line["url"], "rb"))
        # print(img.format)  # "PNG"
        yield {
            "image": img,
        }

lines = open(dataset_path, "r")
dataset = Dataset.from_generator(
    generator,
    gen_kwargs={"lines": lines}
)
```

### Expected behavior

Generating split done.

### Environment info

datasets    2.13.0


I think this is due to the speed of reading a `png` image using pillow compared to a `jpg` image.
Notably the same is true with `tiff`, it is even faster than `jpg` in my case.
A Question About the Map Function

### Describe the bug

Hello,

I have a question regarding the map function in the Hugging Face datasets.

The situation is as follows: when I load a jsonl file using load_dataset(..., streaming=False), and then utilize the map function to process it, I specify that the returned example should be of type Torch.tensor. However, I noticed that after applying the map function, the datatype automatically changes to List, which leads to errors in my program.

I attempted to use load_dataset(..., streaming=True), and this issue no longer occurs. I'm not entirely clear on why this happens. Could you please provide some insights into this?

### Steps to reproduce the bug

1.dataset =  load_dataset(xxx, streaming = False)
2. dataset.map(function), function will return torch.Tensor.
3. you will find the format of data in dataset is List.

### Expected behavior

I expected to receieve the format of data is torch.Tensor.

### Environment info

2.18.0


All data is saved in the arrow format on disk.
If you return a tensor it gets converted to arrow before saving to disk when using map.

To get a tensor when you access data elements you can use `dataset.set_format("pt")`.
Note that this just changes how the data is loaded, not how it is stored.

> All data is saved in the arrow format on disk. If you return a tensor it gets converted to arrow before saving to disk when using map.
> 
> To get a tensor when you access data elements you can use `dataset.set_format("pt")`. Note that this just changes how the data is loaded, not how it is stored.

Thank you very much for your explanation, I understand what you mean now. So you're saying that when streaming=True, there's no need to convert it to the arrow format and save it to disk. But if we directly load all formats and then convert them into the arrow format after passing through the map function, it will convert torch.Tensor into a List. I see.
`DatasetBuilder._split_generators` incomplete type annotation

### Describe the bug

The [`DatasetBuilder._split_generators`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/builder.py#L1449) function has currently the following signature:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: DownloadManager):
        ...
```

However, the `dl_manager` argument can also be of type [`StreamingDownloadManager`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/download/streaming_download_manager.py#L962), which has different functionality. For example, the `download` function doesn't download, but rather just returns the given url(s). 

I suggest changing the function signature to:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: Union[DownloadManager, StreamingDownloadManager]):
        ...
```

and also adjust the docstring accordingly.

I would like to create a Pull Request to fix this, and have the following questions:
* Are there also other options than `DownloadManager`, and `StreamingDownloadManager`?
* Should this also be changed in other functions?

### Steps to reproduce the bug

Minimal example to print the different class names:

```python
import tempfile
from datasets import load_dataset

example = b'''
from datasets import GeneratorBasedBuilder, DatasetInfo, Features, Value, SplitGenerator

class Test(GeneratorBasedBuilder):
    def _info(self):
        return DatasetInfo(features=Features({"x": Value("int64")}))
    def _split_generators(self, dl_manager):
        print(type(dl_manager))
        return [SplitGenerator('test')]
    def _generate_examples(self):
        yield 0, {'x': 42}
'''

with tempfile.NamedTemporaryFile(suffix='.py') as f:
    f.write(example)
    f.flush()
    load_dataset(f.name, streaming=False)
    load_dataset(f.name, streaming=True)
```

### Expected behavior

complete type annotations

### Environment info

/


Good catch! Feel free to open a PR with the suggested fix :).

There is also the [`MockDownloadManager`](https://github.com/JonasLoos/datasets/blob/main/src/datasets/download/mock_download_manager.py#L33), which seems like it might get passed here too. However, to me, it doesn't really seem relevant to the users of the datasets library, so I would just ignore it. What do you think, @mariosasko?

The API (`dummy_data` CLI command ) that uses the `MockDownloadManager` has been deprecated, so ignoring it sounds good!
`add_faiss_index` raises ValueError: not enough values to unpack (expected 2, got 1)

### Describe the bug

Calling `add_faiss_index` on a `Dataset` with a column argument raises a ValueError. The following is the trace

```python
    214 def replacement_add(self, x):
    215     """Adds vectors to the index.
    216     The index must be trained before vectors can be added to it.
    217     The vectors are implicitly numbered in sequence. When `n` vectors are
   (...)
    224         `dtype` must be float32.
    225     """
--> 227     n, d = x.shape
    228     assert d == self.d
    229     x = np.ascontiguousarray(x, dtype='float32')

ValueError: not enough values to unpack (expected 2, got 1)
```

### Steps to reproduce the bug

1. Load any dataset like `ds = datasets.load_dataset("wikimedia/wikipedia", "20231101.en")["train"]`
2. Add an FAISS index on any column `ds.add_faiss_index('title')`

### Expected behavior

The index should be created

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.9.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
- `faiss-cpu` version: 1.8.0


I realized I was passing a string column to this instead of float. Is it possible to add a warning or error to prevent users from falsely believing there's a bug?

Hello!

I agree that we could add some safeguards around the type of `ds[column]`. At least for FAISS, we need the column to be made of embeddings as FAISS doesn't perform the embeddings itself.

I can propose a PR sometime this week.

@Dref360 thanks for the initiative!
CI is broken due to hf-internal-testing/dataset_with_script

CI is broken for test_load_dataset_distributed_with_script. See: https://github.com/huggingface/datasets/actions/runs/8614926216/job/23609378127
```
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[None] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0c741de3b0>)
FAILED tests/test_load.py::test_load_dataset_distributed_with_script[force_redownload] - assert False
 +  where False = all(<generator object test_load_dataset_distributed_with_script.<locals>.<genexpr> at 0x7f0be45f6ea0>)
```

Labels: name: bug

Finally:
- the initial issue seems it was temporary
- there is a different issue now: https://github.com/huggingface/datasets/actions/runs/8627153993/job/23646584590?pr=6797
```
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::ModuleFactoryTest::test_HubDatasetModuleFactoryWithParquetExport_errors_on_wrong_sha - datasets.utils._dataset_viewer.DatasetViewerError: No exported Parquet files available.
FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_with_script - AssertionError: assert 'dataset_with_script' == 'parquet'
  
  - parquet
  + dataset_with_script
```

Maybe related to `hf-internal-testing/dataset_with_script` dataset: https://huggingface.co/datasets/hf-internal-testing/dataset_with_script

This URL: https://datasets-server.huggingface.co/parquet?dataset=hf-internal-testing/dataset_with_script
raises:
> {"error":"The dataset viewer doesn't support this dataset because it runs arbitrary python code. Please open a discussion in the discussion tab if you think this is an error and tag @lhoestq and @severo."}

Was there a recent change on the Hub enforcing this behavior?

OK, I just saw this PR:
- https://github.com/huggingface/dataset-viewer/pull/2689

Once merged and deployed, it should fix the issue.

Once the script-dataset has been allowed in the dataset-viewer, we should fix our test to make the CI pass.

I am addressing this.
got fileNotFound

### Describe the bug

When I use load_dataset to load the nyanko7/danbooru2023 data set, the cache is read in the form of a symlink. There may be a problem with the arrow_dataset initialization process and I get FileNotFoundError: [Errno 2] No such file or directory: '2945000.jpg'

### Steps to reproduce the bug

#code show as below
from datasets import load_dataset
data = load_dataset("nyanko7/danbooru2023",cache_dir=<symlink>)
data["train"][0]

### Expected behavior

I should get this result:
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=365x256 at 0x7FB730CB4070>, 'label': 0}

### Environment info

datasets==2.12.0
python==3.10.14



Hi! I'll open a PR on the Hub to fix this, but please use the Hub's [Community tab](https://huggingface.co/datasets/nyanko7/danbooru2023/discussions) to report such issues in the future.

I've opened a [PR](https://huggingface.co/datasets/nyanko7/danbooru2023/discussions/8) in the repo, so let's continue the discussion there
Unexpected behavior when using load_dataset with streaming=True in a for loop

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04


This is normal behavior in python when using `lambda`: the `i` defined in your `lambda` refers to the global variable `i` in your loop, and `i` equals to `1` when you run your `for e in res[0]` line.

You should pass `fn_kwargs` that will be passed to your `lambda` instead of using the global variable:

```python
from datasets import load_dataset

res=[]
for i in [0,1]:
    di = load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
    ).map(lambda x, source: {"source": source}, fn_kwargs={"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

This doesn't happen in non-streaming since in that case `map` is executed while the variable `i` has the right value. In streaming mode, `map` is executed on-the-fly when you iterate on the dataset.

Thank you very much for your answer. I think this issue can be closed now.
Improve typing of Dataset.search, matching definition

Previously, the output of `score, indices = Dataset.search(...)` would be numpy arrays.

The definition in `SearchResult` is a `List[int]` so this PR now matched the expected type.

The previous behavior is a bit annoying as `Dataset.__getitem__` doesn't support `numpy.int64` which forced me to convert `indices` to int eg:

```python
score, indices = ds.search(...)
item = ds[int(indices[0])]
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6816). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi! This is a breaking change. A better solution is to check for "indexable" types in `__getitem__` to support keys such as `np.int64`:
```python
import operator

def _query_table_with_indices_mapping(...):  # or _query_table
    ...
    try:
        operator.index(key)
    except TypeError:
        pass
 
    _raise_bad_key_type(key)
```

Sounds good! We should still update type annotations for SearchResult in my opinion.
Make convert_to_parquet CLI command create script branch

As proposed by @severo, maybe we should add this functionality as well to the CLI command to convert a script-dataset to Parquet. See: https://github.com/huggingface/datasets/pull/6795#discussion_r1562819168
> When providing support, we sometimes suggest that users store their script in a script branch. What do you think of this alternative to deleting the files?

Labels: name: enhancement
Give more details in `DataFilesNotFoundError` when getting the config names

### Feature request

After https://huggingface.co/datasets/cis-lmu/Glot500/commit/39060e01272ff228cc0ce1d31ae53789cacae8c3, the dataset viewer gives the following error:

```
{
  "error": "Cannot get the config names for the dataset.",
  "cause_exception": "DataFilesNotFoundError",
  "cause_message": "No (supported) data files found in cis-lmu/Glot500",
  "cause_traceback": [
    "Traceback (most recent call last):\n",
    " File \"/src/services/worker/src/worker/job_runners/dataset/config_names.py\", line 73, in compute_config_names_response\n config_names = get_dataset_config_names(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py\", line 347, in get_dataset_config_names\n dataset_module = dataset_module_factory(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1873, in dataset_module_factory\n raise e1 from None\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1854, in dataset_module_factory\n return HubDatasetModuleFactoryWithoutScript(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 1245, in get_module\n module_name, default_builder_kwargs = infer_module_for_data_files(\n",
    " File \"/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py\", line 595, in infer_module_for_data_files\n raise DataFilesNotFoundError(\"No (supported) data files found\" + (f\" in {path}\" if path else \"\"))\n",
    "datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in cis-lmu/Glot500\n"
  ]
}
```

because the deleted files were still listed in the README, see https://huggingface.co/datasets/cis-lmu/Glot500/discussions/4

Ideally, the error message would include the name of the first configuration with missing files, to help the user understand how to fix it. Here, it would tell that configuration `aze_Ethi` has no supported data files, instead of telling that the `cis-lmu/Glot500` *dataset* has no supported data files (which is not true).

### Motivation

Giving more detail in the error would help the Datasets Hub users to debug why the dataset viewer does not work.

### Your contribution

Not sure how to best fix this, as there are a lot of loops on the dataset configs in the traceback methods. "maybe" it would be easier to handle if the code was completely isolating each config.

Labels: name: enhancement
`map` with `num_proc` > 1 leads to OOM

### Describe the bug

When running `map` on parquet dataset loaded from local machine, the RAM usage increases linearly eventually leading to OOM. I was wondering if I should I save the `cache_file` after every n steps in order to prevent this?

### Steps to reproduce the bug



```
ds = load_dataset("parquet", data_files=dataset_path, split="train")
ds = ds.shard(num_shards=4, index=0)

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
ds = ds.map(prepare_dataset,
            num_proc=32,
            writer_batch_size=1000,
            keep_in_memory=False,
            desc="preprocess dataset")
```

```
def prepare_dataset(batch):
    # load audio
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=16000)
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(sample["array"].squeeze())

    return batch
```

### Expected behavior

It shouldn't run into OOM problem.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0


Hi ! You can try to reduce `writer_batch_size`. It corresponds to the number of samples that stay in RAM before being flushed to disk
Loading a remote dataset fails in the last release (v2.19.0)

While loading a dataset with multiple splits I get an error saying `Couldn't find file at <URL>`

I am loading the dataset like so, nothing out of the ordinary.
This dataset needs a token to access it.
```
token="hf_myhftoken-sdhbdsjgkhbd"
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token=token)
```

I get the following error
![Screenshot 2024-04-19 at 11 03 07 PM](https://github.com/huggingface/datasets/assets/35369637/8dce757f-08ff-45dd-85b5-890fced7c5bc)

Now you can see that the URL that it is trying to reach has the JSON object of the dataset split appended to the base URL. I think this may be due to a newly introduced issue.

I did not have this issue with the previous version of the datasets. Everything was fine for me yesterday and after the release 12 hours ago, this seems to have broken. Also, the dataset in question runs custom code and I checked and there have been no commits to the dataset on Huggingface in 6 months.

### Steps to reproduce the bug

Since this happened with one particular dataset for me, I am listing steps to use that dataset.
1. Open https://huggingface.co/datasets/speechcolab/gigaspeech and fill the form to get access.
2. Create a token on your huggingface account with read access.
3. Run the following line, substituing `<your_token_here>` with your token.
```
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token="<your_token_here>")
```

### Expected behavior

Be able to load the dataset in question.

### Environment info

datasets == 2.19.0
python == 3.10
kernel == Linux 6.1.58+