kardwalker/my_dataset huggingface dataset:
GitHub issue #5715: Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List
State: closed | Created: 2023-04-06T13:57:48Z

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks
GitHub issue #5714: Fix xnumpy_load for .npz files
State: closed | Created: 2023-04-06T13:01:45Z

PR:
- #5626 

implemented support for streaming `.npy` files by using `numpy.load`.

However, it introduced a bug when used with `.npz` files, within a context manager:
```
ValueError: seek of closed file
```
or in streaming mode:
```
ValueError: I/O operation on closed file.
```

This PR fixes the bug and tests for both `.npy` and `.npz` files.

Fix #5711.
GitHub issue #5713: ArrowNotImplementedError when loading dataset from the hub
State: closed | Created: 2023-04-06T10:27:22Z

### Describe the bug

Hello,

I have created a dataset by using the image loader. Once the dataset is created I try to download it and I get the error:
```
Traceback (most recent call last):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1860, in _prepare_split_single
    for _, table in generator:
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 69, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/load.py", line 1791, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 891, in download_and_prepare
    self._download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 986, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1748, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1893, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

Create the dataset and push it to the hub:
```python
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset.push_to_hub("org/dataset-name", private=True, max_shard_size="1GB")
```

Then use it:
```python
from datasets import load_dataset

dataset = load_dataset("org/dataset-name")
```

### Expected behavior

To properly download and use the pushed dataset.

Something else to note is that I specified to have shards of 1GB max, but at the end, for the train set, it is an almost 7GB single file that is pushed.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue #5710: OSError: Memory mapping file failed: Cannot allocate memory
State: closed | Created: 2023-04-05T14:11:26Z

### Describe the bug

Hello, I have a series of datasets each of 5 GB, 600 datasets in total. So together this makes 3TB. 

When I trying to load all the 600 datasets into memory, I get the above error message. 

Is this normal because I'm hitting the max size of memory mapping of the OS? 


Thank you

```terminal
0_21/cache-e9c42499f65b1881.arrow
load_hf_datasets_from_disk:  82%|████████████████████████████████████████████████████████████████████████████████████████████████████▍                     | 494/600 [07:26<01:35,  1.11it/s]
Traceback (most recent call last):
  File "example_load_genkalm_dataset.py", line 35, in <module>
    multi_ds.post_process(max_node_num=args.max_node_num,max_seq_length=args.max_seq_length,delay=args.delay)
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 142, in post_process
    genkalm_dataset = GenKaLM_Dataset.from_hf_dataset(path_or_name=ds_path, max_seq_length=self.max_seq_length,
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 47, in from_hf_dataset
    hf_ds = load_from_disk(path_or_name)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/load.py", line 1848, in load_from_disk
    return Dataset.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1549, in load_from_disk
    arrow_table = concat_tables(
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1805, in concat_tables
    tables = list(tables)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1550, in <genexpr>
    table_cls.from_file(Path(dataset_path, data_file["filename"]).as_posix())
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1065, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 50, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 950, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 911, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```








### Steps to reproduce the bug

Sorry I can not provide a reproducible code as the data is stored on my server and it's too large to share.

### Expected behavior

I expect the 3TB of data can be fully mapped to memory

### Environment info


- `datasets` version: 2.9.0
- Platform: Linux-4.15.0-204-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyArrow version: 11.0.0
- Pandas version: 1.0.1
GitHub issue #5708: Dataset sizes are in MiB instead of MB in dataset cards
State: closed | Created: 2023-04-05T06:36:03Z

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations
GitHub issue #5706: Support categorical data types for Parquet
State: closed | Created: 2023-04-04T09:45:35Z

### Feature request

Huggingface datasets does not seem to support categorical / dictionary data types for Parquet as of now. There seems to be a `TODO` in the code for this feature but no implementation yet. Below you can find sample code to reproduce the error that is currently thrown when attempting to read a Parquet file with categorical columns:

```python
import pandas as pd
import pyarrow.parquet as pq

from datasets import load_dataset

# Create categorical sample DataFrame
df = pd.DataFrame({'type': ['foo', 'bar']}).astype('category')
df.to_parquet('data.parquet')

# Read back as pyarrow table
table = pq.read_table('data.parquet')
print(table.schema)
# type: dictionary<values=string, indices=int32, ordered=0>

# Load with huggingface datasets
load_dataset('parquet', data_files='data.parquet')
```

Error:
```
Traceback (most recent call last):                                                                                                                                                                                                        
  File ".venv/lib/python3.10/site-packages/datasets/builder.py", line 1875, in _prepare_split_single
    writer.write_table(table)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 566, in write_table
    self._build_writer(inferred_schema=pa_table.schema)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 379, in _build_writer
    inferred_features = Features.from_arrow_schema(inferred_schema)                                                                                     
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1361, in generate_from_arrow_type
    raise NotImplementedError  # TODO(thom) this will need access to the dictionary as well (for labels). I.e. to the py_table
NotImplementedError
```

### Motivation

Categorical data types, as offered by Pandas and implemented with the `DictionaryType` dtype in `pyarrow` can significantly reduce dataset size and are a handy way to turn textual features into numerical representations and back. Lack of support in Huggingface datasets greatly reduces compatibility with a common Pandas / Parquet feature.

### Your contribution

I could provide a PR. However, it would be nice to have an initial complexity estimate from one of the core developers first.
GitHub issue #5705: Getting next item from IterableDataset took forever.
State: closed | Created: 2023-04-04T09:16:17Z

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12
GitHub issue #5704: 5537 speedup load
State: open | Created: 2023-04-04T08:58:14Z

I reimplemented fsspec.spec.glob() in `hffilesystem.py` as `_glob`,  used it in `_resolve_single_pattern_in_dataset_repository` only, and saw a 20% speedup in times to load the config, on average. 

That's not much when usually this step takes only 2-3 seconds for most datasets, but in this particular case,  `bigcode/the-stack-dedup` , the loading time to get the config (not download the entire 6tb dataset, of course), went from ~170 secs to ~20 secs. 

What makes this work is this code in `_glob`:

```  
      if self.dir_cache is not None:
            allpaths = self.dir_cache
        else:
            allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
```

I also had to `import glob.has_magic( )` for `_glob()` (confusing, I know).

I hope there is no issue with copying most of the code from `fsspec.spec.glob`, as it is a BSD 3-Clause License, 
and I left a comment about this in the docstring of` _glob()`, that we may want to delete.

As mentioned, I evaluated the speedup across a random selection of about 1000 datasets (not all 27k+), and verified that old_config.eq(new_method_config) with the build in method, but deleted this test and related code changes on the subsequent commit. It's in the commit history if anyone wants to see it. (Note this does not include the outlier of  `bigcode/the-stack-dedup`

|   |   old_time     |    new _time |     diff   |  pct_diff |
| -- | -- | -- | -- | -- |
| mean  |    3.340 |    2.642 |    0.698 |   18.404 |
| min   |    2.024   |  1.976   |  -0.840  | -37.634 |
| max   |  66.582  |   41.517 |   30.927  |  85.538 |
GitHub issue #5702: Is it possible or how to define a `datasets.Sequence` that could potentially be either a dict, a str, or None?
State: closed | Created: 2023-04-04T03:20:43Z

### Feature request

Hello! Apologies if my question sounds naive:

I was wondering if it’s possible, or how one would go about defining a 'datasets.Sequence' element in datasets.Features that could potentially be either a dict, a str, or None?

Specifically, I’d like to define a feature for a list that contains 18 elements, each of which has been pre-defined as either a `dict or None` or `str or None` - as demonstrated in the slightly misaligned data provided below:

```json
[
  [
    {"text":"老妇人","idxes":[0,1,2]},null,{"text":"跪","idxes":[3]},null,null,null,null,{"text":"在那坑里","idxes":[4,5,6,7]},null,null,null,null,null,null,null,null,null,null],
  [
    {"text":"那些水","idxes":[13,14,15]},null,{"text":"舀","idxes":[11]},null,null,null,null,null,{"text":"在那坑里","idxes":[4,5,6,7]},null,{"text":"出","idxes":[12]},null,null,null,null,null,null,null],
  [
    {"text":"水","idxes":[38]},
    null,
    {"text":"舀","idxes":[40]},
    "假",  // note this is just a standalone string
    null,null,null,{"text":"坑里","idxes":[35,36]},null,null,null,null,null,null,null,null,null,null]]
```

### Motivation

I'm currently working with a dataset of the following structure and I couldn't find a solution in the [documentation](https://huggingface.co/docs/datasets/v2.11.0/en/package_reference/main_classes#datasets.Features).

```json
{"qid":"3-train-1058","context":"桑桑害怕了。从玉米地里走到田埂上，他遥望着他家那幢草房子里的灯光，知道母亲没有让他回家的意思，很伤感，有点想哭。但没哭，转身朝阿恕家走去。","corefs":[[{"text":"桑桑","idxes":[0,1]},{"text":"他","idxes":[17]}]],"non_corefs":[],"outputs":[[{"text":"他","idxes":[17]},null,{"text":"走","idxes":[11]},null,null,null,null,null,{"text":"从玉米地里","idxes":[6,7,8,9,10]},{"text":"到田埂上","idxes":[12,13,14,15]},null,null,null,null,null,null,null,null],[{"text":"他","idxes":[17]},null,{"text":"走","idxes":[66]},null,null,null,null,null,null,null,{"text":"转身朝阿恕家去","idxes":[60,61,62,63,64,65,67]},null,null,null,null,null,null,null],[{"text":"灯光","idxes":[30,31]},null,null,null,null,null,null,{"text":"草房子里","idxes":[25,26,27,28]},null,null,null,null,null,null,null,null,null,null],[{"text":"他","idxes":[17]},{"text":"他家那幢草房子","idxes":[21,22,23,24,25,26,27]},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"远"],[{"text":"他","idxes":[17]},{"text":"阿恕家","idxes":[63,64,65]},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"变近"]]}
```

### Your contribution

I'm going to provide the dataset at https://huggingface.co/datasets/2030NLP/SpaCE2022 .
GitHub issue #5701: Add Dataset.from_spark
State: closed | Created: 2023-04-03T23:51:29Z

Adds static method Dataset.from_spark to create datasets from Spark DataFrames.

This approach alleviates users of the need to materialize their dataframe---a common use case is that the user loads their dataset into a dataframe, uses Spark to apply some transformation to some of the columns, and then wants to train on the dataset.

Related issue: https://github.com/huggingface/datasets/issues/5678
GitHub issue #5698: Add Qdrant as another search index
State: open | Created: 2023-04-03T14:25:19Z

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.
GitHub issue #5696: Shuffle a sharded iterable dataset without seed can lead to duplicate data
State: closed | Created: 2023-04-03T09:40:03Z

As reported in https://github.com/huggingface/datasets/issues/5360

If `seed=None` in `.shuffle()`, shuffled datasets don't use the same shuffling seed across nodes.

Because of that, the lists of shards is not shuffled the same way across nodes, and therefore some shards may be assigned to multiple nodes instead of exactly one.

This can happen only when you have a number of shards that is a factor of the number of nodes.

The current workaround is to always set a `seed` in `.shuffle()`
GitHub issue #5694: Dataset configuration
State: open | Created: 2023-04-01T13:08:05Z

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md
GitHub issue #5691: [docs] Compress data files
State: closed | Created: 2023-03-31T17:17:26Z

This PR addresses the comments in #5687 about compressing text file extensions before uploading to the Hub. Also clarified what "too large" means based on the GitLFS [docs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage).
GitHub issue #5689: Support streaming Beam datasets from HF GCS preprocessed data
State: closed | Created: 2023-03-31T08:44:24Z

This PR implements streaming Apache Beam datasets that are already preprocessed by us and stored in the HF Google Cloud Storage:
- natural_questions
- wiki40b
- wikipedia

This is done by streaming from the prepared Arrow files in HF Google Cloud Storage.

This will fix their corresponding dataset viewers. Related to:
- https://github.com/huggingface/datasets-server/pull/988#discussion_r1150767138

Related to:
- https://huggingface.co/datasets/natural_questions/discussions/4
- https://huggingface.co/datasets/wiki40b/discussions/2
- https://huggingface.co/datasets/wikipedia/discussions/9

CC: @severo
GitHub issue #5690: raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api
State: closed | Created: 2023-03-31T08:22:22Z

### Describe the bug

rta.sh
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api


### Reproduction

_No response_

### Logs

```shell
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```


### System info

```shell
- huggingface_hub version: 0.13.2
- Platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/appuser/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.7.1
- Jinja2: N/A
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.3.0
- hf_transfer: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/appuser/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/appuser/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/appuser/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
```
GitHub issue #5688: Wikipedia download_and_prepare for GCS
State: closed | Created: 2023-03-30T23:43:22Z

### Describe the bug

I am unable to download the wikipedia dataset onto GCS.

When I run the script provided the memory firstly gets eaten up, then it crashes. 
I tried running this on a VM with 128GB RAM and all I got was a two empty files: _data_builder.lock_, _data.incomplete/beam-temp-wikipedia-train-1ab2039acf3611ed87a9893475de0093_

I have troubleshot this for two straight days now, but I am just unable to get the dataset into storage.

### Steps to reproduce the bug

Run this and insert a path:
```
import datasets

builder = datasets.load_dataset_builder(
    "wikipedia", language="en", date="20230320", beam_runner="DirectRunner")

builder.download_and_prepare({path}, file_format="parquet")
```
This is where the problem of it eating RAM occurs.



I have also tried several versions of this, based on the docs:
```
import gcsfs
import datasets

storage_options = {"project": "tdt4310", "token": "cloud"}
fs = gcsfs.GCSFileSystem(**storage_options)

output_dir = "gcs://wikipediadata/"
builder = datasets.load_dataset_builder(
    "wikipedia", date="20230320", language="en",  beam_runner="DirectRunner")
builder.download_and_prepare(
    output_dir, storage_options=storage_options, file_format="parquet")
```
The error message that is received here is:

> ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://wikipediadata/wikipedia-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']

I have ran `pip install apache-beam[gcp]`

### Expected behavior

The wikipedia data loaded into GCS

Everything worked when testing with a smaller demo dataset found somewhere in the docs

### Environment info

Newest published version of datasets. Python 3.9. Also tested with Python 3.7. 128GB RAM Google Cloud VM instance.
GitHub issue #5687: Document to compress data files before uploading
State: closed | Created: 2023-03-30T06:41:07Z

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1
GitHub issue #5681: Add information about patterns search order to the doc about structuring repo
State: closed | Created: 2023-03-29T11:44:49Z

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....
GitHub issue #5680: Fix a description error for interleave_datasets.
State: closed | Created: 2023-03-29T09:50:23Z

There is a description mistake in the annotation of interleave_dataset with "all_exhausted" stopping_strategy.
``` python
d1 = Dataset.from_dict({"a": [0, 1, 2]})
d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
d3 = Dataset.from_dict({"a": [20, 21, 22, 23, 24]})
dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
```
According to the interleave way, the correct output of `dataset["a"]` is `[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 10, 24]`, not `[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 0, 24]`
GitHub issue #5679: Allow load_dataset to take a working dir for intermediate data
State: open | Created: 2023-03-29T07:21:09Z

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.
GitHub issue #5678: Add support to create a Dataset from spark dataframe
State: closed | Created: 2023-03-29T04:36:28Z

### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.
GitHub issue #5677: Dataset.map() crashes when any column contains more than 1000 empty dictionaries
State: closed | Created: 2023-03-29T00:01:31Z

### Describe the bug

`Dataset.map()` crashes any time any column contains more than `writer_batch_size` (default 1000) empty dictionaries, regardless of whether the column is being operated on. The error does not occur if the dictionaries are non-empty.

### Steps to reproduce the bug

Example:

```
import datasets

def add_one(example):
    example["col2"] += 1
    return example

n = 1001  # crashes
# n = 999 # works
ds = datasets.Dataset.from_dict({"col1": [{}] * n, "col2": [1] * n})

ds = ds.map(add_one, writer_batch_size=1000)
```

### Expected behavior

Above code should not crash

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.10
- Python version: 3.8.15
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue #5675: Filter datasets by language code
State: closed | Created: 2023-03-27T09:42:28Z

Hi! I use the language search field on https://huggingface.co/datasets
However, some of the datasets tagged by ISO language code are not accessible by this search form.
For example, [myv_ru_2022](https://huggingface.co/datasets/slone/myv_ru_2022) is has `myv` language tag but it is not included in Languages search form.
I've also noticed the same problem with `mhr` (see https://huggingface.co/datasets/AigizK/mari-russian-parallel-corpora)
GitHub issue #5673: Pass down storage options
State: closed | Created: 2023-03-26T20:09:37Z

Remove implementation-specific kwargs from `file_utils.fsspec_get` and `file_utils.fsspec_head`, instead allowing them to be passed down via `storage_options`. This fixes an issue where s3fs did not recognize a timeout arg as well as fixes an issue mentioned in https://github.com/huggingface/datasets/issues/5281 by allowing users to pass down `storage_options` all the way from `datasets.load_dataset` to support implementation-specific credentials

Supports something like the following to provide credentials explicitly instead of relying on boto's methods of locating them

```
load_dataset(..., data_files=["s3://..."], storage_options={"profile": "..."})
```
GitHub issue #5671: How to use `load_dataset('glue', 'cola')`
State: closed | Created: 2023-03-26T09:40:34Z

### Describe the bug

I'm new to use HuggingFace datasets but I cannot use `load_dataset('glue', 'cola')`.

- I was stacked by the following problem:

```python
from datasets import load_dataset
cola_dataset = load_dataset('glue', 'cola')

---------------------------------------------------------------------------
InvalidVersion                            Traceback (most recent call last)
File <timed exec>:1

(Omit because of long error message)

File /usr/local/lib/python3.8/site-packages/packaging/version.py:197, in Version.__init__(self, version)
    195 match = self._regex.search(version)
    196 if not match:
--> 197     raise InvalidVersion(f"Invalid version: '{version}'")
    199 # Store the parsed out pieces of the version
    200 self._version = _Version(
    201     epoch=int(match.group("epoch")) if match.group("epoch") else 0,
    202     release=tuple(int(i) for i in match.group("release").split(".")),
   (...)
    208     local=_parse_local_version(match.group("local")),
    209 )

InvalidVersion: Invalid version: '0.10.1,<0.11'
```

- You can check this full error message in my repository: [MLOps-Basics/week_0_project_setup/experimental_notebooks/data_exploration.ipynb](https://github.com/makinzm/MLOps-Basics/blob/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup/experimental_notebooks/data_exploration.ipynb)


### Steps to reproduce the bug

- This is my repository to reproduce: [MLOps-Basics/week_0_project_setup](https://github.com/makinzm/MLOps-Basics/tree/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup)

1. cd `/DockerImage` and command `docker build . -t week0`
2. cd `/` and command `docker-compose up`
3. Run `experimental_notebooks/data_exploration.ipynb`

----

Just to be sure, I wrote down Dockerfile and requirements.txt

- Dockerfile
```Dockerfile
FROM python:3.8

WORKDIR /root/working

RUN apt-get update && \
    apt-get install -y python3-dev python3-pip python3-venv && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .

RUN pip3 install --no-cache-dir jupyter notebook && pip install --no-cache-dir -r requirements.txt

CMD ["bash"]
```

- requirements.txt
```txt
pytorch-lightning==1.2.10
datasets==1.6.2
transformers==4.5.1
scikit-learn==0.24.2
```


### Expected behavior

There is no bug to implement `load_dataset('glue', 'cola')`

### Environment info

I already wrote it.
GitHub issue #5669: Almost identical datasets, huge performance difference
State: open | Created: 2023-03-23T18:20:20Z

### Describe the bug

I am struggling to understand (huge) performance difference between two datasets that are almost identical. 

### Steps to reproduce the bug

# Fast (normal) dataset speed:
```python
import cv2
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("beans", split="train")

for x in DataLoader(dataset.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass
```

The above pass over the dataset takes about 1.5 seconds on my computer. 
However, if I re-create (almost) the same dataset, the sweep takes HUGE amount of time: 15 minutes. Steps to reproduce:

```python
def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"])
    return example

dataset2 = dataset.map(transform, remove_columns=["image"])

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass

```

### Expected behavior

Same timings

### Environment info

python==3.10.9
datasets==2.10.1
GitHub issue #5665: Feature request: IterableDataset.push_to_hub
State: open | Created: 2023-03-23T09:53:04Z

### Feature request

It'd be great to have a lazy push to hub, similar to the lazy loading we have with `IterableDataset`.

Suppose you'd like to filter [LAION](https://huggingface.co/datasets/laion/laion400m) based on certain conditions, but as LAION doesn't fit into your disk, you'd like to leverage streaming:
```
from datasets import load_dataset

dataset = load_dataset("laion/laion400m", streaming=True, split="train")
```
Then you could filter the dataset based on certain conditions:
```
filtered_dataset = dataset.filter(lambda example: example['HEIGHT'] > 400)
```

In order to persist this dataset and push it back to the hub, one currently needs to first load the entire filtered dataset on disk and then push:

```
from datasets import Dataset

Dataset.from_generator(filtered_dataset.__iter__).push_to_hub(...)
```
It would be great if we can instead lazy push to the data to the hub (basically stream the data to the hub), not being limited by our disk size:
```
filtered_dataset.push_to_hub("my-filtered-dataset")
```

### Motivation

This feature would be very useful for people that want to filter huge datasets without having to load the entire dataset or a filtered version thereof on their local disk.

### Your contribution

Happy to test out a PR :)
GitHub issue #5663: CI is broken: ModuleNotFoundError: jax requires jaxlib to be installed
State: closed | Created: 2023-03-23T09:39:43Z

CI test_py310 is broken: see https://github.com/huggingface/datasets/actions/runs/4498945505/jobs/7916194236?pr=5662
```
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_in_memory - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_jax_on_disk - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_audio - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_device - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_image - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/test_formatting.py::FormatterTest::test_jax_formatter_jnp_array_kwargs - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
FAILED tests/features/test_features.py::CastToPythonObjectsTest::test_cast_to_python_objects_jax - ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
===== 8 failed, 2147 passed, 10 skipped, 37 warnings in 228.69s (0:03:48) ======
```
GitHub issue #5662: Fix unnecessary dict comprehension
State: closed | Created: 2023-03-23T09:18:58Z

After ruff-0.0.258 release, the C416 rule was updated with unnecessary dict comprehensions. See:
- https://github.com/charliermarsh/ruff/releases/tag/v0.0.258
- https://github.com/charliermarsh/ruff/pull/3605

This PR fixes one unnecessary dict comprehension in our code: no need to unpack and re-pack the tuple values.

Fix #5661
GitHub issue #5660: integration with imbalanced-learn
State: closed | Created: 2023-03-22T11:05:17Z

### Feature request

Wouldn't it be great if the various class balancing operations from imbalanced-learn were available as part of datasets?

### Motivation

I'm trying to use imbalanced-learn to balance a dataset, but it's not clear how to get the two to interoperate - what would be great would be some examples.  I've looked online, asked gpt-4, but so far not making much progress.

### Your contribution

If I can get this working myself I can submit a PR with example code to go in the docs
GitHub issue #5656: Fix `fsspec.open` when using an HTTP proxy
State: closed | Created: 2023-03-21T15:23:29Z

Most HTTP(S) downloads from this library support proxy automatically by reading the `HTTP_PROXY` environment variable (et al.) because `requests` is widely used. However, in some parts of the code, `fsspec` is used, which in turn uses `aiohttp` for HTTP(S) requests (as opposed to `requests`), which in turn doesn't support reading proxy env variables by default. This PR enables reading them automatically.

Read [aiohttp docs on using proxies](https://docs.aiohttp.org/en/stable/client_advanced.html?highlight=trust_env#proxy-support).

For context, [the Python library requests](https://requests.readthedocs.io/en/latest/user/advanced/?highlight=http_proxy#proxies) and [the official Python library via `urllib.urlopen` support this automatically by default](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen). Many (most common ones?) programs also do the same, including cURL, APT, Wget, and many others.
GitHub issue #5655: Improve features decoding in to_iterable_dataset
State: closed | Created: 2023-03-21T14:18:09Z

Following discussion at https://github.com/huggingface/datasets/pull/5589

Right now `to_iterable_dataset` on images/audio hurts iterable dataset performance a lot (e.g. x4 slower because it encodes+decodes images/audios unnecessarily).

I fixed it by providing a generator that yields undecoded examples
GitHub issue #5653: Doc: save_to_disk, `num_proc` will affect `num_shards`, but it's not documented
State: closed | Created: 2023-03-21T05:25:35Z

### Describe the bug

[`num_proc`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_proc) will affect `num_shards`, but it's not documented


### Steps to reproduce the bug

Nothing to reproduce

### Expected behavior

[document of `num_shards`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_shards) explicitly says that it depends on `max_shard_size`, it should also mention `num_proc`.


### Environment info

datasets main document
GitHub issue #5652: Copy features
State: closed | Created: 2023-03-20T17:17:23Z

Some users (even internally at HF) are doing
```python
dset_features = dset.features
dset_features.pop(col_to_remove)
dset = dset.map(..., features=dset_features)
```

Right now this causes issues because it modifies the features dict in place before the map.

In this PR I modified `dset.features` to return a copy of the features, so that users can modify it if they want.
GitHub issue #5651: expanduser in save_to_disk
State: closed | Created: 2023-03-20T12:02:18Z

### Describe the bug

save_to_disk() does not expand `~`

1. `dataset = load_datasets("any dataset")`
2. `dataset.save_to_disk("~/data")`
3. a folder named "~" created in current folder
4. FileNotFoundError is raised, because the expanded path does not exist (`/home/<user>/data`)

related issue https://github.com/huggingface/transformers/issues/10628

### Steps to reproduce the bug

As described above.

### Expected behavior

expanduser correctly

### Environment info

- datasets 2.10.1
- python 3.10
GitHub issue #5650: load_dataset can't work correct with my image data 
State: closed | Created: 2023-03-18T13:59:13Z

I have about 20000 images in my folder which divided into 4 folders with class names.
When i use load_dataset("my_folder_name", split="train") this function create dataset in which there are only 4 images, the remaining 19000 images were not added there. What is the problem and did not understand. Tried converting images and the like but absolutely nothing worked
GitHub issue #5649: The index column created with .to_sql() is dependent on the batch_size when writing
State: closed | Created: 2023-03-18T05:25:17Z

### Describe the bug

It seems like the "index" column is designed to be unique? The values are only unique per batch. The SQL index is not a unique index.

This can be a problem, for instance, when building a faiss index on a dataset and then trying to match up ids with a sql export.

### Steps to reproduce the bug

```
from datasets import Dataset
import sqlite3
db = sqlite3.connect(":memory:")
nice_numbers = Dataset.from_dict({"nice_number": range(101,106)})
nice_numbers.to_sql("nice1", db, batch_size=1)
nice_numbers.to_sql("nice2", db, batch_size=2)
print(db.execute("select * from nice1").fetchall()) # [(0, 101), (0, 102), (0, 103), (0, 104), (0, 105)]
print(db.execute("select * from nice2").fetchall()) # [(0, 101), (1, 102), (0, 103), (1, 104), (0, 105)]
```

### Expected behavior

I expected the "index" column to be unique

### Environment info

```
 % datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2

zsh: segmentation fault  datasets-cli env
```
GitHub issue #5648: flatten_indices doesn't work with pandas format
State: open | Created: 2023-03-17T12:44:25Z

### Describe the bug

Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output

### Steps to reproduce the bug

tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1
GitHub issue #5643: Support PyArrow arrays as column values in `from_dict`
State: closed | Created: 2023-03-15T19:32:40Z

For consistency with `pa.Table.from_pydict`, which supports both Python lists and PyArrow arrays as column values.

"Fixes" https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417
GitHub issue #5642: Bump hfh to 0.11.0
State: closed | Created: 2023-03-15T18:26:07Z

to fix errors like

```
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/...
```

(e.g. from this [failing CI](https://github.com/huggingface/datasets/actions/runs/4428956210/jobs/7769160997))

0.11.0 is the current minimum version in `transformers`

around 5% of users are currently using versions `<0.11.0`
GitHub issue #5641: Features cannot be named "self"
State: closed | Created: 2023-03-15T17:16:40Z

### Describe the bug

Hi,
I noticed that we cannot create a HuggingFace dataset from Pandas DataFrame with a column named `self`.
The error seems to be coming from arguments validation in the `Features.from_dict` function.



### Steps to reproduce the bug

```python
import datasets
dummy_pandas = pd.DataFrame([0,1,2,3], columns = ["self"])
datasets.arrow_dataset.Dataset.from_pandas(dummy_pandas)
```

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.8.0
- Python version: 3.9.5
- PyArrow version: 6.0.1
- Pandas version: 1.4.1
GitHub issue #5640: Less zip false positives
State: closed | Created: 2023-03-15T16:48:59Z

`zipfile.is_zipfile` return false positives for some Parquet files. It causes errors when loading certain parquet datasets, where some files are considered ZIP files by `zipfile.is_zipfile`

This is a known issue: https://github.com/python/cpython/issues/72680

At first I wanted to rely only on magic numbers, but then I found that someone contributed a [fix to is_zipfile](https://github.com/python/cpython/pull/5053) - do you think we should use it @albertvillanova or not ?

IMO it's ok to rely on magic numbers only for now, since in streaming mode we've had no issue checking only the magic number so far.

Close https://github.com/huggingface/datasets/issues/5639