noamaanMulla-03/datasets-issues huggingface dataset:
Option to disable file locking

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os
 
 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version
 
 
+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```



Labels: name: enhancement
Strange slicing behavior

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


As explained in the [docs](https://huggingface.co/docs/datasets/v2.18.0/en/access#slicing), slicing a `Dataset` returns a dictionary that maps its column names to their values. So, `len(dataset[:300])=2` is expected, assuming your dataset has 2 columns (the returned dict has 2 keys, but each value in the dict has 300 items).
`
load only England English dataset from common voice english dataset

training_data = load_dataset("common_voice", "en",split='train[:250]+validation[:250]')
testing_data = load_dataset("common_voice", "en", split="test[:200]")

I'm trying to load only 8% of the English common voice data with accent == "England English." Can somebody assist me with this?

**Typical Voice Accent Proportions:**

- 24% United States English 
- 8% England English 
- 5% India and South Asia (India, Pakistan, Sri Lanka) 
- 3% Australian English 
- 3% Canadian English 
- 2% Scottish English 
- 1% Irish English 
- 1% Southern African (South Africa, Zimbabwe, Namibia) 
- 1% New Zealand English

Can we replicate this for Age as well?

**Age proportions of the common voice:-**

- 24% 19 - 29 
- 14% 30 - 39 
- 10% 40 - 49 
- 6% < 19 
- 4% 50 - 59 
- 4% 60 - 69 
- 1% 70 – 79 

Labels: name: question

Hi @amanjaiswal777,

First note that the dataset you are trying to load is deprecated: it was the Common Voice dataset release as of Dec 2020.

Currently, Common Voice dataset releases are directly hosted on the Hub, under the Mozilla Foundation organization: https://huggingface.co/mozilla-foundation

For example, to get their latest Common Voice relase (8.0):
- Go to the dataset page and request access permission (Mozilla Foundation requires this for people willing to use their datasets): https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Looking at the dataset card, you can check that data instances have, among other fields, the ones you are interested in: "accent", "age",... 
- Then you can load their "en" language dataset as usual, besides passing your authentication token (more info on auth token here: https://huggingface.co/docs/hub/security)
  ```python
  from datasets import load_dataset
  ds_en = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True)
  ```
- Finally, you can filter only the data instances you are interested in (more info on `filter` here: https://huggingface.co/docs/datasets/process#select-and-filter):
  ```python
  ds_england_en = ds_en.filter(lambda item: item["accent"] == "England English")
  ```

Feel free to reopen this issue if you need further assistance.

Hey @albertvillanova trying the same approach as you with the common_voice_16_1 dataset. What I'm trying to do is to filter the valencian accent in the catalan subset. Gave me this error and I have everything it asks for decoding mp3:
![image](https://github.com/huggingface/datasets/assets/96977715/7ec02483-e728-4358-9372-ba74ec1b7fd4)

![image](https://github.com/huggingface/datasets/assets/96977715/c10fcf23-a141-4dba-a88d-89e293acfe67)
AttributeError: module 'datasets' has no attribute 'load_dataset'

## Describe the bug
I have piped install datasets, but this package doesn't have these attributes: load_dataset, load_metric.

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux-5.13.0-44-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.13
- PyArrow version: 6.0.1



Labels: name: bug

I'm having the same issue，Can you tell me how to solve it？

I have the same issue, can you tell me how to solve it? Thanks

I had a folder named 'datasets' so this is why it can't find the import, it's looking in the wrong place

@briandw your comment saved my day 👍
Use 'with' operator for some download functions

Some functions in `streaming_download_manager.py` are not closing the file they open which lead to `Unclosed file` warnings in our code. This fixes a few of them.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I was mistaken on the intent of those functions, closing the PR.
Support SQLite files?

### Feature request

Support loading a dataset from a SQLite file

https://huggingface.co/datasets/severo/test_iris_sqlite/tree/main

### Motivation

SQLite is a popular file format.

### Your contribution

See discussion on slack: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1702481859117909 (internal)

In particular: a SQLite file can contain multiple tables, which might be matched to multiple configs. Maybe the detail of splits and configs should be defined in the README YAML, or use the same format as for ZIP files: `Iris.sqlite::Iris`. 

See dataset here: https://huggingface.co/datasets/severo/test_iris_sqlite

Note: should we also support DuckDB files?

Labels: name: enhancement

You can use `Dataset.from_sql(path_to_sql_file)` already. Though we haven't added the Sql dataset builder to the `_PACKAGED_DATASETS_MODULES` list or in `_EXTENSION_TO_MODULE` to map `.sqlite` to the Sql dataset builder

This would allow to load a dataset repository with a `.sqlite` file using `load_dataset` and enable the Dataset Viewer

Considering `Dataset.from_sql`'s (extremely) low usage, I don't think many users are interested in using this format for their datasets. Also, SQLite files are hard/impossible to stream efficiently and require custom logic to define splits/subsets, so IMO we shouldn't encourage people to use SQLite on the Hub.

@severo Do you have some real-world examples of datasets published in this format?

No. Indeed, it seems better to explicitly not support sqlite
Persistent multi-process Pool

### Feature request

Running .map and filter functions with `num_procs` consecutively instantiates several multiprocessing pools iteratively. 

As instantiating a Pool is very resource intensive it can be a bottleneck to performing iteratively filtering. 

My ideas:
1. There should be an option to declare `persistent_workers` similar to pytorch DataLoader. Downside would be that would be complex to determine the correct resource allocation and deallocation of the pool. i.e. the dataset can outlive the utility of the pool. 
2. Provide a pool as an argument. Downside would be the expertise required by the user. Upside, is that there is better resource management.

### Motivation

Is really slow to iteratively perform map and filter operations on a dataset.

### Your contribution

If approved I could integrate it. I would need to know what method would be most suitable to implement from the two options above.

Labels: name: enhancement
Test disabling transformers containers in docs CI

Related to https://github.com/huggingface/doc-builder/pull/487 and [internal slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1711384899462349?thread_ts=1711041424.720769&cid=C04F8N7FQNL). There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.

cc @mishig25 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

On slack it was mentioned that it was actually slower for `datasets`, should we close this one or am I missing something ?

@lhoestq I converted to draft. Want to make some more tests and will let you know
Support for loading geotiff files as a part of the ImageFolder

### Feature request

Request for adding rasterio support to load geotiff as a part of ImageFolder, instead of using PIL

### Motivation

As of now, there are many datasets in HuggingFace Hub which are predominantly focussed towards RemoteSensing or are from RemoteSensing. The current ImageFolder (if I have understood correctly) uses PIL. This is not really optimized because mostly these datasets have images with many channels and additional metadata. Using PIL makes one loose it unless we provide a custom script. Hence, maybe an API could be added to have this in common?

### Your contribution

If the issue is accepted - i can contribute the code, because I would like to have it automated and generalised.

Labels: name: enhancement
Hi,do you know how to load the dataset from local file now?

              Hi, if I want to load the dataset from local file, then how to specify the configuration name?

_Originally posted by @WHU-gentle in https://github.com/huggingface/datasets/issues/2976#issuecomment-1333455222_
            



@Gera001
# Loading Dataset from Local Files Using 🤗Hugging Face.

To load a dataset from local files using the Hugging Face datasets library, you can use the `load_dataset` function.

```
from datasets import load_dataset
dataset = load_dataset('csv', data_files={'train': 'path/to/train.csv',
                                         'test': 'path/to/test.csv'})
```

Reference to [HF Datasets docs for loading from local](https://huggingface.co/docs/datasets/en/loading#csv).       

@albertvillanova
this issue can be closed here.

like this: from datasets import load_from_disk
dataset = load_from_disk(data_path)


@ge00009 
> like this: from datasets import load_from_disk dataset = load_from_disk(data_path)

Loads a dataset that was previously saved using `save_to_disk()`.

Reference link:
https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_from_disk.example
(Willing to PR) Datasets with custom python objects

### Feature request

Hi thanks for the library! I would like to have a huggingface Dataset, and one of its column is custom (non-serializable) Python objects. For example, a minimal code:

```
class MyClass:
    pass

dataset = datasets.Dataset.from_list([
    dict(a=MyClass(), b='hello'),
])
```

It gives error:

```
ArrowInvalid: Could not convert <__main__.MyClass object at 0x7a852830d050> with type MyClass: did not recognize Python value type when inferring an Arrow data type
```

I guess it is because Dataset forces to convert everything into arrow format. However, is there any ways to make the scenario work? Thanks!

### Motivation

(see above)

### Your contribution

Yes, I am happy to PR!

Cross-posted: https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050?u=fzyzcjy

EDIT: possibly related https://github.com/huggingface/datasets/issues/5766

Labels: name: enhancement
Small typo on the documentation

### Describe the bug

There is a small typo on https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

It should be `caching is enabled`. 

### Steps to reproduce the bug

Please visit 

https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938


### Expected behavior

`caching is enabled`

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0

Labels: name: good first issue

Thanks for reporting @fostiropoulos! I've edited your comment to fix the link to the problematic line.


@mariosasko can i take this up?

#self-assign
remove_columns is not in-place but the doc shows it is in-place

### Describe the bug

The doc of `datasets` v2.17.0/v2.17.1 shows that `remove_columns` is in-place. [link](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)

In the text classification example of transformers v4.38.1, the columns are not removed.
https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Steps to reproduce the bug

https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Expected behavior

Actually remove the columns.

### Environment info

1. datasets v2.17.0
2. transformers v4.38.1


Good catch! I've opened a PR with a fix in the `transformers` repo.

@mariosasko Thanks!

Will the doc of `datasets` be updated?

I find some possible mistakes in doc about whether `remove_columns` is in-place.
1. [You can also remove a column using map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.Dataset.remove_columns)
2. [You can also remove a column using Dataset.map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)
3. [🤗 Datasets also has a remove_columns() function which is faster because it doesn’t copy the data of the remaining columns.](https://huggingface.co/docs/datasets/v2.17.1/en/process#map)

I've linked a PR that will fix the usage in the `datasets` docs.
Can't load dataset

I'm trying to load a wikitext dataset

```
from datasets import load_dataset
raw_datasets = load_dataset("wikitext")
```

ValueError: Config name is missing.
Please pick one among the available configs: ['wikitext-103-raw-v1', 'wikitext-2-raw-v1', 'wikitext-103-v1', 'wikitext-2-v1']
Example of usage:
	`load_dataset('wikitext', 'wikitext-103-raw-v1')`.

If I try
```
from datasets import load_dataset
raw_datasets = load_dataset("wikitext-2-v1")
```

FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.12.1/datasets/wikitext-2-v1/wikitext-2-v1.py

#### Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic (colab)
- Python version: 3.7.12
- PyArrow version: 3.0.0


Labels: name: bug

Hi @mskovalova, 

Some datasets have multiple configurations. Therefore, in order to load them, you have to specify both the *dataset name* and the *configuration name*.

In the error message you got, you have a usage example:
- To load the 'wikitext-103-raw-v1' configuration of the 'wikitext' dataset, you should use: 
  ```python
  load_dataset('wikitext', 'wikitext-103-raw-v1')
  ```

In your case, if you would like to load the 'wikitext-2-v1' configuration of the 'wikitext' dataset, please use:
```python
raw_datasets = load_dataset("wikitext", "wikitext-2-v1")
```

Hi, if I want to load the dataset from local file, then how to specify the configuration name?

> 嗨，如果我想从本地文件加载数据集，那么如何指定配置名称？

请问你知道怎么从本地加载了吗 谢谢

`raw_datasets = load_dataset("/local/path", "wikitext-2-v1", ...)`
I use datasets==2.18.0
load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

### Describe the bug

Running

```
import datasets
ds = datasets.load_dataset('bigcode/the-stack-dedup', streaming=True)
```

takes about 2.5 minutes! 

I would expect this to be near instantaneous. With other datasets, the runtime is one or two seconds.


### Environment info

- `datasets` version: 2.11.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0


This is due to the slow resolution of the data files: https://github.com/huggingface/datasets/issues/5537.

We plan to switch to `huggingface_hub`'s `HfFileSystem` soon to make the resolution faster (will be up to 20x faster once we merge https://github.com/huggingface/huggingface_hub/pull/1443)



You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

> You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

That's unrelated to the problem discussed in this issue. 

> > You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
> 
> That's unrelated to the problem discussed in this issue.

Sorry, I misunderstood it.

Closing this issue as it has been addressed in `huggingface_hub`!

(This now takes 25s to execute on my machine.)

Thanks for the improvements! 🎉🎉

25 seconds is better but still about 2500x slower than this _should_ be! Loading a tiny 1-2KB metadata file is all that would be necessary with a better design.

Once we merge https://github.com/huggingface/huggingface_hub/pull/2103, this should only take a few seconds. 

For the 2500x speed-up (without metadata files with pre-cached results), we wouldn't even be allowed to use `os.path` functions or `requests`/`aiohttp` for HTTP requests, so I don't think this is feasible for us as it would make the code unreadable.

The HF Datasets Hub is (almost) platform-agnostic, so you are free to implement your own library (in a faster language than Python) to achieve this kind of performance, and we would be happy to support it 🙂.
Dataset.to_csv() missing commas in columns with lists

### Describe the bug

The `to_csv()` method does not output commas in lists.  So when the Dataset is loaded back in the data structure of the column with a list is not correct.

Here's an example:

Obviously, it's not as trivial as inserting commas in the list, since its a comma-separated file.  But hopefully there's a way to export the list in a way that it'll be imported by `load_dataset()` correctly.

### Steps to reproduce the bug

Here's some code to reproduce the bug:


```python
from datasets import Dataset

ds = Dataset.from_dict(
    {
        "pokemon": ["bulbasaur", "squirtle"],
        "type": ["grass", "water"]
    }
)

def ascii_to_hex(text):
    return [ord(c) for c in text]

ds = ds.map(lambda x: {"int": ascii_to_hex(x['pokemon'])})

ds.to_csv('../output/temp.csv')
```

temp.csv then contains:
```

### Expected behavior

ACTUAL OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[ 98 117 108  98  97 115  97 117 114]
squirtle,water,[115 113 117 105 114 116 108 101]
```

EXPECTED OUTPUT:
```
pokemon,type,int
bulbasaur,grass,[98, 117, 108,  98,  97, 115,  97, 117, 114]
squirtle,water,[115, 113, 117, 105, 114, 116, 108, 101]
```

or probably something more like this since it's a CSV file:
```
pokemon,type,int
bulbasaur,grass,"[98, 117, 108,  98,  97, 115,  97, 117, 114]"
squirtle,water,"[115, 113, 117, 105, 114, 116, 108, 101]"
```

### Environment info

### Package Version
Name: datasets
Version: 2.16.1

### Python
version: 3.10.12


### OS Info
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
...
UBUNTU_CODENAME=jammy


Hello!

This is due to how pandas write numpy arrays to csv. [Source](https://stackoverflow.com/questions/54753179/to-csv-saves-np-array-as-string-instead-of-as-a-list)
To fix this, you can convert them to list yourselves.

```python
df = ds.to_pandas()
df['int'] = df['int'].apply(lambda arr: list(arr))
df.to_csv(index=False, '../output/temp.csv')
```

I think it would be good if `datasets` would do the conversion itself, but it's a breaking change and I would wait for the greenlight from someone from HF.
Remove get_inferred_type from ArrowWriter write_batch

Inferring the type seems to be unnecessary given that the pyarrow array has already been created.
Because pyarrow array creation is sometimes extremely slow this doubles the time write_batch takes.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6781). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Close in favor of #6786.