helmo/github-issues huggingface dataset:
prepare_module issue when loading from read-only fs
State: closed | Author: Dref360 (CONTRIBUTOR)
Labels: name: bug

## Describe the bug

When we use prepare_module from a readonly file system, we create a FileLock using the `local_path`.
This path is not necessarily writable.

`lock_path = local_path + ".lock"`


## Steps to reproduce the bug

Run `load_dataset` on a readonly python loader file.
```python
ds = load_dataset(
        python_loader, data_files={"train": train_path, "test": test_path}
    )
```

where `python_loader` is a path to a file located in a readonly folder.

## Expected results
This should work I think?

## Actual results

```python
    return load_dataset(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 711, in load_dataset
    module_path, hash, resolved_file_path = prepare_module(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 465, in prepare_module
    with FileLock(lock_path):
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 314, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 263, in acquire
    self._acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 378, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 30] Read-only file system: 'YOUR_FILE.py.lock'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 3.0.0


Comments: Hello, I opened #2887 to fix this.
Optimize Dataset.filter to only compute the indices to keep
State: closed | Author: lhoestq (MEMBER)

Optimize `Dataset.filter` to only compute the indices of the rows to keep, instead of creating a new Arrow table with the rows to keep. Creating a new table was an issue because it could take a lot of disk space.

This will be useful to process audio datasets for example cc @patrickvonplaten 

Comments: Maybe worth updating the docs here as well?

Yup, will do !
Update: timit_asr - make the dataset streamable
State: closed | Author: lhoestq (MEMBER)

The TIMIT ASR dataset had two issues that was preventing it from being streamable:

1. it was missing a call to `open` before `pd.read_csv`
2. it was using `os.path.dirname` which is not supported for streaming

I made the dataset streamable by using `open` to load the CSV, and by adding the support for `os.path.dirname` in dataset scripts to stream data

You can now do
```python
from datasets import load_dataset

timit_asr = load_dataset("timit_asr", streaming=True)
print(next(iter(timit_asr["train"])))
```
prints:

```json
{"file": "zip://data/TRAIN/DR4/MMDM0/SI681.WAV::https://data.deepai.org/timit.zip",
"phonetic_detail": {"start": [0, 1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720],
"utterance": ["h#", "w", "ix", "dcl", "s", "ah", "tcl", "ch", "ix", "n", "ae", "kcl", "t", "ix", "v", "r", "ix", "f", "y", "ux", "zh", "el", "bcl", "b", "iy", "y", "ux", "s", "f", "el", "h#"],
"stop": [1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720, 39920]},
"sentence_type": "SI", "id": "SI681",
"speaker_id": "MMDM0",
"dialect_region": "DR4",
"text": "Would such an act of refusal be useful?",
"word_detail": {
    "start": [1960, 4000, 9400, 10680, 15880, 18297, 27080, 30120],
    "utterance": ["would", "such", "an", "act", "of", "refusal", "be", "useful"],
    "stop": [4000, 9400, 10680, 15880, 18297, 27080, 30120, 37720]
}}
```

cc @patrickvonplaten @vrindaprabhu
IndexError when accessing first element of a Dataset if first RecordBatch is empty
State: closed | Author: lhoestq (MEMBER)

The computation of the offsets of the underlying Table of a Dataset has some issues if the first RecordBatch is empty.

```python
from datasets import Dataset
import pyarrow as pa

pa_table = pa.Table.from_pydict({"a": [1]})
pa_table2 = pa.Table.from_pydict({"a": []}, schema=pa_table.schema)
ds_table = pa.concat_tables([pa_table2, pa_table])

dataset = Dataset(ds_table)

print([len(b) for b in dataset.data._batches])
# [0, 1]

print(dataset.data._offsets)
# [0 0 1] (should be [0, 1])

dataset[0]
```
raises
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/table.py in _interpolation_search(arr, x)
     90         else:
     91             i, j = i, k
---> 92     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
     93 
     94 

IndexError: Invalid query '0' for size 1.
```

This can be fixed by ignoring empty batches when computing `table._batches` and `table._offsets`

cc @SaulLu
Logging levels not taken into account
State: closed | Author: LysandreJik (MEMBER)
Labels: name: bug

## Describe the bug

The `logging` module isn't working as intended relative to the levels to set.

## Steps to reproduce the bug

```python
from datasets import logging

logging.set_verbosity_debug()
logger = logging.get_logger()

logger.error("ERROR")
logger.warning("WARNING")
logger.info("INFO")
logger.debug("DEBUG"
```

## Expected results

I expect all logs to be output since I'm putting a `debug` level.

## Actual results

Only the two first logs are output.

## Environment info

- `datasets` version: 1.11.0
- Platform: Linux-5.13.9-arch1-1-x86_64-with-glibc2.33
- Python version: 3.9.6
- PyArrow version: 5.0.0

## To go further

This logging issue appears in `datasets` but not in `transformers`. It happens because there is no handler defined for the logger. When no handler is defined, the `logging` library will output a one-off error to stderr, using a `StderrHandler` with level `WARNING`.

`transformers` sets a default `StreamHandler` [here](https://github.com/huggingface/transformers/blob/5c6eca71a983bae2589eed01e5c04fcf88ba5690/src/transformers/utils/logging.py#L86)

Comments: I just take a look at all the outputs produced by `datasets` using the different log-levels.
As far as i can tell using `datasets==1.17.0` they overall issue seems to be fixed.

However, I noticed that there is one tqdm based progress indicator appearing on STDERR that I can simply not suppress.
```
Resolving data files: 100%|██████████| 652/652 [00:00<00:00, 1604.52it/s]
```

According to _get_origin_metadata_locally_or_by_urls it shold be supressable by using the `NOTSET` log-level
https://github.com/huggingface/datasets/blob/1406a04c3e911cec2680d8bc513653e0cafcaaa4/src/datasets/data_files.py#L491-L501
Sadly when specifiing the log-level `NOTSET` it seems to has no effect.

But appart from it not having any effect I must admit that it seems unintuitive to me.
I would suggest changing this such that it is only shown when the log-level is greater or equal to INFO.

This would conform better to INFO according to the [documentation](https://huggingface.co/docs/datasets/v1.0.0/package_reference/logging_methods.html#datasets.logging.set_verbosity_info).
> This will display most of the logging information and tqdm bars.

Any inputs on this?
I will be happy to supply a PR if desired 👍 

Hi! This should disable the tqdm output:
```python
import datasets
datasets.set_progress_bar_enabled(False)
```

On a side note: I believe the issue with logging (not tqdm) is still relevant on master.
ArrowInvalid when mapping dataset with missing values
State: open | Author: uniquefine (NONE)
Labels: name: bug

## Describe the bug
I encountered an `ArrowInvalid` when mapping dataset with missing values. 
Here are the files for a minimal example. The exception is only thrown when the first line in the csv has a missing value (if you move the last line to the top it isn't thrown).
[data_small.csv](https://github.com/huggingface/datasets/files/7037838/data_small.csv)
[data.csv](https://github.com/huggingface/datasets/files/7037842/data.csv)

## Steps to reproduce the bug
```python
from datasets import load_dataset

datasets = load_dataset("csv", data_files=['data_small.csv'])

datasets = datasets.map(lambda e: {'labels': e['match']},
                        remove_columns=['id'])
```

## Expected results
No error

## Actual results
```
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid null value
```

## Environment info
- `datasets` version: 1.5.0
- Platform: Linux-5.11.0-25-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments: Hi ! It fails because of the feature type inference.

Because the first 1000 examples all have null values in the "match" field, then it infers that the type for this field is `null` type before writing the data on disk. But as soon as it tries to map an example with a non-null "match" field, then it fails.

To fix that you can either:
- increase the writer_batch_size to >2000 (default is 1000) so that some non-null values will be in the first batch written to disk
```python
datasets = datasets.map(lambda e: {'labels': e['match']}, remove_columns=['id'], writer_batch_size=2000)
```
- OR force the feature type with:
```python
from datasets import Features, Value

features = Features({
    'conflict': Value('int64'),
    'date': Value('string'),
    'headline': Value('string'),
    'match': Value('float64'),
    'label': Value('float64')
})
datasets = datasets.map(lambda e: {'labels': e['match']}, remove_columns=['id'], features=features)
```
Optimize streaming from TAR archives
State: closed | Author: lhoestq (MEMBER)
Labels: name: enhancement

name: streaming

Hi ! As you know TAR has some constraints for data streaming. While it is optimized for buffering, the files in the TAR archive **need to be streamed in order**. It means that we can't choose which file to stream from, and this notation is to be avoided for TAR archives:
```
tar://books_large_p1.txt::https://storage.googleapis.com/huggingface-nlp/datasets/bookcorpus/bookcorpus.tar.bz2
```
Instead, I suggest we implement `iter_archive` for the `StreamingDownloadManager`.
The regular `DownloadManager` already has it.

Then we will have to update the json/txt/csv/etc. loaders to make them use `iter_archive` on TAR archives.

That's also what Tensorflow Datasets is doing in this case.
See this [dataset](https://github.com/tensorflow/datasets/blob/93895059c80a9e05805e8f32a2e310f66a23fc98/tensorflow_datasets/image_classification/flowers.py) for example.

Therefore instead of doing
```python
uncompressed = dl_manager.extract(tar_archive)
filename = "books_large_p1.txt"
with open(os.path.join(uncompressed, filename)) as f:
    for line in f:
        ...
```
we'll do
```python
for filename, f in dl_manager.iter_archive(tar_archive):
    for line in f:
        ...
```

Comments: Closed by: 
- #3066
Add code-mixed Kannada Hope speech dataset
State: closed | Author: adeepH (CONTRIBUTOR)

## Adding a Dataset
- **Name:** *KanHope*
- **Description:** *A code-mixed English-Kannada dataset for Hope speech detection*
- **Paper:** *https://arxiv.org/abs/2108.04616* 
- **Data:** *https://github.com/adeepH/KanHope/tree/main/dataset*
- **Motivation:** *The dataset is amongst the very few resources available for code-mixed low-resourced Dravidian languages of India*
HF_DATASETS_CACHE variable in Windows
State: closed | Author: rp2839 (NONE)

I can't seem to use a custom Cache directory in Windows. I have tried:

set HF_DATASETS_CACHE = "C:\Datasets"
set HF_DATASETS_CACHE = "C:/Datasets"
set HF_DATASETS_CACHE = "C:\\Datasets"
set HF_DATASETS_CACHE = "r'C:\Datasets'"
set HF_DATASETS_CACHE = "\Datasets"
set HF_DATASETS_CACHE = "/Datasets"

In each instance I get the "[WinError 123] The filename, directory name, or volume label syntax is incorrect" error when attempting to load a dataset

Comments: Agh - I'm a muppet. No quote marks are needed.
set HF_DATASETS_CACHE = C:\Datasets
works as intended.
Add url prefix convention for many compression formats
State: closed | Author: lhoestq (MEMBER)

## Intro

When doing dataset streaming, the uncompression of compressed files is done on the fly using `fsspec`.

In particular, the download manager method `download_and_extract` doesn't return a path to the local download and extracted file, but instead a chained URL so that the uncompression can be done when the file is opened. A few examples of chained URLS:
- `gz://file.txt::https://foo.bar/file.txt.gz`
- `bz2://file.txt::https://foo.bar/file.txt.bz2`
- `zip://::https://foo.bar/archive.zip`
- `tar://::https://foo.bar/archive.tar.gz` (the TAR uncompression includes gz, bz2 etc. uncompression in `fsspec`)

This syntax is highly inspired by the `fsspec` URL chaining syntax from https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining

This url prefixing allows `open` to know what kind of uncompression to do in a dataset script when doing
```python
def _generate_examples(self, urlpath):
    with open(urlpath) as f:
        ....
```

## What it changes

This changes the previous behavior from https://github.com/huggingface/datasets/pull/2786 , in which `open` was trying to infer the compression automatically. Infering the compression made it impossible to know whether the user wanted `open` to return compressed data (as the default behavior of the buitin open), or the uncompressed data. By adding uncompression prefixes to the URL, `open` know directly if it has to uncompress or not, and also which protocol to use.

## Additional notes

This PR should close https://github.com/huggingface/datasets/issues/2813

It should also close this PR https://github.com/huggingface/datasets/pull/2811 since the oscar dataset script won't try to uncompress twice anymore

Note that I had to temporarily remove the support for passing tar and zip files to `data_files` for streaming to make it work, since it makes it ambiguous whether a zip file passed as `data_files` should be uncompressed or not. IMO we can make it work again by changing the syntax to make the glob explicit:
```python
load_dataset("json", data_files="zip://*.jsonl::https://foo.bar/archive.zip")
```
This is the exact same convention as fsspec and it removes all ambiguities

cc @albertvillanova @lewtun 

Comments: Thanks for the feedback :) I will also complete the documentation to explain this convention

I just added some documentation about how streaming works with chained URLs.

I will also add some docs about how to use chained URLs directly in `load_dataset` in #2662, since #2662 does change the documentation already and to avoid having to resolve conflicts.

Merging this one now, next step is resolve the conflicts in #2662 and update the docs for URL chaining :)

There is also the glob feature of zip files that I need to add, to be able to do this for example:
```python
load_dataset("json", data_files="zip://*::https://foo.bar/archive.zip")
```
Cannot load linnaeus dataset
State: closed | Author: NielsRogge (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
The [linnaeus](https://huggingface.co/datasets/linnaeus) dataset cannot be loaded. To reproduce:
```
from datasets import load_dataset

datasets = load_dataset("linnaeus")
```
This results in:
```
Downloading and preparing dataset linnaeus/linnaeus (download: 17.36 MiB, generated: 8.74 MiB, post-processed: Unknown size, total: 26.10 MiB) to /root/.cache/huggingface/datasets/linnaeus/linnaeus/1.0.0/2ff05dbc256108233262f596e09e322dbc3db067202de14286913607cd9cb704...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-4-7ef3a88f6276> in <module>()
      1 from datasets import load_dataset
      2 
----> 3 datasets = load_dataset("linnaeus")

11 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    603             raise FileNotFoundError("Couldn't find file at {}".format(url))
    604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
--> 605         raise ConnectionError("Couldn't reach {}".format(url))
    606 
    607     # Try a second time

ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/
```

Comments: Thanks for reporting ! #2852 fixed this error

We'll do a new release of `datasets` soon :)
cannot load data from my loacal path
State: closed | Author: yang-collect (NONE)
Labels: name: bug

## Describe the bug
I just want to directly load data from my local path,but find a bug.And I compare it with pandas to provide my local path is real.

here is my code
```python3
# print my local path
print(config.train_path)
# read data and print data length
tarin=pd.read_csv(config.train_path)
print(len(tarin))

# loading data by load_dataset 
data = load_dataset('csv',data_files=config.train_path)

print(len(data))
```


## Steps to reproduce the bug
```python
C:\Users\wie\Documents\项目\文本分类\data\train.csv
7613
Traceback (most recent call last):
  File "c:/Users/wie/Documents/项目/文本分类/lib/DataPrecess.py", line 17, in <module>
    data = load_dataset('csv',data_files=config.train_path)
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\load.py", line 830, in load_dataset
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\load.py", line 710, in load_dataset_builder
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 271, in __init__
    **config_kwargs,
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 386, in _create_builder_config
    config_kwargs, custom_features=custom_features, use_auth_token=self.use_auth_token
  File "C:\Users\wie\Miniconda3\lib\site-packages\datasets\builder.py", line 156, in create_config_id
    raise ValueError("Please provide a valid `data_files` in `DatasetBuilder`")
ValueError: Please provide a valid `data_files` in `DatasetBuilder`
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:  1.11.0
- Platform: win10
- Python version: 3.7.9
- PyArrow version: 5.0.0


Comments: Hi ! The `data_files` parameter must be a string, a list/tuple or a python dict.

Can you check the type of your `config.train_path` please ? Or use `data_files=str(config.train_path)` ?
Rename The Pile subsets
State: closed | Author: lhoestq (MEMBER)

After discussing with @yjernite we think it's better to have the subsets of The Pile explicitly have "the_pile" in their names.

I'm doing the changes for the subsets that @richarddwang added:
- [x] books3 -> the_pile_books3 https://github.com/huggingface/datasets/pull/2801
- [x] stack_exchange -> the_pile_stack_exchange https://github.com/huggingface/datasets/pull/2803
- [x] openwebtext2 -> the_pile_openwebtext2 https://github.com/huggingface/datasets/pull/2802

For consistency we should also rename `bookcorpusopen` to `the_pile_bookcorpus` IMO, but let me know what you think.
(we can just add a deprecation message to `bookcorpusopen` for now and add `the_pile_bookcorpus`)

Comments: Sounds good. Should we also have a “the_pile” dataset with the subsets as configuration?

I think the main `the_pile` datasets will be the one that is the mix of all the subsets: https://the-eye.eu/public/AI/pile/

We can also add configurations for each subset, and even allow users to specify the subsets they want:
```python
from datasets import load_dataset

load_dataset("the_pile", subsets=["openwebtext2", "books3", "hn"])
```

We're alrady doing something similar for mC4, where users can specify the list of languages they want to load.
Add Mostly Basic Python Problems Dataset
State: open | Author: osanseviero (CONTRIBUTOR)
Labels: name: dataset request

## Adding a Dataset
- **Name:** Mostly Basic Python Problems Dataset
- **Description:** The benchmark consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://github.com/google-research/google-research/tree/master/mbpp
- **Motivation:** Simple, small dataset related to coding problems.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: I started working on that.
Bump tqdm version
State: closed | Author: mariosasko (COLLABORATOR)

The recently released tqdm 4.62.1 includes a fix for PermissionError on Windows (submitted by me in https://github.com/tqdm/tqdm/pull/1207), which means we can remove expensive `gc.collect` calls by bumping tqdm to that version. This PR does exactly that and, additionally, fixes a `disable_tqdm` definition that would previously, if used, raise a PermissionError on Windows.
Fix stream oscar
State: closed | Author: albertvillanova (MEMBER)

Previously, an additional `open` was added to oscar to make it stream-compatible: 587bbb94e891b22863b312b99696e32708c379f4.

This was argued that might be problematic: https://github.com/huggingface/datasets/pull/2786#discussion_r690045921

This PR:
- removes that additional `open`
- patches `gzip.open` with `xopen` + `compression="gzip"`

Comments: One additional note: if we can try to not change the code of oscar.py too often, I'm sure users that have it in their cache directory will be happy to not have to redownload it every time they update the library ;)

(since changing the code changes the cache directory of the dataset)

I don't think this is confusing for users because users don't even know we have patched `open`. The only thing users care is that if the pass `streaming=True`, they want to be able to load the dataset in streaming mode.

I don't see any other dataset where patching `open` with `fsspec.open`+`compression` is an "underlying issue". Are there other datasets where this is an issue?

The only dataset where this was an issue is in oscar and the issue is indeed due to the additional `open` you added inside `zip.open`.

Closing this one since https://github.com/huggingface/datasets/pull/2822 reverted the change of behavior of `open`