noamaanMulla-03/datasets-issues huggingface dataset:
Lossy json serialization - deserialization of dataset info 

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead. 

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
    
test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}, 
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```


### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3


Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.
Multiprocessing with FileLock fails in python 3.9

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock


def run(i):
    print(f"got the lock in multi process [{i}]")


with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c 

Labels: name: bug

Closing this one because it must be used this way actually:
```python
def main():
    with FileLock("tmp.lock"):
        with Pool(2) as pool:
            pool.map(run, range(2))

if __name__ == "__main__":
    main()
```
Unable to rename columns in streaming dataset

### Describe the bug

Trying to rename column in a streaming datasets, destroys the features object.

### Steps to reproduce the bug

The following code illustrates the error:

```
from datasets import load_dataset
dataset = load_dataset('mc4', 'en', streaming=True, split='train')
dataset.info.features
# {'text': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None), 'url': Value(dtype='string', id=None)}
dataset = dataset.rename_column("text", "content")
dataset.info.features
# This returned object is now None!
```



### Expected behavior

This should just alter the renamed column.

### Environment info

datasets 2.6.1


Hi @peregilk this bug is directly related to https://github.com/huggingface/datasets/issues/3888, and still not fixed... But I'll try to have a look!

Thanks @alvarobartt. It is great if you are able to fix it, but when reading the explanation it seems like it is possible to work around it.

We also tried keeping the 'info.features' and then adding a modified version back after the remove/rename. Unforutunately that leads to a dataset that is not possible to iterate over.

So if you iterate over the `IterableDataset` as `next(iter(ds))` and then run `rename_columns` when checking that data it will work, but in the end, it's just renaming the column one example/batch at a time, not renaming the column name for all the entries in the dataset, which is the ideal.

@alvarobartt Thanks. My use case was that I wanted to do multiple things, ie removing all unnecessary columns, renaming some valid columns, and then using cast (in my case checking if the audio is not 16K and casting it). It is just convenient to look into the info.features between each of these operations. Alternatively, I will just plan ahead...;) To me it seems like all the operations are working.

Thanks for the advice. It was very useful.

If we know the features before renaming, then we know the features after renaming, so we can pass the new features to the returned dataset in `rename_column` indeed ! If anyone is interested in contributing, feel free to open a PR and I'd be happy to help / give some pointers :)

Sure @lhoestq thanks! I’ll try to work on that

#self-assign
Bug in xjoin with Windows pathnames

Currently, `xjoin` function has a bug with local Windows pathnames: instead of returning the OS-dependent join pathname, it always returns it in POSIX format.

```python
from datasets.download.streaming_download_manager import xjoin

path = xjoin("C:\\Users\\USERNAME", "filename.txt")
```

Join path should be:
```python
"C:\\Users\\USERNAME\\filename.txt"
```
However it is:
```python
"C:/Users/USERNAME/filename.txt"
```

Labels: name: bug
Bug in xopen with Windows pathnames

Currently, `xopen` function has a bug with local Windows pathnames:

From its implementation:
```python
def xopen(file: str, mode="r", *args, **kwargs):
    file = _as_posix(PurePath(file))
    main_hop, *rest_hops = file.split("::")
    if is_local_path(main_hop):
        return open(file, mode, *args, **kwargs)
```

On a Windows machine, if we pass the argument:
```python
xopen("C:\\Users\\USERNAME\\filename.txt")
```
it returns
```python
open("C:/Users/USERNAME/filename.txt")
```

Labels: name: bug
IterableDataset columns and feature types

Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova 

Labels: name: generic discussion

name: streaming

#self-assign

@alvarobartt I've assigned you the issue since I'm not actively working on it.

Cool thanks @mariosasko I'll try to fix it in the upcoming days, thanks!

@lhoestq so in order to address what’s not completed in this issue, do you think it makes sense to add a param `features` to `IterableDataset.map` so that the output features right after the `map` are defined there? 

Yes that would be ideal IMO, thanks again for the help :)

@lhoestq cool then if you agree I can work on that! I’ll also update the docs accordingly once done, thanks!

I've already started with a PR as a draft @lhoestq, should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so? Thanks!

> should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so?

Right now one can use `ds = ds._resolve_features()` do to so. It can be used after `map` or `load_dataset` if the features are not known. Maybe we can make this method public ?
Q: Memory release when removing the column?

### Describe the bug

How do I release memory when I use methods like `.remove_columns()` or `clear()` in notebooks?
```python
from datasets import load_dataset
common_voice = load_dataset("mozilla-foundation/common_voice_11_0", "ja", use_auth_token=True)
# check memory -> RAM Used (GB):  0.704 / Total (GB)  33.670
common_voice = common_voice.remove_columns(column_names=common_voice.column_names['train'])
common_voice.clear()
# check memory -> RAM Used (GB):  0.705 / Total (GB)  33.670
```

I tried `gc.collect()` but did not help

### Steps to reproduce the bug

1. load dataset
2. remove all the columns
3. check memory is reduced or not


[link to reproduce](https://www.kaggle.com/code/bayartsogtya/huggingface-dataset-memory-issue/notebook?scriptVersionId=110630567)

### Expected behavior

Memory released when I remove the column

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Hi ! Datasets are memory mapped from your disk, i.e. they're not loaded in RAM. This is possible thanks to the Arrow data format.

Therefore the column you remove is not in RAM, so removing it doesn't cause the RAM to decrease.

Thanks for the explanation! @lhoestq 
I wonder since it is memory mapped, can we reduce or remove this memory map?

Yes you can `del common_voice` for example or wait for it to be garbage collected
Ci py3.10

Added a CI job for python 3.10

Some dependencies don't work on 3.10 like apache beam, so I remove them from the extras in this case.
I also removed some s3 fixtures that we don't use anymore (and that don't work on 3.10 anyway)


_The documentation is not available anymore as the PR was closed or merged._

Does it sound good to you @albertvillanova ?
Update Overview.ipynb google colab

- removed metrics stuff
- added image example
- added audio example (with ffmpeg instructions)
- updated the "add a new dataset" section


_The documentation is not available anymore as the PR was closed or merged._

WDYT @albertvillanova ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5211). All of your documentation changes will be reflected on that endpoint.
Support xPath for Windows pathnames

This PR implements a string representation of `xPath`, which is valid for local paths (also windows) and remote URLs.

Additionally, some `os.path` methods are fixed for remote URLs on Windows machines.

Now, on Windows machines:
```python
In [2]: str(xPath("C:\\dir\\file.txt"))
Out[2]: 'C:\\dir\\file.txt'
In [3]: str(xPath("http://domain.com/file.txt"))
Out[3]: 'http://domain.com/file.txt'
```


_The documentation is not available anymore as the PR was closed or merged._
Fix description of streaming in the docs

We say that "the data is being downloaded progressively" which is not true, it's just streamed, so I fixed it. Probably I missed some other places where it is written? 

Also changed docstrings for `StreamingDownloadManager`'s `download` and `extract` to reflect the same, as these docstrings are displayed in the documentation cc @lhoestq 


_The documentation is not available anymore as the PR was closed or merged._
Bug in sample_by="paragraph"

### Describe the bug

I think [this line](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/text/text.py#L96) is wrong and should be `batch = f.read(self.config.chunksize)`. Otherwise it will never terminate because even when `f` is finished reading, `batch` will still be truthy from the last iteration.

### Steps to reproduce the bug

```
> cat test.txt
a b c

d e f
````

```python
>>> import datasets
>>> datasets.load_dataset("text", data_files={"train":"test.txt"}, sample_by="paragraph")
```

This will go on forever.

### Expected behavior

Terminates very quickly. 

### Environment info

`version = "2.6.1"` but I think the bug is still there on main. 


Thanks for reporting, @adampauls.

We are having a look at it.
Fix loading from HF GCP cache

As reported in https://discuss.huggingface.co/t/error-loading-wikipedia-dataset/26599/4 it's not possible to download a cached version of Wikipedia from the HF GCP cache

I fixed it and added an integration test (runs in 10sec)


_The documentation is not available anymore as the PR was closed or merged._

> Do you know why this stopped working?

It comes from the changes in https://github.com/huggingface/datasets/pull/5107/files#diff-355ae5c229f95f86895404b72378ecd6e966c41cbeebb674af6fe6e9611bc126
Fix docs building for main

This PR reverts the triggering event for building documentation introduced by:
- #5250

Fix #5326.


_The documentation is not available anymore as the PR was closed or merged._

EDIT
At least the docs for ~~main~~ PR branch are now built:
- https://github.com/huggingface/datasets/actions/runs/3594847760/jobs/6053620813

Build documentation for main branch was triggered after this PR being merged: https://github.com/huggingface/datasets/actions/runs/3603370082/jobs/6071482470
Add num_proc to from_csv/generator/json/parquet/text

Allow multiprocessing to from_* methods


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5239). All of your documentation changes will be reflected on that endpoint.

I ended up moving `num_proc` to `AbstractDatasetReader.__init__` :)

Let me know if it sounds good to you now
Add `features` param to `IterableDataset.map`

## Description

As suggested by @lhoestq in #3888, we should be adding the param `features` to `IterableDataset.map` so that the features can be preserved (not turned into `None` as that's the default behavior) whenever the user passes those as param, so as to be consistent with `Dataset.map`, as it provides the `features` param so that those are not inferred by default, but specified by the user, and later validated by `ArrowWriter`.

This is internally handled already by the functions relying on `IterableDataset.map` such as `rename_column`, `rename_columns`, and `remove_columns` as described in #5287.

## Usage Example

```python
from datasets import load_dataset, Features

ds = load_dataset("rotten_tomatoes", split="validation", streaming=True)
print(ds.info.features)
ds = ds.map(
    lambda x: {"target": x["label"]},
    features=Features(
        {"target": ds.info.features["label"], "label": ds.info.features["label"], "text": ds.info.features["text"]}
    ),
)
print(ds.info.features)
```


_The documentation is not available anymore as the PR was closed or merged._
Clarify imagefolder is for small datasets

Based on feedback from [here](https://github.com/huggingface/datasets/issues/5317#issuecomment-1334108824), this PR adds a note to the `imagefolder` loading and creating docs that `imagefolder` is designed for small scale image datasets.


_The documentation is not available anymore as the PR was closed or merged._

I think it's also reasonable to add the same note to the AudioFolder decription

Thank you ! I think "regular" is more appropriate than "small". It can easily scale to a few thousands of images - just not millions x)

Replaced "small" with "several thousand" since what is considered "regular" and even "small" can be kind of vague!
Use same `num_proc` for dataset download and generation

Use the same `num_proc` value for data download and generation. Additionally, do not set `num_proc` to 16 in `DownloadManager` by default (`num_proc` now has to be specified explicitly).


_The documentation is not available anymore as the PR was closed or merged._

I noticed this bug the other day and was going to look into it! "Where are these processes coming from?" ;-)
Set `IterableDataset.map` param `batch_size` typing as optional

This PR solves #5325 

~Indeed we're using the typing for optional values as `Union[type, None]` as it's similar to how Python 3.10 handles optional values as `type | None`, instead of using `Optional[type]`.~

~Do we want to start using `Union[type, None]` for type-hinting optional values or just keep on using `Optional`?~ -> Keeping `Optional` still for consistency with the rest of the code in `datasets`

Also we now allow `batch_size` to be `None` for `IterableDataset.map` and `IterableDataset.filter`e.g. `MappedExamplesIterable` as `map` is internally instantiating those and propagating the `batch_size` param so if it can be `None` for `map` it should also do so for `MappedExamplesIterable`, as well as for `FilteredExamplesIterable` when calling `IterableDataset.filter`.

## TODOs

- [x] Add integration tests
- [x] Handle scenario where `batched=True` and `batch_size=None` or `batch_size<=0`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5336). All of your documentation changes will be reflected on that endpoint.

Hi @mariosasko, @lhoestq I was wondering whether we should include `batched` as a `pytest.mark` param for the functions testing `IterableDataset.map` so as to ensure that the changes done in this PR work fine without breaking anything of the actual functionality.

I've pushed updated tests just for one of the unit testing functions to be run as `pytest tests/test_iterable_dataset.py::test_mapped_examples_iterable -s --durations 0`, but some are still missing `batched` param, it was just to ask you whether we're supposed to do this for the rest of the functions or not, if it's a yes I'll push the commit as it's ready, but didn't want to push extra stuff that may be discarded later!

Thanks :hugs:

Thanks for the feedback @lhoestq, I agree with keeping `Optional` instead of `Union[type, None]` for now 👍🏻
map(...batch_size=None) for IterableDataset

### Feature request

Dataset.map(...) allows batch_size to be None. It would be nice if IterableDataset did too.

### Motivation

Although it may seem a bit of a spurious request given that `IterableDataset` is meant for larger than memory datasets, but there are a couple of reasons why this might be nice.

One is that load_dataset(...) can return either IterableDataset or Dataset. mypy will then complain if batch_size=None even if we know it is Dataset. Of course we can do:

    assert isinstance(d, datasets.DatasetDict)

But it is a mild inconvenience. What's more annoying is that whenever we use something like e.g. `combine_datasets(...)`, we end up with the union again, and so have to do the assert again.

Another is that we could actually end up with an IterableDataset small enough for memory in normal/correct usage, e.g. by filtering a massive IterableDataset.

For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

### Your contribution

Not this time.

Labels: name: enhancement

name: good first issue

Hi! I agree it makes sense for `IterableDataset.map` to support the `batch_size=None` case. This should be super easy to fix.

@mariosasko as this is something simple maybe I can include it as part of https://github.com/huggingface/datasets/pull/5311? Let me know :+1:

#self-assign

Feel free to close this @lhoestq as part of https://github.com/huggingface/datasets/pull/5336 :hugs:

Thanks again :)

> For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

This is interesting as well, if anyone wants to explore
fix: dataset path should be absolute

cache_file_name depends on dataset's path.

A simple way where this could cause a problem:
```
import os
import datasets


def add_prefix(example):
    example["text"] = "Review: " + example["text"]
    return example


ds = datasets.load_from_disk("a/relative/path")
os.chdir("/tmp")

ds_1 = ds.map(add_prefix)
```

while it may feel that the `chdir` is quite constructed, there are many scenarios when the current working dir can/will change...


Good catch thanks ! Have you tried to use the absolue path in `MemoryMappedTable.__init__` in `table.py`?

I think it can fix issues with relative paths at more levels than just fixing it `load_from_disk`. If it works I think it would be a more robust fix to this issue

@lhoestq right, that actually fixed it indeed. I've pushed the changes (one-liner). lemme know if there's anything else you need for this fix

_The documentation is not available anymore as the PR was closed or merged._
Clean up docstrings

As raised by @polinaeterna in #5324, some of the docstrings are a bit of a mess because it has both Markdown and Sphinx syntax. This PR fixes the docstring for `DatasetBuilder`. 

I'll start working on cleaning up the rest of the docstrings and removing the old Sphinx syntax (let me know if you prefer one big PR with all the cleaned changes or multiple smaller ones)! 🧼 

Labels: name: documentation

_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Let us know if we can help :)

Small pref for having multiple PRs

Awesome, thanks! Sorry this one is a little big, I'll open some smaller ones next :)
Unable to resolve any data file after loading once

when I rerun my program, it occurs this error
" Unable to resolve any data file that matches '['**train*']' at /data2/whr/lzy/open_domain_data/retrieval/wiki_dpr with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'zip']", so how could i deal with this problem?
thx.
And below is my code .
![image](https://user-images.githubusercontent.com/84694183/146023446-d75fdec8-65c1-484f-80d8-6c20ff5e994b.png)



Hi ! `load_dataset` accepts as input either a local dataset directory or a dataset name from the Hugging Face Hub.

So here you are getting this error the second time because it tries to load the local `wiki_dpr` directory, instead of `wiki_dpr` from the Hub. It doesn't work since it's a **cache** directory, not a **dataset** directory in itself.

To fix that you can use another cache directory like `cache_dir="/data2/whr/lzy/open_domain_data/retrieval/cache"`

thx a lot