SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [GEM] add WikiLingua cross-lingual abstractive summarization dataset

Comment:
Hi @KMFODA ! A version of WikiLingua is actually already accessible in the [GEM dataset](https://huggingface.co/datasets/gem)

You can use it for example to load the French to English translation with:
```python
from datasets import load_dataset
wikilingua = load_dataset("gem", "wiki_lingua_french_fr")
```

Closed by https://github.com/huggingface/datasets/pull/1807
Issue: [GEM] MultiWOZ dialogue dataset

Comment:
Hi @yjernite can I help in adding this dataset? 

I am excited about this because this will be my first contribution to the datasets library as well as to hugginface.
Issue: Discussion using datasets in offline mode

Comment:
I think it would be very cool. I'm currently working on a cluster from Compute Canada, and I have internet access only when I'm not in the nodes where I run the scripts. So I was expecting to be able to use the wmt14 dataset until I realized I needed internet connection even if I downloaded the data already. I'm going to try option 2 you mention for now though! Thanks ;)
Issue: Discussion using datasets in offline mode

Comment:
Requiring online connection is a deal breaker in some cases unfortunately so it'd be great if offline mode is added similar to how `transformers` loads models offline fine.

@mandubian's second bullet point suggests that there's a workaround allowing you to use your offline (custom?) dataset with `datasets`. Could you please elaborate on how that should look like?
Issue: Discussion using datasets in offline mode

Comment:
here is my way to load a dataset offline, but it **requires** an online machine
1. (online machine)
```
import datasets
data = datasets.load_dataset(...)
data.save_to_disk(/YOUR/DATASET/DIR)
```
2. copy the dir from online to the offline machine
3. (offline machine)
```
import datasets
data = datasets.load_from_disk(/SAVED/DATA/DIR)
```

HTH.
Issue: Discussion using datasets in offline mode

Comment:
> here is my way to load a dataset offline, but it **requires** an online machine
> 
> 1. (online machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_dataset(...)
> 
> data.save_to_disk(/YOUR/DATASET/DIR)
> 
> ```
> 
> 2. copy the dir from online to the offline machine
> 
> 3. (offline machine)
> 
> ```
> 
> import datasets
> 
> data = datasets.load_from_disk(/SAVED/DATA/DIR)
> 
> ```
> 
> 
> 
> HTH.
Issue: Discussion using datasets in offline mode

Comment:
I opened a PR that allows to reload modules that have already been loaded once even if there's no internet.

Let me know if you know other ways that can make the offline mode experience better. I'd be happy to add them :) 

I already note the "freeze" modules option, to prevent local modules updates. It would be a cool feature.

----------

> @mandubian's second bullet point suggests that there's a workaround allowing you to use your offline (custom?) dataset with `datasets`. Could you please elaborate on how that should look like?

Indeed `load_dataset` allows to load remote dataset script (squad, glue, etc.) but also you own local ones.
For example if you have a dataset script at `./my_dataset/my_dataset.py` then you can do
```python
load_dataset("./my_dataset")
```
and the dataset script will generate your dataset once and for all.

----------

About I'm looking into having `csv`, `json`, `text`, `pandas` dataset builders already included in the `datasets` package, so that they are available offline by default, as opposed to the other datasets that require the script to be downloaded.
cf #1724
Issue: Discussion using datasets in offline mode

Comment:
The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
You can now use them offline
```python
datasets = load_dataset('text', data_files=data_files)
```

We'll do a new release soon
Issue: how processing in batch works in datasets 

Comment:
Hi I don’t think this is a request for a dataset like you labeled it.

I also think this would be better suited for the forum at https://discuss.huggingface.co. we try to keep the issue for the repo for bug reports and new features/dataset requests and have usage questions discussed on the forum. Thanks.
Issue: how processing in batch works in datasets 

Comment:
Hi Thomas,
what I do not get from documentation is that why when you set batched=True,
this is processed in batch, while data is not divided to batched
beforehand, basically this is a question on the documentation and I do not
get the batched=True, but sure, if you think this is more appropriate in
forum I will post it there.
thanks
Best
Rabeeh

On Tue, Nov 10, 2020 at 12:21 PM Thomas Wolf <notifications@github.com>
wrote:

> Hi I don’t think this is a request for a dataset like you labeled it.
>
> I also think this would be better suited for the forum at
> https://discuss.huggingface.co. we try to keep the issue for the repo for
> bug reports and new features/dataset requests and have usage questions
> discussed on the forum. Thanks.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/823#issuecomment-724639476>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH4FIPFHVVUHANAE4F3SPEO2JANCNFSM4TQQVEXQ>
> .
>
Issue: how processing in batch works in datasets 

Comment:
Yes the forum is perfect for that. You can post in the `datasets` section.
Thanks a lot!
Issue: datasets freezes 

Comment:
Pytorch is unable to convert strings to tensors unfortunately.
You can use `set_format(type="torch")` on columns that can be converted to tensors, such as token ids.

This makes me think that we should probably raise an error or at least a warning when one tries to create pytorch tensors out of text columns
Issue: datasets freezes 

Comment:
Ultimately, we decided to return a list instead of an error when formatting a string column with the format type `"torch"`.

If you think an error would be more appropriate, please open a new issue.
Issue: [Caching] Dill globalvars() output order is not deterministic and can cause cache issues.

Comment:
To show the issue:
```
python -c "from datasets.fingerprint import Hasher; a=[]; func = lambda : len(a); print(Hasher.hash(func))"
```
doesn't always return the same ouput since `globs` is a dictionary with "a" and "len" as keys but sometimes not in the same order
Issue: Is dataset iterative or not?

Comment:
Hello !
Could you give more details ?

If you mean iter through one dataset then yes, `Dataset` object does implement the `__iter__` method so you can use 
```python
for example in dataset:
    # do something
```

If you want to iter through several datasets you can first concatenate them
```python
from datasets import concatenate_datasets

new_dataset = concatenate_datasets([dataset1, dataset2])
```
Let me know if this helps !
Issue: Is dataset iterative or not?

Comment:
Hi Huggingface/Datasets team,
I want to use the datasets inside Seq2SeqDataset here
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
and there I need to return back each line from the datasets and I am not
sure how to access each line and implement this?
It seems it also has get_item attribute? so I was not sure if this is
iterative dataset? or if this is non-iterable datasets?
thanks.



On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hello !
> Could you give more details ?
>
> If you mean iter through one dataset then yes, Dataset object does
> implement the __iter__ method so you can use
>
> for example in dataset:
>     # do something
>
> If you want to iter through several datasets you can first concatenate them
>
> from datasets import concatenate_datasets
> new_dataset = concatenate_datasets([dataset1, dataset2])
>
> Let me know if this helps !
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
> .
>
Issue: Is dataset iterative or not?

Comment:
could you tell me please if datasets also has __getitem__ any idea on how
to integrate it with Seq2SeqDataset is appreciated thanks

On Mon, Nov 9, 2020 at 10:22 AM Rabeeh Karimi Mahabadi <rabeeh@google.com>
wrote:

> Hi Huggingface/Datasets team,
> I want to use the datasets inside Seq2SeqDataset here
> https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
> and there I need to return back each line from the datasets and I am not
> sure how to access each line and implement this?
> It seems it also has get_item attribute? so I was not sure if this is
> iterative dataset? or if this is non-iterable datasets?
> thanks.
>
>
>
> On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
> wrote:
>
>> Hello !
>> Could you give more details ?
>>
>> If you mean iter through one dataset then yes, Dataset object does
>> implement the __iter__ method so you can use
>>
>> for example in dataset:
>>     # do something
>>
>> If you want to iter through several datasets you can first concatenate
>> them
>>
>> from datasets import concatenate_datasets
>> new_dataset = concatenate_datasets([dataset1, dataset2])
>>
>> Let me know if this helps !
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
>> .
>>
>
Issue: Is dataset iterative or not?

Comment:
`datasets.Dataset` objects implement indeed `__getitem__`. It returns a dictionary with one field per column.

We've not added the integration of the datasets library for the seq2seq utilities yet. The current seq2seq utilities are based on text files.

However as soon as you have a `datasets.Dataset` with columns "tgt_texts" (str), "src_texts" (str), and "id" (int) you should be able to implement your own Seq2SeqDataset class that wraps your dataset object. Does that make sense to you ?
Issue: Is dataset iterative or not?

Comment:
Hi
I am sorry for asking it multiple times but I am not getting the dataloader
type, could you confirm if the dataset library returns back an iterable
type dataloader or a mapping type one where one has access to __getitem__,
in the former case, one can iterate with __iter__, and how I can configure
it to return the data back as the iterative type? I am dealing with
large-scale datasets and  I do not want to bring all in memory
thanks for your help
Best regards
Rabeeh

On Mon, Nov 9, 2020 at 11:17 AM Quentin Lhoest <notifications@github.com>
wrote:

> datasets.Dataset objects implement indeed __getitem__. It returns a
> dictionary with one field per column.
>
> We've not added the integration of the datasets library for the seq2seq
> utilities yet. The current seq2seq utilities are based on text files.
>
> However as soon as you have a datasets.Dataset with columns "tgt_texts"
> (str), "src_texts" (str), and "id" (int) you should be able to implement
> your own Seq2SeqDataset class that wraps your dataset object. Does that
> make sense ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723915556>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYOC22EM7F666BZSOTSO66R3ANCNFSM4TPB7OWA>
> .
>
Issue: Is dataset iterative or not?

Comment:
`datasets.Dataset` objects are both iterative and mapping types: it has both `__iter__` and `__getitem__`
For example you can do
```python
for example in dataset:
    # do something
```
or
```python
for i in range(len(dataset)):
    example = dataset[i]
    # do something
```
When you do that, one and only one example is loaded into memory at a time.
Issue: Is dataset iterative or not?

Comment:
Hi there, 
Here is what I am trying, this is not working for me in map-style datasets, could you please tell me how to use datasets with being able to access ___getitem__ ? could you assist me please correcting this example? I need map-style datasets which is formed from concatenation of two datasets from your library. thanks 


```
import datasets
dataset1 = load_dataset("squad", split="train[:10]")
dataset1 = dataset1.map(lambda example: {"src_texts": "question: {0} context: {1} ".format(
    example["question"], example["context"]),
    "tgt_texts": example["answers"]["text"][0]}, remove_columns=dataset1.column_names)
dataset2 = load_dataset("imdb", split="train[:10]")
dataset2 = dataset2.map(lambda example: {"src_texts": "imdb: " + example["text"],
            "tgt_texts": str(example["label"])}, remove_columns=dataset2.column_names)
train_dataset = datasets.concatenate_datasets([dataset1, dataset2])
train_dataset.set_format(type='torch', columns=['src_texts', 'tgt_texts'])
dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
for id, batch in enumerate(dataloader):
    print(batch)

```
Issue: How to implement DistributedSampler with datasets 

Comment:
Hi Apparently I need to shard the data and give one host a chunk, could you provide me please with examples on how to do it? I want to use it jointly with finetune_trainer.py in huggingface repo seq2seq examples. thanks.
Issue: Too much logging 

Comment:
Hi ! Thanks for reporting :) 
I agree these one should be hidden when the logging level is warning, we'll fix that
Issue: Too much logging 

Comment:
+1, the amount of logging is excessive.

Most of it indeed comes from `filelock.py`, though there are occasionally messages from other sources too. Below is an example (all of these messages were logged after I already called `datasets.logging.set_verbosity_error()`)

```
I1109 21:26:01.742688 139785006901056 filelock.py:318] Lock 139778216292192 released on /home/kitaev/.cache/huggingface/datasets/9ed4f2e133395826175a892c70611f68522c7bc61a35476e8b51a31afb76e4bf.e6f3e3f3e3875a07469d1cfd32e16e1d06b149616b11eef2d081c43d515b492d.py.lock
I1109 21:26:01.747898 139785006901056 filelock.py:274] Lock 139778216290176 acquired on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748258 139785006901056 filelock.py:318] Lock 139778216290176 released on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748412 139785006901056 filelock.py:274] Lock 139778215853024 acquired on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748497 139785006901056 filelock.py:318] Lock 139778215853024 released on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:07:17.029001 140301730502464 filelock.py:274] Lock 140289479304360 acquired on /home/kitaev/.cache/huggingface/datasets/b16d3a04bf2cad1346896852bf120ba846ea1bebb1cd60255bb3a1a2bbcc3a67.ec871b06a00118091ec63eff0a641fddcb8d3c7cd52e855bbb2be28944df4b82.py.lock
I1109 21:07:17.029341 140301730502464 filelock.py:318] Lock 140289479304360 released on /home/kitaev/.cache/huggingface/datasets/b16d3a04bf2cad1346896852bf120ba846ea1bebb1cd60255bb3a1a2bbcc3a67.ec871b06a00118091ec63eff0a641fddcb8d3c7cd52e855bbb2be28944df4b82.py.lock
I1109 21:07:17.058964 140301730502464 filelock.py:274] Lock 140251889388120 acquired on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.060933 140301730502464 filelock.py:318] Lock 140251889388120 released on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.061067 140301730502464 filelock.py:274] Lock 140296072521488 acquired on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.069736 140301730502464 metric.py:400] Removing /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow
I1109 21:07:17.069949 140301730502464 filelock.py:318] Lock 140296072521488 released on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
```
Issue: Too much logging 

Comment:
In the latest version of the lib the logs about locks are at the DEBUG level so you won't see them by default.
Also `set_verbosity_warning` does take into account these logs now.
Can you try to update the lib ?
```
pip install --upgrade datasets
```
Issue: Too much logging 

Comment:
Thanks. For some reason I have to use the older version. Is that possible I can fix this by some surface-level trick?

I'm still using 1.13 version datasets.
Issue: Add Google Taskmaster dataset

Comment:
Hey @yjernite. Was going to start working on this but found taskmaster 1,2 & 3 in the datasets library already so think this can be closed now?
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:
Hi !
The url works on my side.

Is the url working in your navigator ?
Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:
> Hi !
> The url works on my side.
> 
> Is the url working in your navigator ?
> Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?

I tried another server, it's working now. Thanks a lot.

And I'm curious about why download things from "github" when I load dataset from local files ?  Dose datasets work if my network crashed?
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:



> > Hi !
> > The url works on my side.
> > Is the url working in your navigator ?
> > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> 
> I tried another server, it's working now. Thanks a lot.
> 
> And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?

I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？ 

Thanks :D
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:
hello, how did you solve this problems?

> > > Hi !
> > > The url works on my side.
> > > Is the url working in your navigator ?
> > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > 
> > 
> > I tried another server, it's working now. Thanks a lot.
> > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> 
> I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> 
> Thanks :D

hello, I tried this. but it still failed. how do you fix this error?
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:
> hello, how did you solve this problems?
> 
> > > > Hi !
> > > > The url works on my side.
> > > > Is the url working in your navigator ?
> > > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > > 
> > > 
> > > I tried another server, it's working now. Thanks a lot.
> > > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> > 
> > 
> > I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> > Thanks :D
> 
> hello, I tried this. but it still failed. how do you fix this error?

你把那个脚本下载到你本地安装目录下，然后 `load_dataset(csv_script_path, data_fiels)`
Issue: load_dataset for LOCAL CSV files report CONNECTION ERROR

Comment:
> > hello, how did you solve this problems?
> > > > > Hi !
> > > > > The url works on my side.
> > > > > Is the url working in your navigator ?
> > > > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > > > 
> > > > 
> > > > I tried another server, it's working now. Thanks a lot.
> > > > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> > > 
> > > 
> > > I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> > > Thanks :D
> > 
> > 
> > hello, I tried this. but it still failed. how do you fix this error?
> 
> 你把那个脚本下载到你本地安装目录下，然后 `load_dataset(csv_script_path, data_fiels)`

好的好的！解决了，感谢感谢！！！