helmo/github-issues huggingface dataset:
Offline loading
State: closed | Author: lhoestq (MEMBER)

As discussed in #824 it would be cool to make the library work in offline mode.
Currently if there's not internet connection then modules (datasets or metrics) that have already been loaded in the past can't be loaded and it raises a ConnectionError.
This is because `prepare_module` fetches online for the latest version of the module.

To make it work in offline mode one suggestion was to reload the latest local version of the module.
I implemented that and I also raise a warning saying that the module that is loaded is the latest local version.
```python
logger.warning(
    f"Using the latest cached version of the module from {cached_module_path} since it "
    f"couldn't be found locally at {input_path} or remotely ({error_type_that_prevented_reaching_out_remote_stuff})."
)
```

I added tests to make sure it works as expected and I needed to do a few changes in the code to be able to test things properly. In particular I added a parameter `hf_modules_cache` to `init_dynamic_modules` for testing purposes. It makes it possible to have temporary modules caches for testing.

I also added a `offline` context utility that allows to test part of the code by making all the requests fail as if there was no internet.

Close #824, close #761.

Comments: It's maybe a bit annoying to add but could we maybe have as well a version of the local data loading scripts in the package?
The `text`, `json`, `csv`. Thinking about people like in #1725 who are expecting to be able to work with local data without downloading anything.

Maybe we can add them to package_data or something?

Yes I mentioned this in #824 as well. I'm looking into it

Alright now `csv`, `json`, `text` and `pandas` are "packaged datasets", i.e. they're part of the `datasets` package, which makes them available in offline mode without any change in terms of API:
```python
from datasets import load_dataset

d = load_dataset("csv", data_files=["path/to/data.csv"])
```

Instead of loading the dataset script from the module cache, it's loaded from inside the `datasets` package.

I updated the test to still be able to fetch the dummy data files for those datasets from `datasets/{text|csv|pandas|json}/dummy` in the repo.

Alright now all test pass :)
(I don't thank you windows)

LGTM! Since you're getting the local script's last modification date anyways do you think it might be a good idea to show it in the warning?

> LGTM! Since you're getting the local script's last modification date anyways do you think it might be a good idea to show it in the warning?

Yep good idea. I added the date in the warning. For example `(last modified on Mon Nov 30 11:01:56 2020)`
load the local dataset
State: closed | Author: xinjicong (NONE)

your guidebook's example is like
>>>from datasets import load_dataset
>>> dataset = load_dataset('json', data_files='my_file.json')
but the first arg is path...
so how should i do if i want to load the local dataset for model training?
i will be grateful if you can help me handle this problem!
thanks a lot!

Comments: You should rephrase your question or give more examples and details on what you want to do.

it’s not possible to understand it and help you with only this information.

sorry for that.
i want to know how could i load the train set and the test set from the local ,which api or function should i use .


Did you try to follow the instructions in the documentation?
Here: https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files

thanks a lot 
i find that the problem is i dont use vpn...
so i have to keep my net work even if i want to load the local data ?

We will solve this soon (cf #1724)

thanks a lot

Hi! `json` is a packaged dataset now, which means its script comes with the library and doesn't require an internet connection.
could not run models on a offline server successfully
State: closed | Author: lkcao (NONE)

Hi, I really need your help about this.
I am trying to fine-tuning a RoBERTa on a remote server, which is strictly banning internet. I try to install all the packages by hand and try to run run_mlm.py on the server. It works well on colab, but when I try to run it on this offline server, it shows:
![image](https://user-images.githubusercontent.com/49967236/104276256-25a88600-546a-11eb-9776-8ec695dfa24e.png)

is there anything I can do? Is it possible to download all the things in cache and upload it to the server? Please help me out...

Comments: Transferred to `datasets` based on the stack trace.

Hi @lkcao !
Your issue is indeed related to `datasets`. In addition to installing the package manually, you will need to download the `text.py` script on your server. You'll find it (under `datasets/datasets/text`: https://github.com/huggingface/datasets/blob/master/datasets/text/text.py.
Then you can change the line 221 of `run_mlm_new.py` into:
```python
  datasets = load_dataset('/path/to/text.py', data_files=data_files)
```
Where `/path/to/text.py` is the path on the server where you saved the `text.py` script.

We're working on including the local dataset builders (csv, text, json etc.) directly in the `datasets` package so that they can be used offline

The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
You can now use them offline
```python
datasets = load_dataset('text', data_files=data_files)
```

We'll do a new release soon

> The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
> You can now use them offline
> 
> ```python
> datasets = load_dataset('text', data_files=data_files)
> ```
> 
> We'll do a new release soon

so the new version release now?

Yes it's been available since datasets 1.3.0 !
[Scientific papers] Mirror datasets zip
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

Datasets were uploading to https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/arxiv-dataset.zip and https://s3.amazonaws.com/datasets.huggingface.co/scientific_papers/1.1.1/pubmed-dataset.zip respectively to escape google drive quota and enable faster download. 

Comments: > Nice !
> 
> Could you try to reduce the size of the dummy_data.zip files ? they're quite big (300KB)

Yes, I think it might make sense to enhance the tool a tiny bit to prevent this automatically

That's the lightest I can make it...it's long-range summarization so a single sample has ~11000 tokens. 

Ok thanks :)

Awesome good to merge for me :-)
Fix column list comparison in transmit format
State: closed | Author: lhoestq (MEMBER)

As noticed in #1718 the cache might not reload the cache files when new columns were added.
This is because of an issue in `transmit_format` where the column list comparison fails because the order was not deterministic. This causes the `transmit_format` to apply an unnecessary `set_format` transform with shuffled column names.

I fixed that by sorting the columns for the comparison and added a test.

To properly test that I added a third column `col_3` to the dummy_dataset used for tests.
SciFact dataset - minor changes
State: closed | Author: dwadden (CONTRIBUTOR)

Hi,

SciFact dataset creator here. First of all, thanks for adding the dataset to Huggingface, much appreciated!

I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

Thanks,

Dave

Comments: Hi Dave,
You are more than welcome to open a PR to make these changes! 🤗
You will find the relevant information about opening a PR in the [contributing guide](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md) and in the [dataset addition guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Pinging also @lhoestq for the Google cloud matter.

> I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

Sure ! Also feel free to ping us for reviews or if we can help :)

> It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

What makes you think that ?
Afaik there's no scifact on our google storage




> > I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?
> 
> Sure ! Also feel free to ping us for reviews or if we can help :)
> 
OK! We're organizing a [shared task](https://sdproc.org/2021/sharedtasks.html#sciver) based on the dataset, and I made some updates and changed the download URL - so the current code points to a dead URL. I'll update appropriately once the task is finalized and make a PR.

> > It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?
> 
> What makes you think that ?
> Afaik there's no scifact on our google storage

You're right, I had the data cached on my machine somewhere. 



I opened a PR about this: https://github.com/huggingface/datasets/pull/1780. Closing this issue, will continue there.
Adding adversarialQA dataset
State: closed | Author: maxbartolo (CONTRIBUTOR)

Adding the adversarialQA dataset (https://adversarialqa.github.io/) from Beat the AI (https://arxiv.org/abs/2002.00293)

Comments: Oh that's a really cool one, we'll review/merge it soon!

In the meantime, do you have any specific positive/negative feedback on the process of adding a datasets Max?
Did you follow the instruction in the [detailed step-by-step](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)?

Thanks Thom, been a while, hope all is well!

Yes, I followed the step by step instructions and found them pretty straightforward. The only things I wasn't sure of were what should go into the YAML tags field for the dataset card, and whether there was a list of options somewhere (maybe akin to the metrics?) of the possible supported tasks. I found the rest very intuitive and the automated metadata and dummy data generation very handy. Thanks!

Good point! pinging @yjernite here so he can improve this part!

@maxbartolo cool addition!

For the YAML tag, you should use the tagging app we provide to choose from a drop-down menu:
https://github.com/huggingface/datasets-tagging

The process is described toward the end of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card), do you have any suggestions for making it easier to find?

Otherwise, the dataset card is really cool, thanks for making it so complete!


@yjernite

Thanks, YAML tags added. I think my main issue was with the flow of the [step-by-step guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). For example, the [card creator](https://huggingface.co/datasets/card-creator/) is introduced in Step 4, right after creating an empty directory for your dataset. The first field it requires are the YAML tags, which (at least for me) was the last step of the process.

I'd suggest having the guide structured in the same order as the creation process. For me it was something like:
- Step 1: Preparing your env
- Step 2: Write the loading/processing code
- Step 3: Automatically generate dummy data and `dataset_infos.json`
- Step 4: Tag the dataset
- Step 5: Write the dataset card using the [card creator](https://huggingface.co/datasets/card-creator/)
- Step 6: Open a Pull Request on the main HuggingFace repo and share your work!!

Thanks again!
Installation using conda
State: closed | Author: pranav-s (NONE)

Will a conda package for installing datasets be added to the huggingface conda channel? I have installed transformers using conda and would like to use the datasets library to use some of the scripts in the transformers/examples folder but am unable to do so at the moment as datasets can only be installed using pip and using pip in a conda environment is generally a bad idea in my experience.

Comments: Yes indeed the idea is to have the next release on conda cc @LysandreJik 

Great! Did you guys have a timeframe in mind for the next release?

Thank you for all the great work in developing this library.

I think we can have `datasets` on conda by next week. Will see what I can do!

Thank you. Looking forward to it.

`datasets` has been added to the huggingface channel thanks to @LysandreJik :)
It depends on conda-forge though

```
conda install -c huggingface -c conda-forge datasets
```
Silicone
State: closed | Author: eusip (CONTRIBUTOR)

My collaborators and I within the Affective Computing team at Telecom Paris would like to push our spoken dialogue dataset for publication.

Comments: When should we expect to see our dataset appear in the search dropdown at huggingface.co?

Hi @eusip,

> When should we expect to see our dataset appear in the search dropdown at huggingface.co?

when this PR is merged.

Thanks!

I've implemented all the changes requested by @lhoestq but I made the mistake of trying to change the remote branch name. 

Hopefully the changes are seen on your end as both branches `silicone` and `main` should be up-to-date.

It looks like the PR includes changes about many other files than the ones for Silicone (+30,000 line changes)

Maybe you can try to create another branch and another PR ?

> It looks like the PR includes changes about many other files than the ones for Silicone (+30,000 line changes)
> 
> Maybe you can try to create another branch and another PR ?

Sure. I will make a new pull request.
Databases
State: closed | Author: JimmyJim1 (NONE)

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
<html dir="ltr" lang="en" class="focus-outline-visible"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
State: closed | Author: Louiejay54 (NONE)

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Added generated READMEs for datasets that were missing one.
State: closed | Author: madlag (CONTRIBUTOR)

This is it: we worked on a generator with Yacine @yjernite , and we generated dataset cards for all missing ones (161), with all the information we could gather from datasets repository, and using dummy_data to generate examples when possible.

Code is available here for the moment: https://github.com/madlag/datasets_readme_generator .
We will move it to a Hugging Face repository and to https://huggingface.co/datasets/card-creator/ later.


Comments: Looks like we need to trim the ones with too many configs, will look into it tomorrow!
Some datasets miss dataset_infos.json or dummy_data.zip
State: closed | Author: madlag (CONTRIBUTOR)

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json : 

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip : 

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .



Comments: Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.


Closing since the dummy data generation is deprecated now (and the issue with missing metadata seems to be addressed).
Update DBRD dataset card and download URL
State: closed | Author: benjaminvdb (CONTRIBUTOR)

I've added the Dutch Bood Review Dataset (DBRD) during the recent sprint. This pull request makes two minor changes:

1.  I'm changing the download URL from Google Drive to the dataset's GitHub release package. This is now possible because of PR #1316.
2. I've updated the dataset card.

Cheers! 😄

Comments: not sure why the CI was not triggered though
fix ner_tag bugs in thainer
State: closed | Author: cstorm125 (CONTRIBUTOR)

fix bug that results in `ner_tag` always equal to 'O'.

Comments: > Thanks :)
> 
> Apparently the dummy_data.zip got removed. Is this expected ?
> Also can you remove the `data-pos.conll` file that you added ?

Not expected. I forgot to remove the `dummy_data` folder used to create `dummy_data.zip`. 
Changed to only `dummy_data.zip`.
Updated HuggingFace Datasets README (fix typos)
State: closed | Author: 8bitmp3 (CONTRIBUTOR)

Awesome work on 🤗 Datasets. I found a couple of small typos in the README. Hope this helps.



![](https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/google/56/hugging-face_1f917.png)
Fast start up
State: closed | Author: lhoestq (MEMBER)

Currently if optional dependencies such as tensorflow, torch, apache_beam, faiss and elasticsearch are installed, then it takes a long time to do `import datasets` since it imports all of these heavy dependencies.

To make a fast start up for `datasets` I changed that so that they are not imported when `datasets` is being imported. On my side it changed the import time of `datasets` from 5sec to 0.5sec, which is enjoyable.

To be able to check if optional dependencies are available without importing them I'm using `importlib_metadata`, which is part of the standard lib in python>=3.8 and was backported. The difference with `importlib` is that it also enables to get the versions of the libraries without importing them.
I added this dependency in `setup.py`.
Fix ade_corpus_v2 config names
State: closed | Author: lhoestq (MEMBER)

There are currently some typos in the config names of the `ade_corpus_v2` dataset, I fixed them:

- Ade_corpos_v2_classificaion -> Ade_corpus_v2_classification
- Ade_corpos_v2_drug_ade_relation -> Ade_corpus_v2_drug_ade_relation
- Ade_corpos_v2_drug_dosage_relation -> Ade_corpus_v2_drug_dosage_relation
Question: Shouldn't .info be a part of DatasetDict?
State: open | Author: KennethEnevoldsen (CONTRIBUTOR)

Currently, only `Dataset` contains the .info or .features, but as many datasets contains standard splits (train, test) and thus the underlying information is the same (or at least should be) across the datasets. 

For instance:
```
>>> ds = datasets.load_dataset("conll2002", "es")
>>> ds.info
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'DatasetDict' object has no attribute 'info'
```

I could imagine that this wouldn't work for datasets dicts which hold entirely different datasets (multimodal datasets), but it seems odd that splits of the same dataset is treated the same as what is essentially different datasets. 

Intuitively it would also make sense that if a dataset is supplied via. the load_dataset that is have a common .info which covers the entire dataset.

It is entirely possible that I am missing another perspective

Comments: We could do something. There is a part of `.info` which is split specific (cache files, split instructions) but maybe if could be made to work.

Yes this was kinda the idea I was going for. DatasetDict.info would be the shared info amongs the datasets (maybe even some info on how they differ).
Dataset Error: DaNE contains empty samples at the end
State: closed | Author: KennethEnevoldsen (CONTRIBUTOR)

The dataset DaNE, contains empty samples at the end. It is naturally easy to remove using a filter but should probably not be there, to begin with as it can cause errors.

```python
>>> import datasets
[...]
>>> dataset = datasets.load_dataset("dane")
[...]
>>> dataset["test"][-1]
{'dep_ids': [], 'dep_labels': [], 'lemmas': [], 'morph_tags': [], 'ner_tags': [], 'pos_tags': [], 'sent_id': '', 'text': '', 'tok_ids': [], 'tokens': []}
>>> dataset["train"][-1]
{'dep_ids': [], 'dep_labels': [], 'lemmas': [], 'morph_tags': [], 'ner_tags': [], 'pos_tags': [], 'sent_id': '', 'text': '', 'tok_ids': [], 'tokens': []}
```

Best,
Kenneth

Comments: Thanks for reporting, I opened a PR to fix that

One the PR is merged the fix will be available in the next release of `datasets`.

If you don't want to wait the next release you can still load the script from the master branch with

```python
load_dataset("dane", script_version="master")
```

If you have other questions feel free to reopen :)
Update README.md of covid-tweets-japanese
State: closed | Author: forest1988 (CONTRIBUTOR)

Update README.md of covid-tweets-japanese added by PR https://github.com/huggingface/datasets/pull/1367 and https://github.com/huggingface/datasets/pull/1402.

- Update "Data Splits" to be more precise that no information is provided for now.
  - old: [More Information Needed]
  - new: No information about data splits is provided for now.

- The automatic generation of links seemed not working properly, so I added a space before and after the URL to make the links work correctly.

Comments: Thanks for reviewing and merging!
Don't use xlrd for xlsx files
State: closed | Author: lhoestq (MEMBER)

Since the latest release of `xlrd` (2.0), the support for xlsx files stopped.
Therefore we needed to use something else.
A good alternative is `openpyxl` which has also an integration with pandas si we can still call `pd.read_excel`.

I left the unused import of `openpyxl` in the dataset scripts to show users that this is a required dependency to use the scripts.

I tested the different datasets using `datasets-cli test` and the tests are successful (no missing examples).
added TurkishProductReviews dataset
State: closed | Author: basakbuluz (CONTRIBUTOR)

This PR added **Turkish Product Reviews Dataset contains 235.165 product reviews collected online. There are 220.284 positive, 14881 negative reviews**.

- **Repository:** [turkish-text-data](https://github.com/fthbrmnby/turkish-text-data)
- **Point of Contact:** Fatih Barmanbay - @fthbrmnby

Comments: @lhoestq, can you please review this PR?

Thanks for the suggestions. Updates were made and dataset_infos.json file was created again.