noamaanMulla-03/datasets-issues huggingface dataset:
Can't filter language:EN on https://huggingface.co/datasets

When visiting https://huggingface.co/datasets, I don't see an obvious way to filter only English datasets. This is unexpected for me, am I missing something? I'd expect English to be selectable in the language widget. This problem reproduced on Mozilla Firefox and MS Edge:

![screenshot](https://user-images.githubusercontent.com/4547987/102792244-892e1f00-43a8-11eb-9e89-4826ca201a87.png)



cc'ing @mapmeld 

Full language list is now deployed to https://huggingface.co/datasets ! Recommend close

Cool @mapmeld ! My 2 cents (for a next iteration), it would be cool to have a small search widget in the filter dropdown as you have a ton of languages now here! Closing this in the meantime.
Add CLIMATE-FEVER dataset

As suggested by @SBrandeis , fresh PR that adds CLIMATE-FEVER. Replaces PR #1579.

---

A dataset adopting the FEVER methodology that consists of 1,535 real-world claims regarding climate-change collected on the internet. Each claim is accompanied by five manually annotated evidence sentences retrieved from the English Wikipedia that support, refute or do not give enough information to validate the claim totalling in 7,675 claim-evidence pairs. The dataset features challenging claims that relate multiple facets and disputed cases of claims where both supporting and refuting evidence are present.

More information can be found at:

* Homepage: http://climatefever.ai
* Paper: https://arxiv.org/abs/2012.00614


Thank you @lhoestq for your comments! 😄 I added your suggested changes, ran the tests and regenerated `dataset_infos.json` and `dummy_data`.
Add Pubmed (citation + abstract) dataset (2020).



LGTM! ftp addition looks fine but maybe have a look @thomwolf ?

It's not finished yet, I need to run the tests on the full dataset (it was running this weekend, there is an error somewhere deep)


@yjernite Ready for review !
@thomwolf 

So I tried to follow closely the original format that means I still had to drop information (namely tags on elements are simply discarded for now but they don't seem to carry critical information).
Some elements are also discarded they tend to not come up often:
 - The most notable is Author affiliation, which seems to be all over the place in terms of what it look meaning it's hard to actually get a consistent format.
 - Journal is the same, all the elements in there can be wildly different so I decided to drop it for now instead of trying to figure out a way to have a common presentation. (the DOI and medline ID are present so it can be recovered).

I think this PR could go as it but we probably should add a way to get easier information to use with a config.
For instance `{"title": "string", "abstract": "string", "authors": List[str], "substances": List[str]}` maybe ? (substances for instance is a tricky one, some substances only have an international identifier, some have simply a common name, some both)

It's relatively easy to do I think it's mostly discarding other fields and renaming some deep structure into a flat one.

Look ok to me but @lhoestq is the master on the Download Manager side
added air_dialogue

UPDATE2 (3797ce5): Updated for multi-configs 

UPDATE (7018082): manually created the dummy_datasets. All tests were cleared locally. Pushed it to origin/master

DRAFT VERSION (57fdb20):  (_no longer draft_)
Uploaded the air_dialogue database. 
dummy_data creation was failing in local, since the original downloaded file has some nested folders. Pushing it since the tests with real data was cleared. Will re-check & update via manually creating some dummy_data
Can't call shape on the output of select()

I get the error `TypeError: tuple expected at most 1 argument, got 2` when calling `shape` on the output of `select()`.
It's line 531 in shape in arrow_dataset.py that causes the problem:
``return tuple(self._indices.num_rows, self._data.num_columns)``
This makes sense, since `tuple(num1, num2)` is not a valid call.
 
Full code to reproduce:

```python
dataset = load_dataset("cnn_dailymail", "3.0.0")
train_set = dataset["train"]
t = train_set.select(range(10))
print(t.shape)


Indeed that's a typo, do you want to open a PR to fix it?

Yes, created a PR
How to join two datasets?

Hi,

I'm wondering if it's possible to join two (preprocessed) datasets with the same number of rows but different labels? 

I'm currently trying to create paired sentences for BERT from `wikipedia/'20200501.en`, and I couldn't figure out a way to create a paired sentence using `.map()` where the second sentence is **not** the next sentence (i.e., from a different article) of the first sentence.

Thanks!


Hi this is also my question. thanks 

Hi ! Currently the only way to add new fields to a dataset is by using `.map` and picking items from the other dataset


Closing this one. Feel free to re-open if you have other questions about this issue.

Also linking another discussion about joining datasets: #853
CUDA out of memory

In your dataset ,cuda run out of memory as long as the trainer begins:
however, without changing any other element/parameter,just switch dataset to `LineByLineTextDataset`,everything becames OK.



Could you provide more details ? What's the code you ran ?

```python
tokenizer = FunnelTokenizer.from_pretrained('funnel-transformer/small')

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True,max_length=512)

dataset = load_dataset("bookcorpus",split='train[:1000]').shuffle()
dataset = dataset.map(tokenize, batched=True, batch_size=512)

# dataset = LineByLineTextDataset(
#     tokenizer=tokenizer,
#     file_path="./wiki1000.txt",
#     block_size=128
# )

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

config=FunnelConfig(
    return_dict=True
)

model= FunnelForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="./checkpoints",
    overwrite_output_dir=True,
    do_train=True,
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    save_steps=10000,
    logging_dir='./ptlogs'
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
trainer.train()
```

`RuntimeError: CUDA out of memory. Tried to allocate 954.00 MiB (GPU 0; 15.90 GiB total capacity; 14.35 GiB already allocated; 753.75 MiB free; 14.39 GiB reserved in total by PyTorch)
Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):`

part of error output

from funnel model to bert model : error still happened

from your dataset to LineByLineTextDataset : error disapeared

notice i just loaded 1000 rows of data

the error happens when executing loss.backward()

Since you're using a data collator you don't need to tokenizer the dataset using `map`. Could you try not to use `map` and only the data collator instead ? The data collator is supposed to pad to the longest sequence in each batch afaik, instead of padding to 512.

Also cc @sgugger 

Closing this one.
Feel free to re-open if you have other questions about this issue
Shall we change the hashing to encoding to reduce potential replicated cache files?

Hi there. For now, we are using `xxhash` to hash the transformations to fingerprint and we will save a copy of the processed dataset to disk if there is a new hash value. However, there are some transformations that are idempotent or commutative to each other. I think that encoding the transformation chain as the fingerprint may help in those cases, for example, use `base64.urlsafe_b64encode`. In this way, before we want to save a new copy, we can decode the transformation chain and normalize it to prevent omit potential reuse. As the main targets of this project are the really large datasets that cannot be loaded entirely in memory, I believe it would save a lot of time if we can avoid some write.

If you have interest in this, I'd love to help :).

Labels: name: enhancement

name: generic discussion

This is an interesting idea !
Do you have ideas about how to approach the decoding and the normalization ?

@lhoestq
I think we first need to save the transformation chain to a list in `self._fingerprint`. Then we can
- decode all the current saved datasets to see if there is already one that is equivalent to the transformation we need now.
- or, calculate all the possible hash value of the current chain for comparison so that we could continue to use hashing.
If we find one, we can adjust the list in `self._fingerprint` to it.

As for the transformation reordering rules, we can just start with some manual rules, like two sort on the same column should merge to one, filter and select can change orders.

And for encoding and decoding, we can just manually specify `sort` is 0, `shuffling` is 2 and create a base-n number or use some general algorithm like `base64.urlsafe_b64encode`.

Because we are not doing lazy evaluation now, we may not be able to normalize the transformation to its minimal form. If we want to support that, we can provde a `Sequential` api and let user input a list or transformation, so that user would not use the intermediate datasets. This would look like tf.data.Dataset.
added TurkishMovieSentiment dataset

This PR adds the **TurkishMovieSentiment: This dataset contains turkish movie reviews.**

- **Homepage:** [https://www.kaggle.com/mustfkeskin/turkish-movie-sentiment-analysis-dataset/tasks](https://www.kaggle.com/mustfkeskin/turkish-movie-sentiment-analysis-dataset/tasks)
- **Point of Contact:** [Mustafa Keskin](https://www.linkedin.com/in/mustfkeskin/)


> I just generated the dataset_infos.json file
> 
> Thanks for adding this one !

Thank you very much for your support.
Dataset social_bias_frames 404

```
>>> from datasets import load_dataset
>>> dataset = load_dataset("social_bias_frames")
...
Downloading and preparing dataset social_bias_frames/default
...
~/.pyenv/versions/3.7.6/lib/python3.7/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag)
    484             )
    485         elif response is not None and response.status_code == 404:
--> 486             raise FileNotFoundError("Couldn't find file at {}".format(url))
    487         raise ConnectionError("Couldn't reach {}".format(url))
    488 

FileNotFoundError: Couldn't find file at https://homes.cs.washington.edu/~msap/social-bias-frames/SocialBiasFrames_v2.tgz
```
[Here](https://homes.cs.washington.edu/~msap/social-bias-frames/) we find button `Download data` with the correct URL for the data: https://homes.cs.washington.edu/~msap/social-bias-frames/SBIC.v2.tgz


I see, master is already fixed in https://github.com/huggingface/datasets/commit/9e058f098a0919efd03a136b9b9c3dec5076f626
added conv_ai_3 dataset

Dataset : https://github.com/aliannejadi/ClariQ/



@lhoestq Thank you for suggesting changes. I fixed all the changes you suggested. Can you please review it again? 

@lhoestq Thank you for reviewing and suggesting changes. I made the requested changes. Can you please review it again?

Thanks @lhoestq for reviewing it again. I made the required changes. Can you please have a look ?

merging since the CI is fixed on master
Lama

This the LAMA dataset for probing facts and common sense from language models. 

See https://github.com/facebookresearch/LAMA for more details.


Let me know why the pyarrow test is failing. For one of the config "trex", I had to load an initial datafile for a dictionary which is used to augment the rest of the datasets. In the dummy data, the dictionary file was truncated so I had to fudge that. I'm not sure if that is the issue.


@ontocord it just needs a rerun and it will be good to go.

THanks @tanmoyio. How do I do a rerun?

@ontocord contributor can’t rerun it, the maintainers will rerun it, it may take lil bit of time as there are so many PRs left to be reviewed and merged 

@lhoestq not sure why it is failing. i've made all modifications. 

merging since the CI is fixed on master
data loader for reading comprehension task

added doc2dial data loader and dummy data for reading comprehension task.


Thank you for all the feedback! I have updated the dummy data with a zip under 30KB, which needs to include at least one data instance from both document domain and dialog domain. Please let me know if it is still too big. Thanks!

Thank you again for the feedback! I am not too sure what the preferable style for data instance in readme, but still added my edits. Thanks!
ADD COVID-QA dataset

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA


merging since the CI is fixed on master

Wow, thanks for including this dataset from my side as well!
Add Senti_Lex Dataset

TODO:
Fix feature format issue
Create dataset_info.json file
Run pytests
Make Style


Better will be if you close this PR and make a fresh PR

Feel free to ping me if you also have questions about the dummy data

also it looks like this PR includes changes about dummy_data.zip files in the ./datasets//un_pc folder. Can you remove them ?

Thanks for all the advice @lhoestq. I've implemented the changes you kindly highlighted and have made sure the scripts pass all the test. I've also marked this as ready for review as I believe it's in a good place to be merged now.

Great suggestion I fixed the dummy data and the file paths.
Fix "'BertTokenizerFast' object has no attribute 'max_len'"

Tensorflow 2.3.0 gives:
 FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.

Tensorflow 2.4.0 gives:
AttributeError 'BertTokenizerFast' object has no attribute 'max_len'
social_i_qa wrong format of labels

Hi,
there is extra "\n" in labels of social_i_qa datasets, no big deal, but I was wondering if you could remove it to make it consistent.
so label is 'label': '1\n', not '1'
thanks

```
>>> import datasets 
>>> from datasets import load_dataset
>>> dataset = load_dataset(
...    'social_i_qa')
cahce dir  /julia/cache/datasets
Downloading: 4.72kB [00:00, 3.52MB/s]                                                                                                  
cahce dir /julia/cache/datasets
Downloading: 2.19kB [00:00, 1.81MB/s]                                                                                                  
Using custom data configuration default
Reusing dataset social_i_qa (/julia/datasets/social_i_qa/default/0.1.0/4a4190cc2d2482d43416c2167c0c5dccdd769d4482e84893614bd069e5c3ba06)
>>> dataset['train'][0]
{'answerA': 'like attending', 'answerB': 'like staying home', 'answerC': 'a good friend to have', 'context': 'Cameron decided to have a barbecue and gathered her friends together.', 'label': '1\n', 'question': 'How would Others feel as a result?'}

```




@lhoestq, should I raise a PR for this? Just a minor change while reading labels text file

Sure feel free to open a PR thanks !
wiki_dpr dataset pre-processesing performance

I've been working with wiki_dpr and noticed that the dataset processing is seriously impaired in performance [1]. It takes about 12h to process the entire dataset. Most of this time is simply loading and processing the data, but the actual indexing is also quite slow (3h).

I won't repeat the concerns around multiprocessing as they are addressed in other issues (#786), but this is the first obvious thing to do. Using cython to speed up the text manipulation may be also help. Loading and processing a dataset of this size in under 15 minutes does not seem unreasonable on a modern multi-core machine. I have hit such targets myself on similar tasks. Would love to see this improve.

The other issue is that it takes 3h to construct the FAISS index. If only we could use GPUs with HNSW, but we can't. My sharded GPU indexing code can build an IVF + PQ index in 10 minutes on 20 million vectors. Still, 3h seems slow even for the CPU.

It looks like HF is adding only 1000 vectors at a time by default [2], whereas the faiss benchmarks adds 1 million vectors at a time (effectively) [3]. It's possible the runtime could be reduced with a larger batch. Also, it looks like project dependencies ultimately use OpenBLAS, but this is known to have issues when combined with OpenMP, which HNSW does [3]. A workaround is to set the environment variable `OMP_WAIT_POLICY=PASSIVE` via `os.environ` or similar.

References:
[1] https://github.com/huggingface/datasets/blob/master/datasets/wiki_dpr/wiki_dpr.py
[2] https://github.com/huggingface/datasets/blob/master/src/datasets/search.py
[3] https://github.com/facebookresearch/faiss/blob/master/benchs/bench_hnsw.py
[4] https://github.com/facebookresearch/faiss/issues/422


Sorry, double posted.
Fix NER metric example in Overview notebook

Fix errors in `NER metric example` section in `Overview.ipynb`.

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-37-ee559b166e25> in <module>()
----> 1 ner_metric = load_metric('seqeval')
      2 references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      3 predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      4 ner_metric.compute(predictions, references)


/usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    340     if needs_to_be_installed:
    341         raise ImportError(
--> 342             f"To be able to use this {module_type}, you need to install the following dependencies"
    343             f"{[lib_name for lib_name, lib_path in needs_to_be_installed]} using 'pip install "
    344             f"{' '.join([lib_path for lib_name, lib_path in needs_to_be_installed])}' for instance'"

ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'
```

```
ValueError                                Traceback (most recent call last)
<ipython-input-39-ee559b166e25> in <module>()
      2 references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
      3 predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
----> 4 ner_metric.compute(predictions, references)

/usr/local/lib/python3.6/dist-packages/datasets/metric.py in compute(self, *args, **kwargs)
    378         """
    379         if args:
--> 380             raise ValueError("Please call `compute` using keyword arguments.")
    381 
    382         predictions = kwargs.pop("predictions", None)

ValueError: Please call `compute` using keyword arguments.
```
Use full released xsum dataset

#672 Fix xsum to expand coverage and include IDs
Code based on parser from older version of `datasets/xsum/xsum.py`
@lhoestq 


@lhoestq I took a shot at addressing your comments but the build scripts seem to be complaining about not being able to open dummy files. How do I resolve those errors without copying the full dataset into the dummy dir?

Could you check that the names of the dummy data files are right ?
You can use 
```
datasets-cli dummy_data ./datasets/xum
```
to print the expected file names

Ok @lhoestq looks like I got the tests to pass :)
Loss not decrease with Datasets and Transformers

HI,

The following script is used to fine-tune a BertForSequenceClassification model on SST2.

The script is adapted from [this colab](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) that presents an example of fine-tuning BertForQuestionAnswering using squad dataset. In that colab, loss works fine. When I adapt it to SST2, the loss fails to decrease as it should. I attach the adapted script below and appreciate anyone pointing out what I miss?

```python
import torch
from datasets import load_dataset
from transformers import BertForSequenceClassification
from transformers import BertTokenizerFast
# Load our training dataset and tokenizer
dataset = load_dataset("glue", 'sst2')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
del dataset["test"] # let's remove it in this demo

# Tokenize our training dataset
def convert_to_features(example_batch):
    encodings = tokenizer(example_batch["sentence"])
    encodings.update({"labels": example_batch["label"]})
    return encodings

encoded_dataset = dataset.map(convert_to_features, batched=True)
# Format our dataset to outputs torch.Tensor to train a pytorch model
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'labels']
encoded_dataset.set_format(type='torch', columns=columns)

# Instantiate a PyTorch Dataloader around our dataset
# Let's do dynamic batching (pad on the fly with our own collate_fn)
def collate_fn(examples):
    return tokenizer.pad(examples, return_tensors='pt')

dataloader = torch.utils.data.DataLoader(encoded_dataset['train'], collate_fn=collate_fn, batch_size=8)
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# Let's load a pretrained Bert model and a simple optimizer
model = BertForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')


```
In case needed.

- datasets == 1.0.2
- transformers == 3.2.0


And I tested it on T5ForConditionalGeneration, that works no problem.

Hi did you manage to fix your issue ?

If so feel free to share your fix and close this thread
adding  ted_talks_iwslt

UPDATE2: (2nd Jan) Wrote a long writeup on the slack channel. I don't think this approach is correct. Basically this created language pairs (109*108) 
Running the `pytest `went for more than 40+ hours and it was still running!  
So working on a different approach, such that the number of configs = number of languages. Will make a new pull request with that. 

UPDATE: This requires manual download dataset

This is a draft version 


Closing this with reference to the new approach #1676
Switchboard Dialog Act Corpus added under `datasets/swda`

Pleased to announced that I added my first dataset **Switchboard Dialog Act Corpus**.


I think this is an important datasets to be added since it is the only one related to dialogue act classification. 

Hope the pull request is ok. Wasn't able to see any special formatting for the pull request form.


The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2,
with turn/utterance-level dialog-act tags. The tags summarize syntactic, semantic, and pragmatic information
about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.


[webpage](http://compprag.christopherpotts.net/swda.html)

[repo](https://github.com/NathanDuran/Switchboard-Corpus/raw/master/swda_data/)

Please contact me for any support!

All tests passed and followed all steps in the contribution guide!




Need to fix code formatting.