SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: [Question & Bug Report] Can we preprocess a dataset on the fly?

Comment:
Indeed I will submit a PR in a fez days to enable processing on-the-fly :)
This can be useful in language modeling for tokenization, padding etc.
Issue: [Question & Bug Report] Can we preprocess a dataset on the fly?

Comment:
Hi @acul3,

Please look at the discussion on a related Issue #1825. I think using `set_transform` after building from source should do.
Issue: Efficient ways to iterate the dataset

Comment:
It seems that selecting a subset of colums directly from the dataset, i.e., dataset["column"], is slow.
Issue: is it possible to make slice to be more compatible like python list and numpy?

Comment:
Hi ! Thanks for reporting.
I am working on changes in the way data are sliced from arrow. I can probably fix your issue with the changes I'm doing.
If you have some code to reproduce the issue it would be nice so I can make sure that this case will be supported :)
I'll make a PR in a few days
Issue: bug in loading datasets 

Comment:
Looks like an issue with your csv file. Did you use the right delimiter ?
Apparently at line 37 the CSV reader from pandas reads 2 fields instead of 1.
Issue: bug in loading datasets 

Comment:
Note that you can pass any argument you would pass to `pandas.read_csv` as kwargs to `load_dataset`. For example you can do
```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=data_files, sep="\t")
```

for example to use a tab separator.

You can see the full list of arguments here: https://github.com/huggingface/datasets/blob/master/src/datasets/packaged_modules/csv/csv.py

(I've not found the list in the documentation though, we definitely must add them !)
Issue: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

Comment:
Indeed in 1.2.1 the script to process csv file is downloaded. Starting from the next release though we include the csv processing directly in the library.
See PR #1726 
We'll do a new release soon :)
Issue: how can I combine 2 dataset with different/same features?

Comment:
Hi ! Currently we don't have a way to `zip` datasets but we plan to add this soon :)
For now you'll need to use `map` to add the fields from one dataset to the other. See the comment here for more info : https://github.com/huggingface/datasets/issues/853#issuecomment-727872188
Issue: how can I combine 2 dataset with different/same features?

Comment:
Good to hear.
Currently I did not use map , just fetch src and tgt from the 2 dataset and merge them.
It will be a release if you can deal with it at the backend.
Thanks.
Issue: how can I combine 2 dataset with different/same features?

Comment:
Hi! You can rename the columns and concatenate the datasets along `axis=1` to get the desired result as follows:
```python
ds1 = ds1.rename_column("text", "src")
ds2 = ds2.rename_column("text", "tgt")
ds = datasets.concatenate_datasets(["ds1", "ds2"], axis=1)
```
Issue: _pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union when calling datasets.map with num_proc=2

Comment:
Hi ! What version of python and datasets do you have ? And also what version of dill and pickle ?
Issue: _pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union when calling datasets.map with num_proc=2

Comment:
> Hi ! What version of python and datasets do you have ? And also what version of dill and pickle ?

python==3.6.10
datasets==1.2.1
dill==0.3.2
pickle.format_version==4.0
Issue: _pickle.PicklingError: Can't pickle typing.Union[str, NoneType]: it's not the same object as typing.Union when calling datasets.map with num_proc=2

Comment:
Multiprocessing in python require all the functions to be picklable. More specifically, functions need to be picklable with `dill`.

However objects like `typing.Union[str, NoneType]` are not picklable in python <3.7.
Can you try to update your python version to python>=3.7 ?
Issue: Issues when run two programs compute the same metrics

Comment:
Hi ! To avoid collisions you can specify a `experiment_id` when instantiating your metric using `load_metric`. It will replace "default_experiment" with the experiment id that you provide in the arrow filename. 

Also when two `experiment_id` collide we're supposed to detect it using our locking mechanism. Not sure why it didn't work in your case. Could you share some code that reproduces the issue ? This would help us investigate.
Issue: Issues when run two programs compute the same metrics

Comment:
Thank you for your response. I fixed the issue by set "keep_in_memory=True" when load_metric. 
I cannot share the entire source code but below is the wrapper I wrote:

```python
class Evaluation:
    def __init__(self, metric='sacrebleu'):
        # self.metric = load_metric(metric, keep_in_memory=True)
        self.metric = load_metric(metric)

    def add(self, predictions, references):
        self.metric.add_batch(predictions=predictions, references=references)

    def compute(self):
        return self.metric.compute()['score']
```

Then call the given wrapper as follows:

```python
eval = Evaluation(metric='sacrebleu')
for query, candidates,  labels in tqdm(dataset):
    predictions = net.generate(query)
    references = [[s] for s in labels]
    eval.add(predictions, references)
    if n % 100 == 0:
        bleu += eval.compute()
        eval = Evaluation(metric='sacrebleu')
Issue: Error iterating over Dataset with DataLoader

Comment:
Instead of:
```python
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
```
It should be:
```python
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32)
```

`batch_sampler` accepts a Sampler object or an Iterable, so you get an error.
Issue: Error iterating over Dataset with DataLoader

Comment:
@mariosasko I thought that would fix it, but now I'm getting a different error:

```
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py:851: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.tensor(x, **format_kwargs)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-3af1d82bf93a> in <module>()
      1 dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32)
----> 2 next(iter(dataloader))

5 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: stack expects each tensor to be equal size, but got [7] at entry 0 and [10] at entry 1
```

Any thoughts what this means?I Do I need padding?
Issue: Error iterating over Dataset with DataLoader

Comment:
Yes, padding is an answer. 

This can be solved easily by passing a callable to the collate_fn arg of DataLoader that adds padding.
Issue: Error iterating over Dataset with DataLoader

Comment:
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=4)
batch = next(iter(dataloader))

getting 
ValueError: cannot reshape array of size 8192 into shape (1,512,4)

I had put padding as 2048 for encoded_dataset
kindly help
Issue: Error iterating over Dataset with DataLoader

Comment:
data_loader_val = torch.utils.data.DataLoader(val_dataset, batch_size=32, shuffle=True, drop_last=False, num_workers=0)
dataiter = iter(data_loader_val)
images, _ = next(dataiter)

getting -> TypeError: 'list' object is not callable

Cannot iterate through the data. Kindly suggest.
Issue: Unable to format dataset to CUDA Tensors

Comment:
Hi ! You can get CUDA tensors with

```python
dataset.set_format("torch", columns=columns, device="cuda")
```

Indeed `set_format` passes the `**kwargs` to `torch.tensor`
Issue: Unable to format dataset to CUDA Tensors

Comment:
Hi @lhoestq,

Thanks a lot. Is this true for all format types?

As in, for 'torch', I can have `**kwargs` to `torch.tensor` and for 'tf' those args are passed to `tf.Tensor`, and the same for 'numpy' and 'pandas'?
Issue: Unable to format dataset to CUDA Tensors

Comment:
Yes the keywords arguments are passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
We don't support the kwargs for pandas on the other hand.
Issue: Unable to format dataset to CUDA Tensors

Comment:
Thanks @lhoestq,
Would it be okay if I added this to the docs and made a PR?
Issue: wikipedia dataset incomplete

Comment:
Hi !
From what pickle file fo you get this ?
I guess you mean the dataset loaded using `load_dataset` ?
Issue: wikipedia dataset incomplete

Comment:
yes sorry, I used the `load_dataset`function and saved the data to a pickle file so I don't always have to reload it and are able to work offline.
Issue: wikipedia dataset incomplete

Comment:
The wikipedia articles are processed using the `mwparserfromhell` library. Even if it works well in most cases, such issues can happen unfortunately. You can find the repo here: https://github.com/earwig/mwparserfromhell

There also exist other datasets based on wikipedia that were processed differently (and are often cleaner) such as `wiki40b`.
Issue: dataset.search() (elastic) cannot reliably retrieve search results

Comment:
Hi !
I tried your code on my side and I was able to workaround this issue by waiting a few seconds before querying the index.
Maybe this is because the index is not updated yet on the ElasticSearch side ?
Issue: dataset.search() (elastic) cannot reliably retrieve search results

Comment:
Thanks for the feedback! I added a 30 second "sleep" and that seemed to work well!
Issue: FewRel

Comment:
@dspoka Please check the following link : https://github.com/thunlp/FewRel
This link mentions two versions of the datasets. Also, this one seems to be the official link.

I am assuming this is the correct link and implementing based on the same.
Issue: Using select/reordering datasets slows operations down immensely

Comment:
Thanks for the input! I gave that a try by adding this after my selection / reordering operations, but before the big computation task of `score_squad`

```
examples = examples.flatten_indices()
features = features.flatten_indices()
```

That helped quite a bit!
Issue: datasets slicing with seed 

Comment:
Hi :) 
The slicing API from https://huggingface.co/docs/datasets/splits.html doesn't shuffle the data.
You can shuffle and then take a subset of your dataset with
```python
# shuffle and take the first 100 examples
dataset = dataset.shuffle(seed=42).select(range(100))
```

You can find more information about shuffling and selecting rows in the documentation: https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
Issue: datasets slicing with seed 

Comment:
thank you so much

On Mon, Jan 18, 2021 at 3:17 PM Quentin Lhoest <notifications@github.com>
wrote:

> Hi :)
> The slicing API doesn't shuffle the data.
> You can shuffle and then take a subset of your dataset with
>
> # shuffle and take the first 100 examplesdataset = dataset.shuffle(seed=42).select(range(100))
>
> You can find more information about shuffling and selecting rows in the
> documentation:
> https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1747#issuecomment-762278134>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM5D5MDPLJGI4IG3UADS2Q7GPANCNFSM4WHLOZJQ>
> .
>
Issue: difference between wsc and wsc.fixed for superglue

Comment:
From the description given in the dataset script for `wsc.fixed`:
```
This version fixes issues where the spans are not actually substrings of the text.
```
Issue: Issue while Creating Custom Metric

Comment:
Currently it's only possible to define the features for the two columns `references` and `predictions`.
The data for these columns can then be passed to `metric.add_batch` and `metric.compute`.
Instead of defining more columns `text`, `offset_mapping` and `ground` you must include them in either references and predictions.

For example 
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence({
        "references_ids": datasets.Value("int32"),
        "offset_mapping": datasets.Value("int32"),
        'text': datasets.Value('string'),
        "ground": datasets.Value("int32")
    }),
})
```

Another option would be to simply have the two features like 
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence(datasets.Value("int32")),
})
```
and keep `offset_mapping`, `text` and `ground` as as parameters for the computation (i.e. kwargs when calling `metric.compute`).


What is the metric you would like to implement ?

I'm asking since we consider allowing additional fields as requested in the `Comet` metric (see PR and discussion [here](https://github.com/huggingface/datasets/pull/1577)) and I'd like to know if it's something that can be interesting for users.

What do you think ?
Issue: Issue while Creating Custom Metric

Comment:
Hi @lhoestq,

I am doing text segmentation and the metric is effectively dice score on character offsets. So I need to pass the actual spans and I want to be able to get the spans based on predictions using offset_mapping.

Including them in references seems like a good idea. I'll try it out and get back to you. If there's a better way to write a metric function for the same, please let me know.
Issue: connection issue with glue, what is the data url for glue? 

Comment:
Hello @juliahane, which config of GLUE causes you trouble?
The URLs are defined in the dataset script source code: https://github.com/huggingface/datasets/blob/master/datasets/glue/glue.py
Issue: Couldn't reach swda.py

Comment:
Hi @yangp725,
The SWDA has been added very recently and has not been released yet, thus it is not available in the `1.2.0` version of 🤗`datasets`.
You can still access it by installing the latest version of the library (master branch), by following instructions in [this issue](https://github.com/huggingface/datasets/issues/1641#issuecomment-751571471).
Let me know if this helps !
Issue: Is there support for Deep learning datasets?

Comment:
Hi @ZurMaD!
Thanks for your interest in 🤗 `datasets`. Support for image datasets is at an early stage, with CIFAR-10 added in #1617 
MNIST is also on the way: #1730 

If you feel like adding another image dataset, I would advise starting by reading the [ADD_NEW_DATASET.md](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) guide. New datasets are always very much appreciated 🚀
Issue: Add an entry to an arrow dataset

Comment:
Hi @ameet-1997,
I think what you are looking for is the `concatenate_datasets` function: https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#concatenate-several-datasets

For your use case, I would use the [`map` method](https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#processing-data-with-map) to transform the SQuAD sentences and the `concatenate` the original and mapped dataset.

Let me know If this helps!
Issue: Add an entry to an arrow dataset

Comment:
That's a great idea! Thank you so much!

When I try that solution, I get the following error when I try to concatenate `datasets` and `modified_dataset`. I have also attached the output I get when I print out those two variables. Am I missing something?

Code:
``` python
combined_dataset = concatenate_datasets([datasets, modified_dataset])
```

Error:
```
AttributeError: 'DatasetDict' object has no attribute 'features'
```

Output:
```
(Pdb) datasets
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
(Pdb) modified_dataset
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
```

The error is stemming from the fact that the attribute `datasets.features` does not exist. Would it not be possible to use `concatenate_datasets` in such a case? Is there an alternate solution?
Issue: Add an entry to an arrow dataset

Comment:
You should do `combined_dataset = concatenate_datasets([datasets['train'], modified_dataset['train']])`

Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.
Issue: Add an entry to an arrow dataset

Comment:
> Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.

My opinion is that users should always know in advance what type of objects they're going to get. Otherwise the development workflow on their side is going to be pretty chaotic with sometimes unexpected behaviors.
For instance is `split=` is not specified it's currently always returning a DatasetDict. And if `split="train"` is given for example it's always returning a Dataset.
Issue: BLEURT score calculation raises UnrecognizedFlagError

Comment:
And I have the same error with TF 2.4.1. I believe this issue should be reopened. Any ideas?!
Issue: BLEURT score calculation raises UnrecognizedFlagError

Comment:
I'm seeing the same issue with TF 2.4.1 when running the following in https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb:
```
!pip install git+https://github.com/google-research/bleurt.git
references = ["foo bar baz", "one two three"]
bleurt_metric = load_metric('bleurt')
predictions =  ["foo bar", "four five six"]
bleurt_metric.compute(predictions=predictions, references=references)
```
Issue: BLEURT score calculation raises UnrecognizedFlagError

Comment:
@aleSuglia @oscartackstrom - Are you getting the error when running your code in a Jupyter notebook ?

I tried reproducing this error again, and was unable to do so from the python command line console in a virtual environment similar to the one I originally used (and unfortunately no longer have access to) when I first got the error. 
However, I've managed to reproduce the error by running the same code in a Jupyter notebook running a kernel from the same virtual environment.
This made me suspect that the problem is somehow related to the Jupyter notebook.

More environment details:
```
OS: Ubuntu Linux 18.04
conda==4.8.3
python==3.8.5
datasets==1.3.0
tensorflow==2.4.0
BLEURT==0.0.1
notebook==6.2.0
```
Issue: BLEURT score calculation raises UnrecognizedFlagError

Comment:
This happens when running the notebook on colab. The issue seems to be that colab populates sys.argv with arguments not handled by bleurt.

Running this before calling bleurt fixes it:
```
import sys
sys.argv = sys.argv[:1]
```

Not the most elegant solution. Perhaps it needs to be fixed in the bleurt code itself rather than huggingface?

This is the output of `print(sys.argv)` when running on colab:
```
['/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py', '-f', '/root/.local/share/jupyter/runtime/kernel-a857a78c-44d6-4b9d-b18a-030b858ee327.json']
```
Issue: BLEURT score calculation raises UnrecognizedFlagError

Comment:
I got the error when running it from the command line. It looks more like an error that should be fixed in the BLEURT codebase.
Issue: load the local dataset

Comment:
You should rephrase your question or give more examples and details on what you want to do.

it’s not possible to understand it and help you with only this information.
Issue: load the local dataset

Comment:
sorry for that.
i want to know how could i load the train set and the test set from the local ,which api or function should i use .
Issue: load the local dataset

Comment:
thanks a lot 
i find that the problem is i dont use vpn...
so i have to keep my net work even if i want to load the local data ?
Issue: load the local dataset

Comment:
Hi! `json` is a packaged dataset now, which means its script comes with the library and doesn't require an internet connection.
Issue: could not run models on a offline server successfully

Comment:
Hi @lkcao !
Your issue is indeed related to `datasets`. In addition to installing the package manually, you will need to download the `text.py` script on your server. You'll find it (under `datasets/datasets/text`: https://github.com/huggingface/datasets/blob/master/datasets/text/text.py.
Then you can change the line 221 of `run_mlm_new.py` into:
```python
  datasets = load_dataset('/path/to/text.py', data_files=data_files)
```
Where `/path/to/text.py` is the path on the server where you saved the `text.py` script.
Issue: could not run models on a offline server successfully

Comment:
We're working on including the local dataset builders (csv, text, json etc.) directly in the `datasets` package so that they can be used offline
Issue: could not run models on a offline server successfully

Comment:
The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
You can now use them offline
```python
datasets = load_dataset('text', data_files=data_files)
```

We'll do a new release soon
Issue: could not run models on a offline server successfully

Comment:
> The local dataset builders (csv, text , json and pandas) are now part of the `datasets` package since #1726 :)
> You can now use them offline
> 
> ```python
> datasets = load_dataset('text', data_files=data_files)
> ```
> 
> We'll do a new release soon

so the new version release now?
Issue: Possible cache miss in datasets

Comment:
Thanks for reporting !
I was able to reproduce thanks to your code and find the origin of the bug.
The cache was not reusing the same file because one object was not deterministic. It comes from a conversion from `set` to `list` in the `datasets.arrrow_dataset.transmit_format` function, where the resulting list would not always be in the same order and therefore the function that computes the hash used by the cache would not always return the same result.
I'm opening a PR to fix this.

Also we plan to do a new release in the coming days so you can expect the fix to be available soon.
Note that you can still specify `cache_file_name=` in the second `map()` call to name the cache file yourself if you want to.