SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Some error inside nlp.load_dataset()

Comment:
Google colab has an old version of Apache Arrow built-in.
Be sure you execute the "pip install" cell and restart the notebook environment if the colab asks for it.
Issue: 🐛 `map` not working

Comment:
I didn't assign the output 🤦‍♂️

```python
dataset.map(test)
```

should be :

```python
dataset = dataset.map(test)
```
Issue: 🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'

Comment:
It's strange, after installing `nlp` on Colab, the `pyarrow` version seems fine from `pip` but not from python :

```python
import pyarrow

!pip show pyarrow
print("version = {}".format(pyarrow.__version__))
```

> Name: pyarrow
Version: 0.17.0
Summary: Python library for Apache Arrow
Home-page: https://arrow.apache.org/
Author: None
Author-email: None
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: numpy
Required-by: nlp, feather-format
> 
> version = 0.14.1
Issue: 🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'

Comment:
Ok I just had to restart the runtime after installing `nlp`. After restarting, the version of `pyarrow` is fine.
Issue: ❓ How to remove specific rows of a dataset ?

Comment:
you can convert dataset object to pandas and remove a feature and convert back to dataset .
Issue: ❓ How to remove specific rows of a dataset ?

Comment:
That's what I ended up doing too. but it feels like a workaround to a feature that should be added to the datasets class.
Issue: 🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

Comment:
Sure, [here is a Colab notebook](https://colab.research.google.com/drive/1uiS89fnHMG7HV_cYxp3r-_LqJQvNNKs9?usp=sharing) reproducing the error.

> ArrowInvalid: Column 1 named references expected length 36 but got length 56
Issue: 🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

Comment:
This is because `add` takes as input a batch of elements and you provided only one. I think we should have `add` for one prediction/reference and `add_batch` for a batch of predictions/references. This would make it more coherent with the way we use Arrow.

Let me do this change
Issue: AttributeError: 'dict' object has no attribute 'info'

Comment:
I could access the info by first accessing the different splits :

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm['train'].info)
```

Information seems to be duplicated between the subsets :

```python
print(cnn_dm["train"].info == cnn_dm["test"].info == cnn_dm["validation"].info)
# True
```

Is it expected ?
Issue: AttributeError: 'dict' object has no attribute 'info'

Comment:
Good point @Colanim ! What happens under the hood when running:

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
```

is that for every split in `cnn_dailymail`, a different dataset object (which all holds the same info) is created. This has the advantages that the datasets are easily separable in a training setup. 
Also note that you can load e.g. only the `train` split of the dataset via:

```python
cnn_dm_train = nlp.load_dataset('cnn_dailymail', split="train")
print(cnn_dm_train.info)
```

I think we should make the `info` object slightly different when creating the dataset for each split - at the moment it contains for example the variable `splits` which should maybe be renamed to `split` and contain only one `SplitInfo` object ...
Issue: [Checksums] Error for some datasets

Comment:
Fixed with 06882b4

Now your command works :)
Note that you can also do
```
nlp-cli test datasets/nlp/xnli --save_checksums
```
So that it will save the checksums directly in the right directory.
Issue: Error when citation is not given in the DatasetInfo

Comment:
Yes looks good to me.
Note that we may refactor quite strongly the `info.py` to make it a lot simpler (it's very complicated for basically a dictionary of info I think)
Issue: ValueError when a split is empty

Comment:
Yes sounds good to me!
Do you want to make a PR? or I can do it as well
Issue: [Feature] Keep the list of labels of a dataset as metadata

Comment:
Yes! I see mostly two options for this:
- a `Feature` approach like currently (but we might deprecate features)
- wrapping in a smart way the Dictionary arrays of Arrow: https://arrow.apache.org/docs/python/data.html?highlight=dictionary%20encode#dictionary-arrays
Issue: [Feature] Keep the list of labels of a dataset as metadata

Comment:
This should be accessible now as a feature in dataset.info.features (and even have the mapping methods).
Issue: [Feature] Keep the list of labels of a dataset as metadata

Comment:
Hi,
I hope we could get a better documentation.
It took me more than 1 hour to found this way to get the label information.
Issue: [Feature] Keep the list of labels of a dataset as metadata

Comment:
Yes we are working on the doc right now, should be in the next release quite soon.
Issue: [Feature] More dataset outputs

Comment:
Yes!
- pandas will be a one-liner in `arrow_dataset`: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas
- for Spark I have no idea. let's investigate that at some point
Issue: [Feature] More dataset outputs

Comment:
For Spark it looks to be pretty straightforward as well https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html but looks to be having a dependency to Spark is necessary, then nevermind we can skip it
Issue: Issue to read a local dataset

Comment:
Ok great, so as discussed today, let's:
- have a main dataset directory inside the lib with sub-directories hashed by the content of the file
- keep a cache for downloading the scripts from S3 for now
- later: add methods to list and clean the local versions of the datasets (and the distant versions on S3 as well)

Side question: do you often use `builder_kwargs` for other things than supplying file paths? I was thinking about having a more easy to read and remember `data_files` argument maybe.
Issue: Issue to read a local dataset

Comment:
Good plan!

Yes I do use `builder_kwargs` for other things such as:
- dataset name
- properties to know how to properly read a CSV file: do I have to skip the first line in a CSV, which delimiter is used, and the columns ids to use.
- properties to know how to properly read a JSON file: which properties in a JSON object to read