SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Yes right now `ArrayXD` can only be used as a column feature type, not a subtype.
With the current Arrow limitations I don't think we'll be able to make it work as a subtype, however it should be possible to allow dimensions of dynamic sizes (`Array3D(shape=(None, 137, 2), dtype="float32")` for example since the [underlying arrow  type](https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L236) allows dynamic sizes.

For now I'd suggest the use of nested `Sequence` types. Once we have the dynamic sizes you can update the dataset.
What do you think ?
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
> Yes right now ArrayXD can only be used as a column feature type, not a subtype. 

Meaning it can't be nested under `Sequence`?
If so, for now I'll just make it a python list and make it with the nested `Sequence` type you suggested.
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Yea unfortunately..
That's a current limitation with Arrow ExtensionTypes that can't be used in the default Arrow Array objects.
We already have an ExtensionArray that allows us to use them as column types but not for subtypes.
Maybe we can extend it, I haven't experimented with that yet
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Cool
So please consider this issue as a feature request for:
```
Array3D(shape=(None, 137, 2), dtype="float32")
```

its a way to represent videos, poses, and other cool sequences
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
@lhoestq well, so sequence of sequences doesn't work either...

```
pyarrow.lib.ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648
```
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Working with Arrow can be quite fun sometimes.
You can fix this issue by trying to reduce the writer batch size (same trick than the one used to reduce the RAM usage in https://github.com/huggingface/datasets/issues/741).

Let me know if it works.
I haven't investigated yet on https://github.com/huggingface/datasets/issues/741 since I was preparing this week's sprint to add datasets but this is in my priority list for early next week.
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
The batch size fix doesn't work... not for #741 and not for this dataset I'm trying (DGS corpus)
Loading the DGS corpus takes 400GB of RAM, which is fine with me as my machine is large enough
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Not yet, I've been pretty busy with the dataset sprint lately but this is something that's been asked several times already. So I'll definitely work on this as soon as I'm done with the sprint and with the RAM issue you reported.
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Hi @lhoestq,
Any chance you have some updates on the supporting `ArrayXD` as a subtype or support of dynamic sized arrays?

e.g.:
`datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))`
`Array3D(shape=(None, 137, 2), dtype="float32")`
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Hi ! We haven't worked in this lately and it's not in our very short-term roadmap since it requires a bit a work to make it work with arrow. Though this will definitely be added at one point.
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
@lhoestq, thanks for the update.

I actually tried to modify some piece of code to make it work. Can you please tell if I missing anything here?
I think that for vast majority of cases it's enough to make first dimension of the array dynamic i.e. `shape=(None, 100, 100)`. For that, it's enough to modify class [ArrayExtensionArray](https://github.com/huggingface/datasets/blob/9ca24250ea44e7611c4dabd01ecf9415a7f0be6c/src/datasets/features.py#L397) to output list of arrays of different sizes instead of list of arrays of same sizes (current version)
Below are my modifications of this class.

```
class ArrayExtensionArray(pa.ExtensionArray):
    def __array__(self):
        zero_copy_only = _is_zero_copy_only(self.storage.type)
        return self.to_numpy(zero_copy_only=zero_copy_only)

    def __getitem__(self, i):
        return self.storage[i]

    def to_numpy(self, zero_copy_only=True):
        storage: pa.ListArray = self.storage
        size = 1
        for i in range(self.type.ndims):
            size *= self.type.shape[i]
            storage = storage.flatten()
        numpy_arr = storage.to_numpy(zero_copy_only=zero_copy_only)
        numpy_arr = numpy_arr.reshape(len(self), *self.type.shape)
        return numpy_arr

    def to_list_of_numpy(self, zero_copy_only=True):
        storage: pa.ListArray = self.storage
        shape = self.type.shape
        arrays = []
        for dim in range(1, self.type.ndims):
            assert shape[dim] is not None, f"Support only dynamic size on first dimension. Got: {shape}"

        first_dim_offsets = np.array([off.as_py() for off in storage.offsets])
        for i in range(len(storage)):
            storage_el = storage[i:i+1]
            first_dim = first_dim_offsets[i+1] - first_dim_offsets[i]
            # flatten storage
            for dim in range(self.type.ndims):
                storage_el = storage_el.flatten()

            numpy_arr = storage_el.to_numpy(zero_copy_only=zero_copy_only)
            arrays.append(numpy_arr.reshape(first_dim, *shape[1:]))

        return arrays

    def to_pylist(self):
        zero_copy_only = _is_zero_copy_only(self.storage.type)
        if self.type.shape[0] is None:
            return self.to_list_of_numpy(zero_copy_only=zero_copy_only)
        else:
            return self.to_numpy(zero_copy_only=zero_copy_only).tolist()
```

I ran few tests and it works as expected. Let me know what you think.
Issue: pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>

Comment:
Thanks for diving into this !

Indeed focusing on making the first dimensions dynamic make total sense (and users could still re-order their dimensions to match this constraint).
Your code looks great :) I think it can even be extended to support several dynamic dimensions if we want to.

Feel free to open a PR to include these changes, then we can update our test suite to make sure it works in all use cases.
In particular I think we might need a few tweaks to allow it to be converted to pandas (though I haven't tested yet):

```python
from datasets import Dataset, Features, Array3D

# this works
matrix = [[1, 0], [0, 1]]
features = Features({"a": Array3D(dtype="int32", shape=(1, 2, 2))})
d = Dataset.from_dict({"a": [[matrix], [matrix]]})
print(d.to_pandas())

# this should work as well
matrix = [[1, 0], [0, 1]]
features = Features({"a": Array3D(dtype="int32", shape=(None, 2, 2))})
d = Dataset.from_dict({"a": [[matrix], [matrix] * 2]})
print(d.to_pandas())
```

I'll be happy to help you on this :)
Issue: Downloading/caching only a part of a datasets' dataset.

Comment:
I think it would be a very helpful feature, because sometimes one only wants to evaluate models on the dev set, and the whole training data may be many times bigger.
This makes the task impossible with limited memory resources.
Issue: Add SQA

Comment:
I’ll take this one to test the workflow for the sprint next week cc @yjernite @lhoestq
Issue: Add SQA

Comment:
@thomwolf here's a slightly adapted version of the code from the [official Tapas repository](https://github.com/google-research/tapas/blob/master/tapas/utils/interaction_utils.py) that is used to turn the `answer_coordinates` and `answer_texts` columns into true Python lists of tuples/strings:

```
import pandas as pd
import ast

data = pd.read_csv("/content/sqa_data/random-split-1-dev.tsv", sep='\t')

def _parse_answer_coordinates(answer_coordinate_str):
  """Parses the answer_coordinates of a question.
  Args:
    answer_coordinate_str: A string representation of a Python list of tuple
      strings.
      For example: "['(1, 4)','(1, 3)', ...]"
  """

  try:
    answer_coordinates = []
    # make a list of strings
    coords = ast.literal_eval(answer_coordinate_str)
    # parse each string as a tuple
    for row_index, column_index in sorted(
        ast.literal_eval(coord) for coord in coords):
      answer_coordinates.append((row_index, column_index))
  except SyntaxError:
    raise ValueError('Unable to evaluate %s' % answer_coordinate_str)
  
  return answer_coordinates


def _parse_answer_text(answer_text):
  """Populates the answer_texts field of `answer` by parsing `answer_text`.
  Args:
    answer_text: A string representation of a Python list of strings.
      For example: "[u'test', u'hello', ...]"
  """
  try:
    answer = []
    for value in ast.literal_eval(answer_text):
      answer.append(value)
  except SyntaxError:
    raise ValueError('Unable to evaluate %s' % answer_text)

  return answer

data['answer_coordinates'] = data['answer_coordinates'].apply(lambda coords_str: _parse_answer_coordinates(coords_str))
data['answer_text'] = data['answer_text'].apply(lambda txt: _parse_answer_text(txt))
```

Here I'm using Pandas to read in one of the TSV files (the dev set).
Issue: boolq does not load 

Comment:
Hi ! It runs on my side without issues. I tried
```python
from datasets import load_dataset
load_dataset("boolq")
```

What version of datasets and tensorflow are your runnning ?
Also if you manage to get a minimal reproducible script (on google colab for example) that would be useful.
Issue: boolq does not load 

Comment:
hey
i do the exact same commands. for me it fails i guess might be issues with
caching maybe?
thanks
best
rabeeh

On Tue, Nov 24, 2020, 10:24 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hi ! It runs on my side without issues. I tried
>
> from datasets import load_datasetload_dataset("boolq")
>
> What version of datasets and tensorflow are your runnning ?
> Also if you manage to get a minimal reproducible script (on google colab
> for example) that would be useful.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/879#issuecomment-732769114>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCGGDR2FUMRKZTIY5CTSRN3VXANCNFSM4T7R3U6A>
> .
>
Issue: boolq does not load 

Comment:
Could you check if it works on the master branch ?
You can use `load_dataset("boolq", script_version="master")` to do so.
We did some changes recently in boolq to remove the TF dependency and we changed the way the data files are downloaded in https://github.com/huggingface/datasets/pull/881
Issue: Loading Data From S3 Path in Sagemaker

Comment:
> neat feature

I dint get these clearly, can you please elaborate like how to work on these
Issue: Loading Data From S3 Path in Sagemaker

Comment:
It could maybe work almost out of the box just by using `cached_path` in the text/csv/json scripts, no?
Issue: Loading Data From S3 Path in Sagemaker

Comment:
Thanks thomwolf and julien-c

I'm still confusion on what you guys said, 

I have solved the problem as follows:

1. read the csv file using pandas from s3 
2. Convert to dictionary key as column name and values as list column data
3. convert it to Dataset using 
`from datasets import Dataset`
`train_dataset = Dataset.from_dict(train_dict)`
Issue: Loading Data From S3 Path in Sagemaker

Comment:
We were brainstorming around your use-case.

Let's keep the issue open for now, I think this is an interesting question to think about.
Issue: Loading Data From S3 Path in Sagemaker

Comment:
> We were brainstorming around your use-case.
> 
> Let's keep the issue open for now, I think this is an interesting question to think about.

Sure thomwolf, Thanks for your concern
Issue: Loading Data From S3 Path in Sagemaker

Comment:
I agree it would be cool to have that feature. Also that's good to know that pandas supports this.
For the moment I'd suggest to first download the files locally as thom suggested and then load the dataset by providing paths to the local files
Issue: Loading Data From S3 Path in Sagemaker

Comment:
Any updates on this issue?
I face a similar issue. I have many parquet files in S3 and I would like to train on them. 
To be honest I even face issues with only getting the last layer embedding out of them.
Issue: Loading Data From S3 Path in Sagemaker

Comment:
Hi dorlavie, 
You can find one solution that i have mentioned above, that can help you. 
And there is one more solution also which is downloading files locally
Issue: Loading Data From S3 Path in Sagemaker

Comment:
> Hi dorlavie,
> You can find one solution that i have mentioned above, that can help you.
> And there is one more solution also which is downloading files locally

mahesh1amour, thanks for the fast reply

Unfortunately, in my case I can not read with pandas. The dataset is too big (50GB). 
In addition, due to security concerns I am not allowed to save the data locally
Issue: Loading Data From S3 Path in Sagemaker

Comment:
@dorlavie could use `boto3` to download the data to your local machine and then load it with `dataset`

boto3 example [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html)
```python
import boto3

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
```

datasets example [documentation](https://huggingface.co/docs/datasets/loading_datasets.html)

```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])
```
Issue: Loading Data From S3 Path in Sagemaker

Comment:
Thanks @philschmid for the suggestion.
As I mentioned in the previous comment, due to security issues I can not save the data locally.
I need to read it from S3 and process it directly.

I guess that many other people try to train / fit those models on huge datasets (e.g entire Wiki), what is the best practice in those cases?
Issue: Loading Data From S3 Path in Sagemaker

Comment:
If I understand correctly you're not allowed to write data on disk that you downloaded from S3 for example ?
Or is it the use of the `boto3` library that is not allowed in your case ?
Issue: Loading Data From S3 Path in Sagemaker

Comment:
@lhoestq yes you are correct.
I am not allowed to save the "raw text" locally - The "raw text" must be saved only on S3.
I am allowed to save the output of any model locally. 
It doesn't matter how I do it boto3/pandas/pyarrow, it is forbidden
Issue: Loading Data From S3 Path in Sagemaker

Comment:
@dorlavie are you using sagemaker for training too? Then you could use S3 URI, for example `s3://my-bucket/my-training-data` and pass it within the `.fit()` function when you start the sagemaker training job. Sagemaker would then download the data from s3 into the training runtime and you could load it from disk

**sagemaker start training job**
```python
pytorch_estimator.fit({'train':'s3://my-bucket/my-training-data','eval':'s3://my-bucket/my-evaluation-data'})
```

**in the train.py script**
```python
from datasets import load_from_disk

train_dataset = load_from_disk(os.environ['SM_CHANNEL_TRAIN'])
```

I have created an example of how to use transformers and datasets with sagemaker. 
https://github.com/philschmid/huggingface-sagemaker-example/tree/main/03_huggingface_sagemaker_trainer_with_data_from_s3

The example contains a jupyter notebook `sagemaker-example.ipynb` and an `src/` folder. The sagemaker-example is a jupyter notebook that is used to create the training job on AWS Sagemaker. The `src/` folder contains the `train.py`, our training script, and `requirements.txt` for additional dependencies.
Issue: DataLoader(datasets) become more and more slowly within iterations

Comment:
Hi ! Thanks for reporting.
Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
It would be nice to know whether it comes from the dataloader or not
Issue: DataLoader(datasets) become more and more slowly within iterations

Comment:
> Hi ! Thanks for reporting.
> Do you have the same slowdown when you iterate through the raw dataset object as well ? (no dataloader)
> It would be nice to know whether it comes from the dataloader or not

I did not iter data from raw dataset, maybe I will test later. Now I iter all files directly from `open(file)`,  around 20000it/s.
Issue: imdb dataset cannot be loaded 

Comment:
It looks like there was an issue while building the imdb dataset.
Could you provide more information about your OS and the version of python and `datasets` ?

Also could you try again with 
```python
dataset = datasets.load_dataset("imdb", split="train", download_mode="force_redownload")
```
to make sure it's not a corrupted file issue ?
Issue: imdb dataset cannot be loaded 

Comment:
Hi ! I just tried in 1.8.0 and it worked fine. Can you try again ? Maybe the dataset host had some issues that are fixed now
Issue: load_dataset('cnn_dalymail', '3.0.0') gives a 'Not a directory' error

Comment:
I see the issue happening again today - 

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602...

---------------------------------------------------------------------------

NotADirectoryError                        Traceback (most recent call last)

<ipython-input-9-cd4bf8bea840> in <module>()
     22 
     23 
---> 24 train = load_dataset('cnn_dailymail', '3.0.0', split='train')
     25 validation = load_dataset('cnn_dailymail', '3.0.0', split='validation')
     26 test = load_dataset('cnn_dailymail', '3.0.0', split='test')

5 frames

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
    132     else:
    133         logging.fatal("Unsupported publisher: %s", publisher)
--> 134     files = sorted(os.listdir(top_dir))
    135 
    136     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'

Can someone please take a look ?
Issue: load_dataset('cnn_dalymail', '3.0.0') gives a 'Not a directory' error

Comment:
> atal("Unsupported publisher: %s", publisher) --> 134 files = sorted(os.listdir(top_dir)) 135 136 ret_files = []
> 
> NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
> 
> Can someone please take a look ?

2 short-term workarounds:

1.  Use this line instead `dataset = load_dataset('ccdv/cnn_dailymail', '3.0.0')`. [In a related issue](https://github.com/huggingface/datasets/issues/996#issuecomment-997343101), this person mentioned another data source copy that just works.
2.  Use the same data source, but edit the urls.  Instead of google drive quota problems mentioned in #996, I was getting the "can't scan this file for viruses" problem, which results in that prompted html getting downloaded instead of the files.  You can get around this by:
    1. Look at the traceback and find out where `cnn_dailymail.py` is on your computer.
    2. Edit the `cnn_stories` and `dm_stories` url's by adding the following to the end of them `&confirm=t`.  This should be around line 67.
    3. You may have to remove those confirmation html files in your download directory (`~/.cache/huggingface/datasets/downloads` for me) so that they don't get in the way of the new download attempts.

Either method works for me.  I would've made a PR, but not sure if they want to go with the new ccdv/cnn_dailymail source or not.
Issue: load_dataset('cnn_dalymail', '3.0.0') gives a 'Not a directory' error

Comment:
experience the same problem, ccdv/cnn_dailymail not working either. 

Solve this problem by installing datasets library from the master branch:
python -m pip install git+https://github.com/huggingface/datasets.git@master
Issue: load_dataset('cnn_dalymail', '3.0.0') gives a 'Not a directory' error

Comment:
> > atal("Unsupported publisher: %s", publisher) --> 134 files = sorted(os.listdir(top_dir)) 135 136 ret_files = []
> > NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
> > Can someone please take a look ?
> 
> 2 short-term workarounds:
> 
> 1. Use this line instead `dataset = load_dataset('ccdv/cnn_dailymail', '3.0.0')`. [In a related issue](https://github.com/huggingface/datasets/issues/996#issuecomment-997343101), this person mentioned another data source copy that just works.
> 2. Use the same data source, but edit the urls.  Instead of google drive quota problems mentioned in [NotADirectoryError while loading the CNN/Dailymail dataset #996](https://github.com/huggingface/datasets/issues/996), I was getting the "can't scan this file for viruses" problem, which results in that prompted html getting downloaded instead of the files.  You can get around this by:
>    
>    1. Look at the traceback and find out where `cnn_dailymail.py` is on your computer.
>    2. Edit the `cnn_stories` and `dm_stories` url's by adding the following to the end of them `&confirm=t`.  This should be around line 67.
>    3. You may have to remove those confirmation html files in your download directory (`~/.cache/huggingface/datasets/downloads` for me) so that they don't get in the way of the new download attempts.
> 
> Either method works for me. I would've made a PR, but not sure if they want to go with the new ccdv/cnn_dailymail source or not.

Thankyou, editing the urls helped me than the loading dataset line.
Issue: terminate called after throwing an instance of 'google::protobuf::FatalException'

Comment:
Loading the iwslt2017-en-nl config of iwslt2017 works fine on my side. 
Maybe you can open an issue on transformers as well ? And also add more details about your environment (OS, python version, version of transformers and datasets etc.)
Issue: terminate called after throwing an instance of 'google::protobuf::FatalException'

Comment:
closing now, figured out this is because the max length of decoder was set smaller than the input_dimensions. thanks
Issue: [Feature Request] Add optional parameter in text loading script to preserve linebreaks

Comment:
Hi ! Thanks for your message.
Indeed it's a free feature we can add and that can be useful.
If you want to contribute, feel free to open a PR to add it to the text dataset script :)