Question/problem with dataset labels
State: closed | Author: ioana-blue (NONE)

Hi, I'm using a dataset with two labels "nurse" and "not nurse". For whatever reason (that I don't understand), I get an error that I think comes from the datasets package (using csv). Everything works fine if the labels are "nurse" and "surgeon". 

This is the trace I get:

```
File "../../../models/tr-4.3.2/run_puppets.py", line 523, in <module>
    main()
  File "../../../models/tr-4.3.2/run_puppets.py", line 249, in main
    datasets = load_dataset("csv", data_files=data_files)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 572, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 1028, in _prepare_split
    writer.write_table(table)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/arrow_writer.py", line 292, in write_table
    pa_table = pa_table.cast(self._schema)
  File "pyarrow/table.pxi", line 1311, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 265, in pyarrow.lib.ChunkedArray.cast
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/pyarrow/compute.py", line 87, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to parse string: not nurse
```

Any ideas how to fix this? For now, I'll probably make them numeric. 

Comments: It seems that I get parsing errors for various fields in my data. For example now I get this:
```
  File "../../../models/tr-4.3.2/run_puppets.py", line 523, in <module>
    main()
  File "../../../models/tr-4.3.2/run_puppets.py", line 249, in main
    datasets = load_dataset("csv", data_files=data_files)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 572, in download_and_prepare
    self._download_and_prepare(
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/builder.py", line 1028, in _prepare_split
    writer.write_table(table)
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/datasets/arrow_writer.py", line 292, in write_table
    pa_table = pa_table.cast(self._schema)
  File "pyarrow/table.pxi", line 1311, in pyarrow.lib.Table.cast
  File "pyarrow/table.pxi", line 265, in pyarrow.lib.ChunkedArray.cast
  File "/dccstor/redrug_ier/envs/last-tr/lib/python3.8/site-packages/pyarrow/compute.py", line 87, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 298, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 192, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Failed to parse string: https://www.netgalley.com/catalog/book/121872
```

Not sure if this helps, this is how I load my files (as in the sample scripts on transformers):

```
    if data_args.train_file.endswith(".csv"):
        # Loading a dataset from local csv files
        datasets = load_dataset("csv", data_files=data_files)
```

Since this worked out of the box in a few examples before, I wonder if it's some quoting issue or something else. 

Hi @ioana-blue,
Can you share a sample from your .csv? A dummy where you get this error will also help.

I tried this csv:
```csv
feature,label
1.2,not nurse
1.3,nurse
1.5,surgeon
```
and the following snippet:
```python
from datasets import load_dataset

d = load_dataset("csv",data_files=['test.csv'])

print(d)
print(d['train']['label'])
```
and this works perfectly fine for me:
```sh
DatasetDict({
    train: Dataset({
        features: ['feature', 'label'],
        num_rows: 3
    })
})
['not nurse', 'nurse', 'surgeon']
```
I'm sure your csv is more complicated than this one. But it is hard to tell where the issue might be without looking at a sample.

I've had versions where it worked fain. For this dataset, I had all kind of parsing issues that I couldn't understand. What I ended up doing is strip all the columns that I didn't need and also make the label 0/1. 

I think one line that may have caused a problem was the csv version of this:

```crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz        Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job. ^M ('Rose', '', 'Blakey')  journalist      F       38      journalist      https://www.netgalley.com/catalog/book/121872   _ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job.    She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.```

The error I got in this case is this one: https://github.com/huggingface/datasets/issues/1989#issuecomment-790842771

Note, this line was part of a much larger file and until this line I guess it was working fine. 

Hi @ioana-blue,

What is the separator you're using for the csv? I see there are only two commas in the given line, but they don't seem like appropriate points. Also, is this a string part of one line, or an entire line? There should also be a label, right?

Sorry for the confusion, the sample above was from a tsv that was used to derive the csv. Let me construct the csv again (I had remove it). 

This is the line in the csv - this is the whole line:
```crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz,Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead,"('Rose', '', 'Blakey')",journalist,F,38,journalist,https://www.netgalley.com/catalog/book/121872,_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job., She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.```

Hi,
Just in case you want to use tsv directly, you can use the separator argument while loading the dataset.
```python
d = load_dataset("csv",data_files=['test.csv'],sep="\t")
```

Additionally, I don't face the issues with the following csv (same as the one you provided):

```sh
link1,text1,info1,info2,info3,info4,info5,link2,text2,text3
crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz,Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead,"('Rose', '', 'Blakey')",journalist,F,38,journalist,https://www.netgalley.com/catalog/book/121872,_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job., She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.
```
Output after loading:
```sh
{'link1': 'crawl-data/CC-MAIN-2017-47/segments/1510934806225.78/wet/CC-MAIN-20171120203833-20171120223833-00571.warc.wet.gz', 'text1': 'Rose Blakey is an aspiring journalist. She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead', 'info1': "('Rose', '', 'Blakey')", 'info2': 'journalist', 'info3': 'F', 'info4': 38, 'info5': 'journalist', 'link2': 'https://www.netgalley.com/catalog/book/121872', 'text2': '_ is desperate to escape the from the small Australian town in which _ lives. Rejection after rejection mean _ is stuck in what _ sees as a dead-end waitressing job.', 'text3': ' She is desperate to escape the from the small Australian town in which she lives. Rejection after rejection mean she is stuck in what she sees as a dead-end waitressing job.'}
```
Can you check once if the tsv works for you directly using the separator argument? The conversion from tsv to csv could create issues, I'm only guessing though.

thanks for the tip. very strange :/ I'll check my datasets version as well. 

I will have more similar experiments soon so I'll let you know if I manage to get rid of this. 

No problem at all. I thought I'd be able to solve this but I'm unable to replicate the issue :/