GitHub issue #1422: Can't map dataset (loaded from csv)
State: closed | Created: 2020-12-09T22:05:42Z

Hello! I am trying to load single csv file with two columns: ('label': str, 'text' str), where is label is str of two possible classes.

Below steps are similar with [this notebook](https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM?usp=sharing), where bert model and tokenizer are used to classify lmdb loaded dataset. Only one difference it is the dataset loaded from .csv file.
Here is how I load it:

```python
data_path = 'data.csv'
data = pd.read_csv(data_path)

# process class name to indices
classes = ['neg', 'pos']
class_to_idx = { cl: i for i, cl in enumerate(classes) }

# now data is like {'label': int, 'text' str}
data['label'] = data['label'].apply(lambda x: class_to_idx[x])

# load dataset and map it with defined `tokenize` function
features = Features({
  target: ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None),
  feature: Value(dtype='string', id=None),
})
dataset = Dataset.from_pandas(data, features=features)
dataset.map(tokenize, batched=True, batch_size=len(dataset))
```

It ruins on the last line with following error:
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-112-32b6275ce418> in <module>()
      9 })
     10 dataset = Dataset.from_pandas(data, features=features)
---> 11 dataset.map(tokenizer, batched=True, batch_size=len(dataset))

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1237         test_inputs = self[:2] if batched else self[0]
   1238         test_indices = [0, 1] if batched else 0
-> 1239         update_data = does_function_return_dict(test_inputs, test_indices)
   1240         logger.info("Testing finished, running the mapping function on the dataset")
   1241 

/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1208             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1209             processed_inputs = (
-> 1210                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1211             )
   1212             does_return_dict = isinstance(processed_inputs, Mapping)

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py in __call__(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2281             )
   2282         ), (
-> 2283             "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) "
   2284             "or `List[List[str]]` (batch of pretokenized examples)."
   2285         )

AssertionError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

which I think is not expected. I also tried the same steps using `Dataset.from_csv` which resulted in the same error.

For reproducing this, I used [this dataset from kaggle](https://www.kaggle.com/team-ai/spam-text-message-classification)