TypeError: Couldn't cast array of type string to null
Opened by TJ-Solergibert (closed)

### Describe the bug

Processing a dataset I alredy uploaded to the Hub (https://huggingface.co/datasets/tj-solergibert/Europarl-ST) I found that for some splits and some languages (test split, source_lang = "nl") after applying a map function I get the mentioned error. 

I alredy tried reseting the shorter strings (reset_cortas function). It only happends with NL, PL, RO and PT. It does not make sense since when processing the other languages I also use the corpus of those that fail and it does not cause any errors.

I suspect that the error may be in this direction:

We use cast_array_to_feature to support casting to custom types like Audio and Image # Also, when trying type "string", we don't want to convert integers or floats to "string". # We only do it if trying_type is False - since this is what the user asks for.

### Steps to reproduce the bug

Here I link a colab notebook to reproduce the error:

https://colab.research.google.com/drive/1JCrS7FlGfu_kFqChMrwKZ_bpabnIMqbP?authuser=1#scrollTo=FBAvlhMxIzpA

### Expected behavior

Data processing does not fail. A correct example can be seen here: https://huggingface.co/datasets/tj-solergibert/Europarl-ST-processed-mt-en

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

Comments:
Thanks for reporting, @TJ-Solergibert.

We cannot access your Colab notebook: `There was an error loading this notebook. Ensure that the file is accessible and try again.`
Could you please make it publicly accessible?


I swear it's public, I've checked the settings and I've been able to open it in incognito mode.

Notebook: https://colab.research.google.com/drive/1JCrS7FlGfu_kFqChMrwKZ_bpabnIMqbP?usp=sharing

Anyway, this is the code to reproduce the error:

```python3
from datasets import ClassLabel
from datasets import load_dataset

europarl_ds = load_dataset("tj-solergibert/Europarl-ST")

source_lang = "nl"
languages = list(europarl_ds["train"][0]["transcriptions"].keys())
ClassLabels = ClassLabel(num_classes = len(languages), names = languages)

def map_label2id(example):
    example['dest_lang'] = ClassLabels.str2int(example['dest_lang'])
    return example

def unfold_transcriptions(example):
  for lang in languages:
    example[lang] = example["transcriptions"][lang]
  return example

def unroll(batch, src_lang, dest_langs):
  source_t, dest_t, dest_l = [], [], []
  for lang in dest_langs: 
    source_t += batch[src_lang]
    dest_t += batch[lang]
    dest_l += [lang]
  return_dict = {"source_text": source_t, "dest_text": dest_t, "dest_lang": dest_l}
  return return_dict

def preprocess_split(ds_split, src_lang):
  dest_langs = [x for x in languages if x != src_lang]

  ds_split = ds_split.map(unroll, fn_kwargs= {"src_lang": src_lang, "dest_langs": dest_langs}, batched = True, batch_size = 1, remove_columns= list(languages))
  ds_split = ds_split.filter(lambda x: x["source_text"] != None and x["dest_text"] != None) # Remove incomplete translations
  ds_split = ds_split.filter(lambda x: x["source_text"] != "None" and x["dest_text"] != "None")
  ds_split = ds_split.map(map_label2id) 
  ds_split = ds_split.cast_column("dest_lang", ClassLabels)
  return ds_split

def reset_cortas(example):
  for lang in languages:
    if isinstance(example[lang], str):
      if example[lang].isnumeric () or len(example[lang]) <= 5:
        example[lang] = "None"
  return example

def clean_dataset(dataset):
  # Remove columns
  dataset = dataset.remove_columns(["original_speech", "original_language", "audio_path", "segment_start", "segment_end"])
  # Unfold
  dataset = dataset.map(unfold_transcriptions, remove_columns = ["transcriptions"])
  dataset = dataset.map(reset_cortas)
  return dataset

processed_europarl = clean_dataset(europarl_ds["test"])
new_train_ds = preprocess_split(processed_europarl, 'nl')
```

Thanks, @TJ-Solergibert. I can access your notebook now. Maybe it was just a temporary issue.

At first sight, it seems something related to your data: maybe some of the examples do not have all the transcriptions for all the languages. Then, some of them are null when unrolled. And when trying to concatenate with the other rows containing strings, the cast issue is raised (the arrays to be concatenated have different types).

Do you think this could be the case?

See, in this example, "nl" and "ro" transcripts are null:
```python
>>> europarl_ds["test"][:1]
{'original_speech': ['− Señor Presidente, en primer lugar, quisiera felicitar al señor Seeber por el trabajo realizado, porque en su informe se recogen muchas de las preocupaciones manifestadas en esta'],
 'original_language': ['es'],
 'audio_path': ['es/audios/en.20081008.24.3-238.m4a'],
 'segment_start': [0.6200000047683716],
 'segment_end': [11.319999694824219],
 'transcriptions': [{'de': '− Herr Präsident! Zunächst möchte ich Richard Seeber zu der von ihm geleisteten Arbeit gratulieren, denn sein Bericht greift viele der in diesem Haus zum Ausdruck gebrachten Anliegen',
   'en': '− Mr President, firstly I would like to congratulate Mr Seeber on the work he has done, because his report picks up many of the concerns expressed in this',
   'es': '− Señor Presidente, en primer lugar, quisiera felicitar al señor Seeber por el trabajo realizado, porque en su informe se recogen muchas de las preocupaciones manifestadas en esta',
   'fr': '− Monsieur le Président, je voudrais tout d ’ abord féliciter M. Seeber pour le travail qu ’ il a effectué, parce que son rapport reprend beaucoup des inquiétudes exprimées au sein de cette',
   'it': "− Signor Presidente, mi congratulo innanzi tutto con l'onorevole Seeber per il lavoro svolto, perché la sua relazione accoglie molti dei timori espressi da quest'Aula",
   'nl': None,
   'pl': '− Panie przewodniczący! Po pierwsze chciałabym pogratulować panu posłowi Seeberowi wykonanej pracy, ponieważ jego sprawozdanie podejmuje szereg podnoszonych w tej Izbie',
   'pt': '− Senhor Presidente, começo por felicitar o senhor deputado Seeber pelo trabalho que desenvolveu em torno deste relatório, que retoma muitas das preocupações expressas nesta',
   'ro': None}]}
```
```python
>>> processed_europarl[0]
{'de': '− Herr Präsident! Zunächst möchte ich Richard Seeber zu der von ihm geleisteten Arbeit gratulieren, denn sein Bericht greift viele der in diesem Haus zum Ausdruck gebrachten Anliegen',
 'en': '− Mr President, firstly I would like to congratulate Mr Seeber on the work he has done, because his report picks up many of the concerns expressed in this',
 'es': '− Señor Presidente, en primer lugar, quisiera felicitar al señor Seeber por el trabajo realizado, porque en su informe se recogen muchas de las preocupaciones manifestadas en esta',
 'fr': '− Monsieur le Président, je voudrais tout d ’ abord féliciter M. Seeber pour le travail qu ’ il a effectué, parce que son rapport reprend beaucoup des inquiétudes exprimées au sein de cette',
 'it': "− Signor Presidente, mi congratulo innanzi tutto con l'onorevole Seeber per il lavoro svolto, perché la sua relazione accoglie molti dei timori espressi da quest'Aula",
 'nl': None,
 'pl': '− Panie przewodniczący! Po pierwsze chciałabym pogratulować panu posłowi Seeberowi wykonanej pracy, ponieważ jego sprawozdanie podejmuje szereg podnoszonych w tej Izbie',
 'pt': '− Senhor Presidente, começo por felicitar o senhor deputado Seeber pelo trabalho que desenvolveu em torno deste relatório, que retoma muitas das preocupações expressas nesta',
 'ro': None}
```

You can fix this issue by forcing the cast of None to str by hand:
- If you replace this line:
```python
source_t += batch[src_lang]
```
- With this line (because the batch size is 1):
```python
source_t += [str(batch[src_lang][0])]
```
- Or with this line (if the batch size were larger than 1):
```python
source_t += [str(text) for text in batch[src_lang]]
```

Problem solved! Thanks @albertvillanova, now I have even increased the batch size and it's crazy fast :rocket: !