audio column not saved correctly after resampling
State: closed | Author: laphang (NONE)
Labels: name: bug

## Describe the bug
After resampling the audio column, saving with save_to_disk doesn't seem to save with the correct type. 

## Steps to reproduce the bug
- load a subset of common voice dataset (48Khz)
- resample audio column to 16Khz
- save with save_to_disk()
- load with load_from_disk()

## Expected results
I expected that after saving the data, and then loading it back in, the audio column has the correct dataset.Audio type (i.e. same as before saving it)
{'accent': Value(dtype='string', id=None),
 'age': Value(dtype='string', id=None),
 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None),
 'client_id': Value(dtype='string', id=None),
 'down_votes': Value(dtype='int64', id=None),
 'gender': Value(dtype='string', id=None),
 'locale': Value(dtype='string', id=None),
 'path': Value(dtype='string', id=None),
 'segment': Value(dtype='string', id=None),
 'sentence': Value(dtype='string', id=None),
 'up_votes': Value(dtype='int64', id=None)}

## Actual results
Audio column does not have the right type
{'accent': Value(dtype='string', id=None),
 'age': Value(dtype='string', id=None),
 'audio': {'bytes': Value(dtype='binary', id=None),
  'path': Value(dtype='string', id=None)},
 'client_id': Value(dtype='string', id=None),
 'down_votes': Value(dtype='int64', id=None),
 'gender': Value(dtype='string', id=None),
 'locale': Value(dtype='string', id=None),
 'path': Value(dtype='string', id=None),
 'segment': Value(dtype='string', id=None),
 'sentence': Value(dtype='string', id=None),
 'up_votes': Value(dtype='int64', id=None)}

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: linux
- Python version:
- PyArrow version:


Comments: Hi ! We just released a new version of `datasets` that should fix this.

I tested resampling and using save/load_from_disk afterwards and it seems to be fixed now

Hi @lhoestq, 

Just tested the latest datasets version, and confirming that this is fixed for me. 

Thanks!

Also, just an FYI, data that I had saved (with save_to_disk) previously from common voice using datasets==1.17.0 now give the error below when loading (with load_from disk) using datasets==1.18.0. 

However, when starting fresh using load_dataset, then doing the resampling, the save/load_from disk worked fine. 

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed exec> in <module>

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1747         return Dataset.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1748     elif fs.isfile(Path(dest_dataset_path, config.DATASETDICT_JSON_FILENAME).as_posix()):
-> 1749         return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1750     else:
   1751         raise FileNotFoundError(

/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py in load_from_disk(dataset_dict_path, fs, keep_in_memory)
    769                 else Path(dest_dataset_dict_path, k).as_posix()
    770             )
--> 771             dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
    772         return dataset_dict
    773 

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1118             info=dataset_info,
   1119             split=split,
-> 1120             fingerprint=state["_fingerprint"],
   1121         )
   1122 

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in __init__(self, arrow_table, info, split, indices_table, fingerprint)
    655         if self.info.features.type != inferred_features.type:
    656             raise ValueError(
--> 657                 f"External features info don't match the dataset:\nGot\n{self.info.features}\nwith type\n{self.info.features.type}\n\nbut expected something like\n{inferred_features}\nwith type\n{inferred_features.type}"
    658             )
    659 

ValueError: External features info don't match the dataset:
Got
{'accent': Value(dtype='string', id=None), 'age': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=48000, mono=True, id=None), 'client_id': Value(dtype='string', id=None), 'down_votes': Value(dtype='int64', id=None), 'gender': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None)}
with type
struct<accent: string, age: string, audio: struct<bytes: binary, path: string>, client_id: string, down_votes: int64, gender: string, locale: string, path: string, segment: string, sentence: string, up_votes: int64>

but expected something like
{'accent': Value(dtype='string', id=None), 'age': Value(dtype='string', id=None), 'audio': {'path': Value(dtype='string', id=None), 'bytes': Value(dtype='binary', id=None)}, 'client_id': Value(dtype='string', id=None), 'down_votes': Value(dtype='int64', id=None), 'gender': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None)}
with type
struct<accent: string, age: string, audio: struct<path: string, bytes: binary>, client_id: string, down_votes: int64, gender: string, locale: string, path: string, segment: string, sentence: string, up_votes: int64> 
```