ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures
State: closed | Author: timothyjlaurent (NONE)

I tried using the Json dataloader to load some JSON lines files. but get an exception in the parse_schema function.

```
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-23-9aecfbee53bd> in <module>
     55 from nlp import load_dataset
     56 
---> 57 ds = load_dataset("../text2struct/model/dataset_builder.py", data_files=rel_datafiles)
     58 
     59 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    736                     schema_dict[field.name] = Value(str(field.type))
    737 
--> 738         parse_schema(writer.schema, features)
    739         self.info.features = Features(features)
    740 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)
    734                     parse_schema(field.type.value_type, schema_dict[field.name])
    735                 else:
--> 736                     schema_dict[field.name] = Value(str(field.type))
    737 
    738         parse_schema(writer.schema, features)

<string> in __init__(self, dtype, id, _type)

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)
     55 
     56     def __post_init__(self):
---> 57         self.pa_type = string_to_arrow(self.dtype)
     58 
     59     def __call__(self):

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)
     32         if str(type_str + "_") not in pa.__dict__:
     33             raise ValueError(
---> 34                 f"Neither {type_str} nor {type_str + '_'} seems to be a pyarrow data type. "
     35                 f"Please make sure to use a correct data type, see: "
     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"

ValueError: Neither list<item: string> nor list<item: string>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions
```

If I create the dataset imperatively, using a pyarrow table, the dataset is created correctly. If I override the `_prepare_split` method to avoid calling the validate schema, the dataset can load as well. 

Comments: Hi, it depends on what it is in your `dataset_builder.py` file. Can you share it?

If you are just loading `json` files, you can also directly use the `json` script (which will find the schema/features from your JSON structure):

```python
from nlp import load_dataset
ds = load_dataset("json", data_files=rel_datafiles)
```

The behavior I'm seeing is from the `json` script. 
I hacked this together to overcome the error with the `JSON` dataloader

```
class DatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes
        # this is where the error is coming from
        # def parse_schema(schema, schema_dict):
        #     for field in schema:
        #         if pa.types.is_struct(field.type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type, schema_dict[field.name])
        #         elif pa.types.is_list(field.type) and pa.types.is_struct(field.type.value_type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type.value_type, schema_dict[field.name])
        #         else:
        #             schema_dict[field.name] = Value(str(field.type))
        # 
        # parse_schema(writer.schema, features)
        # self.info.features = Features(features)

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table
```

So I basically just don't populate the `self.info.features` though this doesn't seem to cause any problems in my downstream applications. 

The other workaround I was doing was to just use pyarrow.json to build a table and then to create the Dataset with its constructor or from_table methods. `load_dataset` has nice split logic, so I'd prefer to use that.



Also noticed that if you for example in a loader script

```
from nlp import ArrowBasedBuilder

class MyBuilder(ArrowBasedBuilder):
...

```
and use that in the subclass, it will be on the module's __dict__ and will be selected before the `MyBuilder` subclass, and it will raise `NotImplementedError` on its `_generate_examples` method... In the code it check for abstract classes but Builder and ArrowBasedBuilder aren't abstract classes, they're regular classes with `@abstract_methods`.

Indeed this is part of a more general limitation which is the fact that we should generate and update the `features` from the auto-inferred Arrow schema when they are not provided (also happen when a user change the schema using `map()`, the features should be auto-generated and guessed as much as possible to keep the `features` synced with the underlying Arrow table schema).

We will try to solve this soon.