Narrativeqa (with full text)
State: closed | Author: ghomasHudson (CONTRIBUTOR)

Following the uploading of the full text data in #309, I've added the full text to the narrativeqa dataset.

Few notes:
- Had some encoding issues using the default `open` so am using `open(encoding="latin-1"...` which seems to fix it. Looks fine.
-  Can't get the dummy data to work. Currently putting stuff at: 
    ```
    dummy
    |---- 0.0.0
            |- dummy_data.zip
                |-master.zip
                |   |- narrativeqa-master
                |       |- documents.csv
                |       |- qaps.csv
                |       |- third_party ......
                | 
                | - narrativeqa_full_text.zip
                |       | - 001.content
                |       | - ....
    ```
    Not sure what I'm messing up here (probably something obvious).

Comments: I took a look at the dummy data creation for this dataset.

Maybe it didn't work on your side might be because `master.zip` and `narrativeqa_full_text.zip` are supposed to be directories and not acutal zip files in the dummy data folder.

I managed to make it work with this `dummy_data.zip` file:
https://drive.google.com/file/d/1G9ZHAjelazNApbFI0ep2dnSAWklXgGMd/view?usp=sharing

@lhoestq Hmmm wasn't that. Must have been something else I missed.

Have committed your working version though now.

Ok thanks.
Could you rebase from master to fix the CI please ?

Hi @ghomasHudson, did you get the chance to add the test split and regenerate the dataset_infos.json file  ?

> Hi @ghomasHudson, did you get the chance to add the test split and regenerate the dataset_infos.json file ?

Have added the test set code but getting an OverflowError when trying to regen the dataset_infos.json:

---
OverflowError: There was an overflow in the <class 'pyarrow.lib.StructArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

---


Thanks for reporting @ghomasHudson , I'll look into it

It looks like it's an issue with Pyarrow.
By changing the `DEFAULT_MAX_BATCH_SIZE` to 1000 instead of 10 000 in `arrow_writer.py` I was able to run the command.

Basically it seems that is an Arrow StructArray has more than 1-2GB of data, then it shuffles some of its content.
I can't find any issue on Apache Arrow's JIRA about this problem. It will require more investigation.

Maybe we can simply automatically decrease the writer's batch size when this happens. We can just check if the arrow array is more than a certain amount of bytes. 

@lhoestq I've finally got round to regenerating the `dataset_infos.json` for this and adding all 3 splits. I've done this and updated for the new version of datasets.

The CI tests still aren't passing though (they pass on my machine). `test_load_dataset_narrativeqa` seems to fail but I have no idea how. Would appreciate if you have any ideas - would be great to finally finish this one!

The dummy data test fails, apparently it's because no examples are yielded for the dummy data.

Also it looks like the PR now show changes in many other files than the ones for NarrativeQA, could you create another branch and another PR please ?

Feel free to ping me on the new PR so we can fi the dummy data together