Large dataset in Squad2-format

At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ?


I'm pretty sure you can get some inspiration from the squad_v2 script. It looks like the dataset is quite big so it will take some time for the users to generate it, but it should be reasonable.

Also you are saying that you are still making the dataset grow in size right ?
It's probably good practice to let the users do their training/evaluations with the exact same version of the dataset.
We allow for each dataset to specify a version (ex: 1.0.0) and increment this number every time there are new samples in the dataset for example. Does it look like a good solution for you ? Or would you rather have one final version with the full dataset ?

It would also be good if there is any possibility for versioning, I think this way is much better than the dynamic way.
If you mean that part to put the tiles into one is the generation it would take up to 15-20 minutes on home computer hardware.
Are there any compression or optimization algorithms while generating the dataset ?
Otherwise the hardware limit is around 32 GB ram at the moment.
If everything works well we will add some more gigabytes of data in future what would make it pretty memory costly.

15-20 minutes is fine !
Also there's no RAM limitations as we save to disk every 1000 elements while generating the dataset by default.
After generation, the dataset is ready to use with (again) no RAM limitations as we do memory-mapping.

Wow, that sounds pretty cool.
Actually I have the problem of running out of memory while tokenization on our local machine.
That wouldn't happen again, would it ?

You can do the tokenization step using `my_tokenized_dataset = my_dataset.map(my_tokenize_function)` that writes the tokenized texts on disk as well. And then `my_tokenized_dataset` will be a memory-mapped dataset too, so you should be fine :)

Does it have an affect to the trainings speed ?

In your training loop, loading the tokenized texts is going to be fast and pretty much negligible compared to a forward pass. You shouldn't expect any slow down.

Closing this one. Feel free to re-open if you have other questions