Implement proper checkpointing for dataset uploading with resume function that does not require remapping shards that have already been uploaded
Opened by AntreasAntoniou (open)

Context: issue #5990

In order to implement the checkpointing, I introduce a metadata folder that keeps one yaml file for each set that one is uploading. This yaml keeps track of what shards have already been uploaded, and which one the idx of the latest one was. Using this information I am then able to easily get the push_to_hub function to retrieve on demand past history of uploads and continue mapping and uploading from where it was left off. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6056). All of your documentation changes will be reflected on that endpoint.

@lhoestq Reading the filenames is something I tried earlier, but I decided to use the yaml direction because:

1. The yaml file name is constructed to retain information about the shard_size, and total number of shards, hence ensuring that the files uploaded are not just files that have the same name but actually represent a different configuration of shard_size, and total number of shards. 
2. Remembering the total file size is done easily in the yaml, whereas alternatively I am not sure how one could access the file size of the uploaded files without downloading them.
3. I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it. 

If 1 and 2 can be achieved without an additional yaml, then I would be willing to make those changes. Let me know of any ideas. 1. could be done by changing the data file names, but I'd rather not do that as to prevent breaking existing datasets that try to upload updates to their data. 

If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo

> I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.

what was the issue ?

> If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo
> 
> > I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.
> 
> what was the issue ?

You are right. I was having some other issue earlier that I need more input from you guys to overcome, and when I overcame it the yaml was just legacy from before. I'll update the PR. 

> If the file name depends on the shard's fingerprint **before** mapping then we can know if a shard has been uploaded before mapping and without requiring an extra YAML file. It should do the job imo
> 
> > I also had an issue earlier with the hashes not being consistent with which the yaml helped -- but this is no longer an issue as I found a way around it.
> 
> what was the issue ?

I remembered what it was, and why I needed the yaml. I needed it so it could remember the progress for a particular num_shards setup, as different num_shards would lead to different number of splits, and a user might switch between them while uploading, and I did not want the index to be conflated with one of another num_shards setup. 

Any idea how we deal with that without a yaml?

If the user changes the num_shards parameters then we should re-upload everything.

It happens that the num_shards is part of the parquet file names, so it restarts the upload from scratch without having to write additional logic :)