Incremental dataset (e.g. `.push_to_hub(..., append=True)`)

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)

Labels: name: enhancement

Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2. 

Is there any update on this?

Is there any update on this?

No update so far on this feature request but for broader context, this announce will help with incremental datasets https://huggingface.co/blog/xethub-joins-hf :)

Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

> Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right

@tolgadur bro, you may set 'data_dir' as a temporary solution like that:

`Dataset.push_to_hub('your_repo_aame', data_dir='data_{index}')`


Related to this feature request: `pyarrow` 21 is out and added content defined chunking for parquet, which enables deduped uploads to Xet.

Therefore we can have a logic in append=True that modifies an existing Parquet file to add more rows without having to reupload the full file.

will be exciting to see this

> > Still no update? Whats the current recommended way to upload large datasets to the hub? I can't load it all to memory after some limit right
> 
> [@tolgadur](https://github.com/tolgadur) bro, you may set 'data_dir' as a temporary solution like that:
> 
> `Dataset.push_to_hub('your_repo_aame', data_dir='data_{index}')`

This seems like will make the first dir not shwon in online viewer

I prototyped all three approaches to measure actual Xet chunk reuse. Full notebook: https://github.com/lex00/parquet-xet/blob/main/append-prototype.ipynb

**Setup**: 100K-row synthetic dataset (10 shards × 10K rows), appending 1K new rows (1% growth). Chunk simulation uses fixed 64KB blocks + Blake2b — conservative proxy for Xet CDC.

| Approach | Xet chunk reuse | Upload (1K new rows) | Notes |
|----------|----------------|---------------------|-------|
| Reshard (current behavior) | 0% | ~26 MB | Row ranges shift across all shards |
| Add-shard (keep existing shards, append new shard) | 98.8% | ~320 KB | Only new shard uploads |
| CDC-rewrite (single file, `row_group_size` stable) | 98.8% | ~320 KB | 406/411 chunks identical; 5 changed at tail |

**Why CDC-rewrite works for append**: Parquet writes row group data sequentially before the footer. When you append rows with a consistent `row_group_size`, the existing row groups produce byte-identical column chunks — pyarrow's deterministic output guarantees this. Xet reuses those chunks; only the new row group(s) + updated footer are new bytes.

**The tradeoff**: Add-shard is simpler and available today, but shards accumulate and become unequal over time (compaction eventually needed). CDC-rewrite avoids shard fragmentation but requires managing the single-file ↔ multi-shard lifecycle. Pyarrow 21's `use_content_defined_chunking=True` extends the benefit to mid-dataset inserts, not just end-appends.

A possible hybrid: use add-shard for `append=True` now and offer `compact=True` to trigger a CDC-rewrite pass when the user decides to pay the compaction cost.

Thanks @lex00 :) we could have the CDC approach which would download the last Parquet file of the dataset, append the row (and possibly make a new file if we cross the max shard size), and let CDC + xet only upload the updated files chunks

Btw we are working on an API to make files edits on HF easier, it will take a bit of time but I'll share more info once it's ready for people who want to be able to append to Parquet files on HF without having to download+reupload.