Issue: Pushing a large dataset on the hub consistently hangs

Comment:
I tried trying to catch the hanging issue in action again

```
Pushing dataset shards to the dataset hub:  65%|█████████████████████████████████████████████████████████████████▊                                   | 127/195 [2:28:02<1:19:15, 69.94s/it]                                               
Error while uploading 'data/train-00127-of-00195-3f8d036ade107c27.parquet' to the Hub.                                                                                                                                                    
Pushing split train to the Hub.                                                                                                                                                                                                           
Pushing dataset shards to the dataset hub:  64%|████████████████████████████████████████████████████████████████▏                                    | 124/195 [2:06:10<1:12:14, 61.05s/it]C^[^C^C^C                                      
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮                                                                                                                                      
│ /TALI/tali/scripts/validate_dataset.py:127 in <module>                                           │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│   124 │                                                                                          │                                                                                                                                      
│   125 │   while not succesful_competion:                                                         │                                                                                                                                      
│   126 │   │   try:                                                                               │                                                                                                                                      
│ ❱ 127 │   │   │   train_dataset.push_to_hub(repo_id="Antreas/TALI-base", max_shard_size="5GB")   │                                                                                                                                      
│   128 │   │   │   succesful_competion = True                                                     │                                                                                                                                      
│   129 │   │   except Exception as e:                                                             │                                                                                                                                      
│   130 │   │   │   print(e)                                                                       │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/dataset_dict.py:1583 in push_to_hub   │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│   1580 │   │   for split in self.keys():                                                         │                                                                                                                                      
│   1581 │   │   │   logger.warning(f"Pushing split {split} to the Hub.")                          │                                                                                                                                      
│   1582 │   │   │   # The split=key needs to be removed before merging                            │                                                                                                                                      
│ ❱ 1583 │   │   │   repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parq  │                                                                                                                                      
│   1584 │   │   │   │   repo_id,                                                                  │                                                                                                                                      
│   1585 │   │   │   │   split=split,                                                              │                                                                                                                                      
│   1586 │   │   │   │   private=private,                                                          │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:5263 in              │                                                                                                                                      
│ _push_parquet_shards_to_hub                                                                      │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│   5260 │   │                                                                                     │                                                                                                                                      
│   5261 │   │   uploaded_size = 0                                                                 │                                                                                                                                      
│   5262 │   │   shards_path_in_repo = []                                                          │                                                                                                                                      
│ ❱ 5263 │   │   for index, shard in logging.tqdm(                                                 │                                                                                                                                      
│   5264 │   │   │   enumerate(itertools.chain([first_shard], shards_iter)),                       │                                                                                                                                      
│   5265 │   │   │   desc="Pushing dataset shards to the dataset hub",                             │                                                                                                                                      
│   5266 │   │   │   total=num_shards,                                                             │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│ /opt/conda/envs/main/lib/python3.10/site-packages/tqdm/std.py:1178 in __iter__                   │                                                                                                                                      
│                                                                                                  │                                                                                                                                      
│   1175 │   │   time = self._time                                                                 │                                                                                                                                      
│   1176 │   │                                                                                     │                                                                                                                                      
│   1177 │   │   try:                                                                              │
│ ❱ 1178 │   │   │   for obj in iterable:                                                          │
│   1179 │   │   │   │   yield obj                                                                 │
│   1180 │   │   │   │   # Update and possibly print the progressbar.                              │
│   1181 │   │   │   │   # Note: does not call self.update(1) for speed optimisation.              │
│                                                                                                  │
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:5238 in              │
│ shards_with_embedded_external_files                                                              │
│                                                                                                  │
│   5235 │   │   │   │   for shard in shards:                                                      │
│   5236 │   │   │   │   │   format = shard.format                                                 │
│   5237 │   │   │   │   │   shard = shard.with_format("arrow")                                    │
│ ❱ 5238 │   │   │   │   │   shard = shard.map(                                                    │
│   5239 │   │   │   │   │   │   embed_table_storage,                                              │
│   5240 │   │   │   │   │   │   batched=True,                                                     │
│   5241 │   │   │   │   │   │   batch_size=1000,                                                  │
│                                                                                                  │
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:578 in wrapper       │
│                                                                                                  │
│    575 │   │   else:                                                                             │
│    576 │   │   │   self: "Dataset" = kwargs.pop("self")                                          │
│    577 │   │   # apply actual function                                                           │
│ ❱  578 │   │   out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)                │                                         
│    579 │   │   datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [ou  │                                         
│    580 │   │   for dataset in datasets:                                                          │                                         
│    581 │   │   │   # Remove task templates if a column mapping of the template is no longer val  │                                         
│                                                                                                  │                                         
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:543 in wrapper       │                                         
│                                                                                                  │                                         
│    540 │   │   │   "output_all_columns": self._output_all_columns,                               │                                         
│    541 │   │   }                                                                                 │                                         
│    542 │   │   # apply actual function                                                           │                                                                  
│ ❱  543 │   │   out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)                │                                                                  
│    544 │   │   datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [ou  │                                                                  
│    545 │   │   # re-apply format to the output                                                   │                                                                  
│    546 │   │   for dataset in datasets:                                                          │                                                                  
│                                                                                                  │                                                                  
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:3073 in map          │                                                                  
│                                                                                                  │                                                                  
│   3070 │   │   │   │   │   leave=False,                                                          │                                                                  
│   3071 │   │   │   │   │   desc=desc or "Map",                                                   │                                                                  
│   3072 │   │   │   │   ) as pbar:                                                                │                                                                  
│ ❱ 3073 │   │   │   │   │   for rank, done, content in Dataset._map_single(**dataset_kwargs):     │                                                                  
│   3074 │   │   │   │   │   │   if done:                                                          │                                                                  
│   3075 │   │   │   │   │   │   │   shards_done += 1                                              │                                                                                                     
│   3076 │   │   │   │   │   │   │   logger.debug(f"Finished processing shard number {rank} of {n  │                                                                                                     
│                                                                                                  │                                                                                                     
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py:3464 in _map_single  │                                                                                                     
│                                                                                                  │                                                                                                     
│   3461 │   │   │   │   │   │   │   │   buf_writer, writer, tmp_file = init_buffer_and_writer()   │                                                                                                     
│   3462 │   │   │   │   │   │   │   │   stack.enter_context(writer)                               │                                                                                                     
│   3463 │   │   │   │   │   │   │   if isinstance(batch, pa.Table):                               │                                                                                                     
│ ❱ 3464 │   │   │   │   │   │   │   │   writer.write_table(batch)                                 │                                                                                                     
│   3465 │   │   │   │   │   │   │   else:                                                         │                                                                                                     
│   3466 │   │   │   │   │   │   │   │   writer.write_batch(batch)                                 │                                                                                                     
│   3467 │   │   │   │   │   │   num_examples_progress_update += num_examples_in_batch             │                                                                                                     
│                                                                                                  │                                                                                                     
│ /opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_writer.py:567 in write_table    │                                                                                                     
│                                                                                                  │                                                                                                     
│   564 │   │   │   writer_batch_size = self.writer_batch_size                                     │                                                                                                     
│   565 │   │   if self.pa_writer is None:                                                         │                                                                                                     
│   566 │   │   │   self._build_writer(inferred_schema=pa_table.schema)                            │                                                                                                     
│ ❱ 567 │   │   pa_table = pa_table.combine_chunks()                                               │                                                                                                     
│   568 │   │   pa_table = table_cast(pa_table, self._schema)                                      │                                                                                                     
│   569 │   │   if self.embed_local_files:                                                         │                                                                                                     
│   570 │   │   │   pa_table = embed_table_storage(pa_table)                                       │                                                                                                     
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                     
KeyboardInterrupt                                                                                   
```