Issue: Pushing a large dataset on the hub consistently hangs

Comment:
Right. So I did the trick @lhoestq suggested. Here is where things seem to hang

```
Error while uploading 'data/train-00120-of-00195-466c2dbab2eb9989.parquet' to the Hub.                                                                                                     
Pushing split train to the Hub.                                                                                                                                                            
Creating parquet from Arrow format: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.15s/ba]
Upload 1 LFS files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:52<00:00, 52.12s/it]
Creating parquet from Arrow format: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.08s/ba]
Upload 1 LFS files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:45<00:00, 45.54s/it]
Creating parquet from Arrow format: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.08s/ba]
Creating parquet from Arrow format: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00,  1.03s/ba^Upload 1 LFS files:   0%|                                                                                                                                         | 0/1 [
21:27:35<?, ?it/s]                                                                                                                                                                         
Pushing dataset shards to the dataset hub:  63%|█████████████████████████████████████████████████████████████▎                                    | 122/195 [23:37:11<14:07:59, 696.98s/it]
^CError in sys.excepthook:                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                         
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1699, in print                                                                                            
    extend(render(renderable, render_options))                                                                                                                                             
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1335, in render                                                                                           
    yield from self.render(render_output, _options)                                                                                                                                        
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1331, in render                                                                                           
    for render_output in iter_render:                                                                                                                                                      
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/constrain.py", line 29, in __rich_console__                                                                                 
    yield from console.render(self.renderable, child_options)                                                                                                                              
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1331, in render                                                                                           
    for render_output in iter_render:                                                                                                                                                      
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/panel.py", line 220, in __rich_console__                                                                                    
    lines = console.render_lines(renderable, child_options, style=style)                                                                                                                   
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1371, in render_lines                                                                                     
    lines = list(                                                                                                                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/segment.py", line 292, in split_and_crop_lines                                                                              
    for segment in segments:                                                                                                                                                               
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1331, in render                                                                                           
    for render_output in iter_render:                                                                                                                                                      
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/padding.py", line 97, in __rich_console__                                                                                   
    lines = console.render_lines(                                                                                                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1371, in render_lines                                                                                     
    lines = list(                                                                                                                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/segment.py", line 292, in split_and_crop_lines                                                                              
    for segment in segments:                                                                                                                                                               
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1335, in render                                                                                           
    yield from self.render(render_output, _options)                                                                                                                                        
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/console.py", line 1331, in render                                                                                           
    for render_output in iter_render:                                                                                                                                                      
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/syntax.py", line 611, in __rich_console__                                                                                   
    segments = Segments(self._get_syntax(console, options))                                                                                                                                
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/segment.py", line 668, in __init__                                                                                          
    self.segments = list(segments)                                                                                                                                                         
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/syntax.py", line 674, in _get_syntax                                                                                        
    lines: Union[List[Text], Lines] = text.split("\n", allow_blank=ends_on_nl)                                                                                                             
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/text.py", line 1042, in split                                                                                               
    lines = Lines(                                                                                                                                                                         
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/containers.py", line 70, in __init__                                                                                        
    self._lines: List["Text"] = list(lines)                                                                                                                                                
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/text.py", line 1043, in <genexpr>                                                                                           
    line for line in self.divide(flatten_spans()) if line.plain != separator                                                                                                               
  File "/opt/conda/envs/main/lib/python3.10/site-packages/rich/text.py", line 385, in plain                                                    
    if len(self._text) != 1:                                                                                                                                                               
KeyboardInterrupt                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                            
Original exception was:                                                                                                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map                                                                                                                                                                               
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))                                                                                                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/tqdm/std.py", line 1178, in __iter__                                                                                                                                                                                                 
    for obj in iterable:                                                                                                                                                                   
  File "/opt/conda/envs/main/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator                                                                                                                                                                                         
    yield _result_or_cancel(fs.pop())                                                                                                                                                      
  File "/opt/conda/envs/main/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel                                                                                                                                                                                       
    return fut.result(timeout)                                                                                                                                                             
  File "/opt/conda/envs/main/lib/python3.10/concurrent/futures/_base.py", line 453, in result                                                                                                                                                                                                  
    self._condition.wait(timeout)                                                                                                                                                                                                           
  File "/opt/conda/envs/main/lib/python3.10/threading.py", line 320, in wait                                                                                                                                                                                                                   
    waiter.acquire()                                                                                                                                                                                                                        
KeyboardInterrupt                                                                                                                                                                                                                                                                              
                                                                                                                      
During handling of the above exception, another exception occurred:                                                                                                                                                                                                                            
                                                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                                                                             
  File "/TALI/tali/scripts/validate_dataset.py", line 127, in <module>                                                                                                            
    train_dataset.push_to_hub(repo_id="Antreas/TALI-base", max_shard_size="5GB")                                                                                                                                                                                                               
  File "/opt/conda/envs/main/lib/python3.10/site-packages/datasets/dataset_dict.py", line 1583, in push_to_hub                                                                                                                                                                                                                                                      
    repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parquet_shards_to_hub(                                                                                                                                                                                             
  File "/opt/conda/envs/main/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5275, in _push_parquet_shards_to_hub                                                                                                                                                                                                                                     
    _retry(                                                                                                                                                                                                                                                                                    
  File "/opt/conda/envs/main/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 282, in _retry                                                                                                                                                                                                                                                        
    return func(*func_args, **func_kwargs)                                                                                                                                                                                                                                                     
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn                                                                                                                                                                                                                                             
    return fn(*args, **kwargs)                                                                                                                 
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 826, in _inner                                                                                                                                                                                                                                                           
    return fn(self, *args, **kwargs)                                                                                                                                                                                                                                                           
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 3205, in upload_file                                                                                                                                                                                                                                                     
    commit_info = self.create_commit(                                  
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn                                                                                                                                                                                                                                             
    return fn(*args, **kwargs)                                                           
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 826, in _inner                                                                                                                                                                                                                                                           
    return fn(self, *args, **kwargs)                                   
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2680, in create_commit                                                                                                                                                                                                                                                   
    upload_lfs_files(                                                                    
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn                                                                                                                                                                                                                                             
    return fn(*args, **kwargs)                                                           
  File "/opt/conda/envs/main/lib/python3.10/site-packages/huggingface_hub/_commit_api.py", line 353, in upload_lfs_files                                                                                                                                                                                                                                            
    thread_map(                                                                          
  File "/opt/conda/envs/main/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map                                                                                                                                                                                                                                                       
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)                                                                                                       
  File "/opt/conda/envs/main/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 49, in _executor_map                                                                                                                                                                                                                                                    
    with PoolExecutor(max_workers=max_workers, initializer=tqdm_class.set_lock,                                                                                                   
  File "/opt/conda/envs/main/lib/python3.10/concurrent/futures/_base.py", line 649, in __exit__                                                                                                                                                                                                                                                                     
    self.shutdown(wait=True)                                                             
  File "/opt/conda/envs/main/lib/python3.10/concurrent/futures/thread.py", line 235, in shutdown                                                                                                                                                                                                                                                                    
    t.join()                                                                             
  File "/opt/conda/envs/main/lib/python3.10/threading.py", line 1096, in join                                                                                                     
    self._wait_for_tstate_lock()                                                         
  File "/opt/conda/envs/main/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock                                                                                                                                                                                                                                                                      
    if lock.acquire(block, timeout):                                                     
KeyboardInterrupt                                                                        
```