Repository: huggingface/datasets
Pull request: Deprecate num_proc parameter in  DownloadManager.extract
fixes #5132 : Deprecated the `num_proc` parameter in `DownloadManager.extract` by passing `num_proc` parameter to `map_nested` .

Problem statement:
Depracate `num_proc` parameter in `DownloadManager.extract`
The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.

Hints from issue discussion:
I can take this! #self-assign
#self-assign
@lazarust i'm already working on this issue :smile: 
#self-assign

Commit message: deprecate num_proc

Patch to review:
diff --git a/src/datasets/download/download_manager.py b/src/datasets/download/download_manager.py
index ef06e6a7fe3..06f7534fcaa 100644
--- a/src/datasets/download/download_manager.py
+++ b/src/datasets/download/download_manager.py
@@ -377,7 +377,7 @@ def iter_files(self, paths: Union[str, List[str]]):
         """
         return FilesIterable.from_paths(paths)
 
-    def extract(self, path_or_paths, num_proc=None):
+    def extract(self, path_or_paths):
         """Extract given path(s).
 
         Args:
@@ -402,10 +402,12 @@ def extract(self, path_or_paths, num_proc=None):
         # Extract downloads the file first if it is not already downloaded
         if download_config.download_desc is None:
             download_config.download_desc = "Downloading data"
+        if download_config.num_proc is None:
+            download_config.num_proc = 16
         extracted_paths = map_nested(
             partial(cached_path, download_config=download_config),
             path_or_paths,
-            num_proc=num_proc,
+            num_proc=download_config.num_proc,
             disable_tqdm=not is_progress_bar_enabled(),
             desc="Extracting data files",
         )


Code review comments:
diff_hunk: @@ -377,7 +377,7 @@ def iter_files(self, paths: Union[str, List[str]]):
         """
         return FilesIterable.from_paths(paths)
 
-    def extract(self, path_or_paths, num_proc=None):
+    def extract(self, path_or_paths):
path: src/datasets/download/download_manager.py
text: @user1:
"Deprecating a param" does not mean we can remove the parameter abruptly. This is what we should do instead:
```python
def extract(self, path_or_paths, num_proc="deprecated"):
    """Extract given path(s).

    Args:
        path_or_paths: path or `list`/`dict` of path of file to extract. Each
            path is a `str`.
        num_proc: Use multi-processing if `num_proc` > 1 and the length of
            `path_or_paths` is larger than `num_proc`

            <Deprecated version="2.6.2">

            Pass `DownloadConfig(num_proc=<num_proc>)` to the initializer instead.

            </Deprecated>

    Returns:
        extracted_path(s): `str`, The extracted paths matching the given input
            path_or_paths.

    Example:

    ```py
    >>> downloaded_files = dl_manager.download('https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz')
    >>> extracted_files = dl_manager.extract(downloaded_files)
    ```
    """
    if num_proc != "deprecated":
        warnings.warn(
            "'num_proc' was deprecated in version 2.6.2 and will be removed in 3.0.0. Pass `DownloadConfig(num_proc=<num_proc>)` to the initializer instead.",
            FutureWarning,
        )
    # the rest of the code ...
```

@author:
Thanks @user1 for your help. I commited the changes as you said.