Repository: dirty-cat/dirty_cat
Pull request #501 (2023-02-20 14:31:03)

Support paths as strings in public fetching API
Reintroduces dirty-cat/dirty_cat#453


Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index dde52a7..220c89b 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -3,11 +3,11 @@ version: 2
 jobs:
   python3:
     docker:
-      - image: cimg/python:3.8
+      - image: cimg/python:3.9
     environment:
       - MINICONDA_PATH: ~/miniconda
       - CONDA_ENV_NAME: testenv
-      - PYTHON_VERSION: 3.8
+      - PYTHON_VERSION: 3.9
       - OMP_NUM_THREADS: 2 # Avoid over-commit with HistGradientBoosting
     steps:
       - checkout
@@ -31,7 +31,7 @@ jobs:
 
   deploy:
     docker:
-      - image: cimg/python:3.8
+      - image: cimg/python:3.9
     steps:
       - checkout
       - run: ./build_tools/circle/checkout_merge_commit.sh
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
index 268b600..37258d8 100644
--- a/.github/workflows/codeql-analysis.yml
+++ b/.github/workflows/codeql-analysis.yml
@@ -4,11 +4,6 @@
 # You may wish to alter this file to override the set of languages analyzed,
 # or to provide custom queries or build logic.
 #
-# ******** NOTE ********
-# We have attempted to detect the languages in your repository. Please check
-# the `language` matrix defined below to confirm you have the correct set of
-# supported CodeQL languages.
-#
 name: "CodeQL"
 
 on:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 31af2ae..3adfbf3 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -13,4 +13,4 @@ jobs:
                   repo-token: ${{ secrets.GITHUB_TOKEN  }}
                   artifact-path: 0/doc/index.html
                   circleci-jobs: python3
-                  jobtitle: Check the rendered docs here!
+                  job-title: Check the rendered docs here!
diff --git a/CHANGES.rst b/CHANGES.rst
index 54a01d7..af4d8fe 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -47,6 +47,10 @@ Minor changes
   which can be used to specify where to save and load from datasets.
   :pr:`432` by :user:`Lilian Boulard <LilianBoulard>`
 
+* Fetching functions now have an additional argument ``directory``,
+  which can be used to specify where to save and load from datasets.
+  :pr:`432` and :pr:`453` by :user:`Lilian Boulard <LilianBoulard>`
+
 * The :class:`TableVectorizer`'s default `OneHotEncoder` for low cardinality categorical variables now defaults
   to `handle_unknown="ignore"` instead of `handle_unknown="error"` (for sklearn >= 1.0.0).
   This means that categories seen only at test time will be encoded by a vector of zeroes instead of raising an error. :pr:`473` by :user:`Leo Grinsztajn <LeoGrin>`
diff --git a/RELEASE_PROCESS.rst b/RELEASE_PROCESS.rst
index 0d0f9a4..ded8170 100644
--- a/RELEASE_PROCESS.rst
+++ b/RELEASE_PROCESS.rst
@@ -24,24 +24,23 @@ the main steps and appropriate resources:
 1.  Update ``dirty_cat/CHANGES.rst``. It should be updated at each PR,
     but double-checking before the release is good practice.
 2.  Create a branch by running ``git checkout -b 0.<version>.X``
-    (e.g. ``0.1.X``).
+    (e.g. ``0.2.X``).
 3.  Update ``dirty_cat/dirty_cat/VERSION.txt`` with the new version
-    number (e.g. ``1.0``).
+    number (e.g. ``0.2.0``).
 4.  Update ``dirty_cat/setup.py`` (e.g. Python version supported and dependencies).
-5.  You have to commit the changes with a new tag: the version you're
-    going to push with the commands
-    ``git commit -m "Bump version 0.1.0"``. Push the branch to the ``dirty_cat``
-    repository. This will push and create an associated folder in the documentation
-    repository.
+5.  Commit the changes with a new tag: the version you're going to push,
+    with the commands ``git commit -m "Bump to version 0.2.0"``.
+    Push the branch to the ``dirty_cat`` repository.
+    The CI will automatically create an associated folder in the documentation repo.
 6.  In the documentation repository (e.g. ``https://github.com/dirty-cat/dirty-cat.github.io``),
     update the ``stable`` symlink to the latest stable version: first, unlink ``stable``
-    (i.e. ``unlink stable``); then, create a new symlink (i.e. ``ln -s 0.1 stable``);
+    (i.e. ``unlink stable``); then, create a new symlink (i.e. ``ln -s 0.2 stable``);
     finally, commit and push the changes into the repository.
 7.  Create a new release and tag via GitHub: ``https://github.com/dirty-cat/dirty_cat/releases/new``.
-    Provide the tag using the current version, e.g. ``0.1.0`` and make sure to select
-    the target branch created earlier (e.g. ``0.1.X``).
-8.  Next, you will need to install the ``twine`` package with
-    ``pip install --upgrade twine``
+    Provide the tag using the current version, e.g. ``0.2.0`` and make sure to select
+    the target branch created earlier (e.g. ``0.2.X``).
+8.  Next, you will need to install the ``twine`` and ``setuptools`` packages with
+    ``pip install --upgrade twine setuptools``.
 9.  Build the source with ``python setup.py bdist_wheel sdist``
 10. `Check if today is a good day for releasing <https://shouldideploy.today/>`__
 11. It is advised to first push the version on the test package index
@@ -52,18 +51,20 @@ the main steps and appropriate resources:
     the package indexer is working.
 12. Install the new release from the test package index on a dedicated
     environment with the command
-    ``pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://test.pypi.org/simple/ dirty_cat``
+    ``pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://test.pypi.org/simple/ dirty_cat``.
+    If there are other package install errors, it probably means those packages are not on test.pypi.org.
+    Download them from pypi.org directly with just ``pip install``.
 13. Finally, if that works fine, you can push to the official package
     index with ``twine upload dist/*``
 14. It is also good practice at this point to create a new environment
     and try installing and using the library (for example by launching examples).
     Be sure to install it with the command ``pip install dirty_cat==<version>``
-    (e.g. ``pip install dirty_cat==1.0``), otherwise some package/env managers
+    (e.g. ``pip install dirty_cat==0.2.0``), otherwise some package/env managers
     such as conda might use a cached version.
 15. To finish the procedure, create a new release on the GitHub repository.
 
-For the bug fix release (e.g. 0.1.0 -> 0.1.1), the process is similar. You don't need
-to create the branch ``0.1.X`` because it exists already. You need to cherry-pick the
+For the bug fix release (e.g. 0.2.0 -> 0.2.1), the process is similar. You don't need
+to create the branch ``0.2.X`` because it exists already. You need to cherry-pick the
 commits from ``master`` into this branch and then follow the same steps as above:
 bumping the version, update the setup, commit and push the changes and finally create
 and update the wheel.
diff --git a/dirty_cat/datasets/_fetching.py b/dirty_cat/datasets/_fetching.py
index 7bcfbe7..3fa5479 100644
--- a/dirty_cat/datasets/_fetching.py
+++ b/dirty_cat/datasets/_fetching.py
@@ -18,7 +18,7 @@ import warnings
 from dataclasses import dataclass
 from itertools import chain
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, TextIO, Union
 from urllib.error import URLError
 from zipfile import BadZipFile, ZipFile
 
@@ -530,6 +530,7 @@ def _export_gz_data_to_csv(
     atdata_found = False
     with destination_file.open(mode="w", encoding="utf8") as csv:
         with gzip.open(compressed_dir_path, mode="rt", encoding="utf8") as gz:
+            gz: TextIO  # Clarify for IDEs
             csv.write(_features_to_csv_format(features))
             csv.write("\n")
             # We will look at each line of the file until we find
@@ -552,7 +553,7 @@ def _fetch_dataset_as_dataclass(
     dataset_id: Union[int, str],
     target: Optional[str],
     load_dataframe: bool,
-    data_directory: Optional[Path] = None,
+    data_directory: Optional[Union[Path, str]] = None,
     read_csv_kwargs: Optional[dict] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """
@@ -582,6 +583,9 @@ def _fetch_dataset_as_dataclass(
         If `load_dataframe=False`
 
     """
+    if isinstance(data_directory, str):
+        data_directory = Path(data_directory)
+
     if source == "openml":
         info = _fetch_openml_dataset(dataset_id, data_directory)
     elif source == "world_bank":
@@ -635,7 +639,7 @@ def fetch_employee_salaries(
     load_dataframe: bool = True,
     drop_linked: bool = True,
     drop_irrelevant: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the employee_salaries dataset (regression), available at
     https://openml.org/d/42125
@@ -689,7 +693,7 @@ def fetch_employee_salaries(
 
 def fetch_road_safety(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the road safety dataset (classification), available at
     https://openml.org/d/42803
@@ -723,7 +727,7 @@ def fetch_road_safety(
 
 def fetch_medical_charge(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the medical charge dataset (regression), available at
     https://openml.org/d/42720
@@ -762,7 +766,7 @@ def fetch_medical_charge(
 
 def fetch_midwest_survey(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the midwest survey dataset (classification), available at
     https://openml.org/d/42805
@@ -794,7 +798,7 @@ def fetch_midwest_survey(
 
 def fetch_open_payments(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the open payments dataset (classification), available at
     https://openml.org/d/42738
@@ -828,7 +832,7 @@ def fetch_open_payments(
 
 def fetch_traffic_violations(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the traffic violations dataset (classification), available at
     https://openml.org/d/42132
@@ -864,7 +868,7 @@ def fetch_traffic_violations(
 
 def fetch_drug_directory(
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches the drug directory dataset (classification), available at
     https://openml.org/d/43044
@@ -898,7 +902,7 @@ def fetch_drug_directory(
 def fetch_world_bank_indicator(
     indicator_id: str,
     load_dataframe: bool = True,
-    directory: Optional[Path] = None,
+    directory: Optional[Union[Path, str]] = None,
 ) -> Union[DatasetAll, DatasetInfoOnly]:
     """Fetches a dataset of an indicator from the World Bank
     open data platform.


Test patch:
diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
index 02f8639..fc78fc7 100644
--- a/.github/workflows/testing.yml
+++ b/.github/workflows/testing.yml
@@ -14,7 +14,7 @@ jobs:
     outputs:
       skip: ${{ steps.result_step.outputs.ci-skip }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           fetch-depth: 0
       - id: result_step
@@ -31,13 +31,14 @@ jobs:
         fail-fast: false
         matrix:
             os: ["ubuntu-latest", "macos-latest"]
+            os-type: ["Ubuntu", "MacOS"]
             python-version: ["3.8", "3.9", "3.10"]
             # Install the min or latest dependencies for dirty_cat
             # as defined in setup.cfg at [options.extra_require].
             #
             # Installing these extra dependencies will overwrite the already
             # installed version defined at install_requires.
-            dependencies-version-type: ["min", "latest"]
+            dependencies-version-type: ["minimal", "latest"]
             include:
               - dependencies-version: "dev"
               - dependencies-version: "dev, min-py38"
@@ -49,13 +50,13 @@ jobs:
               - dependencies-version: "dev, min-py310"
                 python-version: "3.10"
                 dependencies-version-type: "min"
-    name: ${{ matrix.os }} with Python ${{ matrix.python-version }} and ${{ matrix.dependencies-version-type }} package versions
+    name: ${{ matrix.os-type }} with Python ${{ matrix.python-version }} and ${{ matrix.dependencies-version-type }} package versions
     defaults:
       run:
         shell: bash
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
         with:
             python-version: ${{ matrix.python-version }}
         name: 'Setup python'
@@ -67,6 +68,6 @@ jobs:
       - shell: bash {0}
         run: ./build_tools/github/test.sh
         name: 'Run tests'
-      - uses: codecov/codecov-action@v1
+      - uses: codecov/codecov-action@v3
         if: success()
         name: 'Upload coverage to CodeCov'
diff --git a/dirty_cat/datasets/tests/test_fetching.py b/dirty_cat/datasets/tests/test_fetching.py
index 82ec9ba..2eb3092 100644
--- a/dirty_cat/datasets/tests/test_fetching.py
+++ b/dirty_cat/datasets/tests/test_fetching.py
@@ -1,4 +1,3 @@
-from pathlib import Path
 from tempfile import TemporaryDirectory
 from typing import Callable, Tuple
 from unittest import mock
@@ -41,10 +40,6 @@ def test_openml_fetching(
     Test a function that loads data from OpenML.
     """
     with TemporaryDirectory() as temp_dir_1, TemporaryDirectory() as temp_dir_2:
-        # Convert to path objects
-        temp_dir_1 = Path(temp_dir_1).absolute()
-        temp_dir_2 = Path(temp_dir_2).absolute()
-
         # Fetch without loading into memory
         try:
             dataset_wo_load: _fetching.DatasetInfoOnly = fetching_function(
@@ -126,7 +121,6 @@ def test_fetch_world_bank_indicator():
     }
 
     with TemporaryDirectory() as temp_dir:
-        temp_dir = Path(temp_dir).absolute()
         try:
             # First, we want to purposefully test FileNotFoundError exceptions.
             with pytest.raises(FileNotFoundError):