Repository: altair-viz/altair
Pull request #3130 (2023-07-31 13:53:33)

Fix accidental requirement of Pandas 1.5. Bump minimum Pandas version to 0.25. Run tests with it
[This import](https://github.com/altair-viz/altair/pull/3114/files#diff-d110e8643c6feff37c22c1a5c7d213b8384177718b8e26edd8e1b8f2ed027a0aR18) introduced in #3114 is not available in Pandas < 1.5 and Altair currently supports Pandas >= 0.18. This PR adds a test run with the lowest supported Pandas version.

@jonmmease Do you know of a good way to make that type check independent of the pandas import? Else, we could fall back on not type-hinting it if necessary. Feel free to push directly to this branch if you have an idea and want to try it out in the GitHub actions workflow.

Pandas 0.18 is from 2016 and we could for sure consider bumping it but I don't think we can go to Pandas 1.5 just yet. Especially not in a minor release of Altair.

jonmmease: Ahh, thanks for flagging this @binste, and for working on testing with older pandas.

I'll look into alternatives for the type signature, it might be enough to wrap the import in [`typing.TYPE_CHECKING`](https://mypy.readthedocs.io/en/stable/runtime_troubles.html#typing-type-checking), although maybe the mypy tests would fail with the older version of pandas in this case.

In terms of pandas versions, I personally think we could bump the pandas requirement up to 1.0 (released 3.5 years ago) in an Altair minor version, but I agree that 1.5 (released less than 1 year ago) would be too aggressive.





jonmmease: huh, looks like the `if` condition in the GitHub actions isn't working
jonmmease: I'm really stumped as to why GitHub actions is always evaluating the "Maybe uninstall optional dependencies" and "Maybe install lowest supported Pandas version" jobs, ignoring the `if:` conditions (or always evaluating them to true).
binste: I think I had the same issue recently over in #3118. I'll give it a try.
binste: Seems like you're trying it already. In #3118 I got it working for jsonschema with this line https://github.com/altair-viz/altair/pull/3118/files#diff-5c3fa597431eda03ac3339ae6bf7f05e1a50d6fc7333679ec38e21b337cb6721R32
jonmmease: I'll try that again (I think again 😄 )
jonmmease: Oh, looks like I needed single quotes. The issue now is with installing pandas 0.18.0 in Python 3.8:

```
Run pip install pandas==0.18
Collecting pandas==0.18
  Downloading pandas-0.18.0.tar.gz (7.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 24.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [4 lines of output]
      <string>:39: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      error in pandas setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
          pytz >= 20[11](https://github.com/altair-viz/altair/actions/runs/5716874306/job/15489393829?pr=3130#step:6:12)k
               ~~~~~~~^
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.
```

Looks like pandas 0.18.1 didn't ship wheels for Python 3.8. I'll check what the oldest version of pandas is that includes wheels for 3.8 is.
binste: Thanks for figuring out to use single quotes, weird...

Bumping to minimum version 0.25.3 sounds very reasonable to me too. We can rename this PR and do it all in here. I think   there is some code logic somewhere which is for Pandas 0.18. I'll try to find and remove that one too while we're at it.

I can offer to do the Pandas bump and cleanup  later this week if you prefer to focus on the issue with the type hint.
jonmmease: Sounds good. I'll take this PR and update the min pandas and numpy versions and fix the type hint issue (I'll need to update these versions to get CI working).  I won't look for any pandas/numpy compatibility code that could be removed, so if you could do a pass on that at some point that would be great!
binste: Great, I removed some part of the code which is now redundant and clarified another one which we need to keep but for other reasons compared to why it was originally implemented.

@jonmmease Could you do a review of my last commits? Feel free to merge afterwards. Thanks!

Patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 75aa922c..4fa75275 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -31,11 +31,18 @@ jobs:
       - name: Maybe uninstall optional dependencies
         # We uninstall pyarrow and vegafusion for one job to test that we have not
         # accidentally introduced a hard dependency on these libraries.
-        # Uninstalling for Python 3.9 is an arbitrary choice.
+        # Uninstalling for Python 3.8 is an arbitrary choice.
         # Also see https://github.com/altair-viz/altair/pull/3114
-        if: ${{ matrix.python-version }}=="3.9"
+        if: ${{ matrix.python-version == '3.8' }}
         run: |
           pip uninstall -y pyarrow vegafusion vegafusion-python-embed
+      - name: Maybe install lowest supported Pandas version
+        # We install the lowest supported Pandas version for one job to test that
+        # it still works. Downgrade to the oldest versions of pandas and numpy that include
+        # Python 3.8 wheels, so only run this job for Python 3.8
+        if: ${{ matrix.python-version == '3.8' }}
+        run: |
+          pip install pandas==0.25.3 numpy==1.17.5
       - name: Test that schema generation has no effect
         run: |
           python tools/generate_schema_wrapper.py
diff --git a/altair/utils/core.py b/altair/utils/core.py
index 082db4cc..3a155812 100644
--- a/altair/utils/core.py
+++ b/altair/utils/core.py
@@ -15,7 +15,7 @@ from types import ModuleType
 import jsonschema
 import pandas as pd
 import numpy as np
-from pandas.core.interchange.dataframe_protocol import Column as PandasColumn
+from pandas.api.types import infer_dtype
 
 from altair.utils.schemapi import SchemaBase
 from altair.utils._dfi_types import Column, DtypeKind, DataFrame as DfiDataFrame
@@ -25,13 +25,10 @@ if sys.version_info >= (3, 10):
 else:
     from typing_extensions import ParamSpec
 
-from typing import Literal, Protocol
+from typing import Literal, Protocol, TYPE_CHECKING
 
-try:
-    from pandas.api.types import infer_dtype as _infer_dtype
-except ImportError:
-    # Import for pandas < 0.20.0
-    from pandas.lib import infer_dtype as _infer_dtype  # type: ignore[no-redef]
+if TYPE_CHECKING:
+    from pandas.core.interchange.dataframe_protocol import Column as PandasColumn
 
 _V = TypeVar("_V")
 _P = ParamSpec("_P")
@@ -42,26 +39,6 @@ class _DataFrameLike(Protocol):
         ...
 
 
-def infer_dtype(value: object) -> str:
-    """Infer the dtype of the value.
-
-    This is a compatibility function for pandas infer_dtype,
-    with skipna=False regardless of the pandas version.
-    """
-    if not hasattr(infer_dtype, "_supports_skipna"):
-        try:
-            _infer_dtype([1], skipna=False)
-        except TypeError:
-            # pandas < 0.21.0 don't support skipna keyword
-            infer_dtype._supports_skipna = False  # type: ignore[attr-defined]
-        else:
-            infer_dtype._supports_skipna = True  # type: ignore[attr-defined]
-    if infer_dtype._supports_skipna:  # type: ignore[attr-defined]
-        return _infer_dtype(value, skipna=False)
-    else:
-        return _infer_dtype(value)
-
-
 TYPECODE_MAP = {
     "ordinal": "O",
     "nominal": "N",
@@ -214,7 +191,7 @@ def infer_vegalite_type(
     ----------
     data: object
     """
-    typ = infer_dtype(data)
+    typ = infer_dtype(data, skipna=False)
 
     if typ in [
         "floating",
@@ -348,9 +325,9 @@ def sanitize_dataframe(df: pd.DataFrame) -> pd.DataFrame:  # noqa: C901
     for col_name, dtype in df.dtypes.items():
         dtype_name = str(dtype)
         if dtype_name == "category":
-            # Work around bug in to_json for categorical types in older versions of pandas
-            # https://github.com/pydata/pandas/issues/10778
-            # https://github.com/altair-viz/altair/pull/2170
+            # Work around bug in to_json for categorical types in older versions
+            # of pandas as they do not properly convert NaN values to null in to_json.
+            # We can probably remove this part once we require Pandas >= 1.0
             col = df[col_name].astype(object)
             df[col_name] = col.where(col.notnull(), None)
         elif dtype_name == "string":
@@ -630,7 +607,7 @@ def parse_shorthand(
 
 
 def infer_vegalite_type_for_dfi_column(
-    column: Union[Column, PandasColumn],
+    column: Union[Column, "PandasColumn"],
 ) -> Union[_InferredVegaLiteType, Tuple[_InferredVegaLiteType, list]]:
     from pyarrow.interchange.from_dataframe import column_to_array
 
diff --git a/doc/releases/changes.rst b/doc/releases/changes.rst
index 3ba6fb67..dad0fb6a 100644
--- a/doc/releases/changes.rst
+++ b/doc/releases/changes.rst
@@ -18,7 +18,8 @@ Bug Fixes
 
 Backward-Incompatible Changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- Drop support for Python 3.7 which is end-of-life (#3100).
+- Drop support for Python 3.7 which is end-of-life (#3100)
+- Increase minimum required Pandas version to 0.25 (#3130)
 
 Version 5.0.1 (released May 26, 2023)
 -------------------------------------
diff --git a/pyproject.toml b/pyproject.toml
index c3da1bfd..cb0d5894 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -18,9 +18,11 @@ authors = [ {name = "Vega-Altair Contributors"} ]
 dependencies = [
     "typing_extensions>=4.0.1; python_version<\"3.11\"",
     "jinja2",
+    # If you update the minimum required jsonschema version, also update it in build.yml
     "jsonschema>=3.0",
     "numpy",
-    "pandas>=0.18",
+    # If you update the minimum required pandas version, also update it in build.yml
+    "pandas>=0.25",
     "toolz"
 ]
 description = "Vega-Altair: A declarative statistical visualization library for Python."


Test patch:
diff --git a/tests/utils/test_core.py b/tests/utils/test_core.py
index b5ce8e76..b6fd7cdb 100644
--- a/tests/utils/test_core.py
+++ b/tests/utils/test_core.py
@@ -74,7 +74,7 @@ class StrokeWidthValue(ValueChannel, schemapi.SchemaBase):
     ],
 )
 def test_infer_dtype(value, expected_type):
-    assert infer_dtype(value) == expected_type
+    assert infer_dtype(value, skipna=False) == expected_type
 
 
 def test_parse_shorthand():