Repository: modin-project/modin

Issue:
Interchange `Column.null_count` is a NumPy scalar, not a builtin `int`
A `PandasProtocolColumn` returns a `null_count` as a 0d integer array (specifically a NumPy scalar), as opposed to [`int` as specified in the interchange protocol](https://github.com/data-apis/dataframe-api/blob/4f7c1e0c425643d57120c5b73434d992b3e83595/protocol/dataframe_protocol.py#L216).

```python
>>> from modin import pandas as mpd
>>> df = mpd.DataFrame({"foo": [42]})
>>> interchange_df = df.__dataframe__()
>>> interchange_col = interchange_df.get_column_by_name("foo")
>>> interchange_col.null_count
0
>>> type(interchange_col.null_count)
numpy.int64  # should be Python's int
```

This seems to be because the `null_count` implementation uses `DataFrame.squeeze()`, which returns a NumPy scalar rather than an `int`.

https://github.com/modin-project/modin/blob/9b33451648a3192e93c46ac6961627ed2858c7fd/modin/core/dataframe/pandas/exchange/dataframe_protocol/column.py#L222-L245

Related https://github.com/pandas-dev/pandas/issues/47789

Discussion:

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b20b6ab2..a7edbf65 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -234,11 +234,7 @@ jobs:
           miniforge-version: latest
           use-mamba: true
       - name: ASV installation
-        run: |
-          # FIXME: use the tag or release version of ASV as soon as it appears;
-          # The ability to build a conda environment by specifying yml file has not
-          # yet appeared in the release versions;
-          pip install git+https://github.com/airspeed-velocity/asv.git@ef016e233cb9a0b19d517135104f49e0a3c380e9
+        run: pip install asv==0.5.1
       - name: Running benchmarks
         run: |
           git remote add upstream https://github.com/modin-project/modin.git
@@ -272,7 +268,7 @@ jobs:
               # TODO: Remove manual environment creation after fix https://github.com/airspeed-velocity/asv/issues/1310
               mamba env create -f ../requirements/env_hdk.yml
               conda activate modin_on_hdk
-              pip install git+https://github.com/airspeed-velocity/asv.git@ef016e233cb9a0b19d517135104f49e0a3c380e9
+              pip install asv==0.5.1
               pip install ..
 
               # check Modin on HDK
diff --git a/asv_bench/asv.conf.dask.json b/asv_bench/asv.conf.dask.json
index 92eb25fb..6da9cf24 100644
--- a/asv_bench/asv.conf.dask.json
+++ b/asv_bench/asv.conf.dask.json
@@ -13,31 +13,9 @@
     // project being benchmarked
     "repo": "..",
 
-    // The Python project's subdirectory in your repo.  If missing or
-    // the empty string, the project is assumed to be located at the root
-    // of the repository.
-    // "repo_subdir": "",
-
     // Customizable commands for building, installing, and
     // uninstalling the project. See asv.conf.json documentation.
-    //
     "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[dask]"],
-    // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
-    // "build_command": [
-    //     "python setup.py build",
-    //     "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
-    // ],
-
-    // List of branches to benchmark. If not provided, defaults to "master"
-    // (for git) or "default" (for mercurial).
-    // "branches": ["master"], // for git
-    // "branches": ["default"],    // for mercurial
-
-    // The DVCS being used.  If not set, it will be automatically
-    // determined from "repo" by looking at the protocol in the URL
-    // (if remote), or by looking for special directories, such as
-    // ".git" (if local).
-    // "dvcs": "git",
 
     // The tool to use to create environments.  May be "conda",
     // "virtualenv" or other value depending on the plugins in use.
@@ -61,56 +39,6 @@
     // dependency packages in the specified order
     "conda_channels": ["conda-forge", "defaults"],
 
-    // The matrix of dependencies to test.  Each key is the name of a
-    // package (in PyPI) and the values are version numbers.  An empty
-    // list or empty string indicates to just test against the default
-    // (latest) version. null indicates that the package is to not be
-    // installed. If the package to be tested is only available from
-    // PyPi, and the 'environment_type' is conda, then you can preface
-    // the package name by 'pip+', and the package will be installed via
-    // pip (with all the conda available packages installed first,
-    // followed by the pip installed packages).
-    // "matrix": {
-    //    "pip+ray": ["1.0.1"],
-    //    "pyarrow": ["1.0"]
-    // },
-    // Combinations of libraries/python versions can be excluded/included
-    // from the set to test. Each entry is a dictionary containing additional
-    // key-value pairs to include/exclude.
-    //
-    // An exclude entry excludes entries where all values match. The
-    // values are regexps that should match the whole string.
-    //
-    // An include entry adds an environment. Only the packages listed
-    // are installed. The 'python' key is required. The exclude rules
-    // do not apply to includes.
-    //
-    // In addition to package names, the following keys are available:
-    //
-    // - python
-    //     Python version, as in the *pythons* variable above.
-    // - environment_type
-    //     Environment type, as above.
-    // - sys_platform
-    //     Platform, as in sys.platform. Possible values for the common
-    //     cases: 'linux2', 'win32', 'cygwin', 'darwin'.
-    //
-    // "exclude": [
-    //     {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
-    //     {"environment_type": "conda", "six": null}, // don't run without six on conda
-    // ],
-    //
-    // "include": [
-    //     // additional env for python2.7
-    //     {"python": "2.7", "numpy": "1.8"},
-    //     // additional env if run on windows+conda
-    //     {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
-    // ],
-
-    // The directory (relative to the current directory) that benchmarks are
-    // stored in.  If not provided, defaults to "benchmarks"
-    // "benchmark_dir": "benchmarks",
-
     // The directory (relative to the current directory) to cache the Python
     // environments in.  If not provided, defaults to "env"
     "env_dir": ".asv/env",
@@ -122,36 +50,4 @@
     // The directory (relative to the current directory) that the html tree
     // should be written to.  If not provided, defaults to "html".
     "html_dir": ".asv/html",
-
-    // The number of characters to retain in the commit hashes.
-    // "hash_length": 8,
-
-    // `asv` will cache results of the recent builds in each
-    // environment, making them faster to install next time.  This is
-    // the number of builds to keep, per environment.
-    // "build_cache_size": 2,
-
-    // The commits after which the regression search in `asv publish`
-    // should start looking for regressions. Dictionary whose keys are
-    // regexps matching to benchmark names, and values corresponding to
-    // the commit (exclusive) after which to start looking for
-    // regressions.  The default is to start from the first commit
-    // with results. If the commit is `null`, regression detection is
-    // skipped for the matching benchmark.
-    //
-    // "regressions_first_commits": {
-    //    "some_benchmark": "352cdf",  // Consider regressions only after this commit
-    //    "another_benchmark": null,   // Skip regression detection altogether
-    // },
-
-    // The thresholds for relative change in results, after which `asv
-    // publish` starts reporting regressions. Dictionary of the same
-    // form as in ``regressions_first_commits``, with values
-    // indicating the thresholds.  If multiple entries match, the
-    // maximum is taken. If no entry matches, the default is 5%.
-    //
-    // "regressions_thresholds": {
-    //    "some_benchmark": 0.01,     // Threshold of 1%
-    //    "another_benchmark": 0.5,   // Threshold of 50%
-    // },
 }
diff --git a/asv_bench/asv.conf.json b/asv_bench/asv.conf.json
index e1f7c5ff..9b7ea10f 100644
--- a/asv_bench/asv.conf.json
+++ b/asv_bench/asv.conf.json
@@ -13,31 +13,9 @@
     // project being benchmarked
     "repo": "..",
 
-    // The Python project's subdirectory in your repo.  If missing or
-    // the empty string, the project is assumed to be located at the root
-    // of the repository.
-    // "repo_subdir": "",
-
     // Customizable commands for building, installing, and
     // uninstalling the project. See asv.conf.json documentation.
-    //
     "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[ray]"],
-    // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
-    // "build_command": [
-    //     "python setup.py build",
-    //     "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
-    // ],
-
-    // List of branches to benchmark. If not provided, defaults to "master"
-    // (for git) or "default" (for mercurial).
-    // "branches": ["master"], // for git
-    // "branches": ["default"],    // for mercurial
-
-    // The DVCS being used.  If not set, it will be automatically
-    // determined from "repo" by looking at the protocol in the URL
-    // (if remote), or by looking for special directories, such as
-    // ".git" (if local).
-    // "dvcs": "git",
 
     // The tool to use to create environments.  May be "conda",
     // "virtualenv" or other value depending on the plugins in use.
@@ -61,56 +39,6 @@
     // dependency packages in the specified order
     "conda_channels": ["conda-forge", "defaults"],
 
-    // The matrix of dependencies to test.  Each key is the name of a
-    // package (in PyPI) and the values are version numbers.  An empty
-    // list or empty string indicates to just test against the default
-    // (latest) version. null indicates that the package is to not be
-    // installed. If the package to be tested is only available from
-    // PyPi, and the 'environment_type' is conda, then you can preface
-    // the package name by 'pip+', and the package will be installed via
-    // pip (with all the conda available packages installed first,
-    // followed by the pip installed packages).
-    // "matrix": {
-    //    "pip+ray": ["1.0.1"],
-    //    "pyarrow": ["1.0"]
-    // },
-    // Combinations of libraries/python versions can be excluded/included
-    // from the set to test. Each entry is a dictionary containing additional
-    // key-value pairs to include/exclude.
-    //
-    // An exclude entry excludes entries where all values match. The
-    // values are regexps that should match the whole string.
-    //
-    // An include entry adds an environment. Only the packages listed
-    // are installed. The 'python' key is required. The exclude rules
-    // do not apply to includes.
-    //
-    // In addition to package names, the following keys are available:
-    //
-    // - python
-    //     Python version, as in the *pythons* variable above.
-    // - environment_type
-    //     Environment type, as above.
-    // - sys_platform
-    //     Platform, as in sys.platform. Possible values for the common
-    //     cases: 'linux2', 'win32', 'cygwin', 'darwin'.
-    //
-    // "exclude": [
-    //     {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
-    //     {"environment_type": "conda", "six": null}, // don't run without six on conda
-    // ],
-    //
-    // "include": [
-    //     // additional env for python2.7
-    //     {"python": "2.7", "numpy": "1.8"},
-    //     // additional env if run on windows+conda
-    //     {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
-    // ],
-
-    // The directory (relative to the current directory) that benchmarks are
-    // stored in.  If not provided, defaults to "benchmarks"
-    // "benchmark_dir": "benchmarks",
-
     // The directory (relative to the current directory) to cache the Python
     // environments in.  If not provided, defaults to "env"
     "env_dir": ".asv/env",
@@ -122,36 +50,4 @@
     // The directory (relative to the current directory) that the html tree
     // should be written to.  If not provided, defaults to "html".
     "html_dir": ".asv/html",
-
-    // The number of characters to retain in the commit hashes.
-    // "hash_length": 8,
-
-    // `asv` will cache results of the recent builds in each
-    // environment, making them faster to install next time.  This is
-    // the number of builds to keep, per environment.
-    // "build_cache_size": 2,
-
-    // The commits after which the regression search in `asv publish`
-    // should start looking for regressions. Dictionary whose keys are
-    // regexps matching to benchmark names, and values corresponding to
-    // the commit (exclusive) after which to start looking for
-    // regressions.  The default is to start from the first commit
-    // with results. If the commit is `null`, regression detection is
-    // skipped for the matching benchmark.
-    //
-    // "regressions_first_commits": {
-    //    "some_benchmark": "352cdf",  // Consider regressions only after this commit
-    //    "another_benchmark": null,   // Skip regression detection altogether
-    // },
-
-    // The thresholds for relative change in results, after which `asv
-    // publish` starts reporting regressions. Dictionary of the same
-    // form as in ``regressions_first_commits``, with values
-    // indicating the thresholds.  If multiple entries match, the
-    // maximum is taken. If no entry matches, the default is 5%.
-    //
-    // "regressions_thresholds": {
-    //    "some_benchmark": 0.01,     // Threshold of 1%
-    //    "another_benchmark": 0.5,   // Threshold of 50%
-    // },
 }
diff --git a/asv_bench/asv.conf.unidist.json b/asv_bench/asv.conf.unidist.json
index 351fd3fb..49d5e953 100644
--- a/asv_bench/asv.conf.unidist.json
+++ b/asv_bench/asv.conf.unidist.json
@@ -13,31 +13,9 @@
     // project being benchmarked
     "repo": "..",
 
-    // The Python project's subdirectory in your repo.  If missing or
-    // the empty string, the project is assumed to be located at the root
-    // of the repository.
-    // "repo_subdir": "",
-
     // Customizable commands for building, installing, and
     // uninstalling the project. See asv.conf.json documentation.
-    //
     "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}[unidist]"],
-    // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
-    // "build_command": [
-    //     "python setup.py build",
-    //     "PIP_NO_BUILD_ISOLATION=false python -mpip wheel --no-deps --no-index -w {build_cache_dir} {build_dir}"
-    // ],
-
-    // List of branches to benchmark. If not provided, defaults to "master"
-    // (for git) or "default" (for mercurial).
-    // "branches": ["master"], // for git
-    // "branches": ["default"],    // for mercurial
-
-    // The DVCS being used.  If not set, it will be automatically
-    // determined from "repo" by looking at the protocol in the URL
-    // (if remote), or by looking for special directories, such as
-    // ".git" (if local).
-    // "dvcs": "git",
 
     // The tool to use to create environments.  May be "conda",
     // "virtualenv" or other value depending on the plugins in use.
@@ -61,56 +39,6 @@
     // dependency packages in the specified order
     "conda_channels": ["conda-forge", "defaults"],
 
-    // The matrix of dependencies to test.  Each key is the name of a
-    // package (in PyPI) and the values are version numbers.  An empty
-    // list or empty string indicates to just test against the default
-    // (latest) version. null indicates that the package is to not be
-    // installed. If the package to be tested is only available from
-    // PyPi, and the 'environment_type' is conda, then you can preface
-    // the package name by 'pip+', and the package will be installed via
-    // pip (with all the conda available packages installed first,
-    // followed by the pip installed packages).
-    // "matrix": {
-    //    "pip+ray": ["1.0.1"],
-    //    "pyarrow": ["1.0"]
-    // },
-    // Combinations of libraries/python versions can be excluded/included
-    // from the set to test. Each entry is a dictionary containing additional
-    // key-value pairs to include/exclude.
-    //
-    // An exclude entry excludes entries where all values match. The
-    // values are regexps that should match the whole string.
-    //
-    // An include entry adds an environment. Only the packages listed
-    // are installed. The 'python' key is required. The exclude rules
-    // do not apply to includes.
-    //
-    // In addition to package names, the following keys are available:
-    //
-    // - python
-    //     Python version, as in the *pythons* variable above.
-    // - environment_type
-    //     Environment type, as above.
-    // - sys_platform
-    //     Platform, as in sys.platform. Possible values for the common
-    //     cases: 'linux2', 'win32', 'cygwin', 'darwin'.
-    //
-    // "exclude": [
-    //     {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows
-    //     {"environment_type": "conda", "six": null}, // don't run without six on conda
-    // ],
-    //
-    // "include": [
-    //     // additional env for python2.7
-    //     {"python": "2.7", "numpy": "1.8"},
-    //     // additional env if run on windows+conda
-    //     {"platform": "win32", "environment_type": "conda", "python": "2.7", "libpython": ""},
-    // ],
-
-    // The directory (relative to the current directory) that benchmarks are
-    // stored in.  If not provided, defaults to "benchmarks"
-    // "benchmark_dir": "benchmarks",
-
     // The directory (relative to the current directory) to cache the Python
     // environments in.  If not provided, defaults to "env"
     "env_dir": ".asv/env",
@@ -122,36 +50,4 @@
     // The directory (relative to the current directory) that the html tree
     // should be written to.  If not provided, defaults to "html".
     "html_dir": ".asv/html",
-
-    // The number of characters to retain in the commit hashes.
-    // "hash_length": 8,
-
-    // `asv` will cache results of the recent builds in each
-    // environment, making them faster to install next time.  This is
-    // the number of builds to keep, per environment.
-    // "build_cache_size": 2,
-
-    // The commits after which the regression search in `asv publish`
-    // should start looking for regressions. Dictionary whose keys are
-    // regexps matching to benchmark names, and values corresponding to
-    // the commit (exclusive) after which to start looking for
-    // regressions.  The default is to start from the first commit
-    // with results. If the commit is `null`, regression detection is
-    // skipped for the matching benchmark.
-    //
-    // "regressions_first_commits": {
-    //    "some_benchmark": "352cdf",  // Consider regressions only after this commit
-    //    "another_benchmark": null,   // Skip regression detection altogether
-    // },
-
-    // The thresholds for relative change in results, after which `asv
-    // publish` starts reporting regressions. Dictionary of the same
-    // form as in ``regressions_first_commits``, with values
-    // indicating the thresholds.  If multiple entries match, the
-    // maximum is taken. If no entry matches, the default is 5%.
-    //
-    // "regressions_thresholds": {
-    //    "some_benchmark": 0.01,     // Threshold of 1%
-    //    "another_benchmark": 0.5,   // Threshold of 50%
-    // },
 }
diff --git a/environment-dev.yml b/environment-dev.yml
index 9d133116..b5c8d06d 100644
--- a/environment-dev.yml
+++ b/environment-dev.yml
@@ -62,12 +62,12 @@ dependencies:
   - pandas-stubs>=2.0.0
 
   - pip:
+      - asv==0.5.1
       # no conda package for windows so we install it with pip
       - connectorx>=0.2.6a4
       # experimental version of fuzzydata requires at least 0.0.6 to successfully resolve all dependencies
       - fuzzydata>=0.0.6
       # Fixes breaking ipywidgets changes, but didn't release yet.
       - git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
-      - git+https://github.com/airspeed-velocity/asv.git@ef016e233cb9a0b19d517135104f49e0a3c380e9
       # The `numpydoc` version should match the version installed in the `lint-pydocstyle` job of the CI.
       - numpydoc==1.1.0
diff --git a/modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py b/modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py
index 01471ad4..08185f38 100644
--- a/modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py
+++ b/modin/core/dataframe/pandas/interchange/dataframe_protocol/column.py
@@ -251,7 +251,7 @@ class PandasProtocolColumn(ProtocolColumn):
         # Otherwise, we get mismatching internal and external indices for both axes
         intermediate_df.index = pandas.RangeIndex(1)
         intermediate_df.columns = pandas.RangeIndex(1)
-        self._null_count_cache = intermediate_df.to_pandas().squeeze()
+        self._null_count_cache = intermediate_df.to_pandas().squeeze(axis=1).item()
         return self._null_count_cache
 
     @property
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
index 9604db58..c3752d8e 100644
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -37,7 +37,6 @@ from modin.utils import (
 from modin.pandas.utils import cast_function_modin2pandas
 from modin.core.storage_formats.base.query_compiler import BaseQueryCompiler
 from modin.core.dataframe.algebra.default2pandas.groupby import GroupBy
-from modin.config import IsExperimental
 from .series import Series
 from .window import RollingGroupby
 from .utils import is_label
@@ -182,31 +181,19 @@ class DataFrameGroupBy(ClassLogger):
                 return self.__getitem__(key)
             raise err
 
-    # TODO: `.__getattribute__` overriding is broken in experimental mode. We should
-    # remove this branching one it's fixed:
-    # https://github.com/modin-project/modin/issues/5536
-    if not IsExperimental.get():
+    def __getattribute__(self, item):
+        attr = super().__getattribute__(item)
+        if item not in _DEFAULT_BEHAVIOUR and not self._query_compiler.lazy_execution:
+            # We default to pandas on empty DataFrames. This avoids a large amount of
+            # pain in underlying implementation and returns a result immediately rather
+            # than dealing with the edge cases that empty DataFrames have.
+            if callable(attr) and self._df.empty and hasattr(self._pandas_class, item):
 
-        def __getattribute__(self, item):
-            attr = super().__getattribute__(item)
-            if (
-                item not in _DEFAULT_BEHAVIOUR
-                and not self._query_compiler.lazy_execution
-            ):
-                # We default to pandas on empty DataFrames. This avoids a large amount of
-                # pain in underlying implementation and returns a result immediately rather
-                # than dealing with the edge cases that empty DataFrames have.
-                if (
-                    callable(attr)
-                    and self._df.empty
-                    and hasattr(self._pandas_class, item)
-                ):
-
-                    def default_handler(*args, **kwargs):
-                        return self._default_to_pandas(item, *args, **kwargs)
+                def default_handler(*args, **kwargs):
+                    return self._default_to_pandas(item, *args, **kwargs)
 
-                    return default_handler
-            return attr
+                return default_handler
+        return attr
 
     @property
     def ngroups(self):
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 75d69e08..3d8d94cd 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -41,10 +41,10 @@ PyGithub>=1.58.0
 pygit2>=1.9.2
 
 ## test dependencies
+asv==0.5.1
 coverage>=7.1.0
 # experimental version of fuzzydata requires at least 0.0.6 to successfully resolve all dependencies
 fuzzydata>=0.0.6
-git+https://github.com/airspeed-velocity/asv.git@ef016e233cb9a0b19d517135104f49e0a3c380e9
 # The `numpydoc` version should match the version installed in the `lint-pydocstyle` job of the CI.
 numpydoc==1.1.0
 moto>=4.1.0
diff --git a/requirements/requirements-no-engine.yml b/requirements/requirements-no-engine.yml
index 7d8f339a..e03e0be3 100644
--- a/requirements/requirements-no-engine.yml
+++ b/requirements/requirements-no-engine.yml
@@ -45,10 +45,10 @@ dependencies:
   - flake8-print>=5.0.0
 
   - pip:
+      - asv==0.5.1
       # no conda package for windows
       - connectorx>=0.2.6a4
       # Fixes breaking ipywidgets changes, but didn't release yet.
       - git+https://github.com/modin-project/modin-spreadsheet.git@49ffd89f683f54c311867d602c55443fb11bf2a5
-      - git+https://github.com/airspeed-velocity/asv.git@ef016e233cb9a0b19d517135104f49e0a3c380e9
       # The `numpydoc` version should match the version installed in the `lint-pydocstyle` job of the CI.
       - numpydoc==1.1.0


Test patch:
diff --git a/modin/test/interchange/dataframe_protocol/test_general.py b/modin/test/interchange/dataframe_protocol/test_general.py
index 5a1aa6f9..b4373835 100644
--- a/modin/test/interchange/dataframe_protocol/test_general.py
+++ b/modin/test/interchange/dataframe_protocol/test_general.py
@@ -82,6 +82,14 @@ def test_na_float(df_from_dict):
     assert colX.null_count == 1
 
 
+def test_null_count(df_from_dict):
+    df = df_from_dict({"foo": [42]})
+    dfX = df.__dataframe__()
+    colX = dfX.get_column_by_name("foo")
+    null_count = colX.null_count
+    assert null_count == 0 and type(null_count) is int
+
+
 def test_noncategorical(df_from_dict):
     df = df_from_dict({"a": [1, 2, 3]})
     dfX = df.__dataframe__()