Repository: modin-project/modin (python)

Issue:
REFACTOR: Provide overridable method for estimated row count
We have a number of properties in BaseQueryCompiler to direct and restrict the calls to axis=0 queries, but we do not have a method where an estimated row count can be retrieved. Initially this will only be used for the auto switch code where the row count is a necessary part of figuring out if a dataset can be moved. The method is internal but overridable by sub-classes. We can decide later if we want to make it a public method.

Pull request:
REFACTOR-#7534: Provide internal, overridable method for max_shape
Existing query compilers can call get_axis_len(axis=0) to get a row count, but this may be expensive on lazily evaluated systems. Instead we add a new function which can be overridden on a per-query compiler basis for the max row count, which can be sometimes calculated more cheaply.

The default implementation of this is to simply call get_axis_len but sub-classes may have cheaper ways of estimating the length of the dataset.

<!--
Thank you for your contribution!
Please review the contributing docs: https://modin.readthedocs.io/en/latest/development/contributing.html
if you have questions about contributing.
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] first commit message and PR title follow format outlined [here](https://modin.readthedocs.io/en/latest/development/contributing.html#commit-message-formatting)
  > **_NOTE:_**  If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
- [ ] passes `flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [ ] passes `black --check modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [ ] Resolves #7534 <!-- issue must be created for each patch -->
- [ ] tests added and passing
- [x] module layout described at `docs/development/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->


Patch:
diff --git a/modin/core/storage_formats/base/query_compiler.py b/modin/core/storage_formats/base/query_compiler.py
index 8431a114..a1375677 100644
--- a/modin/core/storage_formats/base/query_compiler.py
+++ b/modin/core/storage_formats/base/query_compiler.py
@@ -361,7 +361,7 @@ class BaseQueryCompiler(
         cost = int(
             (
                 QCCoercionCost.COST_IMPOSSIBLE
-                * self.get_axis_len(axis=0)
+                * self._max_shape()[0]
                 / self._TRANSFER_THRESHOLD
             )
         )
@@ -445,7 +445,7 @@ class BaseQueryCompiler(
             Cost of doing this operation on the current backend.
         """
         return self._stay_cost_rows(
-            self.get_axis_len(axis=0),
+            self._max_shape()[0],
             self._OPERATION_PER_ROW_OVERHEAD,
             self._MAX_SIZE_THIS_ENGINE_CAN_HANDLE,
             self._OPERATION_INITIALIZATION_OVERHEAD,
@@ -493,8 +493,10 @@ class BaseQueryCompiler(
             Cost of migrating the data from other_qc to this qc or
             None if the cost cannot be determined.
         """
+        row_count = other_qc._max_shape()[0]
+
         return cls._stay_cost_rows(
-            other_qc.get_axis_len(axis=0),
+            row_count,
             cls._OPERATION_PER_ROW_OVERHEAD,
             cls._MAX_SIZE_THIS_ENGINE_CAN_HANDLE,
             cls._OPERATION_INITIALIZATION_OVERHEAD,
@@ -523,6 +525,21 @@ class BaseQueryCompiler(
     lazy_column_labels = False
     lazy_column_count = False
 
+    def _max_shape(self) -> tuple[int, int]:
+        """
+        Return the maximum dimensions of the frame.
+
+        For lazily evaluated engines the shape of the dataset may be expensive to
+        determine (see lazy_shape), but the maximum shape can be calculated
+        inexpensively.
+
+        Returns
+        -------
+        Tuple
+            Maximum shape of the dataframe (height, width).
+        """
+        return self.get_axis_len(axis=0), self.get_axis_len(axis=1)
+
     @property
     def lazy_shape(self):
         """


Test patch:
diff --git a/modin/tests/pandas/native_df_interoperability/test_compiler_caster.py b/modin/tests/pandas/native_df_interoperability/test_compiler_caster.py
index 92d37f81..7257bd2e 100644
--- a/modin/tests/pandas/native_df_interoperability/test_compiler_caster.py
+++ b/modin/tests/pandas/native_df_interoperability/test_compiler_caster.py
@@ -647,6 +647,11 @@ def test_stay_or_move_evaluation(cloud_df, default_df):
     assert move_cost is None
 
 
+def test_max_shape(cloud_df):
+    # default implementation matches df.shape
+    assert cloud_df.shape == cloud_df._query_compiler._max_shape()
+
+
 @contextlib.contextmanager
 def backend_test_context(test_backend: str, choices: set) -> Iterator[None]: