Repository: modin-project/modin

Issue:
DOC: which classes are immutable?
Doing some performance troubleshooting I'm finding a call to `PandasOnRayDataframePartition.drain_call_queue` is overriding a already-evaluated `self._width_cache` with a ObjectIDType object, which later takes an absurd amount of time to access via ray.get.

Is it the case that a Partition's shape should not change following `__init__`?  if so, it would be nice to have that clearly documented. 


Discussion:
@jbrockmendel 

> Is it the case that a Partition's shape should not change following __init__? 

No. For one thing `_length_cache` and `_width_cache` are often `None` when the frame is initialized, and `apply` or `drain_call_queue` can set them. I think you might be thinking of cases where `_length_cache` and `_width_cache` are already known. In that case, yes, they should never change. However, I don't think it would help performance much to make `drain_call_queue` stop recomputing them. `width()` and `length()` only drain the call queue when the dimension they're computing is unknown: https://github.com/modin-project/modin/blob/4ec7f6347903f9133c65ebc5b6e0e15553b98577/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py#L301-L307

So if a partition decides to recompute known length and width with a remote call in `drain_call_queue`, presumably it's because the new `_data` is needed for something, and waiting on the `_data` alone would probably not be much faster than also waiting on the `_data` shapes computed with `len`: https://github.com/modin-project/modin/blob/4ec7f6347903f9133c65ebc5b6e0e15553b98577/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py#L430-L431

@mvashishtha thanks for the quick response.  For context, I'm perf-troubleshooting a script in which about a third of the runtime is currently in `width` calls, essentially all of that in `ray.get`.  So the two things I'm focusing on are 1) trying to figure out why the heck ray.get takes so long (on which I'm making little progress) and 2) tracking down where _length_cache/_width_cache are set.

> width() and length() only drain the call queue when the dimension they're computing is unknown:

In the script I'm profiling/troubleshooting, I'm seeing cases where _length_cache and _width_cache are already integers and drain_call_queue overwrites them with ObjectIDType objects.

(I'm also seeing places where the constructor is called without length/width being passed where it _may_ be possible to infer the output shape, but that can be considered out-of-scope for this issue)
@jbrockmendel 

>  For context, I'm perf-troubleshooting a script in which about a third of the runtime is currently in width calls, essentially all of that in ray.get. 

I'm curious to see the script where you see this bottleneck. 

> In the script I'm profiling/troubleshooting, I'm seeing cases where _length_cache and _width_cache are already integers and drain_call_queue overwrites them with ObjectIDType objects.

I spent a while trying to imagine a scenario where this happens, but I couldn't. In all the scenarios I could think of where we ended up with something on the call queue, the partition shape was unknown.

> (I'm also seeing places where the constructor is called without length/width being passed where it may be possible to infer the output shape, but that can be considered out-of-scope for this issue)

Yep I saw that while investigating your question e.g. `transpose` should be able to tell the new partitions their shapes if the old partitions already know them them, but it calls `add_to_apply_calls`, which constructs a new partition with no cached shape. Similarly, `map` should preserve dimensions of each partition, but `PandasDataframePartitionManager .map_partitions` [calls `apply`](https://github.com/modin-project/modin/blob/88c3c33cf9b8aab5f5b7077555bea9f2d8466534/modin/core/dataframe/pandas/partitioning/partition_manager.py#L499), which also discards the old caches. I think this hasn't been an issue so far because the `PandasDataframe` size caches normally cover us, and we don't have to query the partitions for sizes. e.g. for transpose we set sizes: https://github.com/modin-project/modin/blob/88c3c33cf9b8aab5f5b7077555bea9f2d8466534/modin/core/dataframe/pandas/dataframe/dataframe.py#L2877-L2884

 (really we should transfer the shape caches, not compute the shapes until they're required)

same with map: 

https://github.com/modin-project/modin/blob/88c3c33cf9b8aab5f5b7077555bea9f2d8466534/modin/core/dataframe/pandas/dataframe/dataframe.py#L1522-L1529

Resolving patch:
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
@@ -159,8 +159,8 @@ def drain_call_queue(self):
             logger.debug(f"SUBMIT::_apply_list_of_funcs::{self._identity}")
             (
                 self._data,
-                self._length_cache,
-                self._width_cache,
+                new_length,
+                new_width,
                 self._ip_cache,
             ) = _apply_list_of_funcs.remote(call_queue, data)
         else:
@@ -170,13 +170,20 @@ def drain_call_queue(self):
             logger.debug(f"SUBMIT::_apply_func::{self._identity}")
             (
                 self._data,
-                self._length_cache,
-                self._width_cache,
+                new_length,
+                new_width,
                 self._ip_cache,
             ) = _apply_func.remote(data, func, *args, **kwargs)
         logger.debug(f"EXIT::Partition.drain_call_queue::{self._identity}")
         self.call_queue = []
 
+        # GH#4732 if we already have evaluated width/length cached as ints,
+        #  don't overwrite that cache with non-evaluated values.
+        if not isinstance(self._length_cache, int):
+            self._length_cache = new_length
+        if not isinstance(self._width_cache, int):
+            self._width_cache = new_width
+
     def wait(self):
         """Wait completing computations on the object wrapped by the partition."""
         self.drain_call_queue()


Test patch: