Repository: modin-project/modin

Issue:
Modin on ray 1.11.0 can't start on macs
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS Monterey 12.2.21
- **Modin version** (`modin.__version__`): latest source
- **Python version**: 3.9.10
- **Code we can use to reproduce**:

```python
import modin.pandas as pd
pd.DataFrame()
```

### Describe the problem
In [ray 1.11.0](https://www.anyscale.com/blog/ray-version-1-11-released), released 2022-03-09, all macs have their object store memory capped at 2 GB due to https://github.com/ray-project/ray/issues/20388. Setting environment variable `RAY_ENABLE_MAC_LARGE_OBJECT_STORE=1` can override this limit. However, Modin doesn't set that variable and [sets a custom memory](https://github.com/modin-project/modin/blob/e7cb2e82f8b9c7a68f82abdd3b6011d661230b7e/modin/core/execution/ray/common/utils.py#L181) that is by default, at least for my mac and that of @jeffreykennethli , 9.313GiB. Consequently, `ray` raises a `ValueError` on `init()` and Modin doesn't work at all.

One solution is to always set `RAY_ENABLE_MAC_LARGE_OBJECT_STORE=1` before initializing Ray. 

Note that the though the object store bug only seemed to apply to macs with 16 GB of RAM, the limitation applies to all macs.

### Source code / logs
<details>
<summary>Stack trace</summary>

```
UserWarning: Ray execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    import ray
    ray.init()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [2], in <module>
----> 1 df = pd.DataFrame()

File ~/modin/modin/pandas/dataframe.py:114, in DataFrame.__init__(self, data, index, columns, dtype, copy, query_compiler)
    102 def __init__(
    103     self,
    104     data=None,
   (...)
    111     # Siblings are other dataframes that share the same query compiler. We
    112     # use this list to update inplace when there is a shallow copy.
    113     self._siblings = []
--> 114     Engine.subscribe(_update_engine)
    115     if isinstance(data, (DataFrame, Series)):
    116         self._query_compiler = data._query_compiler.copy()

File ~/modin/modin/config/pubsub.py:213, in Parameter.subscribe(cls, callback)
    204 """
    205 Add `callback` to the `_subs` list and then execute it.
    206
   (...)
    210     Callable to execute.
    211 """
    212 cls._subs.append(callback)
--> 213 callback(cls)

File ~/modin/modin/pandas/__init__.py:124, in _update_engine(publisher)
    121     if _is_first_update.get("Ray", True):
    122         from modin.core.execution.ray.common import initialize_ray
--> 124         initialize_ray()
    125 elif publisher.get() == "Native":
    126     # With OmniSci storage format there is only a single worker per node
    127     # and we allow it to work on all cores.
    128     if StorageFormat.get() == "Omnisci":

File ~/modin/modin/core/execution/ray/common/utils.py:183, in initialize_ray(override_is_cluster, override_redis_address, override_redis_password)
    172         object_store_memory = int(object_store_memory)
    174     ray_init_kwargs = {
    175         "num_cpus": CpuCount.get(),
    176         "num_gpus": GpuCount.get(),
   (...)
    181         "_memory": object_store_memory,
    182     }
--> 183     ray.init(**ray_init_kwargs)
    185 if StorageFormat.get() == "Cudf":
    186     from modin.core.execution.ray.implementations.cudf_on_ray.partitioning import (
    187         GPUManager,
    188         GPU_MANAGERS,
    189     )

File /usr/local/lib/python3.9/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /usr/local/lib/python3.9/site-packages/ray/worker.py:933, in init(address, num_cpus, num_gpus, resources, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, _enable_object_reconstruction, _redis_max_memory, _plasma_directory, _node_ip_address, _driver_object_store_memory, _memory, _redis_password, _temp_dir, _metrics_export_port, _system_config, _tracing_startup_hook, **kwargs)
    895     ray_params = ray._private.parameter.RayParams(
    896         node_ip_address=node_ip_address,
    897         raylet_ip_address=raylet_ip_address,
   (...)
    927         metrics_export_port=_metrics_export_port,
    928         tracing_startup_hook=_tracing_startup_hook)
    929     # Start the Ray processes. We set shutdown_at_exit=False because we
    930     # shutdown the node in the ray.shutdown call that happens in the atexit
    931     # handler. We still spawn a reaper process in case the atexit handler
    932     # isn't called.
--> 933     _global_node = ray.node.Node(
    934         head=True,
    935         shutdown_at_exit=False,
    936         spawn_reaper=True,
    937         ray_params=ray_params)
    938 else:
    939     # In this case, we are connecting to an existing cluster.
    940     if num_cpus is not None or num_gpus is not None:

File /usr/local/lib/python3.9/site-packages/ray/node.py:274, in Node.__init__(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only)
    268         self.get_gcs_client().internal_kv_put(
    269             b"tracing_startup_hook",
    270             ray_params.tracing_startup_hook.encode(), True,
    271             ray_constants.KV_NAMESPACE_TRACING)
    273 if not connect_only:
--> 274     self.start_ray_processes()
    275     # we should update the address info after the node has been started
    276     try:

File /usr/local/lib/python3.9/site-packages/ray/node.py:1114, in Node.start_ray_processes(self)
   1110 # Make sure we don't call `determine_plasma_store_config` multiple
   1111 # times to avoid printing multiple warnings.
   1112 resource_spec = self.get_resource_spec()
   1113 plasma_directory, object_store_memory = \
-> 1114     ray._private.services.determine_plasma_store_config(
   1115         resource_spec.object_store_memory,
   1116         plasma_directory=self._ray_params.plasma_directory,
   1117         huge_pages=self._ray_params.huge_pages
   1118     )
   1119 self.start_raylet(plasma_directory, object_store_memory)
   1120 if self._ray_params.include_log_monitor:

File /usr/local/lib/python3.9/site-packages/ray/_private/services.py:1951, in determine_plasma_store_config(object_store_memory, plasma_directory, huge_pages)
   1942     raise ValueError("Attempting to cap object store memory usage at {} "
   1943                      "bytes, but the minimum allowed is {} bytes.".format(
   1944                          object_store_memory,
   1945                          ray_constants.OBJECT_STORE_MINIMUM_MEMORY_BYTES))
   1947 if sys.platform == "darwin" \
   1948         and object_store_memory > \
   1949         ray_constants.MAC_DEGRADED_PERF_MMAP_SIZE_LIMIT \
   1950         and os.environ.get("RAY_ENABLE_MAC_LARGE_OBJECT_STORE") != "1":
-> 1951     raise ValueError(
   1952         "The configured object store size ({:.4}GiB) exceeds "
   1953         "the optimal size on Mac ({:.4}GiB). "
   1954         "This will harm performance! There is a known issue where "
   1955         "Ray's performance degrades with object store size greater"
   1956         " than {:.4}GB on a Mac."
   1957         "To reduce the object store capacity, specify"
   1958         "`object_store_memory` when calling ray.init() or ray start."
   1959         "To ignore this warning, "
   1960         "set RAY_ENABLE_MAC_LARGE_OBJECT_STORE=1.".format(
   1961             object_store_memory / 2**30,
   1962             ray_constants.MAC_DEGRADED_PERF_MMAP_SIZE_LIMIT / 2**30,
   1963             ray_constants.MAC_DEGRADED_PERF_MMAP_SIZE_LIMIT / 2**30))
   1965 # Print the object store memory using two decimal places.
   1966 logger.debug(
   1967     "Determine to start the Plasma object store with {} GB memory "
   1968     "using {}.".format(
   1969         round(object_store_memory / 10**9, 2), plasma_directory))

ValueError: The configured object store size (9.313GiB) exceeds the optimal size on Mac (2.0GiB). This will harm performance! There is a known issue where Ray's performance degrades with object store size greater than 2.0GB on a Mac.To reduce the object store capacity, specify`object_store_memory` when calling ray.init() or ray start.To ignore this warning, set RAY_ENABLE_MAC_LARGE_OBJECT_STORE=1.

In [3]: import ray

In [4]: ray.available_resources()
---------------------------------------------------------------------------
RaySystemError                            Traceback (most recent call last)
Input In [4], in <module>
----> 1 ray.available_resources()

File /usr/local/lib/python3.9/site-packages/ray/_private/client_mode_hook.py:105, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    103     if func.__name__ != "init" or is_client_mode_enabled_by_default:
    104         return getattr(ray, func.__name__)(*args, **kwargs)
--> 105 return func(*args, **kwargs)

File /usr/local/lib/python3.9/site-packages/ray/state.py:904, in available_resources()
    890 @DeveloperAPI
    891 @client_mode_hook(auto_init=False)
    892 def available_resources():
    893     """Get the current available cluster resources.
    894
    895     This is different from `cluster_resources` in that this will return idle
   (...)
    902             resource in the cluster.
    903     """
--> 904     return state.available_resources()

File /usr/local/lib/python3.9/site-packages/ray/state.py:716, in GlobalState.available_resources(self)
    704 def available_resources(self):
    705     """Get the current available cluster resources.
    706
    707     This is different from `cluster_resources` in that this will return
   (...)
    714             resource in the cluster.
    715     """
--> 716     self._check_connected()
    718     available_resources_by_id = self._available_resources_per_node()
    720     # Calculate total available resources.

File /usr/local/lib/python3.9/site-packages/ray/state.py:49, in GlobalState._check_connected(self)
     47 # _really_init_global_state should have set self.global_state_accessor
     48 if self.global_state_accessor is None:
---> 49     raise ray.exceptions.RaySystemError(
     50         "Ray has not been started yet. You can start Ray with "
     51         "'ray.init()'.")

RaySystemError: System error: Ray has not been started yet. You can start Ray with 'ray.init()'.
```

</details>



Discussion:

Resolving patch:
diff --git a/modin/core/execution/ray/common/utils.py b/modin/core/execution/ray/common/utils.py
--- a/modin/core/execution/ray/common/utils.py
+++ b/modin/core/execution/ray/common/utils.py
@@ -171,6 +171,26 @@ def initialize_ray(
             else:
                 object_store_memory = int(object_store_memory)
 
+            mac_size_limit = getattr(
+                ray.ray_constants, "MAC_DEGRADED_PERF_MMAP_SIZE_LIMIT", None
+            )
+            if (
+                sys.platform == "darwin"
+                and mac_size_limit is not None
+                and object_store_memory > mac_size_limit
+            ):
+                warnings.warn(
+                    "On Macs, Ray's performance is known to degrade with "
+                    + "object store size greater than "
+                    + f"{mac_size_limit / 2 ** 30:.4} GiB. Ray by default does "
+                    + "not allow setting an object store size greater than "
+                    + "that. Modin is overriding that default limit because "
+                    + "it would rather have a larger, slower object store "
+                    + "than spill to disk more often. To override Modin's "
+                    + "behavior, you can initialize Ray yourself."
+                )
+                os.environ["RAY_ENABLE_MAC_LARGE_OBJECT_STORE"] = "1"
+
             ray_init_kwargs = {
                 "num_cpus": CpuCount.get(),
                 "num_gpus": GpuCount.get(),


Test patch: