Repository: modin-project/modin
Pull request: FIX-#3151: Fix default size of Ray obj store memory
Signed-off-by: Alexey Prutskov <alexey.prutskov@intel.com>

<!--
Thank you for your contribution!
Please review the contributing docs: https://modin.readthedocs.io/en/latest/CONTRIBUTING.html
if you have questions about contributing.
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] commit message follows format outlined [here](https://modin.readthedocs.io/en/latest/contributing.html)
- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #3151  <!-- issue must be created for each patch -->
- [ ] tests added and passing
- [ ] module layout described at `docs/developer/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->


Problem statement:
`ValueError` during initialization of Ray 1.4 in Modin
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu
- **Modin version** (`modin.__version__`): 0.10.0
- **Python version**:  3.8.10
- **Code we can use to reproduce**:

```python
import modin.pandas as pd
df = pd.DataFrame([0])
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
During initialization of Ray in Modin `ValueError` is occurred:
```
ValueError: The configured object store size (486.0 GB) exceeds /dev/shm size (405.07611136 GB). This will harm performance. Consider deleting files in /dev/shm or increasing its size with --shm-size in Docker. To ignore this warning, set RAY_OBJECT_STORE_ALLOW_SLOW_STORAGE=1.
```

Hints from issue discussion:
Ray verifies object-store size and raises such exception when [`plasma_directory` is not specified](https://github.com/ray-project/ray/blob/90805d302f5950244f346f55d253b926f1b5504f/python/ray/_private/services.py#L1751), in this case, ray assumes `/dev/shm` to be the default plasma directory (on Linux) and compares specified `object_store_size` with the available shared memory.

Modin at the ray initialization uses total available memory to compute default object store size, which appears to be greater than the available shared memory:
https://github.com/modin-project/modin/blob/b8dd3b2f1a1b416d425ecfe49227fefbe1774505/modin/engines/ray/utils.py#L139-L140 
Modin should use `ray._private.utils.get_shared_memory_bytes()` to fill `system_memory` value instead.

### Temporary workaround
Specify a valid `object_store_size` in bytes by `MODIN_MEMORY` environment variable or by explicit ray initialization before using Modin:
```python
import ray
ray.init(object_store_size=STORE_SIZE)

import modn.pandas as pd
...
```

Commit message: FIX-#3151: Revert default size for win platform

Signed-off-by: Alexey Prutskov <alexey.prutskov@intel.com>

Patch to review:
diff --git a/modin/engines/ray/utils.py b/modin/engines/ray/utils.py
index 060d464b42d..f9e23187f67 100644
--- a/modin/engines/ray/utils.py
+++ b/modin/engines/ray/utils.py
@@ -136,7 +136,11 @@ def initialize_ray(
             # In case anything failed above, we can still improve the memory for Modin.
             if object_store_memory is None:
                 # Round down to the nearest Gigabyte.
-                system_memory = ray._private.utils.get_system_memory()
+                system_memory = (
+                    ray._private.utils.get_shared_memory_bytes()
+                    if sys.platform == "linux" or sys.platform == "linux2"
+                    else ray._private.utils.get_system_memory()
+                )
                 object_store_memory = int(0.6 * system_memory // 10 ** 9 * 10 ** 9)
                 # If the memory pool is smaller than 2GB, just use the default in ray.
                 if object_store_memory == 0:


Code review comments:
diff_hunk: @@ -136,7 +136,11 @@ def initialize_ray(
             # In case anything failed above, we can still improve the memory for Modin.
             if object_store_memory is None:
                 # Round down to the nearest Gigabyte.
-                system_memory = ray._private.utils.get_system_memory()
+                system_memory = (
+                    ray._private.utils.get_shared_memory_bytes()
path: modin/engines/ray/utils.py
text: @user1:
Previously there was some discussion on #3132 about removing dependency on `ray._private`.

Should we manually check `/dev/shm` size ourselves instead of using this utility?

@author:
I agree, we should refuse to use private API of Ray in my opinion. We can set `system_memory` by manually checking /dev/shm. But I have a pair of questions:
1) `ray._private.utils.get_shared_memory_bytes()` gives size of /dev/shm (it is half of RAM by default) on Linux. But `ray._private.utils.get_system_memory()` gives size of full RAM on Windows. What size should we use on both OS?
2) Why do we use 0.6 of `system_memory` for `object_store_memory`?

@user1:
>What size should we use on both OS?

On Windows and Mac, we should use the full RAM and on Linux we should use size of /dev/shm. In Windows and Mac they use a temporary file for Plasma, and on Linux they use /dev/shm.

>Why do we use 0.6 of system_memory for object_store_memory?

We need to reserve memory for the workers.

@author:
> On Windows and Mac, we should use the full RAM and on Linux we should use size of /dev/shm. In Windows and Mac they use a temporary file for Plasma, and on Linux they use /dev/shm.

Thanks, I understood.

> We need to reserve memory for the workers.

I understand this, thanks. But why are we using *exactly* 0.6? 



@user1:
>But why are we using exactly 0.6?

It seemed rational, but there really isn't a good reason. Do you have ideas on how it can be improved?

@author:
Unfortunately, I haven't concrete ideas related this right now.

@author:
Dependency on `ray._private` was removed