Repository: ray-project/ray

Issue:
Allow greater than memory allocation for plasma store on Mac
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: OSX 10.12
- **Ray installed from (source or binary)**: source
- **Python version**: 3.6
- **Exact command to reproduce**: (See below)

### Describe the problem
<!-- Describe the problem clearly here. -->
In Modin, it would be great if we could specify `plasma_directory="/tmp"` and `object_store_memory=n * physical_mem`. It seems to work fine on Ubuntu, but I am getting the following error on Mac:

```Exception: The requested object store memory size is greater than the total available memory.```

### Source code / logs
```python
from psutil import virtual_memory
mem_bytes = virtual_memory().total
object_store_memory = 8 * mem_bytes
plasma_directory = "/tmp"
ray.init(
    redirect_output=True,
    include_webui=False,
    redirect_worker_output=True,
    use_raylet=True,
    ignore_reinit_error=True,
    plasma_directory=plasma_directory,
    object_store_memory=object_store_memory,
)
```


Discussion:
Really, that works on Linux? I don't think you actually want this behavior because it will start swapping (even before it hits the total amount of memory) and freeze your laptop.
I guess it doesn't on 0.6. On 0.5.X it worked.

I don't understand why this would cause the laptop to freeze. The OS would manage the swapping, and it would be slower than in-memory, would you expect that to break the system?
Maybe not literally freeze, but in the past when I start using too much memory, I've seen things become sufficiently unresponsive that I've had to reboot the machine.
Just to clarify, here is what we need:

* plasma store on disk ("/tmp")
* having larger than memory dataframes supported with the `object_store_memory` parameter

The OS maintains the paging. We have been experimenting with this on 0.5.3, and while it is slower than purely in-memory, it works and allows 10's of GB dataframes on a laptop. This is a very important requirement in Modin.
I think it's a reasonable request for the Modin use case. We already provide the ability to specify the plasma directory as a mount point (e.g., for huge pages). We should try mounting a large file as tmpfs mountpoint, passing that to the object store and evaluate performance. This, combined with explicitly specifying object store memory should just work. If there's some internal check in python that overrides the specified object store memory, capping it to available system memory, I'd say it's a bug, because the plasma dir could point to a larger pool of memory.

Resolving patch:
diff --git a/python/ray/services.py b/python/ray/services.py
--- a/python/ray/services.py
+++ b/python/ray/services.py
@@ -1028,9 +1028,6 @@ def determine_plasma_store_config(object_store_memory=None,
                 "when calling ray.init() or ray start.")
             object_store_memory = MAX_DEFAULT_MEM
 
-    if plasma_directory is not None:
-        plasma_directory = os.path.abspath(plasma_directory)
-
     # Determine which directory to use. By default, use /tmp on MacOS and
     # /dev/shm on Linux, unless the shared-memory file system is too small,
     # in which case we default to /tmp on Linux.
@@ -1055,10 +1052,15 @@ def determine_plasma_store_config(object_store_memory=None,
         else:
             plasma_directory = "/tmp"
 
-    # Do some sanity checks.
-    if object_store_memory > system_memory:
-        raise Exception("The requested object store memory size is greater "
-                        "than the total available memory.")
+        # Do some sanity checks.
+        if object_store_memory > system_memory:
+            raise Exception(
+                "The requested object store memory size is greater "
+                "than the total available memory.")
+    else:
+        plasma_directory = os.path.abspath(plasma_directory)
+        logger.warning("WARNING: object_store_memory is not verified when "
+                       "plasma_directory is set.")
 
     if not os.path.isdir(plasma_directory):
         raise Exception("The file {} does not exist or is not a directory."


Test patch: