Repository: modin-project/modin

Issue:
[Ray] Add unit tests for ray client + modin
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?

Wanted to open discussion about adding unit tests using ray client to this repo's CI. If there are any examples that should be covered that would be handy to know, as well as any guidelines for where to put these tests. Example issue with ray-client on the ray repo https://github.com/ray-project/ray/issues/14857.


@amogkam @devin-petersohn 


Discussion:
Having some CI coverage here would really help with making sure we have native support for Modin with Ray Client. We can also add tests for this on the Ray CI.

Another relevant issue is here https://github.com/modin-project/modin/issues/2688
Thanks @ckw017 and @amogkam. I agree it would be good if we could add tests for basic functionality.
@devin-petersohn any guidance on how to go about adding this to modin CI? I think ideally we can just run the same tests that currently exist for modin on ray, except with ray client.
For this, we would probably want to add a separate job within the existing CI and push for the GitHub Actions (not necessary for our TeamCity tests). 

We would need to start the client before running the tests. Modin will automatically connect to the client if you run with environment variable `MODIN_RAY_CLUSTER`. Here's our current github actions workflow: https://github.com/modin-project/modin/blob/master/.github/workflows/ci.yml
@devin-petersohn Hey Devin, just wanted to check two things that would help get this closed:
1. It looks like to add tests for modin onto Ray's CI, we need a version of Modin with the fix in #2851. It looks like this was a bit after 0.9.1 was released, so wanted to check if it was possible to get 0.9.2 out so that we can run our tests against that.
2. For getting tests for Ray Client onto Modin's CI, we would need to use a nightly ray wheel since ray-project/ray#16278 won't be part of Ray 1.4, and is needed to get the tests to run consistently. It looks like all of the current unit tests run against Ray <= 1.1.0, so I wanted to check if you currently test against nightly anywhere so that we can stick the client tests there. Alternatively, we can add the client tests onto your CI as planned (as a separate job) and install a wheel including the patch just for the client tests
^ First thing should be fine for now, we'll just install the latest commit when testing for now.
@ckw017 thanks for following up!

1.) We are planning to release 0.10 tomorrow (probably 😄). After release it'll probably be good to test the stable release since we are planning major changes for the release after 0.10.
2.) Ray 1.4 will work with Modin given recent patches (e.g., https://github.com/ray-project/ray/issues/15990). We also have a nightly test that runs on the nightly docker container, but that probably needs to be changed to `pip install ray && ray install-nightly` because it's been failing frequently (see here:  https://github.com/modin-project/modin/actions/workflows/push-to-master.yml). Ideally we have Ray 1.4 in 0.10. I will try to make that happen.



Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -239,6 +239,16 @@ class TestDatasetSize(EnvironmentVariable, type=str):
     choices = ("Small", "Normal", "Big")
 
 
+class TestRayClient(EnvironmentVariable, type=bool):
+    """
+    Set to true to start and connect ray client before a testing session
+    starts.
+    """
+
+    varname = "MODIN_TEST_RAY_CLIENT"
+    default = False
+
+
 class TrackFileLeaks(EnvironmentVariable, type=bool):
     """
     Whether to track for open file handles leakage during testing
diff --git a/modin/conftest.py b/modin/conftest.py
--- a/modin/conftest.py
+++ b/modin/conftest.py
@@ -42,7 +42,7 @@ def _saving_make_api_url(token, _make_api_url=modin.utils._make_api_url):
 
 import modin  # noqa: E402
 import modin.config  # noqa: E402
-from modin.config import IsExperimental  # noqa: E402
+from modin.config import IsExperimental, TestRayClient  # noqa: E402
 
 from modin.backends import PandasQueryCompiler, BaseQueryCompiler  # noqa: E402
 from modin.engines.python.pandas_on_python.io import PandasOnPythonIO  # noqa: E402
@@ -452,3 +452,26 @@ def TestReadGlobCSVFixture():
 @pytest.fixture
 def get_generated_doc_urls():
     return lambda: _generated_doc_urls
+
+
+ray_client_server = None
+
+
+def pytest_sessionstart(session):
+    if TestRayClient.get():
+        import ray
+        import ray.util.client.server.server as ray_server
+
+        addr = "localhost:50051"
+        global ray_client_server
+        ray_client_server = ray_server.serve(addr)
+        ray.util.connect(addr)
+
+
+def pytest_sessionfinish(session, exitstatus):
+    if TestRayClient.get():
+        import ray
+
+        ray.util.disconnect()
+        if ray_client_server:
+            ray_client_server.stop(0)


Test patch: