Repository: modin-project/modin

Issue:
test_join_sort.py in CI failed by timeout with new Dask release - 2022.2.0
Error: https://github.com/modin-project/modin/runs/5195622251?check_suite_focus=true

Dask release - https://github.com/dask/dask/releases/tag/2022.02.0

Fastest option here - pin `dask<2022.2.0`, but it also requires an investigation into the cause.


Discussion:
cc @modin-project/modin-core 
Besides fixing the perf issue, I think we can revise `test_join_sort` suites in order to reduce them. [Only this one test](https://github.com/modin-project/modin/blob/b7027597985a442734a57949d72e3fa0625b4d43/modin/pandas/test/dataframe/test_join_sort.py#L393-L517) generates 17.000+ cases (around ~35% of all tests for dataframes).
So new Dask release has caused performance to degrade on these tests to the point where they always timeout?

Is this something that we need to fix, or does it need to be reported to Dask?
> So new Dask release has caused performance to degrade on these tests to the point where they always timeout?

Some kind of memory problem occurs, which leads to a timeout ([example](https://github.com/modin-project/modin/runs/5206389461?check_suite_focus=true)):
```bash
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise TimeoutError("No valid workers found")
E   asyncio.exceptions.TimeoutError: No valid workers found

C:\Miniconda\envs\modin\lib\site-packages\distributed\scheduler.py:5846: TimeoutError
---------------------------- Captured stderr call -----------------------------
distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 448.04 MiB -- Worker memory limit: 476.84 MiB
```

> Is this something that we need to fix, or does it need to be reported to Dask?

I don't know, it's also possible that some `Dask` settings have changed. Can someone take a look? (I don't have the capacity to investigate the problem for now)
Let's pin dask for now to make CI pass.
Agree, let's pin for now.
`test_join_sort` on Ubuntu still takes about 4x long with dask as with ray. e.g. compare [this run](https://github.com/modin-project/modin/runs/5365519294?check_suite_focus=true) of test-ubuntu on dask with [this run](https://github.com/modin-project/modin/runs/6789322001?check_suite_focus=true) on ray. 47 minutes with dask, 12 minutes with ray. Anecdotally, `test-ubuntu` is quite often one of the last things to finish in our CI, and `test_join_sort` is one of the main reasons for that. test_join_sort on dask on windows is also way slower (~1.8x) than on ray on windows, e.g. dask [here](https://github.com/modin-project/modin/runs/6789325888?check_suite_focus=true) took 82 minutes vs ray [here](https://github.com/modin-project/modin/runs/6789324532?check_suite_focus=true) took 46 minutes.

Speeding up `test_join_sort` on dask is likely to substantially reduce the time for our CI to run.
I am trying `MODIN_ENGINE=dask pytest -n 2  --durations=0 modin/pandas/test/dataframe/test_join_sort.py` on my Macbook to get an understanding of the test case time distribution on dask 2022.1.1.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -7,7 +7,7 @@
 with open("README.md", "r", encoding="utf-8") as fh:
     long_description = fh.read()
 
-dask_deps = ["dask>=2.22.0,<2022.2.0", "distributed>=2.22.0,<2022.2.0"]
+dask_deps = ["dask>=2.22.0", "distributed>=2.22.0"]
 if sys.version_info < (3, 8):
     dask_deps.append("pickle5")
 


Test patch: