Repository: modin-project/modin

Issue:
Dask engine can only have 4 partitions - Pull Requests #2492 
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS release 6.5 (Final)
- **Modin version** (`modin.__version__`): 0.8.2 installed from conda-forge channel
- **Python version**: 3.7.3
- **Code we can use to reproduce**:
```python
from distributed import Client
client = Client(n_workers=10)
import modin.pandas as pd
df = pd.read_csv(filename)
```
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

Hi there, I recently got a really huge CSV file to read and found that no matter how I declare the number of the worker by using
```python
from distributed import Client
client = Client(n_workers=10)
import modin.pandas as pd
```
The partition of the Dataframe object is still 4 causing the efficiency really low.

So I reviewed some of the code briefly (actually it takes me a whole afternoon to track the problem) and found `pd.DEFAULT_NPARTITIONS` is 4 defaulted (in file `modin/modin/pandas/__init__.py`) and the init function (`_update_engine`) didn't change the value of it when using Dask engine.

Hence, I change `pd.DEFAULT_NPARTITIONS` after importing `pd` by 
```python
pd.DEFAULT_NPARTITIONS = 10
```
and it works well!

So I decided to add a line of code to solve this problem (See Pull Requests #2492 ). However, it failed in `test_read_csv_datetime`. Really confused about that.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Hi @zhfanrui, thanks for the report!

I can reproduce the issue, and your PR does fix the issue. I am not sure why the tests failed in that way because I do not see the issue when locally running the tests with that change.

Relevant Test Traceback:
```
  File "/modin/modin/pandas/__init__.py", line 138, in _update_engine
    num_cpus = len(dask_client.nthreads())
TypeError: object of type 'coroutine' has no len()
```

Any ideas here?
@devin-petersohn , when running locally I got the same error. Maybe that is related to how we execute `_update_engine` via `Engine.subcribe`. @vnlitvinov , are there any thoughts on this?
The log from the PR can be found here ([link](https://github.com/modin-project/modin/pull/2492#issuecomment-736242901)).
@YarShev which tests did you run to reproduce? I ran the I/O tests and did not have the error.
@devin-petersohn , I ran `pytest modin/pandas/test/test_io.py::TestReadCSV::test_read_csv_datetime`. After some digging it was found out we fall down into this [place](https://github.com/dask/distributed/blob/f018007fba3b5d82e7b1a484ba758d2c302abd25/distributed/client.py#L826). That's why we can see the error.

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -135,6 +135,8 @@ def _update_engine(publisher: Parameter):
 
                 dask_client = Client(n_workers=CpuCount.get())
 
+            num_cpus = len(dask_client.ncores())
+
     elif publisher.get() == "Cloudray":
         from modin.experimental.cloud import get_connection
 


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -587,7 +587,7 @@ def test_read_csv_col_handling(
             None,
             {
                 "col1": lambda x: np.int64(x) * 10,
-                "col2": pd.to_datetime,
+                "col2": pandas.to_datetime,
                 "col4": lambda x: x.replace(":", ";"),
             },
         ],
@@ -748,7 +748,7 @@ def test_read_csv_nans_handling(
     @pytest.mark.parametrize("infer_datetime_format", [True, False])
     @pytest.mark.parametrize("keep_date_col", [True, False])
     @pytest.mark.parametrize(
-        "date_parser", [None, lambda x: pd.datetime.strptime(x, "%Y-%m-%d")]
+        "date_parser", [None, lambda x: pandas.datetime.strptime(x, "%Y-%m-%d")]
     )
     @pytest.mark.parametrize("dayfirst", [True, False])
     @pytest.mark.parametrize("cache_dates", [True, False])