Repository: dask/dask

Issue:
ImportError: cannot import name 'ThreadPoolExecutor' when writing Parquet-file with pyarrow
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:

I would like to reraise my issue #7193 as this is still occurs on Dask 2021.03.0. An error related to threading is raised when exporting a dask-dataframe as parquet when using Python 3.9 (other Python versions seems to be fine):
```python
Traceback (most recent call last):
  File "pyarrow-issue.py", line 9, in <module>
    df2.to_parquet('test.parquet', engine='pyarrow')
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\dataframe\core.py", line 4127, in to_parquet
    return to_parquet(self, path, *args, **kwargs)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\dataframe\io\parquet\core.py", line 671, in to_parquet
    out = out.compute(**compute_kwargs)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\base.py", line 283, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\base.py", line 565, in compute
    results = schedule(dsk, keys, **kwargs)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\threaded.py", line 76, in get
    results = get_async(
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\local.py", line 487, in get_async
    raise_exception(exc, tb)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\local.py", line 317, in reraise
    raise exc
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\utils.py", line 35, in apply
    return func(*args, **kwargs)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\dataframe\io\parquet\arrow.py", line 841, in write_partition
    t = cls._pandas_to_arrow_table(df, preserve_index=preserve_index, schema=schema)
  File "C:\miniforge\envs\pyarrow\lib\site-packages\dask\dataframe\io\parquet\arrow.py", line 814, in _pandas_to_arrow_table
    table = pa.Table.from_pandas(df, preserve_index=preserve_index, schema=schema)
  File "pyarrow\table.pxi", line 1479, in pyarrow.lib.Table.from_pandas
  File "C:\miniforge\envs\pyarrow\lib\site-packages\pyarrow\pandas_compat.py", line 596, in dataframe_to_arrays
    with futures.ThreadPoolExecutor(nthreads) as executor:
  File "C:\miniforge\envs\pyarrow\lib\concurrent\futures\__init__.py", line 49, in __getattr__
    from .thread import ThreadPoolExecutor as te
ImportError: cannot import name 'ThreadPoolExecutor' from partially initialized module 'concurrent.futures.thread' (most likely due to a circular import) (C:\miniforge\envs\pyarrow\lib\concurrent\futures\thread.py)
```

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
import numpy as np
import pandas as pd

df = pd.DataFrame(np.ones((1000000, 2)))
df2 = dd.from_pandas(df, npartitions=4)
df2.to_parquet('test.parquet', engine='pyarrow')
```

**Anything else we need to know?**:

I tried the above example with both versions of pyarrow (2.0.0 and 3.0.0) although [pyarrow officially added support for Python 3.9 with version 3.0.0](https://arrow.apache.org/blog/2021/01/25/3.0.0-release/) and i do not think it is related to pyarrow alone. My guess is it has to do with pyarrow and dask using threads to perform their operations.

I tried to replicate the above error by calling the pyarrow-functions directly (`pa.Table.from_pandas`), however that seems to work (both pyarrow versions):
```python
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame(np.ones((1000000, 2)))
table = pa.Table.from_pandas(df)
pq.write_table(table, 'test.parquet')
```

**Environment**:

- Dask version: 2021.3.0
- Python version: 3.9.2
- pyarrow version: 2.0.0 and 3.0.0
- Operating System: Windows 10 1909
- Install method (conda, pip, source): conda



Discussion:
Thanks for re-raising this. Ping @rjzamora for visibility
Thanks for raising @m-rossi - It looks like pyarrow is running into problems within the default `pyarrow.Table.from_pandas` call.  Perhaps we just need to pass `nthreads=1` to prevent arrow from trying to use a thread pool here.
That's a workaround I also had in mind, however passing it `to_parquet` does not have an effect. The `kwargs` did not get through to the  `pa.Table.from_*`-calls. I found these two which should be the relevant ones:

* https://github.com/dask/dask/blob/de0e6e8c767ea15a8557ae8d1bf58bd179262352/dask/dataframe/io/parquet/arrow.py#L108-L110
* https://github.com/dask/dask/blob/de0e6e8c767ea15a8557ae8d1bf58bd179262352/dask/dataframe/io/parquet/arrow.py#L814

Should we add a generel `nthreads=1` there?
>The kwargs did not get through to the pa.Table.from_*-calls.

Right - Sorry, I didn't mean to imply that you should pass `nthreads=1` to `to_parquet`.  You are right that the easiest solution is probably to hard-code the `nthreads` option in `pa.Table.from_pandas`.  We are already passing `use_threads=False` for all read-related pyarrow operations, so I don't think there is any problem with this.

@jorisvandenbossche - Do you have any thoughts here?  Does pyarrow currenly support Python 3.9?
Yes, pyarrow certainly supports Python 3.9 (but which doesn't mean there can't be a bug related to it, of course).

I can reproduce it using Python 3.9 and pyarrow 3.0. But it only happens the first time I call it, not subsequent times. So it might indeed be an import issue.

Also if I put `import concurrent.futures.thread` in the reproducible snippet, it works fine as well.
I suppose this is the same Python import issue as distributed also ran into (https://github.com/dask/distributed/issues/4168), which seems to be caused by a change in Python? (https://discuss.python.org/t/differences-between-3-8-and-3-9-in-importing-module/5520/2, https://bugs.python.org/issue35943)

I would need to look into more detail to know whether (and how) this can be solved in pyarrow, but indeed I think it's correct that dask can specify to pyarrow to not use threads in `from_pandas`
> I can reproduce it using Python 3.9 and pyarrow 3.0. But it only happens the first time I call it, not subsequent times. So it might indeed be an import issue.

I can confirm that. I first noticed this issue when I tried to add Python 3.9 support to an internal package. It uses parameterized tests and only the first parameter set fails.


Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -106,7 +106,11 @@ def _write_partitioned(
             ]
         )
         subtable = pa.Table.from_pandas(
-            subgroup, preserve_index=preserve_index, schema=subschema, safe=False
+            subgroup,
+            nthreads=1,
+            preserve_index=preserve_index,
+            schema=subschema,
+            safe=False,
         )
         prefix = fs.sep.join([root_path, subdir])
         fs.mkdirs(prefix, exist_ok=True)
@@ -811,7 +815,9 @@ def initialize_write(
     def _pandas_to_arrow_table(
         cls, df: pd.DataFrame, preserve_index=False, schema=None
     ) -> pa.Table:
-        table = pa.Table.from_pandas(df, preserve_index=preserve_index, schema=schema)
+        table = pa.Table.from_pandas(
+            df, nthreads=1, preserve_index=preserve_index, schema=schema
+        )
         return table
 
     @classmethod


Test patch: