Repository: horovod/horovod

Issue:
pyarrow 0.17 changed hdfs.connect api to remove driver parameter
**Environment:**
1. Framework: (TensorFlow, Keras, PyTorch, MXNet) Keras Estimator example
2. Framework version:
3. Horovod version:  master branch
4. MPI version:
5. CUDA version:
6. NCCL version:
7. Python version:
8. OS and version:
9. GCC version:

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**
Please describe errorneous behavior you're observing and steps to reproduce it.

Trying to run with the latest horovod code with pyarrow 0.17.1 against HDFS for the working directory and it fails with:

```
Traceback (most recent call last):
  File "keras_spark_rossmann_estimator.py", line 408, in <module>
    store = Store.create(args.work_dir)
  File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/store.py", line 141, in create
    return HDFSStore(prefix_path, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/store.py", line 337, in __init__
    self._hdfs = self._get_filesystem_fn()()
  File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/store.py", line 416, in fn
    return pa.hdfs.connect(**hdfs_kwargs)
TypeError: connect() got an unexpected keyword argument 'driver'
```

It looks like pyarrow removed that parameter with commit:
https://github.com/apache/arrow/commit/4e53749097ba687afd5e000067925def2e2802c9#diff-72abd78694ddde2b1a059b194978b77b

Note I called it like:
python keras_spark_rossmann_estimator.py --num-proc 2 --batch-size 1000 --epochs 2 --master spark://3ee9bf36f06b:7077 --work-dir hdfs:///rossmann



Discussion:
Note this is setup here: https://github.com/horovod/horovod/blob/master/horovod/spark/common/store.py#L335
it looks like the older version of pyarrow may also only support Hadoop 2.x. Its failing with class mismatch exception when I run on hdfs 3.x.  I see they have a new api for filesystems, not sure if that supports both

Resolving patch:
diff --git a/horovod/spark/common/store.py b/horovod/spark/common/store.py
--- a/horovod/spark/common/store.py
+++ b/horovod/spark/common/store.py
@@ -20,6 +20,8 @@
 import shutil
 import tempfile
 
+from distutils.version import LooseVersion
+
 import pyarrow as pa
 import pyarrow.parquet as pq
 
@@ -332,8 +334,9 @@ def __init__(self, prefix_path,
                                  port=port,
                                  user=user,
                                  kerb_ticket=kerb_ticket,
-                                 driver=driver,
                                  extra_conf=extra_conf)
+        if LooseVersion(pa.__version__) < LooseVersion('0.17.0'):
+            self._hdfs_kwargs['driver'] = driver
         self._hdfs = self._get_filesystem_fn()()
 
         super(HDFSStore, self).__init__(prefix_path, *args, **kwargs)


Test patch: