Repository: nteract/papermill

Issue:
TypeError: pyarrow._hdfs.HadoopFileSystem__init__() takes at least 1 positional argument (0 given)
## 🐛 Bug
In papermill==2.3.4, when I try to use HadoopFileSystem, I get the following error message.

```
$ papermill Untitled.ipynb hdfs://myhost/tmp.ipynb
Executing:   0%|                                                                     | 0/9 [00:00<?, ?cell/s]
Traceback (most recent call last):
  File "/opt/conda/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/cli.py", line 267, in papermill
    execution_timeout=execution_timeout,
  File "/opt/conda/lib/python3.7/site-packages/papermill/execute.py", line 118, in execute_notebook
    **engine_kwargs
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 357, in execute_notebook
    nb_man.notebook_start()
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 69, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 198, in notebook_start
    self.save()
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 69, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/papermill/engines.py", line 139, in save
    write_ipynb(self.nb, self.output_path)
  File "/opt/conda/lib/python3.7/site-packages/papermill/iorw.py", line 447, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/opt/conda/lib/python3.7/site-packages/papermill/iorw.py", line 143, in write
    return self.get_handler(path).write(buf, path)
  File "/opt/conda/lib/python3.7/site-packages/papermill/iorw.py", line 372, in write
    with self._get_client().open(path, 'wb') as f:
  File "/opt/conda/lib/python3.7/site-packages/papermill/iorw.py", line 361, in _get_client
    self._client = HadoopFileSystem()
  File "pyarrow/_hdfs.pyx", line 55, in pyarrow._hdfs.HadoopFileSystem.__init__
TypeError: __init__() takes at least 1 positional argument (0 given)
```

### Steps to reproduce the behavior
1. Install the latest version of papermill by either of the following steps
- `pip install papermill==2.3.4`.
- Install according to CONTRIBUTING.md

2. Execute the papermill with specifing a URL starting with `hdfs://` for the path. 
e.g.: `$ papermill untitled.ipynb hdfs://myhost/tmp.ipynb`

## Analysis
I found that the `HDFSHandler` in `iorw.py` has not been modified, even though the API has changed with the filesystem change (#615).
- [New filesystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html#pyarrow.fs.HadoopFileSystem)
- [Legacy filesystem](https://arrow.apache.org/docs/python/filesystems_deprecated.html)

## Proposed amendment
As mentioned above, the function interface is different and I would like to fix HDFSHandler to match the new API of `PyArrow.fs.HadoopFileSystem`.

Note that in that case, I think we should delete [the part importing PyArrow.HadoopFileSystem](https://github.com/nteract/papermill/blob/92b02321eb48a1be4297f73ab804e85ee1e20ea0/papermill/iorw.py#L58) and update `requirements/hdfs.txt`.
(If it continues to support the previous filesystem, it needs to determine which was Imported and call the appropriate API.
However, I personally think that it is not necessary to support the deprecated class, considering the future maintenance effort.
Would you mind sharing your thoughts?)


Discussion:

Patch:
diff --git a/papermill/abs.py b/papermill/abs.py
index aa1c88a..700164f 100644
--- a/papermill/abs.py
+++ b/papermill/abs.py
@@ -30,7 +30,7 @@ class AzureBlobStore(object):
         see: https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1  # noqa: E501
         abs://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sastoken
         """
-        match = re.match(r"abs://(.*)\.blob\.core\.windows\.net\/(.*)\/(.*)\?(.*)$", url)
+        match = re.match(r"abs://(.*)\.blob\.core\.windows\.net\/(.*?)\/(.*)\?(.*)$", url)
         if not match:
             raise Exception("Invalid azure blob url '{0}'".format(url))
         else:
diff --git a/papermill/iorw.py b/papermill/iorw.py
index ba973cc..b79c00a 100644
--- a/papermill/iorw.py
+++ b/papermill/iorw.py
@@ -51,13 +51,10 @@ except ImportError:
     GCSFileSystem = missing_dependency_generator("gcsfs", "gcs")
 
 try:
-    try:
-        from pyarrow.fs import HadoopFileSystem
-    except ImportError:
-        # Attempt the older package import pattern in case we're using an old dep version.
-        from pyarrow import HadoopFileSystem
+    from pyarrow.fs import HadoopFileSystem, FileSelector
 except ImportError:
     HadoopFileSystem = missing_dependency_generator("pyarrow", "hdfs")
+
 try:
     from github import Github
 except ImportError:
@@ -358,18 +355,18 @@ class HDFSHandler(object):
 
     def _get_client(self):
         if self._client is None:
-            self._client = HadoopFileSystem()
+            self._client = HadoopFileSystem(host="default")
         return self._client
 
     def read(self, path):
-        with self._get_client().open(path, 'rb') as f:
+        with self._get_client().open_input_stream(path) as f:
             return f.read()
 
     def listdir(self, path):
-        return self._get_client().ls(path)
+        return [f.path for f in self._get_client().get_file_info(FileSelector(path))]
 
     def write(self, buf, path):
-        with self._get_client().open(path, 'wb') as f:
+        with self._get_client().open_output_stream(path) as f:
             return f.write(str.encode(buf))
 
     def pretty_path(self, path):
diff --git a/requirements/hdfs.txt b/requirements/hdfs.txt
index a5cce87..63570d3 100644
--- a/requirements/hdfs.txt
+++ b/requirements/hdfs.txt
@@ -1,1 +1,1 @@
-pyarrow
\ No newline at end of file
+pyarrow >= 2.0
\ No newline at end of file


Test patch:
diff --git a/papermill/tests/test_abs.py b/papermill/tests/test_abs.py
index 2fbc20c..d5b2124 100644
--- a/papermill/tests/test_abs.py
+++ b/papermill/tests/test_abs.py
@@ -50,6 +50,15 @@ class ABSTest(unittest.TestCase):
         self.assertEqual(params["blob"], "sasblob.txt")
         self.assertEqual(params["sas_token"], "sastoken")
 
+    def test_split_url_splits_valid_url_with_prefix(self):
+        params = AzureBlobStore._split_url(
+            "abs://myaccount.blob.core.windows.net/sascontainer/A/B/sasblob.txt?sastoken"
+        )
+        self.assertEqual(params["account"], "myaccount")
+        self.assertEqual(params["container"], "sascontainer")
+        self.assertEqual(params["blob"], "A/B/sasblob.txt")
+        self.assertEqual(params["sas_token"], "sastoken")
+
     def test_listdir_calls(self):
         self.assertEqual(
             self.abs.listdir(
diff --git a/papermill/tests/test_hdfs.py b/papermill/tests/test_hdfs.py
index 74b81a8..fab802e 100644
--- a/papermill/tests/test_hdfs.py
+++ b/papermill/tests/test_hdfs.py
@@ -5,12 +5,14 @@ from ..iorw import HDFSHandler
 
 
 class MockHadoopFileSystem(MagicMock):
-    def ls(self, path):
-        return ['test1.ipynb', 'test2.ipynb']
+    def get_file_info(self, path):
+        return [MockFileInfo('test1.ipynb'), MockFileInfo('test2.ipynb')]
 
-    def open(self, path, *args):
+    def open_input_stream(self, path):
         return MockHadoopFile()
 
+    def open_output_stream(self, path):
+        return MockHadoopFile()
 
 class MockHadoopFile(object):
     def __init__(self):
@@ -29,6 +31,10 @@ class MockHadoopFile(object):
         self._content = new_content
         return 1
 
+class MockFileInfo(object):
+    def __init__(self, path):
+        self.path = path
+
 
 @patch('papermill.iorw.HadoopFileSystem', side_effect=MockHadoopFileSystem())
 class HDFSTest(unittest.TestCase):