Repository: mlflow/mlflow

Issue:
[BUG] HdfsArtifactRepository list_artifacts recursively lists all items under directory
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)
for information on what types of issues we address.

For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

  
Please do not delete this template unless you are sure your issue is outside its scope.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Centos 7
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.1.0
- **Python version**: 3.6.8
- **npm version, if running the dev UI**: N/A
- **Exact command to reproduce**: mlflow artifacts list -r <run id for artifacts stored on hdfs>

### Describe the problem
list_artifacts of an artifact repository is expected to only list the files directly under the provided path (see https://github.com/mlflow/mlflow/blob/4b1868719837d1844f19b6242643222549ee2794/mlflow/store/cli.py#L74 ). HdfsArtifactRepository walks all files under the given path and returns them (see https://github.com/mlflow/mlflow/blob/4b1868719837d1844f19b6242643222549ee2794/mlflow/store/hdfs_artifact_repo.py#L89 ).

This behavior breaks the mflow server as it expects the behavior specified in the cli file.

### Code to reproduce issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks,
please include the full traceback. Large logs and files should be attached.



Discussion:

Resolving patch:
diff --git a/mlflow/store/hdfs_artifact_repo.py b/mlflow/store/hdfs_artifact_repo.py
--- a/mlflow/store/hdfs_artifact_repo.py
+++ b/mlflow/store/hdfs_artifact_repo.py
@@ -74,17 +74,21 @@ def list_artifacts(self, path=None):
 
             :param path: Relative source path. Possible subdirectory existing under
                          hdfs:/some/path/run_id/artifacts
-            :return: List of files and directories under given path -
-                example:
-                    ['conda.yaml', 'MLmodel', 'model.pkl']
+            :return: List of FileInfos under given path
         """
         hdfs_base_path = _resolve_base_path(self.path, path)
-        base_path_len = len(hdfs_base_path) + 1
 
         with hdfs_system(host=self.host, port=self.port) as hdfs:
             paths = []
-            for path, is_dir, size in self._walk_path(hdfs, hdfs_base_path):
-                paths.append(FileInfo(path[base_path_len:], is_dir, size))
+            if hdfs.exists(hdfs_base_path):
+                for file_detail in hdfs.ls(hdfs_base_path, detail=True):
+                    file_name = file_detail.get("name")
+                    # Strip off anything that comes before the artifact root e.g. hdfs://name
+                    offset = file_name.index(self.path)
+                    rel_path = _relative_path_remote(self.path, file_name[offset:])
+                    is_dir = file_detail.get("kind") == "directory"
+                    size = file_detail.get("size")
+                    paths.append(FileInfo(rel_path, is_dir, size))
             return sorted(paths, key=lambda f: paths)
 
     def _walk_path(self, hdfs, hdfs_path):
@@ -202,6 +206,9 @@ def _tmp_dir(local_path):
 
 
 def _download_hdfs_file(hdfs, remote_file_path, local_file_path):
+    # Ensure all required directories exist. Without doing this nested files can't be downloaded.
+    dirs = os.path.dirname(local_file_path)
+    os.makedirs(dirs)
     with open(local_file_path, 'wb') as f:
         f.write(hdfs.open(remote_file_path, 'rb').read())
 


Test patch:
diff --git a/tests/store/test_hdfs_artifact_repo.py b/tests/store/test_hdfs_artifact_repo.py
--- a/tests/store/test_hdfs_artifact_repo.py
+++ b/tests/store/test_hdfs_artifact_repo.py
@@ -105,23 +105,51 @@ def test_log_artifacts(hdfs_system_mock):
 
 
 @mock.patch('pyarrow.hdfs.HadoopFileSystem')
-def test_list_artifacts(hdfs_system_mock):
-    repo = HdfsArtifactRepository('hdfs:/some/path')
+def test_list_artifacts_root(hdfs_system_mock):
+    repo = HdfsArtifactRepository('hdfs://host/some/path')
 
-    expected = [FileInfo('conda.yaml', False, 33),
-                FileInfo('model.pkl', False, 33),
-                FileInfo('MLmodel', False, 33)]
+    expected = [FileInfo('model', True, 0)]
 
-    hdfs_system_mock.return_value.walk.return_value = [
-        ('/some/path', False, ['conda.yaml', 'model.pkl', 'MLmodel'])]
-    hdfs_system_mock.return_value.info.return_value.get.return_value = 33
-    hdfs_system_mock.return_value.isdir.side_effect = [True, False, False, False]
+    hdfs_system_mock.return_value.ls.return_value = [{
+            'kind': 'directory',
+            'name': 'hdfs://host/some/path/model',
+            'size': 0,
+            }]
 
     actual = repo.list_artifacts()
 
     assert actual == expected
 
 
+@mock.patch('pyarrow.hdfs.HadoopFileSystem')
+def test_list_artifacts_nested(hdfs_system_mock):
+    repo = HdfsArtifactRepository('hdfs:://host/some/path')
+
+    expected = [FileInfo('model/conda.yaml', False, 33),
+                FileInfo('model/model.pkl', False, 33),
+                FileInfo('model/MLmodel', False, 33)]
+
+    hdfs_system_mock.return_value.ls.return_value = [{
+            'kind': 'file',
+            'name': 'hdfs://host/some/path/model/conda.yaml',
+            'size': 33,
+            },
+            {
+            'kind': 'file',
+            'name': 'hdfs://host/some/path/model/model.pkl',
+            'size': 33,
+            },
+            {
+            'kind': 'file',
+            'name': 'hdfs://host/some/path/model/MLmodel',
+            'size': 33,
+            }]
+
+    actual = repo.list_artifacts('model')
+
+    assert actual == expected
+
+
 @mock.patch('pyarrow.hdfs.HadoopFileSystem', spec=HadoopFileSystem)
 def test_list_artifacts_empty_hdfs_dir(hdfs_system_mock):
     hdfs_system_mock.return_value.exists.return_value = False