Repository: mlflow/mlflow

Issue:
Error in loading spark ml models from gcs_artifact_storage [BUG] 
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [x] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **MLflow version (run ``mlflow --version``)**: 1.9.1
- **Python version**: 3.6.10


### Describe the problem
While loading/fetching sparkml models/artifacts from GCS bucket, raises the below error because `client.list_artifacts(run_id, artifact_relative_path)` returns the **<FileInfo: file_size=0, is_dir=False, path=artifact_relative_path +'/'>** which(path) does not exist along with other FileInfos. Hence it sends 404 error for file not found.

### Code to reproduce the issue
`mlflow.spark.load_model(gcs_path_to_model)`
or 
`client.download_artifacts(run_id, sparkml_model_relative_gcs_path)`


### Other info / logs
Complete error logs
2020/07/07 13:49:25 INFO mlflow.spark: URI 'gs://gcs-datascience-adhoc-us/ashwar/ecf191ff5c864bc99b75e531b31c3e4a/artifacts/spark-model/sparkml' does not point to the current DFS.
2020/07/07 13:49:25 INFO mlflow.spark: File 'gs://gcs-datascience-adhoc-us/ashwar/ecf191ff5c864bc99b75e531b31c3e4a/artifacts/spark-model/sparkml' not found on DFS. Will attempt to upload the file.
Traceback (most recent call last):
  File "/opt/conda/default/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 951, in download_to_file
    transport, file_obj, download_url, headers, start, end, raw_download
  File "/opt/conda/default/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 834, in _do_download
    download.consume(transport)
  File "/opt/conda/default/lib/python3.6/site-packages/google/resumable_media/requests/download.py", line 153, in consume
    self._process_response(result)
  File "/opt/conda/default/lib/python3.6/site-packages/google/resumable_media/_download.py", line 171, in _process_response
    response, _ACCEPTABLE_STATUS_CODES, self._get_status_code
  File "/opt/conda/default/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/ashwar_gupta_zeotap_com/pyspark/predict.py", line 29, in <module>
    model = mlflow.spark.load_model(model_production_uri, "hdfs:/user/mlflow/temp")
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/spark.py", line 501, in load_model
    return _load_model(model_uri=model_uri, dfs_tmpdir=dfs_tmpdir)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/spark.py", line 452, in _load_model
    model_path = _HadoopFileSystem.maybe_copy_from_uri(model_uri, tmp_path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/spark.py", line 302, in maybe_copy_from_uri
    return cls.maybe_copy_from_local_file(_download_artifact_from_uri(src_uri), dst_path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/tracking/artifact_utils.py", line 74, in _download_artifact_from_uri
    artifact_path=artifact_path, dst_path=output_path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/store/artifact/artifact_repo.py", line 128, in download_artifacts
    return download_artifact_dir(artifact_path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/store/artifact/artifact_repo.py", line 106, in download_artifact_dir
    download_file(file_info.path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/store/artifact/artifact_repo.py", line 90, in download_file
    self._download_file(remote_file_path=fullpath, local_path=local_file_path)
  File "/opt/conda/default/lib/python3.6/site-packages/mlflow/store/artifact/gcs_artifact_repo.py", line 105, in _download_file
    gcs_bucket.blob(remote_full_path).download_to_filename(local_path)
  File "/opt/conda/default/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1024, in download_to_filename
    if_metageneration_not_match=if_metageneration_not_match,
  File "/opt/conda/default/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 954, in download_to_file
    _raise_from_invalid_response(exc)
  File "/opt/conda/default/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 3091, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/gcs-datascience-adhoc-us/o/ashwar%2Fecf191ff5c864bc99b75e531b31c3e4a%2Fartifacts%2Fspark-model%2Fsparkml?alt=media: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
20/07/07 13:49:25 INFO org.spark_project.jetty.server.AbstractConnector: Stopped Spark@5b7269db{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

**PS: I have found the fix and tested the same.**   


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [x] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [x] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support





Discussion:
@ashwar-zeotap Thanks for filing this. Does the same thing occur for other model type (e.g. sklearn)?
@harupy No, was able to load sklearn models from gcs bucket. It is because model here is stored as just one file 

Resolving patch:
diff --git a/mlflow/store/artifact/gcs_artifact_repo.py b/mlflow/store/artifact/gcs_artifact_repo.py
--- a/mlflow/store/artifact/gcs_artifact_repo.py
+++ b/mlflow/store/artifact/gcs_artifact_repo.py
@@ -86,6 +86,10 @@ def list_artifacts(self, path=None):
 
         results = bkt.list_blobs(prefix=prefix, delimiter="/")
         for result in results:
+            # skip blobs matching current directory path as list_blobs api
+            # returns subdirectories as well
+            if result.name == prefix:
+                continue
             blob_path = result.name[len(artifact_path) + 1 :]
             infos.append(FileInfo(blob_path, False, result.size))
 


Test patch: