Repository: mlflow/mlflow

Issue:
[BUG] Artifact Locations become malformed with Windows Client
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [x] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04 Tracking Server (Docker) + Windows Client
- **MLflow installed from (source or binary)**: 
- **MLflow version (run ``mlflow --version``)**: 1.8.0
- **Python version**:  3.7.0
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: 

### Describe the problem
Artifact locations become malformed when using Windows as a client

### Code to reproduce issue
Either:
mlflow.keras.autolog()
or
mlflow.keras.log_model(model, "model")

Produce the artifact locations:

model
model\.\data
model\.\MLmodel
model\.\data\keras_module.txt
model\.\data\model.h5
model_summary.txt


### Other info / logs
No errors produced. Using an SFTP file store.

Looks like Windows folder location is being mixed in with Linux style ones.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [x] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for
Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [x] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations



Discussion:
Can confirm that after using Linux Environment the Locations aren't malformed:

![image](https://user-images.githubusercontent.com/10991802/83669504-97ae1800-a5c9-11ea-8691-7317fb7ff212.png)


Compared to artifacts created using a Windows Environment:

![image](https://user-images.githubusercontent.com/10991802/83669609-bad8c780-a5c9-11ea-893e-9c3b81a39069.png)

I am also facing the same issue with sklearn ElasticNet model from ML Flow Windows Client - 
<img width="337" alt="MLFlowWindowsClient" src="https://user-images.githubusercontent.com/16755538/83883942-2e85ec00-a712-11ea-8ad7-837194156ad1.png">


@mc55boy @sijuveugin Thanks for reporting the issue. Does the same thing occur when you log artifacts in local instead of an SFTP file store?
Hi Harutaka,

While logging artifacts in local it is fine.
This is an issue with pysftp client only.

On Thu, Jul 23, 2020, 4:27 AM Harutaka Kawamura <notifications@github.com>
wrote:

> @mc55boy <https://github.com/mc55boy> @sijuveugin
> <https://github.com/sijuveugin> Thanks for reporting the issue. Does the
> same thing occur when you log artifacts in local instead of an FPT file
> store?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlflow/mlflow/issues/2844#issuecomment-662882680>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AD72WUXWGKD6RWO4EC7B3TDR47X7ZANCNFSM4NDQASWA>
> .
>

@sijuveugin Thanks for the reply. Are artifacts stored correctly in an SFTP file store?
FPT - Hope you mean SFTP.

Only the file path issue is there. Files are not corrupted. If we refactor
file path everything will work fine.

Thanks
Siju Eugin

On Thu, Jul 23, 2020, 8:10 AM Harutaka Kawamura <notifications@github.com>
wrote:

> @sijuveugin <https://github.com/sijuveugin> Thanks for the reply. Are
> files stored correctly in an FPT file store?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlflow/mlflow/issues/2844#issuecomment-662970712>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AD72WURCPW2NULG2KKMTVK3R5ASDZANCNFSM4NDQASWA>
> .
>

I was finally able to reproduce the issue.

<img width="444" alt="Screen Shot 2020-07-24 at 2 16 31" src="https://user-images.githubusercontent.com/17039389/88317144-d3589900-cd53-11ea-8099-e5d3a02c7d7c.png">

Great

On Thu, Jul 23, 2020, 1:17 PM Harutaka Kawamura <notifications@github.com>
wrote:

> I was finally able to reproduce the issue.
>
> [image: Screen Shot 2020-07-24 at 2 16 31]
> <https://user-images.githubusercontent.com/17039389/88317144-d3589900-cd53-11ea-8099-e5d3a02c7d7c.png>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlflow/mlflow/issues/2844#issuecomment-663129136>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AD72WUUADIEPKMNLZZO7TATR5BWEBANCNFSM4NDQASWA>
> .
>

@sijuveugin Looks like the issue has been fixed using [this stackoverflow answer](https://stackoverflow.com/questions/58460718/python-pysftp-put-r-does-not-work-on-windows).

<img width="360" alt="Screen Shot 2020-07-24 at 2 58 05" src="https://user-images.githubusercontent.com/17039389/88321097-92fc1980-cd59-11ea-94f5-b51984469322.png">


Resolving patch:
diff --git a/mlflow/store/artifact/sftp_artifact_repo.py b/mlflow/store/artifact/sftp_artifact_repo.py
--- a/mlflow/store/artifact/sftp_artifact_repo.py
+++ b/mlflow/store/artifact/sftp_artifact_repo.py
@@ -1,4 +1,5 @@
 import os
+import sys
 
 import posixpath
 from six.moves import urllib
@@ -8,6 +9,18 @@
 from mlflow.exceptions import MlflowException
 
 
+# Based on: https://stackoverflow.com/a/58466685
+def _put_r_for_windows(sftp, local_dir, remote_dir, preserve_mtime=False):
+    for entry in os.listdir(local_dir):
+        local_path = os.path.join(local_dir, entry)
+        remote_path = posixpath.join(remote_dir, entry)
+        if os.path.isdir(local_path):
+            sftp.mkdir(remote_path)
+            _put_r_for_windows(sftp, local_path, remote_path, preserve_mtime)
+        else:
+            sftp.put(local_path, remote_path, preserve_mtime=preserve_mtime)
+
+
 class SFTPArtifactRepository(ArtifactRepository):
     """Stores artifacts as files in a remote directory, via sftp."""
 
@@ -70,7 +83,10 @@ def log_artifacts(self, local_dir, artifact_path=None):
         artifact_dir = posixpath.join(self.path, artifact_path) \
             if artifact_path else self.path
         self.sftp.makedirs(artifact_dir)
-        self.sftp.put_r(local_dir, artifact_dir)
+        if sys.platform == "win32":
+            _put_r_for_windows(self.sftp, local_dir, artifact_dir)
+        else:
+            self.sftp.put_r(local_dir, artifact_dir)
 
     def _is_directory(self, artifact_path):
         artifact_dir = self.path


Test patch: