Repository: mlflow/mlflow

Issue:
[BUG] Duplicate _SPARK_MODEL_PATH_SUB in mlflow.spark.log_model
### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

v1.28.0

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Python version**:
- **yarn version, if running the dev UI**:


### Describe the problem

There is a duplicate `_SPARK_MODEL_PATH_SUB` in `_maybe_save_model` function

https://github.com/mlflow/mlflow/blob/c9f64728f93331a4b1d70d685c17694fb2ac45fc/mlflow/spark.py#L226-L250

https://github.com/mlflow/mlflow/blob/c9f64728f93331a4b1d70d685c17694fb2ac45fc/mlflow/spark.py#L291-L298

### Tracking information

_No response_

### Code to reproduce issue

https://github.com/mlflow/mlflow/blob/c9f64728f93331a4b1d70d685c17694fb2ac45fc/mlflow/spark.py#L234

https://github.com/mlflow/mlflow/blob/c9f64728f93331a4b1d70d685c17694fb2ac45fc/mlflow/spark.py#L291-L298

### Stack trace

Up version incompatible.

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [X] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/pipelines`: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:

Resolving patch:
diff --git a/mlflow/spark.py b/mlflow/spark.py
--- a/mlflow/spark.py
+++ b/mlflow/spark.py
@@ -231,7 +231,7 @@ def log_model(
     # artifact repo via Spark. If this fails, we defer to Model.log().
     elif is_local_uri(run_root_artifact_uri) or not _maybe_save_model(
         spark_model,
-        append_to_uri_path(run_root_artifact_uri, artifact_path, _SPARK_MODEL_PATH_SUB),
+        append_to_uri_path(run_root_artifact_uri, artifact_path),
     ):
         return Model.log(
             artifact_path=artifact_path,


Test patch:
diff --git a/tests/spark/test_spark_model_export.py b/tests/spark/test_spark_model_export.py
--- a/tests/spark/test_spark_model_export.py
+++ b/tests/spark/test_spark_model_export.py
@@ -663,7 +663,7 @@ def test_model_is_recorded_when_using_direct_save(spark_model_iris):
             "false",
             True,
             False,
-            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml/sparkml",
+            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml",
         ),
         (
             "dbfs:/databricks/mlflow-tracking/a/b",
@@ -671,7 +671,7 @@ def test_model_is_recorded_when_using_direct_save(spark_model_iris):
             "",
             False,
             True,
-            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml/sparkml",
+            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml",
         ),
         (
             "dbfs:/databricks/mlflow-tracking/a/b",
@@ -679,7 +679,7 @@ def test_model_is_recorded_when_using_direct_save(spark_model_iris):
             "",
             True,
             True,
-            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml/sparkml",
+            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml",
         ),
         (
             "dbfs:/databricks/mlflow-tracking/a/b",
@@ -687,10 +687,10 @@ def test_model_is_recorded_when_using_direct_save(spark_model_iris):
             "true",
             True,
             True,
-            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml/sparkml",
+            "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml",
         ),
-        ("dbfs:/root/a/b", "12.0", "", True, True, "dbfs:/root/a/b/model/sparkml/sparkml"),
-        ("s3://mybucket/a/b", "12.0", "", True, True, "s3://mybucket/a/b/model/sparkml/sparkml"),
+        ("dbfs:/root/a/b", "12.0", "", True, True, "dbfs:/root/a/b/model/sparkml"),
+        ("s3://mybucket/a/b", "12.0", "", True, True, "s3://mybucket/a/b/model/sparkml"),
     ],
 )
 def test_model_logged_via_mlflowdbfs_when_appropriate(
@@ -800,7 +800,7 @@ def mock_get_dbutils():
             mock_save.assert_called_once_with(
                 f"mlflowdbfs:///artifacts?run_id={run_id}&path=/model/sparkml"
                 if dummy_read_shows_mlflowdbfs_available
-                else "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml/sparkml"
+                else "dbfs:/databricks/mlflow-tracking/a/b/model/sparkml"
             )