Repository: mlflow/mlflow

Issue:
[BUG] Model lineage tracking - missing "MLmodel/registered_model_meta" file
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Other

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### MLflow version

- Client: 2.8.0
- Tracking server: 2.3.1


### System information

- Ubuntu 20.04 LTS
- Python 3.8

### Describe the problem

It seems that latest release of MLflow breaks compatibility between client and server when server runs on older version of MLflow.

https://github.com/mlflow/mlflow/commit/85037652d6caf4a78d8ac73ca734e433e86b1fd9


### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
REPLACE_ME
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import mlflow
from mlflow.store.artifact.artifact_repository_registry import get_artifact_repository

# server 2.3.1 + client 2.7.1 -> works
# server 2.3.1 + client 2.8.0 -> throws

mlflow.set_tracking_uri('FILL IN')

model_name = 'FILL IN'
model_version = 'FILL IN'
model_uri = f'models:/{model_name}/{model_version}'

artifact_repository = get_artifact_repository(model_uri)
mlmodel_path = artifact_repository.download_artifacts('MLmodel')
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
Traceback (most recent call last):
  File "/app/miniconda/envs/engine/bin/setup-image", line 33, in <module>
    sys.exit(load_entry_point('engine', 'console_scripts', 'setup-image')())
  File "/app/engine/assembly/scripts.py", line 53, in setup_image
    mlmodel_path = artifact_repository.download_artifacts('MLmodel')
  File "/app/miniconda/envs/engine/lib/python3.8/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 173, in download_artifacts
    self._add_registered_model_meta_file(model_path)
  File "/app/miniconda/envs/engine/lib/python3.8/site-packages/mlflow/store/artifact/models_artifact_repo.py", line 143, in _add_registered_model_meta_file
    write_yaml(
  File "/app/miniconda/envs/engine/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 250, in write_yaml
    raise e
  File "/app/miniconda/envs/engine/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 240, in write_yaml
    with codecs.open(yaml_file_name, mode="w", encoding=ENCODING) as yaml_file:
  File "/app/miniconda/envs/engine/lib/python3.8/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
NotADirectoryError: [Errno 20] Not a directory: '/tmp/tmpv6q_d0ps/MLmodel/registered_model_meta'
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/gateway`: AI Gateway service, Gateway client APIs, third-party Gateway integrations
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [X] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [X] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@MajerMartin Thanks for reporting the issue. Here's a minimum repro:

```python
import mlflow
from mlflow.store.artifact.artifact_repository_registry import get_artifact_repository

mlflow.set_tracking_uri("sqlite:///:memory:")

with mlflow.start_run():
    mlflow.sklearn.log_model("I am a model", "model", registered_model_name="foo")

model_uri = f"models:/foo/1"

artifact_repository = get_artifact_repository(model_uri)
mlmodel_path = artifact_repository.download_artifacts("MLmodel")

```
https://github.com/mlflow/mlflow/pull/9402 added `_add_registered_model_meta_file`.
<!-- assign-maintainer -->
@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

Resolving patch:
diff --git a/mlflow/store/artifact/models_artifact_repo.py b/mlflow/store/artifact/models_artifact_repo.py
--- a/mlflow/store/artifact/models_artifact_repo.py
+++ b/mlflow/store/artifact/models_artifact_repo.py
@@ -1,3 +1,5 @@
+import logging
+import os
 import urllib.parse
 
 import mlflow
@@ -20,6 +22,8 @@
 
 REGISTERED_MODEL_META_FILE_NAME = "registered_model_meta"
 
+_logger = logging.getLogger(__name__)
+
 
 class ModelsArtifactRepository(ArtifactRepository):
     """
@@ -169,8 +173,14 @@ def download_artifacts(self, artifact_path, dst_path=None):
         :return: Absolute path of the local filesystem location containing the desired artifacts.
         """
 
+        from mlflow.models.model import MLMODEL_FILE_NAME
+
         model_path = self.repo.download_artifacts(artifact_path, dst_path)
-        self._add_registered_model_meta_file(model_path)
+        # NB: only add the registered model metadata iff the artifact path is at the root model
+        # directory. For individual files or subdirectories within the model directory, do not
+        # create the metadata file.
+        if os.path.isdir(model_path) and MLMODEL_FILE_NAME in os.listdir(model_path):
+            self._add_registered_model_meta_file(model_path)
 
         return model_path
 


Test patch:
diff --git a/tests/sklearn/test_sklearn_model_export.py b/tests/sklearn/test_sklearn_model_export.py
--- a/tests/sklearn/test_sklearn_model_export.py
+++ b/tests/sklearn/test_sklearn_model_export.py
@@ -28,6 +28,7 @@
 from mlflow.models.utils import _read_example
 from mlflow.protos.databricks_pb2 import INVALID_PARAMETER_VALUE, ErrorCode
 from mlflow.store._unity_catalog.registry.rest_store import UcModelRegistryStore
+from mlflow.store.artifact.artifact_repository_registry import get_artifact_repository
 from mlflow.store.artifact.s3_artifact_repo import S3ArtifactRepository
 from mlflow.tracking.artifact_utils import _download_artifact_from_uri
 from mlflow.types import DataType
@@ -837,3 +838,24 @@ def test_model_size_bytes(sklearn_logreg_model, tmp_path):
 
     mlmodel = yaml.safe_load(tmp_path.joinpath("MLmodel").read_bytes())
     assert mlmodel["model_size_bytes"] == expected_size
+
+
+def test_model_registration_metadata_handling(sklearn_knn_model, tmp_path):
+    artifact_path = "model"
+    with mlflow.start_run():
+        mlflow.sklearn.log_model(
+            sk_model=sklearn_knn_model.model,
+            artifact_path=artifact_path,
+            registered_model_name="test",
+        )
+        model_uri = "models:/test/1"
+
+    artifact_repository = get_artifact_repository(model_uri)
+
+    dst_full = tmp_path.joinpath("full")
+    dst_full.mkdir()
+
+    artifact_repository.download_artifacts("MLmodel", dst_full)
+    # This validates that the models artifact repo will not attempt to create a
+    # "registered model metadata" file if the source of an artifact download is a file.
+    assert os.listdir(dst_full) == ["MLmodel"]
diff --git a/tests/store/artifact/test_models_artifact_repo.py b/tests/store/artifact/test_models_artifact_repo.py
--- a/tests/store/artifact/test_models_artifact_repo.py
+++ b/tests/store/artifact/test_models_artifact_repo.py
@@ -1,5 +1,4 @@
 from unittest import mock
-from unittest.mock import Mock
 
 import pytest
 
@@ -10,6 +9,7 @@
 from mlflow.store.artifact.unity_catalog_models_artifact_repo import (
     UnityCatalogModelsArtifactRepository,
 )
+from mlflow.utils.os import is_windows
 
 from tests.store.artifact.constants import (
     UC_MODELS_ARTIFACT_REPOSITORY,
@@ -111,50 +111,104 @@ def test_models_artifact_repo_init_with_stage_uri_and_not_using_databricks_regis
         get_repo_mock.assert_called_once_with(artifact_location)
 
 
-def test_models_artifact_repo_uses_repo_download_artifacts():
+def test_models_artifact_repo_uses_repo_download_artifacts(tmp_path):
     """
-    ``ModelsArtifactRepository`` should delegate `download_artifacts` to its
-    ``self.repo.download_artifacts`` function.
+    `ModelsArtifactRepository` should delegate `download_artifacts` to its
+    `self.repo.download_artifacts` function.
     """
     artifact_location = "s3://blah_bucket/"
+    dummy_file = tmp_path / "dummy_file.txt"
+    dummy_file.touch()
+
     with mock.patch.object(
         MlflowClient, "get_model_version_download_uri", return_value=artifact_location
     ), mock.patch.object(ModelsArtifactRepository, "_add_registered_model_meta_file"):
         model_uri = "models:/MyModel/12"
         models_repo = ModelsArtifactRepository(model_uri)
-        models_repo.repo = Mock()
-        models_repo.download_artifacts("artifact_path", "dst_path")
-        models_repo.repo.download_artifacts.assert_called_once()
+        models_repo.repo = mock.Mock(**{"download_artifacts.return_value": str(dummy_file)})
+
+        models_repo.download_artifacts("artifact_path", str(tmp_path))
+
+        models_repo.repo.download_artifacts.assert_called_once_with("artifact_path", str(tmp_path))
+
 
+@pytest.mark.skipif(is_windows(), reason="This test fails on Windows")
+def test_models_artifact_repo_download_with_real_files(tmp_path):
+    # Simulate an artifact repository
+    temp_remote_storage = tmp_path / "remote_storage"
+    model_dir = temp_remote_storage / "model_dir"
+    model_dir.mkdir(parents=True)
+    mlmodel_path = model_dir / "MLmodel"
+    mlmodel_path.touch()
+
+    # Mock get_model_version_download_uri to return the path to the temp_remote_storage location
+    with mock.patch.object(
+        MlflowClient, "get_model_version_download_uri", return_value=str(model_dir)
+    ):
+        # Create ModelsArtifactRepository instance
+        models_repo = ModelsArtifactRepository("models:/MyModel/1")
 
-def test_models_artifact_repo_add_registered_model_meta_file():
-    from mlflow.store.artifact.models_artifact_repo import REGISTERED_MODEL_META_FILE_NAME
+        # Use another temporary directory as the download destination
+        temp_local_storage = tmp_path / "local_storage"
+        temp_local_storage.mkdir()
 
-    artifact_path = "artifact_path"
-    dst_path = "dst_path"
+        # Download artifacts
+        models_repo.download_artifacts("", str(temp_local_storage))
+
+        # Check if the files are downloaded correctly
+        downloaded_mlmodel_path = temp_local_storage / "MLmodel"
+        assert downloaded_mlmodel_path.exists()
+
+        # Check if the metadata file is created
+        metadata_file_path = temp_local_storage / "registered_model_meta"
+        assert metadata_file_path.exists()
+
+
+def test_models_artifact_repo_does_not_add_meta_for_file(tmp_path):
+    artifact_path = "artifact_file.txt"
+    model_name = "MyModel"
+    model_version = "12"
     artifact_location = f"s3://blah_bucket/{artifact_path}"
-    artifact_dst_path = f"{dst_path}/{artifact_path}"
+
+    dummy_file = tmp_path / artifact_path
+    dummy_file.touch()
+
+    with mock.patch.object(
+        MlflowClient, "get_model_version_download_uri", return_value=artifact_location
+    ), mock.patch.object(
+        ModelsArtifactRepository, "_add_registered_model_meta_file"
+    ) as add_meta_mock:
+        models_repo = ModelsArtifactRepository(f"models:/{model_name}/{model_version}")
+        models_repo.repo = mock.Mock(**{"download_artifacts.return_value": str(dummy_file)})
+
+        models_repo.download_artifacts(artifact_path, str(tmp_path))
+
+        add_meta_mock.assert_not_called()
+
+
+def test_models_artifact_repo_does_not_add_meta_for_directory_without_mlmodel(tmp_path):
+    artifact_path = "artifact_directory"
     model_name = "MyModel"
     model_version = "12"
+    artifact_location = f"s3://blah_bucket/{artifact_path}"
+
+    # Create a directory without an MLmodel file
+    dummy_dir = tmp_path / artifact_path
+    dummy_dir.mkdir()
+    dummy_file = dummy_dir / "dummy_file.txt"
+    dummy_file.touch()
 
     with mock.patch.object(
         MlflowClient, "get_model_version_download_uri", return_value=artifact_location
-    ), mock.patch("mlflow.store.artifact.models_artifact_repo.write_yaml") as write_yaml_mock:
+    ), mock.patch.object(
+        ModelsArtifactRepository, "_add_registered_model_meta_file"
+    ) as add_meta_mock:
         models_repo = ModelsArtifactRepository(f"models:/{model_name}/{model_version}")
-        models_repo.repo = Mock(**{"download_artifacts.return_value": artifact_dst_path})
-
-        models_repo.download_artifacts(artifact_path, dst_path)
-
-        write_yaml_mock.assert_called_with(
-            artifact_dst_path,
-            REGISTERED_MODEL_META_FILE_NAME,
-            {
-                "model_name": model_name,
-                "model_version": model_version,
-            },
-            overwrite=True,
-            ensure_yaml_extension=False,
-        )
+        models_repo.repo = mock.Mock(**{"download_artifacts.return_value": str(dummy_dir)})
+
+        models_repo.download_artifacts(artifact_path, str(tmp_path))
+
+        add_meta_mock.assert_not_called()
 
 
 def test_split_models_uri():