Repository: mlflow/mlflow

Issue:
[BUG] Internal Server Error on delete
### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

1.19.0

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS 7
- **Python version**: 3.9
- **yarn version, if running the dev UI**:


### Describe the problem

After using mlflow for ~1 year, we've run into this difficult to reproduce bug several times where an experiment tab will show an INTERNAL_SERVER_ERROR and stops displaying the runs that were stored in that experiment.

This issue seems to come up when deleting one or multiple runs from an experiment, but it doesn't happen every time a delete is attempted. But it seems to usually happen on the first delete attempt in an experiment if it's going to fail... The delete command hangs longer than usual then prompts the INTERNAL_SERVER_ERROR.

<img width="1624" alt="Screen Shot 2022-07-26 at 4 30 40 PM" src="https://user-images.githubusercontent.com/93339979/181131068-8ea76d64-f1a8-473d-8c4c-63b7e4d0fad0.png">

Besides a full fix, I would also be interested to learn if it's possible to retrieve the runs that were stored in the broken experiment.

### Steps to reproduce the bug

This is inconsistently reproducible (and I apologize, I know that's not very helpful), but I logged 50-100 runs in a new experiment. Each run had 5 metrics with ~200 values. I then tried deleting a random amount of runs. Sometimes one or the whole page. Sometimes the experiment broke, but most times the delete executed no problem.

### Code to generate data required to reproduce the bug

```python
import mlflow
import boto3
import random

MLFLOW_EXPERIMENT_NAME = "break server"
if not mlflow.get_experiment_by_name(MLFLOW_EXPERIMENT_NAME):
      mlflow.create_experiment(MLFLOW_EXPERIMENT_NAME)
mlflow.set_experiment(MLFLOW_EXPERIMENT_NAME)

mlflow.start_run()

mlflow.log_param("number", 1)

randomlist = random.sample(range(10, 500), 200)
for x in randomlist:
  mlflow.log_metric("rand1", x)
  mlflow.log_metric("rand2", x)
  mlflow.log_metric("rand3", x)
  mlflow.log_metric("rand4", x)
  mlflow.log_metric("rand5", x)

mlflow.end_run()
```
I then ran this 50-100 times to fill the experiment with runs to delete.

### Is the console panel in DevTools showing errors relevant to the bug?

<img width="1680" alt="Screen Shot 2022-07-26 at 5 19 29 PM" src="https://user-images.githubusercontent.com/93339979/181134085-3bd23fb4-c985-488a-b823-ea6ec2cf296a.png">

```
Error: Promised response from onMessage listener went out of scope 5 [background.js:841:170](moz-extension://d6a7add6-de11-4239-898f-0a287b30c3e7/dist/background.js)
XHR failed 
Object { readyState: 4, getResponseHeader: getResponseHeader(e), getAllResponseHeaders: getAllResponseHeaders(), setRequestHeader: setRequestHeader(e, t), overrideMimeType: overrideMimeType(e), statusCode: statusCode(e), abort: abort(e), state: state(), always: always(), catch: catch(e)
, … }
​
abort: function abort(e)​
always: function always()​
catch: function catch(e)​
done: function add()​
fail: function add()​
getAllResponseHeaders: function getAllResponseHeaders()​
getResponseHeader: function getResponseHeader(e)​
overrideMimeType: function overrideMimeType(e)​
pipe: function pipe()​
progress: function add()​
promise: function promise(e)
​
readyState: 4
​
responseText: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n"
​
setRequestHeader: function setRequestHeader(e, t)​
state: function state()
​
status: 500
​
statusCode: function statusCode(e)
​
statusText: "Internal Server Error"
​
then: function then(e, r, o)​
<prototype>: Object { … }
[main.895f3836.chunk.js:1:15852](https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/main.895f3836.chunk.js)
    error https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/main.895f3836.chunk.js:1
    l https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/2.8c754ded.chunk.js:1
    fireWith https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/2.8c754ded.chunk.js:1
    S https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/2.8c754ded.chunk.js:1
    t https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/2.8c754ded.chunk.js:1
Object { xhr: {…} }
​
xhr: Object { readyState: 4, getResponseHeader: getResponseHeader(e), getAllResponseHeaders: getAllResponseHeaders()
, … }
​​
abort: function abort(e)​​
always: function always()​​
catch: function catch(e)​​
done: function add()​​
fail: function add()​​
getAllResponseHeaders: function getAllResponseHeaders()​​
getResponseHeader: function getResponseHeader(e)​​
overrideMimeType: function overrideMimeType(e)​​
pipe: function pipe()​​
progress: function add()​​
promise: function promise(e)
​​
readyState: 4
​​
responseText: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n"
​​
setRequestHeader: function setRequestHeader(e, t)​​
state: function state()
​​
status: 500
​​
statusCode: function statusCode(e)
​​
statusText: "Internal Server Error"
​​
then: function then(e, r, o)​​
<prototype>: Object { … }
​
<prototype>: Object { … }
[main.895f3836.chunk.js:1:8885](https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/main.895f3836.chunk.js)
    value https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/main.895f3836.chunk.js:1
    value https://mlwf-devel-server.mlflow.pnl.gov/static-files/static/js/main.895f3836.chunk.js:1
```

### Does the network panel in DevTools contain failed requests relevant to the bug?

<img width="1680" alt="Screen Shot 2022-07-26 at 5 24 17 PM" src="https://user-images.githubusercontent.com/93339979/181134493-6bcb6f07-3e8c-46e9-ba71-1ac1bff0c04c.png">



Discussion:
@jaelynlitz Thanks for reporting this issue! Have you checked the tracking server logs? When an internal server error occurs, the server usually generates logs that look like this:

```
[2022-07-27 11:18:50 +0900] [62087] [DEBUG] POST /ajax-api/2.0/preview/mlflow/runs/search
2022/07/27 11:18:50 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/runs/search [POST]
Traceback (most recent call last):
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 454, in wrapper
    return func(*args, **kwargs)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 512, in wrapper
    return func(*args, **kwargs)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 858, in _search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/abstract_store.py", line 294, in search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/sqlalchemy_store.py", line 1097, in _search_runs
    raise Exception("Unexpected error")
Exception: Unexpected error
```

`Error: Promised response from onMessage listener went out of scope 5 [background.js:841:170](moz-extension://d6a7add6-de11-4239-898f-0a287b30c3e7/dist/background.js)` is unrelated to MLflow. I think this comes from a chrome extension.
Thanks for the quick response @harupy! I found the logs you're looking for:

```
> WARNING:root:Malformed run '14c113ff02da4f5b9125a70482316277'. Detailed error Yaml file '/data/data/5/14c113ff02da4f5b9125a70482316277/meta.yaml' does not exist.
> Traceback (most recent call last):
>   File "/app/mlflow/store/tracking/file_store.py", line 606, in _list_run_infos
>     run_info = self._get_run_info_from_dir(r_dir)
>   File "/app/mlflow/store/tracking/file_store.py", line 450, in _get_run_info_from_dir
>     meta = read_yaml(run_dir, FileStore.META_DATA_FILE_NAME)
>   File "/app/mlflow/utils/file_utils.py", line 167, in read_yaml
>     raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
> mlflow.exceptions.MissingConfigException: Yaml file '/data/data/5/14c113ff02da4f5b9125a70482316277/meta.yaml' does not exist.

> 2022/07/27 19:03:31 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/runs/search [POST]
> Traceback (most recent call last):
>   File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
>     response = self.full_dispatch_request()
>   File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
>     rv = self.handle_user_exception(e)
>   File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
>     reraise(exc_type, exc_value, tb)
>   File "/opt/conda/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
>     raise value
>   File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
>     rv = self.dispatch_request()
>   File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
>     return self.view_functions[rule.endpoint](**req.view_args)
>   File "/app/mlflow/server/handlers.py", line 155, in wrapper
>     return func(*args, **kwargs)
>   File "/app/mlflow/server/handlers.py", line 394, in _search_runs
>     max_results, order_by, page_token)
>   File "/app/mlflow/store/tracking/abstract_store.py", line 230, in search_runs
>     order_by, page_token)
>   File "/app/mlflow/store/tracking/file_store.py", line 631, in _search_runs
>     run_infos = self._list_run_infos(experiment_id, run_view_type)
>   File "/app/mlflow/store/tracking/file_store.py", line 606, in _list_run_infos
>     run_info = self._get_run_info_from_dir(r_dir)
>   File "/app/mlflow/store/tracking/file_store.py", line 451, in _get_run_info_from_dir
>     run_info = _read_persisted_run_info_dict(meta)
>   File "/app/mlflow/store/tracking/file_store.py", line 69, in _read_persisted_run_info_dict
>     dict_copy = run_info_dict.copy()
> AttributeError: 'NoneType' object has no attribute 'copy'
```
Hi @jaelynlitz, are you deleting runs permanently using [mlflow gc](https://mlflow.org/docs/latest/cli.html#mlflow-gc), or are you soft-deleting them using MLflow's [delete_run](https://mlflow.org/docs/latest/cli.html#mlflow-gc) API / UI?
@dbczumar I was deleting runs using the UI (so I'd check the boxes then hit the delete button)
Got it. The `mlflow.exceptions.MissingConfigException: Yaml file '/data/data/5/14c113ff02da4f5b9125a70482316277/meta.yaml' does not exist.` excerpt from the error message seems to indicate the metadata file associated with the run has been deleted. However, deletion through the UI doesn't actually remove run files, it just writes a `lifecycle_stage: deleted` line to the metadata file. Is anyone else in your organization using `mlflow gc` or manually removing files from the `mlruns` directory?
@jaelynlitz @dbczumar  The stack trace indicates `read_yaml` in `_get_run_info_from_dir` returns `None`:

https://github.com/mlflow/mlflow/blob/56698a23f5996e1d6b605cd714a5917434adeb9b/mlflow/store/tracking/file_store.py#L449-L451

(Note this is the code in MLflow 1.10.0)

`read_yaml` returns `None` when the file `{run_dir}/{FileStore.META_DATA_FILE_NAME}` exists, but it's empty.

During `delete_run`, `meta.yaml` is opened with the `w` mode to update `lifecycle_stage` to `deleted` and becomes empty for a moment:

https://github.com/mlflow/mlflow/blob/56698a23f5996e1d6b605cd714a5917434adeb9b/mlflow/store/tracking/file_store.py#L741

https://github.com/mlflow/mlflow/blob/56698a23f5996e1d6b605cd714a5917434adeb9b/mlflow/utils/file_utils.py#L143-L147

If the read occurs before `yaml.dump` completes, it might cause the error.
This might not be what's really happening but we can reproduce the error with this script:

```python
# Before running this script: log some runs using the file store and launch MLflow UI

import time

path = "mlruns/0/f18249d1f76d445f8324550a74c0d07b/meta.yaml"

with open(path, "r") as f:
    a = f.read()

with open(path, "w") as f:
    print("Fetch some runs on MLflow UI")
    time.sleep(10)
    f.write(a)
```

### UI

<img width="1325" alt="image" src="https://user-images.githubusercontent.com/17039389/181506491-9b7868d0-44c5-49e3-9284-d3b1425f265a.png">


### Tracking server log

```
2022/07/28 21:36:08 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/runs/search [POST]
Traceback (most recent call last):
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/haru/miniconda3/envs/mlflow-dev-env/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 454, in wrapper
    return func(*args, **kwargs)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 512, in wrapper
    return func(*args, **kwargs)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/server/handlers.py", line 858, in _search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/abstract_store.py", line 294, in search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/file_store.py", line 809, in _search_runs
    run_infos = self._list_run_infos(experiment_id, run_view_type)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/file_store.py", line 777, in _list_run_infos
    run_info = self._get_run_info_from_dir(r_dir)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/file_store.py", line 610, in _get_run_info_from_dir
    run_info = _read_persisted_run_info_dict(meta)
  File "/home/haru/Desktop/repositories/mlflow/mlflow/store/tracking/file_store.py", line 112, in _read_persisted_run_info_dict
    dict_copy = run_info_dict.copy()
AttributeError: 'NoneType' object has no attribute 'copy'
```
@dbczumar I doubt any of our other users are using the `mlflow gc` command or touching the mlruns directory. We mainly interact with mlflow by logging runs through code, then looking through the UI.
@jaelynlitz Thank you for clarifying. @harupy awesome investigation! I'm inclined to believe that the race condition noted by @harupy is the cause of transient `SearchRuns` API request errors that cause UI failures. I will investigate a fix. Note that this also impacts operations that update the run's `meta.yaml` file, such as `end_run()`
Awesome! Thank you so much @harupy and @dbczumar! 😄 In the meantime - is there a way to retrieve the runs that were displayed in the broken experiment? I had some users run into this on an important experiment tab
Happy to help! You should be able to view runs again by identifying and removing any empty `meta.yaml` files in your `mlruns` directory. Unfortunately, runs whose `meta.yaml` files are empty may be a bit harder to restore, as you would need to reconstruct the contents of `meta.yaml`; this can be done by copying the contents of a non-empty `meta.yaml` file and changing the run ID field (start and end times will be incorrect, unfortunately).

This is a very interesting class of failure that we'll make sure to address. Thank you for reporting it to us.

Resolving patch:
diff --git a/mlflow/store/tracking/file_store.py b/mlflow/store/tracking/file_store.py
--- a/mlflow/store/tracking/file_store.py
+++ b/mlflow/store/tracking/file_store.py
@@ -3,6 +3,8 @@
 import os
 import sys
 import shutil
+import time
+import tempfile
 
 import uuid
 
@@ -387,7 +389,7 @@ def _get_experiment(self, experiment_id, view_type=ViewType.ALL):
                 "Could not find experiment with ID %s" % experiment_id,
                 databricks_pb2.RESOURCE_DOES_NOT_EXIST,
             )
-        meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
+        meta = FileStore._read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
         if experiment_dir.startswith(self.trash_folder):
             meta["lifecycle_stage"] = LifecycleStage.DELETED
         else:
@@ -464,7 +466,11 @@ def rename_experiment(self, experiment_id, new_name):
                 "Cannot rename experiment in non-active lifecycle stage."
                 " Current stage: %s" % experiment.lifecycle_stage
             )
-        write_yaml(meta_dir, FileStore.META_DATA_FILE_NAME, dict(experiment), overwrite=True)
+        FileStore._overwrite_yaml(
+            root=meta_dir,
+            file_name=FileStore.META_DATA_FILE_NAME,
+            data=dict(experiment),
+        )
 
     def delete_run(self, run_id):
         run_info = self._get_run_info(run_id)
@@ -606,7 +612,7 @@ def _get_run_info(self, run_uuid):
         return run_info
 
     def _get_run_info_from_dir(self, run_dir):
-        meta = read_yaml(run_dir, FileStore.META_DATA_FILE_NAME)
+        meta = FileStore._read_yaml(run_dir, FileStore.META_DATA_FILE_NAME)
         run_info = _read_persisted_run_info_dict(meta)
         return run_info
 
@@ -923,7 +929,11 @@ def delete_tag(self, run_id, key):
     def _overwrite_run_info(self, run_info):
         run_dir = self._get_run_dir(run_info.experiment_id, run_info.run_id)
         run_info_dict = _make_persisted_run_info_dict(run_info)
-        write_yaml(run_dir, FileStore.META_DATA_FILE_NAME, run_info_dict, overwrite=True)
+        FileStore._overwrite_yaml(
+            root=run_dir,
+            file_name=FileStore.META_DATA_FILE_NAME,
+            data=run_info_dict,
+        )
 
     def log_batch(self, run_id, metrics, params, tags):
         _validate_run_id(run_id)
@@ -968,3 +978,57 @@ def record_logged_model(self, run_id, mlflow_model):
             self._set_run_tag(run_info, tag)
         except Exception as e:
             raise MlflowException(e, INTERNAL_ERROR)
+
+    @staticmethod
+    def _overwrite_yaml(root, file_name, data):
+        """
+        Safely overwrites a preexisting yaml file, ensuring that file contents are not deleted or
+        corrupted if the write fails. This is achieved by writing contents to a temporary file
+        and moving the temporary file to replace the preexisting file, rather than opening the
+        preexisting file for a direct write.
+
+        :param root: Directory name.
+        :param file_name: File name. Expects to have '.yaml' extension.
+        :param data: The data to write, represented as a dictionary.
+        """
+        tmp_file_path = None
+        try:
+            tmp_file_fd, tmp_file_path = tempfile.mkstemp(suffix="file.yaml")
+            os.close(tmp_file_fd)
+            write_yaml(
+                root=get_parent_dir(tmp_file_path),
+                file_name=os.path.basename(tmp_file_path),
+                data=data,
+                overwrite=True,
+                sort_keys=True,
+            )
+            shutil.move(
+                tmp_file_path,
+                os.path.join(root, file_name),
+            )
+        finally:
+            if tmp_file_path is not None and os.path.exists(tmp_file_path):
+                os.remove(tmp_file_path)
+
+    @staticmethod
+    def _read_yaml(root, file_name, retries=2):
+        """
+        Read data from yaml file and return as dictionary, retrying up to
+        a specified number of times if the file contents are unexpectedly
+        empty due to a concurrent write.
+
+        :param root: Directory name.
+        :param file_name: File name. Expects to have '.yaml' extension.
+        :param retries: The number of times to retry for unexpected empty content.
+        :return: Data in yaml file as dictionary
+        """
+
+        def _read_helper(root, file_name, attempts_remaining=2):
+            result = read_yaml(root, file_name)
+            if result is not None or attempts_remaining == 0:
+                return result
+            else:
+                time.sleep(0.1 * (3 - attempts_remaining))
+                return _read_helper(root, file_name, attempts_remaining - 1)
+
+        return _read_helper(root, file_name, attempts_remaining=retries)


Test patch:
diff --git a/tests/store/tracking/test_file_store.py b/tests/store/tracking/test_file_store.py
--- a/tests/store/tracking/test_file_store.py
+++ b/tests/store/tracking/test_file_store.py
@@ -331,6 +331,29 @@ def test_get_experiment_int_experiment_id_backcompat(self):
         with safe_edit_yaml(root_dir, "meta.yaml", self._experiment_id_edit_func):
             self._verify_experiment(fs, exp_id)
 
+    def test_get_experiment_retries_for_transient_empty_yaml_read(self):
+        fs = FileStore(self.test_root)
+        exp_name = random_str()
+        exp_id = fs.create_experiment(exp_name)
+
+        mock_empty_call_count = 0
+
+        def mock_read_yaml_impl(*args, **kwargs):
+            nonlocal mock_empty_call_count
+            if mock_empty_call_count < 2:
+                mock_empty_call_count += 1
+                return None
+            else:
+                return read_yaml(*args, **kwargs)
+
+        with mock.patch(
+            "mlflow.store.tracking.file_store.read_yaml", side_effect=mock_read_yaml_impl
+        ) as mock_read_yaml:
+            fetched_experiment = fs.get_experiment(exp_id)
+            assert fetched_experiment.experiment_id == exp_id
+            assert fetched_experiment.name == exp_name
+            assert mock_read_yaml.call_count == 3
+
     def test_get_experiment_by_name(self):
         fs = FileStore(self.test_root)
         for exp_id in self.experiments:
@@ -643,6 +666,28 @@ def test_get_run(self):
             for run_id in runs:
                 self._verify_run(fs, run_id)
 
+    def test_get_run_retries_for_transient_empty_yaml_read(self):
+        fs = FileStore(self.test_root)
+        run = self._create_run(fs)
+
+        mock_empty_call_count = 0
+
+        def mock_read_yaml_impl(*args, **kwargs):
+            nonlocal mock_empty_call_count
+            if mock_empty_call_count < 2:
+                mock_empty_call_count += 1
+                return None
+            else:
+                return read_yaml(*args, **kwargs)
+
+        with mock.patch(
+            "mlflow.store.tracking.file_store.read_yaml", side_effect=mock_read_yaml_impl
+        ) as mock_read_yaml:
+            fetched_run = fs.get_run(run.info.run_id)
+            assert fetched_run.info.run_id == run.info.run_id
+            assert fetched_run.info.artifact_uri == run.info.artifact_uri
+            assert mock_read_yaml.call_count == 3
+
     def test_get_run_int_experiment_id_backcompat(self):
         fs = FileStore(self.test_root)
         exp_id = FileStore.DEFAULT_EXPERIMENT_ID