Repository: mlflow/mlflow

Issue:
[BUG] `File already exists, cannot overwrite` when logging a png file
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

- Client: 2.2.2
- Tracking server: >=2.0.0 (unclear how to find the information from Databricks)


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 12.6.1
- **Python version**: 3.8.11
- **yarn version, if running the dev UI**: N/A


### Describe the problem

Working with the [classification recipes example](https://github.com/mlflow/recipes-examples/tree/main/classification) for the new Recipes feature, in combination with a Tracking Server implemented by Databricks, training (and subsequent evaluation) steps are failing because they try to write their confusion matrix artifacts to the same path, which the Tracking Server implemented by Databricks does not allow (either at all, or with the way it's being called):

```sh
$ mlflow recipes run -p remote
...
make: *** [steps/train/outputs/run_id] Error 1
2023/03/16 13:14:26 INFO mlflow.recipes.utils.step: Opening HTML file at: '~/.mlflow/recipes/d8ec9d1c1e529a5f49b335b7581fe9f61aba4a6e885fd14412317fc2074512ca/steps/train/outputs/card.html'
Traceback (most recent call last):
  File "./env/bin/mlflow", line 8, in <module>
    sys.exit(cli())
  File "./env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "./env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "./env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "./env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "./env/lib/python3.8/site-packages/mlflow/recipes/cli.py", line 46, in run
    Recipe(profile=profile).run(step)
  File "./env/lib/python3.8/site-packages/mlflow/recipes/classification/v1/recipe.py", line 267, in run
    return super().run(step=step)
  File "./env/lib/python3.8/site-packages/mlflow/recipes/recipe.py", line 111, in run
    raise MlflowException(
mlflow.exceptions.MlflowException: Failed to run recipe 'classification'. An error was encountered while running step 'train': Traceback (most recent call last):
  File "./env/lib/python3.8/site-packages/mlflow/recipes/step.py", line 139, in run
    self.step_card = self._run(output_directory=output_directory)
  File "./env/lib/python3.8/site-packages/mlflow/recipes/steps/train.py", line 489, in _run
    eval_result = mlflow.evaluate(
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 1346, in evaluate
    evaluate_result = _evaluate(
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 934, in _evaluate
    eval_result = evaluator.evaluate(
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1261, in evaluate
    evaluation_result = self._evaluate(model, is_baseline_model=False)
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1219, in _evaluate
    self._log_metrics_and_artifacts()
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1168, in _log_metrics_and_artifacts
    self._log_confusion_matrix()
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1079, in _log_confusion_matrix
    self._log_image_artifact(
  File "./env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 588, in _log_image_artifact
    mlflow.log_artifact(artifact_file_local_path)
  File "./env/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 783, in log_artifact
    MlflowClient().log_artifact(run_id, local_path, artifact_path)
  File "./env/lib/python3.8/site-packages/mlflow/tracking/client.py", line 1023, in log_artifact
    self._tracking_client.log_artifact(run_id, local_path, artifact_path)
  File "./env/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 439, in log_artifact
    artifact_repo.log_artifact(local_path, artifact_path)
  File "./env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 120, in log_artifact
    self._databricks_api_request(
  File "./env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 62, in _databricks_api_request
    return http_request_safe(host_creds=host_creds, endpoint=endpoint, method=method, **kwargs)
  File "./env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 200, in http_request_safe
    return verify_rest_response(response, endpoint)
  File "./env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 213, in verify_rest_response
    raise MlflowException(
mlflow.exceptions.MlflowException: API request to endpoint /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/2e0d0322d0874a85918e16fd0e957c64/artifacts/confusion_matrix.png failed with error code 409 != 200. Response body: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 409 </title>
</head>
<body>
<h2>HTTP ERROR: 409</h2>
<p>Problem accessing /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/2e0d0322d0874a85918e16fd0e957c64/artifacts/confusion_matrix.png. Reason:
<pre>    File already exists, cannot overwrite: &apos;/user/[redacted]/experiments/sklearn_classification_experiment/2e0d0322d0874a85918e16fd0e957c64/artifacts/confusion_matrix.png&apos;</pre></p>
<hr />
</body>
</html>
'
```

`./profiles/remote.yaml`:

```yaml
experiment:
  tracking_uri: "https://[redacted].cloud.databricks.com/"
...
```

Commenting out the upload step gets the Recipe to run, and it's evident from the logs that the same `mlflow.models.evaluation.default_evaluator` code is used three times, twice in training and once in evaluation:

```python
def _log_image_artifact(
    self,
    do_plot,
    artifact_name,
):
    from matplotlib import pyplot

    artifact_file_name = f"{artifact_name}.png"
    artifact_file_local_path = self.temp_dir.path(artifact_file_name)

    try:
        pyplot.clf()
        do_plot()
        pyplot.savefig(artifact_file_local_path, bbox_inches="tight")
    finally:
        pyplot.close(pyplot.gcf())

    #mlflow.log_artifact(artifact_file_local_path)
    artifact = ImageEvaluationArtifact(uri=mlflow.get_artifact_uri(artifact_file_name))
    artifact._load(artifact_file_local_path)
    self.artifacts[artifact_name] = artifact
```

```
Run MLFlow Recipe step: train
2023/03/16 13:02:53 INFO mlflow.recipes.step: Running step train...
2023/03/16 13:02:54 INFO mlflow.recipes.steps.train: Training data has less than 5000 rows, skipping rebalancing.
2023/03/16 13:03:10 WARNING mlflow.sklearn: Model was missing function: predict. Not logging python_function flavor!
2023/03/16 13:03:36 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2023/03/16 13:03:36 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as binary dataset, positive label is 1, negative label is 0.
2023/03/16 13:03:40 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2023/03/16 13:03:40 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as binary dataset, positive label is 1, negative label is 0.
Run MLFlow Recipe step: evaluate
2023/03/16 13:03:47 INFO mlflow.recipes.step: Running step evaluate...
2023/03/16 13:03:53 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2023/03/16 13:03:53 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as binary dataset, positive label is 1, negative label is 0.
2023/03/16 13:03:55 INFO mlflow.models.evaluation.default_evaluator: Shap explainer _PatchedKernelExplainer is used.
```

 Either the client that implements the interaction with the Tracking Server should be configured to force the overwrite (if that's possible, and the desired behavior), or the output locations should be properly namespaced by step and/or sub-step, to avoid collision.

### Tracking information

```
System information: Darwin Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64
Python version: 3.8.11
MLflow version: 2.2.2
MLflow module location: ./env/lib/python3.8/site-packages/mlflow/__init__.py
Tracking URI: https://[redacted].cloud.databricks.com/
Registry URI: https://[redacted].cloud.databricks.com/
MLflow environment variables: 
  MLFLOW_EXPERIMENT_NAME: /Users/[redacted]/sklearn_classification_experiment
  MLFLOW_TRACKING_TOKEN: [redacted]
  MLFLOW_TRACKING_URI: https://[redacted].cloud.databricks.com/
MLflow dependencies: 
  Flask: 2.2.3
  Jinja2: 3.1.2
  alembic: 1.10.2
  click: 8.1.3
  cloudpickle: 2.2.1
  databricks-cli: 0.17.5
  docker: 5.0.3
  entrypoints: 0.4
  gitpython: 3.1.31
  gunicorn: 20.1.0
  importlib-metadata: 6.0.0
  markdown: 3.4.1
  matplotlib: 3.7.1
  numpy: 1.23.5
  packaging: 23.0
  pandas: 1.5.3
  protobuf: 3.20.3
  pyarrow: 11.0.0
  pytz: 2022.7.1
  pyyaml: 6.0
  querystring-parser: 1.2.4
  requests: 2.28.2
  scikit-learn: 1.2.2
  scipy: 1.10.1
  shap: 0.41.0
  sqlalchemy: 2.0.5.post1
  sqlparse: 0.4.3
```

### Code to reproduce issue

```sh
git clone https://github.com/mlflow/recipes-examples.git

cd recipes-examples/
python -m venv env
source env/bin/activate
pip install -r requirements.txt

cd classification/
export MLFLOW_TRACKING_TOKEN=[redacted]
# assumes the experiment has already been created via UI / CLI / API
export MLFLOW_EXPERIMENT_NAME=/Users/[redacted]/sklearn_classification_experiment
git apply profiles-local.diff # attached with redactions

mlflow recipes run -p local
```

```diff
diff --git a/classification/profiles/local.yaml b/classification/profiles/local.yaml
index 58c057f..5bdfb60 100644
--- a/classification/profiles/local.yaml
+++ b/classification/profiles/local.yaml
@@ -1,7 +1,5 @@
 experiment:
-  name: "sklearn_classification_experiment"
-  tracking_uri: "sqlite:///metadata/mlflow/mlruns.db"
-  artifact_location: "./metadata/mlflow/mlartifacts"
+  tracking_uri: https://[redacted].cloud.databricks.com/
 
 # Set the registry server URI. This property is especially useful if you have a registry
 # server that’s different from the tracking server.
```

### Stack trace

```
2023/03/16 13:52:10 INFO mlflow.recipes.recipe: Creating MLflow Recipe 'classification' with profile: 'local'
2023/03/16 13:52:18 INFO mlflow.recipes.utils.execution: ingest, split: No changes. Skipping.
Run MLFlow Recipe step: transform
2023/03/16 13:52:21 INFO mlflow.recipes.step: Running step transform...
Run MLFlow Recipe step: train
2023/03/16 13:52:25 INFO mlflow.recipes.step: Running step train...
2023/03/16 13:52:27 INFO mlflow.recipes.steps.train: Detected class imbalance: minority class percentage is 0.25
2023/03/16 13:52:27 INFO mlflow.recipes.steps.train: After downsampling: minority class percentage is 0.30
2023/03/16 13:52:50 WARNING mlflow.sklearn: Model was missing function: predict. Not logging python_function flavor!
2023/03/16 13:53:17 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2023/03/16 13:53:17 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as binary dataset, positive label is 1, negative label is 0.
2023/03/16 13:53:20 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.
2023/03/16 13:53:20 INFO mlflow.models.evaluation.default_evaluator: The evaluation dataset is inferred as binary dataset, positive label is 1, negative label is 0.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/step.py", line 139, in run
    self.step_card = self._run(output_directory=output_directory)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/steps/train.py", line 489, in _run
    eval_result = mlflow.evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 1346, in evaluate
    evaluate_result = _evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 934, in _evaluate
    eval_result = evaluator.evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1261, in evaluate
    evaluation_result = self._evaluate(model, is_baseline_model=False)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1219, in _evaluate
    self._log_metrics_and_artifacts()
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1168, in _log_metrics_and_artifacts
    self._log_confusion_matrix()
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1079, in _log_confusion_matrix
    self._log_image_artifact(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 588, in _log_image_artifact
    mlflow.log_artifact(artifact_file_local_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 783, in log_artifact
    MlflowClient().log_artifact(run_id, local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/client.py", line 1023, in log_artifact
    self._tracking_client.log_artifact(run_id, local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 439, in log_artifact
    artifact_repo.log_artifact(local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 120, in log_artifact
    self._databricks_api_request(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 62, in _databricks_api_request
    return http_request_safe(host_creds=host_creds, endpoint=endpoint, method=method, **kwargs)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 200, in http_request_safe
    return verify_rest_response(response, endpoint)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 213, in verify_rest_response
    raise MlflowException(
mlflow.exceptions.MlflowException: API request to endpoint /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png failed with error code 409 != 200. Response body: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 409 </title>
</head>
<body>
<h2>HTTP ERROR: 409</h2>
<p>Problem accessing /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png. Reason:
<pre>    File already exists, cannot overwrite: &apos;/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png&apos;</pre></p>
<hr />
</body>
</html>
'
make: *** [steps/train/outputs/run_id] Error 1
2023/03/16 13:53:24 INFO mlflow.recipes.utils.step: Opening HTML file at: '/Users/[redacted]/.mlflow/recipes/f59fedfb48fa2bd9aed7f8e32118d8b19cce8c84182eb295d4eeb24f8dd36548/steps/train/outputs/card.html'
Traceback (most recent call last):
  File "/Users/[redacted]/repro/recipes-examples/env/bin/mlflow", line 8, in <module>
    sys.exit(cli())
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/cli.py", line 46, in run
    Recipe(profile=profile).run(step)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/classification/v1/recipe.py", line 267, in run
    return super().run(step=step)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/recipe.py", line 111, in run
    raise MlflowException(
mlflow.exceptions.MlflowException: Failed to run recipe 'classification'. An error was encountered while running step 'train': Traceback (most recent call last):
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/step.py", line 139, in run
    self.step_card = self._run(output_directory=output_directory)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/recipes/steps/train.py", line 489, in _run
    eval_result = mlflow.evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 1346, in evaluate
    evaluate_result = _evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/base.py", line 934, in _evaluate
    eval_result = evaluator.evaluate(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1261, in evaluate
    evaluation_result = self._evaluate(model, is_baseline_model=False)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1219, in _evaluate
    self._log_metrics_and_artifacts()
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1168, in _log_metrics_and_artifacts
    self._log_confusion_matrix()
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 1079, in _log_confusion_matrix
    self._log_image_artifact(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/models/evaluation/default_evaluator.py", line 588, in _log_image_artifact
    mlflow.log_artifact(artifact_file_local_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 783, in log_artifact
    MlflowClient().log_artifact(run_id, local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/client.py", line 1023, in log_artifact
    self._tracking_client.log_artifact(run_id, local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 439, in log_artifact
    artifact_repo.log_artifact(local_path, artifact_path)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 120, in log_artifact
    self._databricks_api_request(
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/store/artifact/dbfs_artifact_repo.py", line 62, in _databricks_api_request
    return http_request_safe(host_creds=host_creds, endpoint=endpoint, method=method, **kwargs)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 200, in http_request_safe
    return verify_rest_response(response, endpoint)
  File "/Users/[redacted]/repro/recipes-examples/env/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 213, in verify_rest_response
    raise MlflowException(
mlflow.exceptions.MlflowException: API request to endpoint /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png failed with error code 409 != 200. Response body: '<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 409 </title>
</head>
<body>
<h2>HTTP ERROR: 409</h2>
<p>Problem accessing /dbfs/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png. Reason:
<pre>    File already exists, cannot overwrite: &apos;/user/[redacted]/experiments/sklearn_classification_experiment/b6e21658be1d4003b57fe560c1d248cd/artifacts/confusion_matrix.png&apos;</pre></p>
<hr />
</body>
</html>
'
```

### Other info / logs

Not available.

### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [X] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [X] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [X] `integrations/databricks`: Databricks integrations


Discussion:
@dmcguire81 Thanks for reportin the issue. I was able to reproduce the issue with the following code:

```python
import mlflow

# Before running this script, create an experiment with artifact_location=dbfs:/path/to/location

mlflow.set_tracking_uri("databricks")
mlflow.set_experiment("/path/to/experiment")

with mlflow.start_run():
    mlflow.log_artifact(__file__)
    mlflow.log_artifact(__file__)
```

We're investigating the cause now.
One workaround is to create a new experiment without specifying the artifact location. If it's unspecifed, it's automatically set to a path under `dbfs:/databricks/` and file overwrite should work.
I'm investigating why the confusiont matrix image is logged twice in the same run. I guess autologging logs it first and the default evaluator logs it again, then hits the error.

Resolving patch:
diff --git a/mlflow/models/evaluation/base.py b/mlflow/models/evaluation/base.py
--- a/mlflow/models/evaluation/base.py
+++ b/mlflow/models/evaluation/base.py
@@ -1041,8 +1041,8 @@ def evaluate(
           For multiclass classification tasks, the maximum number of classes for which to log
           the per-class ROC curve and Precision-Recall curve. If the number of classes is
           larger than the configured maximum, these curves are not logged.
-        - **metric_prefix**: An optional prefix to prepend to the name of each metric produced
-          during evaluation.
+        - **metric_prefix**: An optional prefix to prepend to the name of each metric and artifact
+          produced during evaluation.
         - **log_metrics_with_dataset_info**: A boolean value specifying whether or not to include
           information about the evaluation dataset in the name of each metric logged to MLflow
           Tracking during evaluation, default value is True.
diff --git a/mlflow/models/evaluation/default_evaluator.py b/mlflow/models/evaluation/default_evaluator.py
--- a/mlflow/models/evaluation/default_evaluator.py
+++ b/mlflow/models/evaluation/default_evaluator.py
@@ -575,7 +575,8 @@ def _log_image_artifact(
     ):
         from matplotlib import pyplot
 
-        artifact_file_name = f"{artifact_name}.png"
+        prefix = self.evaluator_config.get("metric_prefix", "")
+        artifact_file_name = f"{prefix}{artifact_name}.png"
         artifact_file_local_path = self.temp_dir.path(artifact_file_name)
 
         try:
diff --git a/mlflow/recipes/steps/evaluate.py b/mlflow/recipes/steps/evaluate.py
--- a/mlflow/recipes/steps/evaluate.py
+++ b/mlflow/recipes/steps/evaluate.py
@@ -38,7 +38,7 @@
 
 
 _FEATURE_IMPORTANCE_PLOT_FILE = "feature_importance.png"
-
+_VALIDATION_METRIC_PREFIX = "val_"
 
 MetricValidationResult = namedtuple(
     "MetricValidationResult", ["metric", "greater_is_better", "value", "threshold", "validated"]
@@ -199,7 +199,7 @@ def my_warn(*args, **kwargs):
                         {
                             "explainability_algorithm": "kernel",
                             "explainability_nsamples": 10,
-                            "metric_prefix": "val_",
+                            "metric_prefix": _VALIDATION_METRIC_PREFIX,
                         },
                     ),
                     (
@@ -343,7 +343,7 @@ def get_icon(validated):
             confusion_matrix_path = os.path.join(
                 output_directory,
                 "eval_validation/artifacts",
-                "confusion_matrix.png",
+                f"{_VALIDATION_METRIC_PREFIX}confusion_matrix.png",
             )
             if os.path.exists(confusion_matrix_path):
                 classifiers_plot_tab.add_html(
@@ -357,7 +357,7 @@ def get_icon(validated):
             lift_curve_path = os.path.join(
                 output_directory,
                 "eval_validation/artifacts",
-                "lift_curve_plot.png",
+                f"{_VALIDATION_METRIC_PREFIX}lift_curve_plot.png",
             )
             if os.path.exists(lift_curve_path):
                 classifiers_plot_tab.add_html(
@@ -369,7 +369,7 @@ def get_icon(validated):
             pr_curve_path = os.path.join(
                 output_directory,
                 "eval_validation/artifacts",
-                "precision_recall_curve_plot.png",
+                f"{_VALIDATION_METRIC_PREFIX}precision_recall_curve_plot.png",
             )
             if os.path.exists(pr_curve_path):
                 classifiers_plot_tab.add_html(
@@ -381,7 +381,7 @@ def get_icon(validated):
             roc_curve_path = os.path.join(
                 output_directory,
                 "eval_validation/artifacts",
-                "roc_curve_plot.png",
+                f"{_VALIDATION_METRIC_PREFIX}roc_curve_plot.png",
             )
             if os.path.exists(roc_curve_path):
                 classifiers_plot_tab.add_html(
@@ -399,12 +399,14 @@ def get_icon(validated):
         )
 
         shap_bar_plot_path = os.path.join(
-            output_directory, "eval_validation/artifacts", "shap_feature_importance_plot.png"
+            output_directory,
+            "eval_validation/artifacts",
+            f"{_VALIDATION_METRIC_PREFIX}shap_feature_importance_plot.png",
         )
         shap_beeswarm_plot_path = os.path.join(
             output_directory,
             "eval_validation/artifacts",
-            "shap_beeswarm_plot.png",
+            f"{_VALIDATION_METRIC_PREFIX}shap_beeswarm_plot.png",
         )
         shap_plot_tab.add_image("SHAP_BAR_PLOT", shap_bar_plot_path, width=800)
         shap_plot_tab.add_image("SHAP_BEESWARM_PLOT", shap_beeswarm_plot_path, width=800)


Test patch: