Repository: mlflow/mlflow

Issue:
[BUG] upload-artifact and prompt engineering don't work with proxied artifacts
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.7.1
- Tracking server: 2.7.1


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS Ventura 13.5.1
- **Python version**: 3.8



### Describe the problem

Prompt engineering doesn't work if experiment uses proxied artifacts (artifact uri: `mlflow-artifacts:/...`).
Run gets created, and the `eval_results_table` file is stored in the storage, but at a location that is not reachable using the `get-artifact` endpoint.
The result is a run in the table but without the prompts themselves:
![image](https://github.com/mlflow/mlflow/assets/28020880/e4b1a170-00f6-48c8-b96d-c493c79f2058)
While the requests to `http://localhost:5000/get-artifact?path=eval_results_table.json&run_uuid=918120b8e6c349eca4eac855d5fc777b` just fail.

According to https://github.com/mlflow/mlflow/blob/7095fb483fd9215c90b796f5a9d3e4001989584f/mlflow/utils/promptlab_utils.py#L134
and to https://github.com/mlflow/mlflow/blob/7095fb483fd9215c90b796f5a9d3e4001989584f/mlflow/server/handlers.py#L1275C18-L1275C18
The artifact location for `eval_results_table.json` is `<artifact-root>/<experiment-id>/<runid>/artifacts/eval_results_table.json`, which disregards the artifact location set in my experiment (`mlflow-artifacts:/<experiment-artifacts-root>`), and assumes that the experiment artifact location is the conventional auto generated location.

Then `get-artifact` uses the actual expected location stored in the run artifact_uri as prefix, and cannot find the file. 

### Tracking information

```shell
mlflow server -h 0.0.0.0 \
    --backend-store-uri <db_uri> \
    --default-artifact-root ./artifacts \
    --serve-artifacts \
    --artifacts-destination  gs://<bucket>
```

### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/gateway`: AI Gateway service, Gateway client APIs, third-party Gateway integrations
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@simonlsk Thanks for reporting this issue! Can you provide the commands you used?
No commands required, reproduces with UI only.
Server runs with the following envs:
```
_MLFLOW_SERVER_ARTIFACT_DESTINATION=./artifacts
_MLFLOW_SERVER_ARTIFACT_ROOT=mlflow-artifacts:/
_MLFLOW_SERVER_SERVE_ARTIFACTS=true
```
Then I create an experiment with the following artifact location: `mlflow-artifacts:/<some-prefix>`.
Last step is create a new prompt engineering run.

How did you run the tracking server?

```
mlflow server ???
```

I want to know the `???` part.
I'm debugging using `flask run` having `FLASK_APP=mlflow.server:app`, and the said env variables or running with:
```
mlflow server -h 0.0.0.0 \
    --backend-store-uri <db_uri> \
    --default-artifact-root ./artifacts \
    --serve-artifacts \
    --artifacts-destination  gs://<bucket>
```
## Rough design
* Calculate the relative location of the artifact from the artifact-root
* Log artifact similarly to the `_upload_artifact` [handler](https://github.com/mlflow/mlflow/blob/2801858f65203a7f3155e412ca3fc2e0bff92803/mlflow/server/handlers.py#L1916)

@harupy WDYT?
Maybe @prithvikannan can shed some light on what I might be missing. 
@simonlsk thanks for reporting this. i was able to repro with the following:

1. run the python server with the following env vars and `mlflow server --serve-artifacts`
```
_MLFLOW_SERVER_ARTIFACT_DESTINATION=./artifacts
_MLFLOW_SERVER_ARTIFACT_ROOT=mlflow-artifacts:/
_MLFLOW_SERVER_SERVE_ARTIFACTS=true
```

2. create an experiment named "test1" with artifact location "mlflow-artifacts:/test1-artifacts"
3. create a prompt eng run and observe the issue
```
2023/09/28 10:17:33 ERROR mlflow.server: Exception on /get-artifact [GET]
Traceback (most recent call last):
  ...
  File "/Users/prithvi.kannan/mlflow/mlflow/server/handlers.py", line 580, in get_artifact_handler
    return _send_artifact(artifact_repo, artifact_path)
  File "/Users/prithvi.kannan/mlflow/mlflow/server/handlers.py", line 478, in _send_artifact
    file_path = os.path.abspath(artifact_repository.download_artifacts(path))
  File "/Users/prithvi.kannan/mlflow/mlflow/store/artifact/local_artifact_repo.py", line 81, in download_artifacts
    raise OSError(f"No such file or directory: '{local_artifact_path}'")
OSError: No such file or directory: './mlartifacts/test1-artifacts/493606ce44a94c48b0d73e4554a9412c/artifacts/eval_results_table.json'
```

the solution you proposed makes sense. we can use something like `_get_proxied_run_artifact_destination_path` https://github.com/mlflow/mlflow/blob/master/mlflow/server/handlers.py#L572-L575
@prithvikannan Thanks for the repro!

<!-- assign-maintainer -->
@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

Resolving patch:
diff --git a/mlflow/utils/promptlab_utils.py b/mlflow/utils/promptlab_utils.py
--- a/mlflow/utils/promptlab_utils.py
+++ b/mlflow/utils/promptlab_utils.py
@@ -101,6 +101,7 @@ def _create_promptlab_run_impl(
         from mlflow._promptlab import save_model
         from mlflow.server.handlers import (
             _get_artifact_repo_mlflow_artifacts,
+            _get_proxied_run_artifact_destination_path,
             _is_servable_proxied_run_artifact_root,
         )
 
@@ -129,12 +130,10 @@ def _create_promptlab_run_impl(
 
             if _is_servable_proxied_run_artifact_root(run.info.artifact_uri):
                 artifact_repo = _get_artifact_repo_mlflow_artifacts()
-                artifact_repo.log_artifacts(
-                    local_dir,
-                    artifact_path=os.path.join(
-                        run.info.experiment_id, run.info.run_id, "artifacts"
-                    ),
+                artifact_path = _get_proxied_run_artifact_destination_path(
+                    proxied_artifact_root=run.info.artifact_uri,
                 )
+                artifact_repo.log_artifacts(local_dir, artifact_path=artifact_path)
             else:
                 artifact_repo = get_artifact_repository(artifact_dir)
                 artifact_repo.log_artifacts(local_dir)


Test patch: