Repository: mlflow/mlflow

Issue:
[BUG] Proxied multipart upload artifact stores artifact at incorrect S3 path
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Other

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.10.1
- Tracking server: 2.10.0


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: debian11
- **Python version**: 3.11
- **yarn version, if running the dev UI**:


### Describe the problem

### Setup:
**MLflow tracking server** 

- version `2.10.0` 
- artifacts store `AWS S3`
- running with this command in a kubernetes pod container-     
      
        mlflow server \
        --backend-store-uri postgresql://$DB_USER:"$DB_PWD"@$DB_HOST:5432/$DB_NAME \
        --artifacts-destination s3://$MLFLOW_ARTIFACTS_BUCKET_NAME \
        --serve-artifacts --host 0.0.0.0
      
**Env Variables set on the server**

- MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD: True
- MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE: 314572800
- MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE: 209715200
- MLFLOW_MULTIPART_DOWNLOAD_CHUNK_SIZE: 209715200

**MLflow client** 

- version `2.10.1` 
- running on `Mac M2 and OS version 14.3.1`

**Env Variables set on the client**
- MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD: True
- MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE: 314572800
- MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE: 209715200
- MLFLOW_MULTIPART_DOWNLOAD_CHUNK_SIZE: 209715200


### Issue
**Actual:**
When logging an artifact using multipart upload from mlflow python client, the `artifact_path` gets created at the root of the artifact store. Hence the artifact doesn't get linked to the experiment run. There is no existence of the artifact in the mlflow backend system.

**Expected:**
When logging an artifact using multipart upload from mlflow python client, the `artifact_path` gets created in the experiment run artifacts directory of the artifact store. Hence the artifact gets linked to the experiment run. 

Steps to reproduce:
1. Host the mlflow tracking server `2.10.0` in **proxied pattern** using `AWS S3` as artifact store and `postgresql` as backend store (use the command mentioned in the setup section above)
2. Use python mlflow client version `2.10.1` to -
    - create an experiment 
    - create a run
    - log artifact that is more than 300MB in size with `artifact_path` as `testdir`
3. Go to the mlflow UI and navigate to the experiment run that was created in step 2, you should **_not_** see the artifact you logged from the code
4. Login to the AWS account and navigate to the S3 bucket which is the `artifact-destination` of the tracking artifact store- you should see the artifact_path under the root directory of the artifact store instead of inside the experiment run artifacts directory

### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): aleta***.api.***.com:443
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): aleta***.api.***.com:443
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:git.cmd:Popen(['git', 'version'], cwd=/Users/***/Documents/***/***examples/***-examples, stdin=None, shell=False, universal_newlines=False)
DEBUG:git.cmd:Popen(['git', 'version'], cwd=/Users/***/Documents/***/***examples/***-examples, stdin=None, shell=False, universal_newlines=False)
DEBUG:git.cmd:Popen(['git', 'check-ignore', '/Users/***/Documents/***/***examples/***-examples'], cwd=/Users/***/Documents/***/***examples, stdin=None, shell=False, universal_newlines=False)
DEBUG:git.cmd:Popen(['git', 'cat-file', '--batch-check'], cwd=/Users/***/Documents/***/***examples, stdin=<valid stream>, shell=False, universal_newlines=False)
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/create with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow/runs/create HTTP/1.1" 200 1568
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/get with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/runs/get?run_uuid=3b40edbd40744bbf90e39edbdee4d4f7&run_id=3b40edbd40744bbf90e39edbdee4d4f7 HTTP/1.1" 200 1586
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow-artifacts/artifacts/178/3b40edbd40744bbf90e39edbdee4d4f7/artifacts/requirements.txt with method PUT from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "PUT /api/2.0/mlflow-artifacts/artifacts/178/3b40edbd40744bbf90e39edbdee4d4f7/artifacts/requirements.txt HTTP/1.1" 200 2
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/log-batch with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow/runs/log-batch HTTP/1.1" 200 2
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/log-batch with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow/runs/log-batch HTTP/1.1" 200 2
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/log-batch with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow/runs/log-batch HTTP/1.1" 200 2
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/experiments/get-by-name with method GET from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "GET /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment-rt-f14-mpu-5 HTTP/1.1" 200 771
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow-artifacts/mpu/create/testdir with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow-artifacts/mpu/create/testdir HTTP/1.1" 200 5970
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.us-west-2.amazonaws.com:443
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.us-west-2.amazonaws.com:443
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.us-west-2.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://s3.us-west-2.amazonaws.com:443 "PUT /mlflow-data-qa/testdir/test1.zip?partNumber=3&uploadId=***.7l_***-&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***us-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240222T075556Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=***-Amz-Signature=*** HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://s3.us-west-2.amazonaws.com:443 "PUT /mlflow-data-qa/testdir/test1.zip?partNumber=1&uploadId=***.7l_***-&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***us-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240222T075556Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=***-Amz-Signature=*** HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://s3.us-west-2.amazonaws.com:443 "PUT /mlflow-data-qa/testdir/test1.zip?partNumber=2&uploadId=***.7l_***-&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***us-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240222T075556Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=***-Amz-Signature=*** HTTP/1.1" 200 0
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow-artifacts/mpu/complete/testdir with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow-artifacts/mpu/complete/testdir HTTP/1.1" 200 2
DEBUG:aleta.aleta:Requesting https://aleta***.api.***.com/api/2.0/mlflow/runs/update with method POST from patched _get_http_response_with_retries
DEBUG:urllib3.connectionpool:https://aleta***.api.***.com:443 "POST /api/2.0/mlflow/runs/update HTTP/1.1" 200 426

Process finished with exit code 0

```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import mlflow


mlflow.set_tracking_uri("REPLACE_THIS_WITH_YOUR_TRACKING_URI")

os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = 'True'
os.environ['MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE'] = '314572800'
os.environ['MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE'] = '209715200'
os.environ['MLFLOW_MULTIPART_DOWNLOAD_CHUNK_SIZE'] = '209715200'

experiment_id = mlflow.create_experiment(name="test-experiment-rt-f14-mpu-5")
with mlflow.start_run(experiment_id=experiment_id) as run:
        mlflow.log_artifact(local_path="PATH_TO_YOUR_LOCAL_FILE_WITH_SIZE_GREATER_THAN_300MB", artifact_path="testdir")
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
[2024-02-20 20:56:33 +0000] [71] [INFO] Starting gunicorn 21.2.0
[2024-02-20 20:56:33 +0000] [71] [INFO] Listening at: http://0.0.0.0:5000 (71)
[2024-02-20 20:56:33 +0000] [71] [INFO] Using worker: sync
[2024-02-20 20:56:33 +0000] [72] [INFO] Booting worker with pid: 72
[2024-02-20 20:56:33 +0000] [73] [INFO] Booting worker with pid: 73
[2024-02-20 20:56:33 +0000] [74] [INFO] Booting worker with pid: 74
[2024-02-20 20:56:33 +0000] [90] [INFO] Booting worker with pid: 90
```


### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:

Resolving patch:
diff --git a/mlflow/store/artifact/http_artifact_repo.py b/mlflow/store/artifact/http_artifact_repo.py
--- a/mlflow/store/artifact/http_artifact_repo.py
+++ b/mlflow/store/artifact/http_artifact_repo.py
@@ -113,11 +113,21 @@ def delete_artifacts(self, artifact_path=None):
         resp = http_request(self._host_creds, endpoint, "DELETE", stream=True)
         augmented_raise_for_status(resp)
 
+    def _construct_mpu_uri_and_path(self, base_endpoint, artifact_path):
+        uri, path = self.artifact_uri.split("/mlflow-artifacts/artifacts", maxsplit=1)
+        path = path.strip("/")
+        endpoint = (
+            posixpath.join(base_endpoint, path, artifact_path)
+            if artifact_path
+            else posixpath.join(base_endpoint, path)
+        )
+        return uri, endpoint
+
     def create_multipart_upload(self, local_file, num_parts=1, artifact_path=None):
-        url, _ = self.artifact_uri.split("/mlflow-artifacts", maxsplit=1)
-        host_creds = get_default_host_creds(url)
-        base_endpoint = "/mlflow-artifacts/mpu/create"
-        endpoint = posixpath.join(base_endpoint, artifact_path) if artifact_path else base_endpoint
+        uri, endpoint = self._construct_mpu_uri_and_path(
+            "/mlflow-artifacts/mpu/create", artifact_path
+        )
+        host_creds = get_default_host_creds(uri)
         params = {
             "path": local_file,
             "num_parts": num_parts,
@@ -127,10 +137,10 @@ def create_multipart_upload(self, local_file, num_parts=1, artifact_path=None):
         return CreateMultipartUploadResponse.from_dict(resp.json())
 
     def complete_multipart_upload(self, local_file, upload_id, parts=None, artifact_path=None):
-        url, _ = self.artifact_uri.split("/mlflow-artifacts", maxsplit=1)
-        host_creds = get_default_host_creds(url)
-        base_endpoint = "/mlflow-artifacts/mpu/complete"
-        endpoint = posixpath.join(base_endpoint, artifact_path) if artifact_path else base_endpoint
+        uri, endpoint = self._construct_mpu_uri_and_path(
+            "/mlflow-artifacts/mpu/complete", artifact_path
+        )
+        host_creds = get_default_host_creds(uri)
         params = {
             "path": local_file,
             "upload_id": upload_id,
@@ -140,10 +150,10 @@ def complete_multipart_upload(self, local_file, upload_id, parts=None, artifact_
         augmented_raise_for_status(resp)
 
     def abort_multipart_upload(self, local_file, upload_id, artifact_path=None):
-        url, _ = self.artifact_uri.split("/mlflow-artifacts", maxsplit=1)
-        host_creds = get_default_host_creds(url)
-        base_endpoint = "/mlflow-artifacts/mpu/abort"
-        endpoint = posixpath.join(base_endpoint, artifact_path) if artifact_path else base_endpoint
+        uri, endpoint = self._construct_mpu_uri_and_path(
+            "/mlflow-artifacts/mpu/abort", artifact_path
+        )
+        host_creds = get_default_host_creds(uri)
         params = {
             "path": local_file,
             "upload_id": upload_id,


Test patch: