Repository: mlflow/mlflow

Issue:
[BUG] extra ghost runs due to failed  connection retries
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.2.2
- Tracking server: ngrok


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Unbuntu 20
- **Python version**: 3.7
- **yarn version, if running the dev UI**:


### Describe the problem

When tracking experiments with MLFlow and ngrok as the web interface, I often get _extra ghost runs_ related to failed url retries that _remain unfinished_. By "extra" I mean that all the jobs get done, but the process spawns an incomplete run too.
Can't provide a minimal working example, but believe this may be known to experienced developers. How to avoid this situation, as in monitoring I ideally want a clean situation? 

### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```shell
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=5, read=5, redirect=5, status=5)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2633)'))': /api/2.0/mlflow/runs/create
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=5, read=4, redirect=5, status=5)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /api/2.0/mlflow/runs/create
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=5, read=5, redirect=5, status=5)) after connection broken by 'SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)'))': /api/2.0/mlflow/runs/create
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
# Run mlflow server
get_ipython().system_raw("mlflow ui --port 5000 &")

# Terminate open tunnels if exist
ngrok.kill()

# Setting the authtoken (optional)
# Get your authtoken from https://dashboard.ngrok.com/auth
NGROK_AUTH_TOKEN = getpass('Enter the ngrok authtoken: ')  
ngrok.set_auth_token(NGROK_AUTH_TOKEN)

# Open an HTTPS tunnel on port 5000 for http://localhost:5000
ngrok_tunnel = ngrok.connect(addr="5000", proto="http", bind_tls=True)

# visit the tracking address to enable it
mlflow.set_tracking_uri(ngrok_tunnel.public_url)
logger.info(f"MLflow Tracking UI:{ngrok_tunnel.public_url}")

# plus standard mlflow logging code
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
appears as a warning, no stack trace
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [X] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@maciejskorski Are you saying that the code like beflow results in creating multiple unfinished runs due to connection retries?

```python
with mlflow.start_run():
    ...
```

> @maciejskorski Are you saying that the code like beflow results in creating multiple unfinished runs due to connection retries?
> 
> ```python
> with mlflow.start_run():
>     ...
> ```

Yup that's what I am saying. 

The usage is pretty standard as you said, except that I am tunnelling and multiprocessing. 
```
experiment_id = mlflow.create_experiment(experiment_name)

def do_job(cfg):
    with mlflow.start_run(experiment_id=experiment_id):
      mlflow.log_param('file',cfg.inputfile)
      # ... do stuff

with mp.Pool(4) as p:
  p.map(do_job, all_cfgs)
  
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=5, read=5, redirect=5, status=5)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2633)'))': /api/2.0/mlflow/runs/log-parameter
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=5, read=5, redirect=5, status=5)) after connection broken by 'SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)'))': /api/2.0/mlflow/runs/create
```
I typically end up with one ghost round more than the length of the task list.

If nothing comes to mind right away, I can try a bit harder and make a fully reproducible example in Google Colab


@maciejskorski We encountered the same issue in https://github.com/mlflow/mlflow/pull/8116 and found reusing a cached `requests.Session` was the cause. See https://stackoverflow.com/questions/3724900/python-ssl-problem-with-multiprocessing.
Relevant code:

https://github.com/mlflow/mlflow/blob/24e7c0701ca31bc0925a6647c97dffdee9ef938b/mlflow/utils/rest_utils.py#L42-L44


We currently don't provide an option to disable session caching.
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
@harupy many thanks for root-causing it! Indeed, without multiprocessing, my use-case is stable. yet, one thing that still puzzles me is that the issue seems to occur somewhat randomly
@maciejskorski 

> one thing that still puzzles me is that the issue seems to occur somewhat randomly

We observed the same. It was flaky :)

Resolving patch:
diff --git a/mlflow/utils/file_utils.py b/mlflow/utils/file_utils.py
--- a/mlflow/utils/file_utils.py
+++ b/mlflow/utils/file_utils.py
@@ -599,7 +599,7 @@ def download_chunk(request_index, chunk_size, headers, download_path, http_uri):
     range_end = range_start + chunk_size - 1
     combined_headers = {**headers, "Range": f"bytes={range_start}-{range_end}"}
     with cloud_storage_http_request(
-        "get", http_uri, stream=False, headers=combined_headers, cached_session=False
+        "get", http_uri, stream=False, headers=combined_headers
     ) as response:
         # File will have been created upstream. Use r+b to ensure chunks
         # don't overwrite the entire file.
diff --git a/mlflow/utils/rest_utils.py b/mlflow/utils/rest_utils.py
--- a/mlflow/utils/rest_utils.py
+++ b/mlflow/utils/rest_utils.py
@@ -1,5 +1,6 @@
 import base64
 import json
+import os
 
 import requests
 from requests.adapters import HTTPAdapter
@@ -40,20 +41,17 @@
 
 
 @lru_cache(maxsize=64)
-def _get_request_session(max_retries, backoff_factor, retry_codes):
-    return _get_request_session_uncached(max_retries, backoff_factor, retry_codes)
-
-
-def _get_request_session_uncached(max_retries, backoff_factor, retry_codes):
+def _cached_get_request_session(
+    max_retries,
+    backoff_factor,
+    retry_codes,
+    # To create a new Session object for each process, we use the process id as the cache key.
+    # This is to avoid sharing the same Session object across processes, which can lead to issues
+    # such as https://stackoverflow.com/q/3724900.
+    _pid,
+):
     """
-    Returns a Requests.Session object for making HTTP request.
-
-    :param max_retries: Maximum total number of retries.
-    :param backoff_factor: a time factor for exponential backoff. e.g. value 5 means the HTTP
-      request will be retried with interval 5, 10, 20... seconds. A value of 0 turns off the
-      exponential backoff.
-    :param retry_codes: a list of HTTP response error codes that qualifies for retry.
-    :return: requests.Session object.
+    This function should not be called directly. Instead, use `_get_request_session` below.
     """
     assert 0 <= max_retries < 10
     assert 0 <= backoff_factor < 120
@@ -80,8 +78,27 @@ def _get_request_session_uncached(max_retries, backoff_factor, retry_codes):
     return session
 
 
+def _get_request_session(max_retries, backoff_factor, retry_codes):
+    """
+    Returns a `Requests.Session` object for making an HTTP request.
+
+    :param max_retries: Maximum total number of retries.
+    :param backoff_factor: a time factor for exponential backoff. e.g. value 5 means the HTTP
+      request will be retried with interval 5, 10, 20... seconds. A value of 0 turns off the
+      exponential backoff.
+    :param retry_codes: a list of HTTP response error codes that qualifies for retry.
+    :return: requests.Session object.
+    """
+    return _cached_get_request_session(
+        max_retries,
+        backoff_factor,
+        retry_codes,
+        _pid=os.getpid(),
+    )
+
+
 def _get_http_response_with_retries(
-    method, url, max_retries, backoff_factor, retry_codes, cached_session=True, **kwargs
+    method, url, max_retries, backoff_factor, retry_codes, **kwargs
 ):
     """
     Performs an HTTP request using Python's `requests` module with an automatic retry policy.
@@ -93,15 +110,11 @@ def _get_http_response_with_retries(
       request will be retried with interval 5, 10, 20... seconds. A value of 0 turns off the
       exponential backoff.
     :param retry_codes: a list of HTTP response error codes that qualifies for retry.
-    :param cached_session: Whether to cache session object. False used for multiprocessing contexts.
     :param kwargs: Additional keyword arguments to pass to `requests.Session.request()`
 
     :return: requests.Response object.
     """
-    if cached_session:
-        session = _get_request_session(max_retries, backoff_factor, retry_codes)
-    else:
-        session = _get_request_session_uncached(max_retries, backoff_factor, retry_codes)
+    session = _get_request_session(max_retries, backoff_factor, retry_codes)
     return session.request(method, url, **kwargs)
 
 
@@ -311,7 +324,6 @@ def cloud_storage_http_request(
     backoff_factor=2,
     retry_codes=_TRANSIENT_FAILURE_RESPONSE_CODES,
     timeout=None,
-    cached_session=True,
     **kwargs,
 ):
     """
@@ -326,7 +338,6 @@ def cloud_storage_http_request(
     :param retry_codes: a list of HTTP response error codes that qualifies for retry.
     :param timeout: wait for timeout seconds for response from remote server for connect and
       read request. Default to None owing to long duration operation in read / write.
-    :param cached_session: Whether to cache session object. False used for multiprocessing contexts.
     :param kwargs: Additional keyword arguments to pass to `requests.Session.request()`
 
     :return requests.Response object.
@@ -340,7 +351,6 @@ def cloud_storage_http_request(
         backoff_factor,
         retry_codes,
         timeout=timeout,
-        cached_session=cached_session,
         **kwargs,
     )
 


Test patch:
diff --git a/tests/utils/test_rest_utils.py b/tests/utils/test_rest_utils.py
--- a/tests/utils/test_rest_utils.py
+++ b/tests/utils/test_rest_utils.py
@@ -4,6 +4,7 @@
 import numpy
 import pytest
 import requests
+from concurrent.futures import ProcessPoolExecutor
 
 from mlflow.environment_variables import MLFLOW_HTTP_REQUEST_TIMEOUT
 from mlflow.exceptions import MlflowException, RestException
@@ -16,6 +17,7 @@
     call_endpoints,
     augmented_raise_for_status,
     _can_parse_as_json_object,
+    _get_request_session,
 )
 from mlflow.tracking.request_header.default_request_header_provider import (
     DefaultRequestHeaderProvider,
@@ -523,3 +525,18 @@ def test_augmented_raise_for_status():
     assert e.value.response == response
     assert e.value.request == response.request
     assert response.text in str(e.value)
+
+
+def call_get_request_session():
+    return _get_request_session(max_retries=0, backoff_factor=0, retry_codes=(403,))
+
+
+def test_get_request_session():
+    sess = call_get_request_session()
+    another_sess = call_get_request_session()
+    assert sess is another_sess
+
+    with ProcessPoolExecutor(max_workers=2) as e:
+        futures = [e.submit(call_get_request_session) for _ in range(2)]
+    sess, another_sess = [f.result() for f in futures]
+    assert sess is not another_sess