Repository: mlflow/mlflow

Issue:
[FR] Customizable timeout and retry for rest_store api calls.
Thank you for submitting a feature request. **Before proceeding, please review MLflow's [Issue Policy for feature requests](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md#feature-requests) and the [MLflow Contributing Guide](https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst)**.

**Please fill in this feature request template to ensure a timely and thorough response.**

## Willingness to contribute
The MLflow Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature (either as an MLflow Plugin or an enhancement to the MLflow code base)?

- [X] Yes. I can contribute this feature independently.
- [ ] Yes. I would be willing to contribute this feature with guidance from the MLflow community.
- [ ] No. I cannot contribute this feature at this time.

## Proposal Summary
Customize defaults for `http_request`.
 
## Motivation
- What is the use case for this feature? Allow users to customize timeout
- Why is this use case valuable to support for MLflow users in general? Currently the default timeout is 2 minutes with 5 retries and a backoff of 2x which goes over an hour before it fails. 
- Why is it currently difficult to achieve this use case? (please be as specific as possible about why related MLflow features and components are insufficient) There is no way to customize the timeout using the Pytho nAPI

### What component(s), interfaces, languages, and integrations does this feature affect?
Components
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interfaces
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Languages
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

## Details
https://github.com/mlflow/mlflow/blob/master/mlflow/utils/rest_utils.py#L94
```python
def http_request(
    host_creds,
    endpoint,
    method,
    max_retries=5,
    backoff_factor=2,
    retry_codes=_TRANSIENT_FAILURE_RESPONSE_CODES,
    timeout=120,
    **kwargs,
):
```

could change to 
```python

_MAX_RETRIES=5
_DEFAULT_TIMEOUT=120
_BACKOFF_FACTOR=2
def http_request(
    host_creds,
    endpoint,
    method,
    max_retries=_MAX_RETRIES,
    backoff_factor=_BACKOFF_FACTOR,
    retry_codes=_TRANSIENT_FAILURE_RESPONSE_CODES,
    timeout=_DEFAULT_TIMEOUT,
    **kwargs,
):
```


Discussion:

Resolving patch:
diff --git a/mlflow/environment_variables.py b/mlflow/environment_variables.py
new file mode 100644
--- /dev/null
+++ b/mlflow/environment_variables.py
@@ -0,0 +1,49 @@
+"""
+This module defines environment variables used in MLflow.
+"""
+import os
+
+
+class _EnvironmentVariable:
+    """
+    Represents an environment variable.
+    """
+
+    def __init__(self, name, type_, default):
+        self.name = name
+        self.type = type_
+        self.default = default
+
+    def get(self):
+        """
+        Reads the value of the environment variable if it exists and converts it to the desired
+        type. Otherwise, returns the default value.
+        """
+        val = os.getenv(self.name)
+        if val:
+            try:
+                return self.type(val)
+            except Exception as e:
+                raise ValueError(f"Failed to convert {val} to {self.type} for {self.name}: {e}")
+        return self.default
+
+    def __str__(self):
+        return f"EnvironmentVariable(name={self.name}, type={self.type}, default={self.default})"
+
+    def __repr__(self):
+        return repr(self.name)
+
+
+#: Specify the maximum number of retries for MLflow http request
+#: (default: ``5``)
+MLFLOW_HTTP_REQUEST_MAX_RETRIES = _EnvironmentVariable("MLFLOW_HTTP_REQUEST_MAX_RETRIES", int, 5)
+
+#: Specify backoff factor for MLflow http request
+#: (default: ``2``)
+MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR = _EnvironmentVariable(
+    "MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR", int, 2
+)
+
+#: Specify timeout in seconds for MLflow http request
+#: (default: ``120``)
+MLFLOW_HTTP_REQUEST_TIMEOUT = _EnvironmentVariable("MLFLOW_HTTP_REQUEST_TIMEOUT", int, 120)
diff --git a/mlflow/utils/rest_utils.py b/mlflow/utils/rest_utils.py
--- a/mlflow/utils/rest_utils.py
+++ b/mlflow/utils/rest_utils.py
@@ -1,5 +1,6 @@
 import base64
 import json
+
 import requests
 import urllib3
 from contextlib import contextmanager
@@ -9,13 +10,18 @@
 from urllib3.util import Retry
 from requests.exceptions import HTTPError
 
-from mlflow import __version__
 from mlflow.protos import databricks_pb2
 from mlflow.protos.databricks_pb2 import INVALID_PARAMETER_VALUE, ENDPOINT_NOT_FOUND, ErrorCode
 from mlflow.utils.proto_json_utils import parse_dict
 from mlflow.utils.string_utils import strip_suffix
 from mlflow.exceptions import get_error_code, MlflowException, RestException
 
+from mlflow.environment_variables import (
+    MLFLOW_HTTP_REQUEST_TIMEOUT,
+    MLFLOW_HTTP_REQUEST_MAX_RETRIES,
+    MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR,
+)
+
 RESOURCE_DOES_NOT_EXIST = "RESOURCE_DOES_NOT_EXIST"
 _REST_API_PATH_PREFIX = "/api/2.0"
 # Response codes that generally indicate transient network failures and merit client retries,
@@ -95,10 +101,10 @@ def http_request(
     host_creds,
     endpoint,
     method,
-    max_retries=5,
-    backoff_factor=2,
+    max_retries=None,
+    backoff_factor=None,
     retry_codes=_TRANSIENT_FAILURE_RESPONSE_CODES,
-    timeout=120,
+    timeout=None,
     **kwargs,
 ):
     """
@@ -122,6 +128,9 @@ def http_request(
 
     :return: requests.Response object.
     """
+    max_retries = max_retries or MLFLOW_HTTP_REQUEST_MAX_RETRIES.get()
+    backoff_factor = backoff_factor or MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR.get()
+    timeout = timeout or MLFLOW_HTTP_REQUEST_TIMEOUT.get()
     hostname = host_creds.host
     auth_str = None
     if host_creds.username and host_creds.password:


Test patch:
diff --git a/tests/utils/test_rest_utils.py b/tests/utils/test_rest_utils.py
--- a/tests/utils/test_rest_utils.py
+++ b/tests/utils/test_rest_utils.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-
+import os
 from unittest import mock
 import numpy
 import pytest
@@ -397,3 +397,43 @@ def test_can_parse_as_json_object():
     assert not _can_parse_as_json_object("[0, 1, 2]")
     assert not _can_parse_as_json_object('"abc"')
     assert not _can_parse_as_json_object("123")
+
+
+def test_http_request_customize_config():
+    with mock.patch(
+        "mlflow.utils.rest_utils._get_http_response_with_retries"
+    ) as mock_get_http_response_with_retries:
+        host_only = MlflowHostCreds("http://my-host")
+        with mock.patch.dict(os.environ, {}, clear=True):
+            http_request(host_only, "/my/endpoint", "GET")
+            mock_get_http_response_with_retries.assert_called_with(
+                mock.ANY,
+                mock.ANY,
+                5,
+                2,
+                mock.ANY,
+                headers=mock.ANY,
+                verify=mock.ANY,
+                timeout=120,
+            )
+        mock_get_http_response_with_retries.reset_mock()
+        with mock.patch.dict(
+            os.environ,
+            {
+                "MLFLOW_HTTP_REQUEST_MAX_RETRIES": "8",
+                "MLFLOW_HTTP_REQUEST_BACKOFF_FACTOR": "3",
+                "MLFLOW_HTTP_REQUEST_TIMEOUT": "300",
+            },
+            clear=True,
+        ):
+            http_request(host_only, "/my/endpoint", "GET")
+            mock_get_http_response_with_retries.assert_called_with(
+                mock.ANY,
+                mock.ANY,
+                8,
+                3,
+                mock.ANY,
+                headers=mock.ANY,
+                verify=mock.ANY,
+                timeout=300,
+            )