Repository: mlflow/mlflow

Issue:
[FR] AzureBlobArtifactRepository to support MLFLOW_TRACKING_SERVER_CERT_PATH env var
### Willingness to contribute

Yes. I can contribute this feature independently.

### Proposal Summary

Providing user control over CA certs used for connection to AzureBlobArtifactRepository would allow use of WASBS storage with self-signed certificates, as can be case using Azurite container with `--cert`/`--key` args. Since an environment variable (MLFLOW_TRACKING_SERVER_CERT_PATH) has already been defined for this purpose, this Feature Request suggests passing its value down to BlobServiceClient.

### Motivation

> #### What is the use case for this feature?
Specific use case for this issue is to enable using Azurite artifact store with self-signed certificates.

> #### Why is this use case valuable to support for MLflow users in general?
Azurite is widely used in test environments in order to emulate Azure Blob Storage backend. Although some connections could be supported on http (e.g. `BlobServiceClient.from_connection_string()`), others require https (e.g. `BlobServiceClient(credential=DefaultAzureCredential())`). Providing unified https connection simplifies integration across use-cases.

> #### Why is this use case valuable to support for your project(s) or organization?
In this case, we use Azurite in local setups as well as in CI testing before deploying our mlflow server integration.

> #### Why is it currently difficult to achieve this use case?
The only available alternative to add custom CA to verify Azurite certs is to manually import it into python's `certifi` store. Depending on integration, it has either to be done at buildtime (e.g. Dockerfile) or by modifying runtime execution to do so, both requiring deeper changes and potential divergence from production setup.


### Details

Requested change is very similar to `HttpArtifactRepository` feature where this environment variable is parsed in `mlflow.tracking._tracking_service.utils._get_default_host_creds(store_uri) -> rest_utils.MlflowHostCreds` then used in `rest_utils.http_request` to then set `request(verify=ca_cert_path)`.

The simplest approach would be either to reuse this method in `AzureBlobArtifactRepository` constructor to parse the environment variable:
```python
        if "AZURE_STORAGE_CONNECTION_STRING" in os.environ:
            self.client = BlobServiceClient.from_connection_string(
                conn_str=os.environ.get("AZURE_STORAGE_CONNECTION_STRING"),
                connection_verify=_get_default_host_creds(self.artifact_uri).server_cert_path,
            )
```
either to parse directly the environment variable:
```python
        if "AZURE_STORAGE_CONNECTION_STRING" in os.environ:
            self.client = BlobServiceClient.from_connection_string(
                conn_str=os.environ.get("AZURE_STORAGE_CONNECTION_STRING"),
                connection_verify=os.environ.get("MLFLOW_TRACKING_SERVER_CERT_PATH"),
            )
```
Same goes for other two instanciations of BlobServiceClient.

### What component(s) does this bug affect?

- [X] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [X] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
Sounds like a good idea @martlaf . 
So the plan is to use the env config value if set (for use in testing), else, use the default host creds value? 
Have you validated what the behavior is if this cert key is present when acquiring auth access to Azure Blob Store? Does it ignore it?
Either way, we'll be looking forward to your PR!
In both snippets in description, if the environment variable is not set, `MlflowHostCreds.server_cert_path` is set to None. In turn this would have set request's `verify` to None (behaves like False); this rather would definitely need to default to True to preserve behavior. The idea is that if variable is unset, it behaves exactly the same (will need to hardcode `requests`'s `verify`'s default value to pass explicitely, could be done by simply changing `MlflowHostCreds.server_cert_path`'s default); if it is set, we enforce validation against that cert instead.

If trying to validate Azure Blob Store with local CA cert (or invalid path) set in `MLFLOW_TRACKING_SERVER_CERT_PATH` environment variable, it will prevent connection (`CERTIFICATE_VERIFY_FAILED`). This will induce a risk that it will break setups already defining this to invalid CA certificate, but as such I'm not sure how much undesirable it is in itself.
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.

Resolving patch:
diff --git a/mlflow/store/artifact/azure_blob_artifact_repo.py b/mlflow/store/artifact/azure_blob_artifact_repo.py
--- a/mlflow/store/artifact/azure_blob_artifact_repo.py
+++ b/mlflow/store/artifact/azure_blob_artifact_repo.py
@@ -6,6 +6,7 @@
 from mlflow.entities import FileInfo
 from mlflow.exceptions import MlflowException
 from mlflow.store.artifact.artifact_repo import ArtifactRepository
+from mlflow.tracking._tracking_service.utils import _get_default_host_creds
 
 from mlflow.environment_variables import MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT
 
@@ -38,12 +39,15 @@ def __init__(self, artifact_uri, client=None):
         (_, account, _, api_uri_suffix) = AzureBlobArtifactRepository.parse_wasbs_uri(artifact_uri)
         if "AZURE_STORAGE_CONNECTION_STRING" in os.environ:
             self.client = BlobServiceClient.from_connection_string(
-                conn_str=os.environ.get("AZURE_STORAGE_CONNECTION_STRING")
+                conn_str=os.environ.get("AZURE_STORAGE_CONNECTION_STRING"),
+                connection_verify=_get_default_host_creds(artifact_uri).verify,
             )
         elif "AZURE_STORAGE_ACCESS_KEY" in os.environ:
             account_url = f"https://{account}.{api_uri_suffix}"
             self.client = BlobServiceClient(
-                account_url=account_url, credential=os.environ.get("AZURE_STORAGE_ACCESS_KEY")
+                account_url=account_url,
+                credential=os.environ.get("AZURE_STORAGE_ACCESS_KEY"),
+                connection_verify=_get_default_host_creds(artifact_uri).verify,
             )
         else:
             try:
@@ -56,7 +60,9 @@ def __init__(self, artifact_uri, client=None):
 
             account_url = f"https://{account}.{api_uri_suffix}"
             self.client = BlobServiceClient(
-                account_url=account_url, credential=DefaultAzureCredential()
+                account_url=account_url,
+                credential=DefaultAzureCredential(),
+                connection_verify=_get_default_host_creds(artifact_uri).verify,
             )
 
     @staticmethod
diff --git a/mlflow/utils/rest_utils.py b/mlflow/utils/rest_utils.py
--- a/mlflow/utils/rest_utils.py
+++ b/mlflow/utils/rest_utils.py
@@ -147,11 +147,6 @@ def http_request(
     if auth_str:
         headers["Authorization"] = auth_str
 
-    if host_creds.server_cert_path is None:
-        verify = not host_creds.ignore_tls_verification
-    else:
-        verify = host_creds.server_cert_path
-
     if host_creds.client_cert_path is not None:
         kwargs["cert"] = host_creds.client_cert_path
 
@@ -171,7 +166,7 @@ def http_request(
             backoff_factor,
             retry_codes,
             headers=headers,
-            verify=verify,
+            verify=host_creds.verify,
             timeout=timeout,
             **kwargs,
         )
@@ -398,3 +393,10 @@ def __eq__(self, other):
         if isinstance(other, self.__class__):
             return self.__dict__ == other.__dict__
         return NotImplemented
+
+    @property
+    def verify(self):
+        if self.server_cert_path is None:
+            return not self.ignore_tls_verification
+        else:
+            return self.server_cert_path


Test patch: