Repository: mlflow/mlflow
Pull request #14839: Support mysql ssl connections with client certs
Author: login: aksylumoed
Labels: name: rn/feature
description: Mention under Features in Changelogs.
name: area/tracking
description: Tracking service, tracking client APIs, autologging
name: area/sqlalchemy
description: Use of SQL alchemy in tracking service or model registry
name: v2.20.4
Created: 2025-03-04T18:09:57Z | Merged: 2025-03-06T10:46:57Z

<details><summary>&#x1F6E0 DevTools &#x1F6E0</summary>
<p>

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/aksylumoed/mlflow/pull/14839?quickstart=1)

#### Install mlflow from this PR

```
# Use `%sh` to run this command on Databricks
OPTIONS=$(if pip freeze | grep -q 'mlflow @ git+https://github.com/mlflow/mlflow.git'; then echo '--force-reinstall --no-deps'; fi)
pip install $OPTIONS git+https://github.com/mlflow/mlflow.git@refs/pull/14839/merge
```

#### Checkout with GitHub CLI

```
gh pr checkout 14839
```

</p>
</details>

Signed-off-by: aksylumoed <adityadndkr@gmail.com>

### Related Issues/PRs
https://github.com/mlflow/mlflow/issues/14812

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #xxx

### What changes are proposed in this pull request?

Added three env vars that can be used to create an sqlalchemy engine via the `connect_args` param:
- MLFLOW_MYSQL_SSL_CA
- MLFLOW_MYSQL_SSL_CERT 
- MLFLOW_MYSQL_SSL_KEY

<!-- Please fill in changes proposed in this PR. -->

### How is this PR tested?

- [X] Existing unit/integration tests
- [X] New unit/integration tests
- [ ] Manual tests

<!-- Attach code, screenshot, video used for manual testing here. -->

### Does this PR require documentation update?

- [ ] No. You can skip the rest of this section.
- [X] Yes. I've updated:
  - [ ] Examples
  - [ ] API references
  - [X] Instructions

### Release Notes

#### Is this a user-facing change?

- [X] No. You can skip the rest of this section.
- [ ] Yes. Give a description of this change to be included in the release notes for MLflow users.

<!-- Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change. -->

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [X] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [X] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes

#### Should this PR be included in the next patch release?

`Yes` should be selected for bug fixes, documentation updates, and other small changes. `No` should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

<details>
<summary>What is a minor/patch release?</summary>

- Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
  Bug fixes, doc updates and new features usually go into minor releases.
- Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
  Bug fixes and doc updates usually go into patch releases.

</details>

<!-- patch -->

- [x] Yes (this PR will be cherry-picked and included in the next patch release)
- [ ] No (this PR will be included in the next minor release)



Diff:
diff --git a/docs/docs/tracking/backend-stores/index.mdx b/docs/docs/tracking/backend-stores/index.mdx
index f3e1d643095d2..5116d246bee00 100644
--- a/docs/docs/tracking/backend-stores/index.mdx
+++ b/docs/docs/tracking/backend-stores/index.mdx
@@ -95,6 +95,29 @@ You can inject some [SQLAlchemy connection pooling options](https://docs.sqlalch
   </tbody>
 </table>
 
+## MySQL SSL Options
+
+When connecting to a MySQL database that requires SSL certificates, you can set the following environment variables:
+
+```bash
+# Path to SSL CA certificate file
+export MLFLOW_MYSQL_SSL_CA=/path/to/ca.pem
+
+# Path to SSL client certificate file (if needed)
+export MLFLOW_MYSQL_SSL_CERT=/path/to/client-cert.pem
+
+# Path to SSL client key file (if needed)
+export MLFLOW_MYSQL_SSL_KEY=/path/to/client-key.pem
+```
+
+Then start the MLflow server with your MySQL URI:
+
+```bash
+mlflow server --backend-store-uri="mysql+pymysql://username@hostname:port/database" --default-artifact-root=s3://your-bucket --host=0.0.0.0 --port=5000
+```
+
+These environment variables will be used to configure the SSL connection to the MySQL server.
+
 ## File Store Performance
 
 MLflow will automatically try to use [LibYAML](https://pyyaml.org/wiki/LibYAML) bindings if they are already installed.
diff --git a/mlflow/environment_variables.py b/mlflow/environment_variables.py
index 2e27b7270c878..a4ed59449c225 100644
--- a/mlflow/environment_variables.py
+++ b/mlflow/environment_variables.py
@@ -722,3 +722,18 @@ def get(self):
 MLFLOW_ARTIFACT_LOCATION_MAX_LENGTH = _EnvironmentVariable(
     "MLFLOW_ARTIFACT_LOCATION_MAX_LENGTH", int, 2048
 )
+
+#: Path to SSL CA certificate file for MySQL connections
+#: Used when creating a SQLAlchemy engine for MySQL
+#: (default: ``None``)
+MLFLOW_MYSQL_SSL_CA = _EnvironmentVariable("MLFLOW_MYSQL_SSL_CA", str, None)
+
+#: Path to SSL certificate file for MySQL connections
+#: Used when creating a SQLAlchemy engine for MySQL
+#: (default: ``None``)
+MLFLOW_MYSQL_SSL_CERT = _EnvironmentVariable("MLFLOW_MYSQL_SSL_CERT", str, None)
+
+#: Path to SSL key file for MySQL connections
+#: Used when creating a SQLAlchemy engine for MySQL
+#: (default: ``None``)
+MLFLOW_MYSQL_SSL_KEY = _EnvironmentVariable("MLFLOW_MYSQL_SSL_KEY", str, None)
diff --git a/mlflow/store/db/utils.py b/mlflow/store/db/utils.py
index f0d067cf39c20..75d11743bfd84 100644
--- a/mlflow/store/db/utils.py
+++ b/mlflow/store/db/utils.py
@@ -20,6 +20,9 @@
 )
 
 from mlflow.environment_variables import (
+    MLFLOW_MYSQL_SSL_CA,
+    MLFLOW_MYSQL_SSL_CERT,
+    MLFLOW_MYSQL_SSL_KEY,
     MLFLOW_SQLALCHEMYSTORE_ECHO,
     MLFLOW_SQLALCHEMYSTORE_MAX_OVERFLOW,
     MLFLOW_SQLALCHEMYSTORE_POOL_RECYCLE,
@@ -27,7 +30,11 @@
     MLFLOW_SQLALCHEMYSTORE_POOLCLASS,
 )
 from mlflow.exceptions import MlflowException
-from mlflow.protos.databricks_pb2 import BAD_REQUEST, INTERNAL_ERROR, TEMPORARILY_UNAVAILABLE
+from mlflow.protos.databricks_pb2 import (
+    BAD_REQUEST,
+    INTERNAL_ERROR,
+    TEMPORARILY_UNAVAILABLE,
+)
 from mlflow.store.db.db_types import SQLITE
 from mlflow.store.model_registry.dbmodels.models import (
     SqlModelVersion,
@@ -257,17 +264,17 @@ def create_sqlalchemy_engine(db_uri):
     pool_recycle = MLFLOW_SQLALCHEMYSTORE_POOL_RECYCLE.get()
     echo = MLFLOW_SQLALCHEMYSTORE_ECHO.get()
     poolclass = MLFLOW_SQLALCHEMYSTORE_POOLCLASS.get()
-    pool_kwargs = {}
+    kwargs = {}
     # Send argument only if they have been injected.
     # Some engine does not support them (for example sqllite)
     if pool_size:
-        pool_kwargs["pool_size"] = pool_size
+        kwargs["pool_size"] = pool_size
     if pool_max_overflow:
-        pool_kwargs["max_overflow"] = pool_max_overflow
+        kwargs["max_overflow"] = pool_max_overflow
     if pool_recycle:
-        pool_kwargs["pool_recycle"] = pool_recycle
+        kwargs["pool_recycle"] = pool_recycle
     if echo:
-        pool_kwargs["echo"] = echo
+        kwargs["echo"] = echo
     if poolclass:
         pool_class_map = {
             "AssertionPool": AssertionPool,
@@ -286,7 +293,22 @@ def create_sqlalchemy_engine(db_uri):
             )
             _logger.warning(err_str)
             raise ValueError(err_str)
-        pool_kwargs["poolclass"] = pool_class_map[poolclass]
-    if pool_kwargs:
-        _logger.info("Create SQLAlchemy engine with pool options %s", pool_kwargs)
-    return sqlalchemy.create_engine(db_uri, pool_pre_ping=True, **pool_kwargs)
+        kwargs["poolclass"] = pool_class_map[poolclass]
+    if kwargs:
+        _logger.info("Create SQLAlchemy engine with pool options %s", kwargs)
+
+    # Handle MySQL SSL certificates via connect_args
+    if db_uri.startswith("mysql"):
+        connect_args = {
+            k: v
+            for k, v in {
+                "ssl_ca": MLFLOW_MYSQL_SSL_CA.get(),
+                "ssl_cert": MLFLOW_MYSQL_SSL_CERT.get(),
+                "ssl_key": MLFLOW_MYSQL_SSL_KEY.get(),
+            }.items()
+            if v
+        }
+        if connect_args:
+            kwargs["connect_args"] = connect_args
+
+    return sqlalchemy.create_engine(db_uri, pool_pre_ping=True, **kwargs)
diff --git a/tests/store/db/test_utils.py b/tests/store/db/test_utils.py
index e687900b8bda0..5d217c8e4d2cc 100644
--- a/tests/store/db/test_utils.py
+++ b/tests/store/db/test_utils.py
@@ -91,7 +91,10 @@ def test_create_sqlalchemy_engine_with_retry_success_after_third_call():
 
 def test_create_sqlalchemy_engine_with_retry_fail():
     with (
-        mock.patch("sqlalchemy.inspect", side_effect=[Exception("failed")] * utils.MAX_RETRY_COUNT),
+        mock.patch(
+            "sqlalchemy.inspect",
+            side_effect=[Exception("failed")] * utils.MAX_RETRY_COUNT,
+        ),
         mock.patch(
             "mlflow.store.db.utils.create_sqlalchemy_engine", return_value="Engine"
         ) as mock_create_sqlalchemy_engine,
@@ -103,3 +106,65 @@ def test_create_sqlalchemy_engine_with_retry_fail():
             mock_create_sqlalchemy_engine.mock_calls
             == [mock.call("mydb://host:port/")] * utils.MAX_RETRY_COUNT
         )
+
+
+def test_mysql_ssl_params(monkeypatch):
+    """Test that MySQL SSL certificate params are correctly passed to create_engine."""
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_CA", "/path/to/ca.pem")
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_CERT", "/path/to/cert.pem")
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_KEY", "/path/to/key.pem")
+    monkeypatch.setenv("MLFLOW_SQLALCHEMYSTORE_POOLCLASS", "NullPool")
+
+    with mock.patch("sqlalchemy.create_engine") as mock_create_engine:
+        utils.create_sqlalchemy_engine("mysql+pymysql://user@host:port/db")
+
+        # Check that create_engine was called with the right arguments
+        mock_create_engine.assert_called_once_with(
+            "mysql+pymysql://user@host:port/db",
+            pool_pre_ping=True,
+            connect_args={
+                "ssl_ca": "/path/to/ca.pem",
+                "ssl_cert": "/path/to/cert.pem",
+                "ssl_key": "/path/to/key.pem",
+            },
+            poolclass=NullPool,
+        )
+
+
+def test_mysql_ssl_params_partial(monkeypatch):
+    """Test that MySQL SSL certificate params work with only CA certificate."""
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_CA", "/path/to/ca.pem")
+    monkeypatch.delenv("MLFLOW_MYSQL_SSL_CERT", raising=False)
+    monkeypatch.delenv("MLFLOW_MYSQL_SSL_KEY", raising=False)
+    monkeypatch.setenv("MLFLOW_SQLALCHEMYSTORE_POOLCLASS", "NullPool")
+
+    with mock.patch("sqlalchemy.create_engine") as mock_create_engine:
+        utils.create_sqlalchemy_engine("mysql+pymysql://user@host:port/db")
+
+        # Check that create_engine was called with the right arguments
+        mock_create_engine.assert_called_once_with(
+            "mysql+pymysql://user@host:port/db",
+            pool_pre_ping=True,
+            connect_args={
+                "ssl_ca": "/path/to/ca.pem",
+            },
+            poolclass=NullPool,
+        )
+
+
+def test_non_mysql_no_ssl_params(monkeypatch):
+    """Test that non-MySQL connections don't get SSL params."""
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_CA", "/path/to/ca.pem")
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_CERT", "/path/to/cert.pem")
+    monkeypatch.setenv("MLFLOW_MYSQL_SSL_KEY", "/path/to/key.pem")
+    monkeypatch.setenv("MLFLOW_SQLALCHEMYSTORE_POOLCLASS", "NullPool")
+
+    with mock.patch("sqlalchemy.create_engine") as mock_create_engine:
+        utils.create_sqlalchemy_engine("postgresql://user@host:port/db")
+
+        # Check that create_engine was called without connect_args
+        mock_create_engine.assert_called_once_with(
+            "postgresql://user@host:port/db",
+            pool_pre_ping=True,
+            poolclass=NullPool,
+        )


Reviews:
user: login: serena-ruan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T07:47:43Z
user: login: serena-ruan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T07:50:00Z
user: login: serena-ruan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T07:50:13Z
user: login: serena-ruan
body: Overall LGTM! Left small nits, could you fix the failing test and we're good to merge!
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T07:52:48Z
user: login: serena-ruan
body: LGTM!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-06T10:46:44Z

Review comments:
diff_hunk: @@ -289,4 +292,32 @@ def create_sqlalchemy_engine(db_uri):
         pool_kwargs["poolclass"] = pool_class_map[poolclass]
     if pool_kwargs:
         _logger.info("Create SQLAlchemy engine with pool options %s", pool_kwargs)
-    return sqlalchemy.create_engine(db_uri, pool_pre_ping=True, **pool_kwargs)
+
+    # Handle MySQL SSL certificates via connect_args
+    connect_args = {}
+    # Check if this is a MySQL connection
+    if db_uri.startswith("mysql"):
+        # Get SSL certificate paths from environment variables
+        ssl_ca = MLFLOW_MYSQL_SSL_CA.get()
+        ssl_cert = MLFLOW_MYSQL_SSL_CERT.get()
+        ssl_key = MLFLOW_MYSQL_SSL_KEY.get()
+
+        # If any SSL certificate paths are provided, set up the SSL connection
+        if ssl_ca or ssl_cert or ssl_key:
+            ssl_args = {}
+            if ssl_ca:
+                ssl_args["ssl_ca"] = ssl_ca
+            if ssl_cert:
+                ssl_args["ssl_cert"] = ssl_cert
+            if ssl_key:
+                ssl_args["ssl_key"] = ssl_key
+
+            connect_args = ssl_args
path: mlflow/store/db/utils.py
user: login: serena-ruan
body: ```suggestion
        connect_args = {k: v for k, v in {"ssl_ca": MLFLOW_MYSQL_SSL_CA.get(), "ssl_cert": MLFLOW_MYSQL_SSL_CERT.get(), "ssl_key": MLFLOW_MYSQL_SSL_KEY.get()}.items() if v}
```
created_at: 2025-03-05T07:47:43Z
author_association: COLLABORATOR
diff_hunk: @@ -289,4 +292,32 @@ def create_sqlalchemy_engine(db_uri):
         pool_kwargs["poolclass"] = pool_class_map[poolclass]
     if pool_kwargs:
         _logger.info("Create SQLAlchemy engine with pool options %s", pool_kwargs)
-    return sqlalchemy.create_engine(db_uri, pool_pre_ping=True, **pool_kwargs)
+
+    # Handle MySQL SSL certificates via connect_args
+    connect_args = {}
+    # Check if this is a MySQL connection
+    if db_uri.startswith("mysql"):
+        # Get SSL certificate paths from environment variables
+        ssl_ca = MLFLOW_MYSQL_SSL_CA.get()
+        ssl_cert = MLFLOW_MYSQL_SSL_CERT.get()
+        ssl_key = MLFLOW_MYSQL_SSL_KEY.get()
+
+        # If any SSL certificate paths are provided, set up the SSL connection
+        if ssl_ca or ssl_cert or ssl_key:
+            ssl_args = {}
+            if ssl_ca:
+                ssl_args["ssl_ca"] = ssl_ca
+            if ssl_cert:
+                ssl_args["ssl_cert"] = ssl_cert
+            if ssl_key:
+                ssl_args["ssl_key"] = ssl_key
+
+            connect_args = ssl_args
path: mlflow/store/db/utils.py
user: login: serena-ruan
body: Let's add `pool_kwargs["connect_args"] = connect_args` here, and remove the if condition at below. You can rename `pool_kwargs` to `kwargs` if that's a more accurate name :)
created_at: 2025-03-05T07:50:00Z
author_association: COLLABORATOR
diff_hunk: @@ -289,4 +292,32 @@ def create_sqlalchemy_engine(db_uri):
         pool_kwargs["poolclass"] = pool_class_map[poolclass]
     if pool_kwargs:
         _logger.info("Create SQLAlchemy engine with pool options %s", pool_kwargs)
-    return sqlalchemy.create_engine(db_uri, pool_pre_ping=True, **pool_kwargs)
+
+    # Handle MySQL SSL certificates via connect_args
+    connect_args = {}
path: mlflow/store/db/utils.py
user: login: serena-ruan
body: ```suggestion
```
created_at: 2025-03-05T07:50:13Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-04T18:10:32Z
body: <!-- documentation preview -->


Documentation preview for 0619d8dffd2daddd3f5437f72353a9f2cbdc0ce3 will be available when [this CircleCI job](https://circleci.com/gh/mlflow/mlflow/148212)
completes successfully. You may encounter a `{"message":"not found"}` error when reloading
a page. If so, add `/index.html` to the URL.

- [Top page](https://output.circle-artifacts.com/output/job/01e02262-6617-4456-a368-68cca586a44c/artifacts/0/docs/build/latest/index.html)
- [Changed pages](https://output.circle-artifacts.com/output/job/01e02262-6617-4456-a368-68cca586a44c/artifacts/0/docs/build/diff.html) (⚠️ only MDX file changes are detected ⚠️)

<details>
<summary>More info</summary>

- Ignore this comment if this PR does not change the documentation.
- It takes a few minutes for the preview to be available.
- The preview is updated when a new commit is pushed to this PR.
- This comment was created by https://github.com/mlflow/mlflow/actions/runs/13672809343.

</details>

author_association: CONTRIBUTOR
user: login: aksylumoed
created_at: 2025-03-05T10:58:29Z
body: @serena-ruan The failing tests seem to be unrelated to my changes:
```
FAILED | MEM 2.8/15.6 GB | DISK 55.9/71.6 GB tests/evaluate/test_default_evaluator.py::test_pipeline_model_kernel_explainer_on_categorical_features - assert False
 +  where False = <built-in method issubset of set object at 0x7f83644342e0>(['confusion_matrix.png', 'per_class_metrics.csv', 'precision_recall_curve_plot.png', 'roc_curve_plot.png'])
 +    where <built-in method issubset of set object at 0x7f83644342e0> = {'explainer', 'shap_feature_importance_plot.png', 'shap_summary_plot.png'}.issubset
 +    and   ['confusion_matrix.png', 'per_class_metrics.csv', 'precision_recall_curve_plot.png', 'roc_curve_plot.png'] = RunData(params={}, metrics={'score': 0.94, 'example_count': 50.0, 'accuracy_score': 0.94, 'recall_score': 0.94, 'precision_score': 0.9490000000000001, 'f1_score': 0.9393289841565704, 'log_loss': 0.1969951641114006, 'roc_auc': 0.9904456327985739}, tags={'mlflow.user': 'runner', 'mlflow.source.name': '/opt/hostedtoolcache/Python/3.9.18/x64/bin/pytest', 'mlflow.source.type': 'LOCAL', 'mlflow.runName': 'righteous-chimp-291', 'mlflow.datasets': '[{"name":"0696bff3914cc1016192cc9af377d540","hash":"0696bff3914cc1016192cc9af377d540","model":"ec7594b0992f4451a4a99080b9c67497"}]'}, artifacts=['confusion_matrix.png', 'per_class_metrics.csv', 'precision_recall_curve_plot.png', 'roc_curve_plot.png']).artifacts
FAILED | MEM 2.8/15.6 GB | DISK 55.9/71.6 GB tests/evaluate/test_default_evaluator.py::test_evaluation_works_with_model_pipelines_that_modify_input_data - AssertionError: assert {'model'} >= {'shap_featur...ary_plot.png'}
  
  Extra items in the right set:
  'shap_feature_importance_plot.png'
  'shap_summary_plot.png'
```

Should I ignore?
author_association: CONTRIBUTOR
user: login: serena-ruan
created_at: 2025-03-05T11:00:08Z
body: Yep, could you manually verify this branch works before I merge it?
author_association: COLLABORATOR
user: login: aksylumoed
created_at: 2025-03-06T10:34:17Z
body: (sorry for the block of text)
Tested this manually, steps:
- Create certs:
```
> mkdir -p ~/mysql-ssl-test && cd ~/mysql-ssl-test
> openssl genrsa 2048 > ca-key.pem
> openssl req -new -x509 -nodes -days 365 -key ca-key.pem -out ca-cert.pem -subj "/CN=MySQL CA"
> openssl genrsa 2048 > server-key.pem
> openssl req -new -key server-key.pem -out server-req.pem -subj "/CN=MySQL Server"
> openssl x509 -req -in server-req.pem -days 365 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
> openssl genrsa 2048 > client-key.pem
> openssl req -new -key client-key.pem -out client-req.pem -subj "/CN=MySQL Client"
> openssl x509 -req -in client-req.pem -days 365 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem
```

- Setup mysql server via docker container:
```
> echo "ssl-ca=/etc/mysql/ssl/ca-cert.pem" >> my.cnf
> echo "ssl-cert=/etc/mysql/ssl/server-cert.pem" >> my.cnf
> echo "ssl-key=/etc/mysql/ssl/server-key.pem" >> my.cnf
> echo "require_secure_transport=ON" >> my.cnf
> docker run --name mysql-ssl -p 3306:3306 -e MYSQL_ROOT_PASSWORD=rootpass -e MYSQL_DATABASE=mlflow -d mysql:8
> sleep 15 && docker logs mysql-ssl | tail -20
> docker exec mysql-ssl mkdir -p /etc/mysql/ssl/
> docker cp ca-cert.pem mysql-ssl:/etc/mysql/ssl/ && docker cp server-cert.pem mysql-ssl:/etc/mysql/ssl/ && docker cp server-key.pem mysql-ssl:/etc/mysql/ssl/
> echo -e "[mysqld]\nssl-ca=/etc/mysql/ssl/ca-cert.pem\nssl-cert=/etc/mysql/ssl/server-cert.pem\nssl-key=/etc/mysql/ssl/server-key.pem\nrequire_secure_transport=ON" > my-ssl.cnf
> docker cp my-ssl.cnf mysql-ssl:/etc/mysql/conf.d/
> docker restart mysql-ssl

> docker exec -it mysql-ssl mysql -u root -prootpass -e "CREATE USER 'mlflow'@'%' IDENTIFIED BY '' REQUIRE SSL; GRANT ALL PRIVILEGES ON mlflow.* TO 'mlflow'@'%'; FLUSH PRIVILEGES;"
D BY '' REQUIRE SSdocker exec -it mysql-ssl mysql -u root -prootpass -e "CREATE USER 'mlflow'@'%' IDENTIFIE
D BY '' REQUIRE SSL; GRANT ALL PRIVILEGES ON mlflow.* TO 'mlflow'@'%'; FLUSH PRIVILEGES;"
mysql: [Warning] Using a password on the command line interface can be insecure.

> docker exec mysql-ssl mysql -u root -prootpass -e "SHOW GLOBAL VARIABLES LIKE '%ssl%';"
mysqlx_ssl_cipher
mysqlx_ssl_crl
mysqlx_ssl_crlpath
mysqlx_ssl_key
performance_schema_show_processlist     OFF
ssl_ca  /etc/mysql/ssl/ca-cert.pem
ssl_capath
ssl_cert        /etc/mysql/ssl/server-cert.pem
ssl_cipher
ssl_crl
ssl_crlpath
ssl_fips_mode   OFF
ssl_key /etc/mysql/ssl/server-key.pem
ssl_session_cache_mode  ON
ssl_session_cache_timeout       300

> docker exec mysql-ssl mysql -u root -prootpass -e "SHOW GLOBAL VARIABLES LIKE 'require_secure_transport';"
mysql: [Warning] Using a password on the command line interface can be insecure.
Variable_name   Value
require_secure_transport        ON
```

- Start mlflow with env vars set:
```
> cd ~/mysrc/mlflow && export MLFLOW_MYSQL_SSL_CA="$HOME/mysql-ssl-test/ca-cert.pem" && export MLFLOW_MYSQL_SSL_CERT="$HOME/mysql-ssl-test/client-cert.pem" && export MLFLOW_MYSQL_SSL_KEY="$HOME/mysql-ssl-test/client-key.pem" && export MLFLOW_SQLALCHEMYSTORE_ECHO=true 

> mlflow server --backend-store-uri="mysql+pymysql://mlflow:mlflow@localhost:3306/mlflow" --default-artifact-root="./mlruns" --host=0.0.0.0 --port=5000
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Create SQLAlchemy engine with pool options {'echo': True}
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CA: /Users/adandekar/mysql-ssl-test/ca-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CERT: /Users/adandekar/mysql-ssl-test/client-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_KEY: /Users/adandekar/mysql-ssl-test/client-key.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Using MySQL SSL with connect_args: {'ssl_ca': '/Users/adandekar/mysql-ssl-test/ca-cert.pem', 'ssl_cert': '/Users/adandekar/mysql-ssl-test/client-cert.pem', 'ssl_key': '/Users/adandekar/mysql-ssl-test/client-key.pem'}
2025-03-06 11:16:57,402 INFO sqlalchemy.engine.Engine SELECT DATABASE()
2025-03-06 11:16:57,402 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,404 INFO sqlalchemy.engine.Engine SELECT @@sql_mode
2025-03-06 11:16:57,404 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,404 INFO sqlalchemy.engine.Engine SELECT @@lower_case_table_names
2025-03-06 11:16:57,404 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,408 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-03-06 11:16:57,408 INFO sqlalchemy.engine.Engine SHOW FULL TABLES FROM `mlflow`
2025-03-06 11:16:57,408 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,409 INFO sqlalchemy.engine.Engine ROLLBACK
2025-03-06 11:16:57,418 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-03-06 11:16:57,418 INFO sqlalchemy.engine.Engine DESCRIBE `mlflow`.`alembic_version`
2025-03-06 11:16:57,418 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,420 INFO sqlalchemy.engine.Engine SELECT alembic_version.version_num 
FROM alembic_version
2025-03-06 11:16:57,420 INFO sqlalchemy.engine.Engine [generated in 0.00007s] {}
2025-03-06 11:16:57,421 INFO sqlalchemy.engine.Engine ROLLBACK
2025-03-06 11:16:57,425 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-03-06 11:16:57,426 INFO sqlalchemy.engine.Engine SELECT experiments.experiment_id, experiments.name, experiments.artifact_location, experiments.lifecycle_stage, experiments.creation_time, experiments.last_update_time 
FROM experiments 
WHERE experiments.lifecycle_stage IN (%(lifecycle_stage_1_1)s, %(lifecycle_stage_1_2)s) ORDER BY experiments.creation_time DESC, experiments.experiment_id ASC 
 LIMIT %(param_1)s, %(param_2)s
2025-03-06 11:16:57,426 INFO sqlalchemy.engine.Engine [generated in 0.00008s] {'param_1': 0, 'param_2': 1001, 'lifecycle_stage_1_1': 'active', 'lifecycle_stage_1_2': 'deleted'}
2025-03-06 11:16:57,429 INFO sqlalchemy.engine.Engine SELECT experiment_tags.`key` AS experiment_tags_key, experiment_tags.value AS experiment_tags_value, experiment_tags.experiment_id AS experiment_tags_experiment_id, anon_1.experiments_experiment_id AS anon_1_experiments_experiment_id 
FROM (SELECT experiments.experiment_id AS experiments_experiment_id 
FROM experiments 
WHERE experiments.lifecycle_stage IN (%(lifecycle_stage_1_1)s, %(lifecycle_stage_1_2)s) ORDER BY experiments.creation_time DESC, experiments.experiment_id ASC 
 LIMIT %(param_1)s, %(param_2)s) AS anon_1 INNER JOIN experiment_tags ON anon_1.experiments_experiment_id = experiment_tags.experiment_id
2025-03-06 11:16:57,429 INFO sqlalchemy.engine.Engine [generated in 0.00006s] {'param_1': 0, 'param_2': 1001, 'lifecycle_stage_1_1': 'active', 'lifecycle_stage_1_2': 'deleted'}
2025-03-06 11:16:57,431 INFO sqlalchemy.engine.Engine COMMIT
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Create SQLAlchemy engine with pool options {'echo': True}
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CA: /Users/adandekar/mysql-ssl-test/ca-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CERT: /Users/adandekar/mysql-ssl-test/client-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_KEY: /Users/adandekar/mysql-ssl-test/client-key.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Using MySQL SSL with connect_args: {'ssl_ca': '/Users/adandekar/mysql-ssl-test/ca-cert.pem', 'ssl_cert': '/Users/adandekar/mysql-ssl-test/client-cert.pem', 'ssl_key': '/Users/adandekar/mysql-ssl-test/client-key.pem'}
2025-03-06 11:16:57,444 INFO sqlalchemy.engine.Engine SELECT DATABASE()
2025-03-06 11:16:57,444 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,445 INFO sqlalchemy.engine.Engine SELECT @@sql_mode
2025-03-06 11:16:57,445 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,447 INFO sqlalchemy.engine.Engine SELECT @@lower_case_table_names
2025-03-06 11:16:57,447 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,450 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-03-06 11:16:57,450 INFO sqlalchemy.engine.Engine SHOW FULL TABLES FROM `mlflow`
2025-03-06 11:16:57,450 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,451 INFO sqlalchemy.engine.Engine ROLLBACK
2025-03-06 11:16:57,453 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2025-03-06 11:16:57,454 INFO sqlalchemy.engine.Engine SHOW FULL TABLES FROM `mlflow`
2025-03-06 11:16:57,454 INFO sqlalchemy.engine.Engine [raw sql] {}
2025-03-06 11:16:57,455 INFO sqlalchemy.engine.Engine ROLLBACK
[2025-03-06 11:16:57 +0100] [81050] [INFO] Starting gunicorn 23.0.0
[2025-03-06 11:16:57 +0100] [81050] [INFO] Listening at: http://0.0.0.0:5000 (81050)
[2025-03-06 11:16:57 +0100] [81050] [INFO] Using worker: sync
[2025-03-06 11:16:57 +0100] [81051] [INFO] Booting worker with pid: 81051
[2025-03-06 11:16:57 +0100] [81052] [INFO] Booting worker with pid: 81052
[2025-03-06 11:16:57 +0100] [81053] [INFO] Booting worker with pid: 81053
[2025-03-06 11:16:57 +0100] [81054] [INFO] Booting worker with pid: 81054
```
Note that I added some logging statements to the utils.py to further verify they are being set (not part of the PR):

```
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Create SQLAlchemy engine with pool options {'echo': True}
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CA: /Users/adandekar/mysql-ssl-test/ca-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CERT: /Users/adandekar/mysql-ssl-test/client-cert.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_KEY: /Users/adandekar/mysql-ssl-test/client-key.pem
2025/03/06 11:16:57 INFO mlflow.store.db.utils: Using MySQL SSL with connect_args: {'ssl_ca': '/Users/adandekar/mysql-ssl-test/ca-cert.pem', 'ssl_cert': '/Users/adandekar/mysql-ssl-test/client-cert.pem', 'ssl_key': '/Users/adandekar/mysql-ssl-test/client-key.pem'}
```

- Test mlflow server without env vars unset (should fail):
```
2025/03/06 11:14:46 INFO mlflow.store.db.utils: Create SQLAlchemy engine with pool options {'echo': True}
2025/03/06 11:14:46 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CA: None
2025/03/06 11:14:46 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_CERT: None
2025/03/06 11:14:46 INFO mlflow.store.db.utils: MLFLOW_MYSQL_SSL_KEY: None
2025/03/06 11:14:46 INFO mlflow.store.db.utils: No MySQL SSL environment variables set, not using SSL
2025/03/06 11:14:46 WARNING mlflow.store.db.utils: SQLAlchemy engine could not be created. The following ex
ception is caught.
(pymysql.err.OperationalError) (3159, 'Connections using insecure transport are prohibited while --require_
secure_transport=ON.')
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Operation will be retried in 0.1 seconds
...
```
author_association: CONTRIBUTOR
user: login: serena-ruan
created_at: 2025-03-06T10:45:52Z
body: Thanks for the verification!
author_association: COLLABORATOR