Repository: mlflow/mlflow

Issue:
[BUG] Alembic migration for metrics table uses incorrect server default
## MLflow Roadmap Item

This is an MLflow Roadmap item that has been prioritized by the MLflow maintainers. We're seeking help with the implementation of roadmap items tagged with the `help wanted` label.

For requirements clarifications and implementation questions, or to request a PR review, please tag @harupy in your communications related to this issue.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: miniconda container - debian buster
- **MLflow installed from (source or binary)**:  mflow from pypi
- **MLflow version (run ``mlflow --version``)**: mlflow 1.14.1  trying to upgrade to 1.16 or 1.17
- **Python version**:  Python 3.9.2
- **npm version, if running the dev UI**: NA
- **Exact command to reproduce**: mlflow db upgrade <MSSQL connection string >
-  Tracking server DB: Azure Microsoft SQL DB

### Describe the problem
When I upgrade the database from 1.14.1 to a higher version I get an error. Currently use an Azure MSFT DB. Would like to upgrade to 1.16 or 1.17

### Code to reproduce issue
mlflow db upgrade "mssql+pyodbc://_rest_of_conn_string"

### Other info / logs
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Column already has a DEFAULT bound to it. (1781) (SQLExecDirectW)')
[SQL: ALTER TABLE metrics ADD DEFAULT '0' FOR is_nan]
(Background on this error at: http://sqlalche.me/e/14/f405)


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [x] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [x] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
@GlamTam Thanks for filing this. Since you want to contribute to it, we welcome and look forward to community help with bug fixes. Please follow the instructions on how to [contribute](https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst#contribution-process). 
Seeing the same issue on Windows (mlflow version 1.17.0) with MSSQL when running `mlflow server --backend-store-uri <uri> --default-artifact-root file:.`.

Error is the same as above (occurs when running the alembic migrations)
found the issue in: versions/c48cb773bb87_reset_default_value_for_is_nan_in_metrics_table_for_mysql.py

``` python
def upgrade():    
# This part of the migration is only relevant for MySQL.    
# In 39d1c3be5f05_add_is_nan_constraint_for_metrics_tables_if_necessary.py    
# (added in MLflow 1.15.0), `alter_column` is called on the `is_nan` column in the `metrics`    
# table without specifying `existing_server_default`. This alters the column default value to    
# NULL in MySQL (see the doc below).    
#    
# https://alembic.sqlalchemy.org/en/latest/ops.html#alembic.operations.Operations.alter_column    
#    
# To revert this change, set the default column value to "0" by specifying `server_default`    
with op.batch_alter_table("metrics") as batch_op:       
batch_op.alter_column(            
"is_nan",            
type_=sa.types.Boolean(create_constraint=True),            nullable=False,            server_default="0",        )
```

server_default should be existing_server_default. Will look into creating a PR for the fix.

Hi @GlamTam, I was wondering if you have managed to find a work around for this issue?
@Fl4v , 

- I replaced `server_default ` with `existing_server_default` on a local mlflow==1.16.0 repo. 
- Pip installed mlflow from the local repo. 
- Executed `mlflow db upgrade ...` with this custom mlflow installation.
- upgrade to 1.17 or 1.18 can be done using the regular pypi packages

Thanks @GlamTam 
Hi, are there any timelines for the release of this fix?
only fix I currently see is editing this old migration. Not sure how @dmatrix  thinks about this?
Facing this issue as I try connect to SQL DB as backend-component too. Any updates on this please?

Resolving patch:
diff --git a/mlflow/store/db_migrations/versions/c48cb773bb87_reset_default_value_for_is_nan_in_metrics_table_for_mysql.py b/mlflow/store/db_migrations/versions/c48cb773bb87_reset_default_value_for_is_nan_in_metrics_table_for_mysql.py
--- a/mlflow/store/db_migrations/versions/c48cb773bb87_reset_default_value_for_is_nan_in_metrics_table_for_mysql.py
+++ b/mlflow/store/db_migrations/versions/c48cb773bb87_reset_default_value_for_is_nan_in_metrics_table_for_mysql.py
@@ -26,13 +26,15 @@ def upgrade():
     # https://alembic.sqlalchemy.org/en/latest/ops.html#alembic.operations.Operations.alter_column
     #
     # To revert this change, set the default column value to "0" by specifying `server_default`
-    with op.batch_alter_table("metrics") as batch_op:
-        batch_op.alter_column(
-            "is_nan",
-            type_=sa.types.Boolean(create_constraint=True),
-            nullable=False,
-            server_default="0",
-        )
+    bind = op.get_bind()
+    if bind.engine.name == "mysql":
+        with op.batch_alter_table("metrics") as batch_op:
+            batch_op.alter_column(
+                "is_nan",
+                type_=sa.types.Boolean(create_constraint=True),
+                nullable=False,
+                server_default="0",
+            )
 
 
 def downgrade():


Test patch:
diff --git a/tests/db/.dockerignore b/tests/db/.dockerignore
--- a/tests/db/.dockerignore
+++ b/tests/db/.dockerignore
@@ -2,3 +2,5 @@
 
 !dist/*.whl
 !log.py
+!init-mssql-db.sh
+!init-mssql-db.sql
diff --git a/tests/db/Dockerfile.mssql b/tests/db/Dockerfile.mssql
new file mode 100644
--- /dev/null
+++ b/tests/db/Dockerfile.mssql
@@ -0,0 +1,27 @@
+FROM python:3.6
+
+WORKDIR /tmp/mlflow
+
+COPY dist ./dist
+
+# apt-get and system utilities
+RUN apt-get update && apt-get install -y \
+	curl apt-transport-https debconf-utils \
+    && rm -rf /var/lib/apt/lists/*
+
+# adding custom MS repository
+RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
+RUN curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
+
+# install SQL Server drivers and tools
+RUN apt-get update && ACCEPT_EULA=Y apt-get install -y mssql-tools unixodbc-dev
+
+RUN pip install dist/*.whl
+RUN pip install pyodbc
+RUN pip list
+
+COPY log.py .
+COPY init-mssql-db.sh .
+COPY init-mssql-db.sql .
+
+RUN chmod +x init-mssql-db.sh
diff --git a/tests/db/docker-compose.yml b/tests/db/docker-compose.yml
--- a/tests/db/docker-compose.yml
+++ b/tests/db/docker-compose.yml
@@ -32,3 +32,19 @@ services:
       context: .
     environment:
       MLFLOW_TRACKING_URI: mysql://mlflowuser:mlflowpassword@mysql:3306/mlflowdb
+
+  mssql:
+    image: mcr.microsoft.com/mssql/server
+    restart: always
+    environment:
+      ACCEPT_EULA: Y
+      SA_PASSWORD: "1Secure*Password1"
+
+  mlflow-mssql:
+    depends_on:
+      - mssql
+    build:
+      context: .
+      dockerfile: Dockerfile.mssql
+    environment:
+      MLFLOW_TRACKING_URI: mssql+pyodbc://mlflowuser:Mlfl*wpassword1@mssql/mlflowdb?driver=ODBC+Driver+17+for+SQL+Server
diff --git a/tests/db/init-mssql-db.sh b/tests/db/init-mssql-db.sh
new file mode 100644
--- /dev/null
+++ b/tests/db/init-mssql-db.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+for BACKOFF in 0 1 2 4 8 16; do
+    if [ "$BACKOFF" -ne "0" ]; then
+        echo "Could not connect to SQL Server"
+        echo "Trying again in ${BACKOFF} seconds"
+        sleep $BACKOFF
+    fi
+    if /opt/mssql-tools/bin/sqlcmd -S mssql -U sa -P 1Secure*Password1 -d master -i init-mssql-db.sql; then
+        exit 0
+    fi
+done
+exit 1
diff --git a/tests/db/init-mssql-db.sql b/tests/db/init-mssql-db.sql
new file mode 100644
--- /dev/null
+++ b/tests/db/init-mssql-db.sql
@@ -0,0 +1,15 @@
+CREATE DATABASE mlflowdb;
+GO
+
+USE mlflowdb;
+
+CREATE LOGIN mlflowuser
+    WITH PASSWORD = 'Mlfl*wpassword1';
+GO
+
+CREATE USER mlflowuser FOR LOGIN mlflowuser;
+GO
+
+ALTER ROLE db_owner
+    ADD MEMBER mlflowuser;
+GO