Repository: mlflow/mlflow

Issue:
[BUG] Deleting the 'Default' Experiment breaks MLflow initialization

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ probably ] Yes. I can contribute a fix for this bug independently.
- [ maybe ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### System information
- MLflow docker image based on 'python:3.9-slim-buster'
- MLflow installed from official pip \ pypi
- MLflow version 1.16.0
- MLflow backend store: AWS RDS MySQL 8.0.23
- MLflow artifact root: AWS S3 bucket
- Python version: 3.9
- command: mlflow server --backend-store-uri mysql+pymysql://root:${mlflow_rds_root_password}@${address}/${stack_env} --default-artifact-root "s3://${s3_bucket}/${s3_bucket_prefix}/${stack_env}/" --host 0.0.0.0:8080

### Describe the problem
After running MLflow for the first time, against an AWS RDS backend store,
a chaos engineer could request MLflow to delete the 'Default' experiment, which is created by the db migration mechanisms used by MLflow ( SQLalchemy migrations ).

after the default experiment (id 0) is deleted, MLflow continues to function fine, **until** the next service restart,
if the service is restarted, then it would fail to initialize successfully with these errors:


```
2021/05/12 17:20:12 ERROR mlflow.cli: Error initializing backend store
2021/05/12 17:20:12 ERROR mlflow.cli: (pymysql.err.IntegrityError) (1062, "Duplicate entry '0' for key 'experiments.PRIMARY'")
[SQL: INSERT INTO experiments (experiment_id, name, artifact_location, lifecycle_stage) VALUES (0, 'Default', '"s3://aura-data-mlflow/MLflow/dev/"/0', 'active');]
(Background on this error at: http://sqlalche.me/e/14/gkpj)
Traceback (most recent call last):
...
...
...
pymysql.err.IntegrityError: (1062, "Duplicate entry '0' for key 'experiments.PRIMARY'")
[SQL: INSERT INTO experiments (experiment_id, name, artifact_location, lifecycle_stage) VALUES (0, 'Default', '"s3://aura-data-mlflow/MLflow/dev/"/0', 'active');]
```

Clearly, the migration process attempts to insert a new default experiment at id 0, while it was already there, but removed due to user request to delete it..

### Code to reproduce issue
User action based, but I guess someone could also do it with:

```
#!/usr/bin/env python
import mlflow
TRACKING_URI = 'https://mlflow-dev.yourdomain.com/'

mlflow.set_tracking_uri(TRACKING_URI)
mlflow.delete_experiment('0')
# OR 
client = mlflow.tracking.MlflowClient(TRACKING_URI)
client.delete_experiment('0')
```

### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ X ] `area/server-infra`: MLflow server, JavaScript dev server
- [ X ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ X ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ X ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry



Discussion:
Short Stack:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/db/utils.py", line 80, in make_managed_session
    yield session
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 130, in __init__
    self._create_default_experiment(session)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 178, in _create_default_experiment
    session.execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1677, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
```
Full Stack - 3 Tracebacks

```Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1770, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/usr/local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1062, "Duplicate entry '0' for key 'experiments.PRIMARY'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/db/utils.py", line 80, in make_managed_session
    yield session
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 130, in __init__
    self._create_default_experiment(session)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 178, in _create_default_experiment
    session.execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1677, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 329, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1451, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1994, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1770, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/usr/local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/usr/local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.IntegrityError: (pymysql.err.IntegrityError) (1062, "Duplicate entry '0' for key 'experiments.PRIMARY'")
[SQL: INSERT INTO experiments (experiment_id, name, artifact_location, lifecycle_stage) VALUES (0, 'Default', '"s3://aura-data-mlflow/MLflow/dev/"/0', 'active');]
(Background on this error at: http://sqlalche.me/e/14/gkpj)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/mlflow/cli.py", line 383, in server
    initialize_backend_stores(backend_store_uri, default_artifact_root)
  File "/usr/local/lib/python3.9/site-packages/mlflow/server/handlers.py", line 145, in initialize_backend_stores
    _get_tracking_store(backend_store_uri, default_artifact_root)
  File "/usr/local/lib/python3.9/site-packages/mlflow/server/handlers.py", line 130, in _get_tracking_store
    _tracking_store = _tracking_store_registry.get_store(store_uri, artifact_root)
  File "/usr/local/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/registry.py", line 38, in get_store
    return builder(store_uri=store_uri, artifact_uri=artifact_uri)
  File "/usr/local/lib/python3.9/site-packages/mlflow/server/handlers.py", line 101, in _get_sqlalchemy_store
    return SqlAlchemyStore(store_uri, artifact_uri)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 130, in __init__
    self._create_default_experiment(session)
  File "/usr/local/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/db/utils.py", line 87, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: (pymysql.err.IntegrityError) (1062, "Duplicate entry '0' for key 'experiments.PRIMARY'")```
It's funny, guess what I saw on the offending function?
https://github.com/mlflow/mlflow/blob/master/mlflow/store/tracking/sqlalchemy_store.py#L156
`ToDo: Identify a less hacky mechanism to create default experiment 0`
PR Created ^ #4352 

Resolving patch:
diff --git a/mlflow/store/tracking/sqlalchemy_store.py b/mlflow/store/tracking/sqlalchemy_store.py
--- a/mlflow/store/tracking/sqlalchemy_store.py
+++ b/mlflow/store/tracking/sqlalchemy_store.py
@@ -125,7 +125,7 @@ def __init__(self, db_uri, default_artifact_root):
         if is_local_uri(default_artifact_root):
             mkdir(local_file_uri_to_path(default_artifact_root))
 
-        if len(self.list_experiments()) == 0:
+        if len(self.list_experiments(view_type=ViewType.ALL)) == 0:
             with self.ManagedSessionMaker() as session:
                 self._create_default_experiment(session)
 


Test patch:
diff --git a/tests/store/tracking/test_sqlalchemy_store.py b/tests/store/tracking/test_sqlalchemy_store.py
--- a/tests/store/tracking/test_sqlalchemy_store.py
+++ b/tests/store/tracking/test_sqlalchemy_store.py
@@ -1852,6 +1852,14 @@ def test_sqlalchemy_store_behaves_as_expected_with_inmemory_sqlite_db():
     assert param.key in fetched_run.data.params
 
 
+def test_sqlalchemy_store_can_be_initialized_when_default_experiment_has_been_deleted(tmpdir):
+    db_uri = "sqlite:///{}/mlflow.db".format(tmpdir.strpath)
+    store = SqlAlchemyStore(db_uri, ARTIFACT_URI)
+    store.delete_experiment("0")
+    assert store.get_experiment("0").lifecycle_stage == entities.LifecycleStage.DELETED
+    SqlAlchemyStore(db_uri, ARTIFACT_URI)
+
+
 class TestSqlAlchemyStoreSqliteMigratedDB(TestSqlAlchemyStoreSqlite):
     """
     Test case where user has an existing DB with schema generated before MLflow 1.0,