Repository: mlflow/mlflow

Issue:
[BUG] Cannot delete experiments with deleted runs
### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

1.27.0

### System information

- **OS Platform and Distribution**: Ubuntu 20.04
- **Python version**: 3.8


### Describe the problem

I am trying to delete experiments, using both the UI and the client, but I am getting the following error ``` INVALID_PARAMETER_VALUE: The run 39a47c2a9df849b2b8a3b210da5478b4 must be in the 'active' state. Current state is deleted.```. This happens only with experiments that I have deleted individual runs, in fact even if I delete a single run, the experiment cannot be deleted later. What I've found is that if I restore the deleted runs, then I am able to remove the experiment.

### Tracking information

_No response_

### Code to reproduce issue

-

### Stack trace

```
INVALID_PARAMETER_VALUE: The run 39a47c2a9df849b2b8a3b210da5478b4 must be in the 'active' state. Current state is deleted.
```

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/pipelines`: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [X] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@nsantavas Thank you for pointing this out. This is definitely a bug, and we are working on a fix.

Resolving patch:
diff --git a/mlflow/entities/run_info.py b/mlflow/entities/run_info.py
--- a/mlflow/entities/run_info.py
+++ b/mlflow/entities/run_info.py
@@ -15,14 +15,6 @@ def check_run_is_active(run_info):
         )
 
 
-def check_run_is_deleted(run_info):
-    if run_info.lifecycle_stage != LifecycleStage.DELETED:
-        raise MlflowException(
-            "The run {} must be in 'deleted' lifecycle_stage.".format(run_info.run_id),
-            error_code=INVALID_PARAMETER_VALUE,
-        )
-
-
 class searchable_attribute(property):
     # Wrapper class over property to designate some of the properties as searchable
     # run attributes
diff --git a/mlflow/store/tracking/file_store.py b/mlflow/store/tracking/file_store.py
--- a/mlflow/store/tracking/file_store.py
+++ b/mlflow/store/tracking/file_store.py
@@ -22,7 +22,7 @@
     ExperimentTag,
 )
 from mlflow.entities.lifecycle_stage import LifecycleStage
-from mlflow.entities.run_info import check_run_is_active, check_run_is_deleted
+from mlflow.entities.run_info import check_run_is_active
 from mlflow.exceptions import MlflowException, MissingConfigException
 from mlflow.protos import databricks_pb2
 from mlflow.protos.databricks_pb2 import (
@@ -480,7 +480,6 @@ def delete_run(self, run_id):
             raise MlflowException(
                 "Run '%s' metadata is in invalid state." % run_id, databricks_pb2.INVALID_STATE
             )
-        check_run_is_active(run_info)
         new_info = run_info._copy_with_overrides(lifecycle_stage=LifecycleStage.DELETED)
         self._overwrite_run_info(new_info, deleted_time=int(time.time() * 1000))
 
@@ -519,7 +518,6 @@ def restore_run(self, run_id):
             raise MlflowException(
                 "Run '%s' metadata is in invalid state." % run_id, databricks_pb2.INVALID_STATE
             )
-        check_run_is_deleted(run_info)
         new_info = run_info._copy_with_overrides(lifecycle_stage=LifecycleStage.ACTIVE)
         self._overwrite_run_info(new_info, deleted_time=None)
 
diff --git a/mlflow/store/tracking/sqlalchemy_store.py b/mlflow/store/tracking/sqlalchemy_store.py
--- a/mlflow/store/tracking/sqlalchemy_store.py
+++ b/mlflow/store/tracking/sqlalchemy_store.py
@@ -478,7 +478,6 @@ def delete_experiment(self, experiment_id):
             self._save_to_db(objs=experiment, session=session)
 
     def _mark_run_deleted(self, session, run):
-        self._check_run_is_active(run)
         run.lifecycle_stage = LifecycleStage.DELETED
         run.deleted_time = int(time.time() * 1000)
         self._save_to_db(objs=run, session=session)
@@ -602,15 +601,6 @@ def _check_experiment_is_active(self, experiment):
                 INVALID_PARAMETER_VALUE,
             )
 
-    def _check_run_is_deleted(self, run):
-        if run.lifecycle_stage != LifecycleStage.DELETED:
-            raise MlflowException(
-                "The run {} must be in the 'deleted' state. Current state is {}.".format(
-                    run.run_uuid, run.lifecycle_stage
-                ),
-                INVALID_PARAMETER_VALUE,
-            )
-
     def update_run_info(self, run_id, run_status, end_time):
         with self.ManagedSessionMaker() as session:
             run = self._get_run(run_uuid=run_id, session=session)
@@ -645,7 +635,6 @@ def get_run(self, run_id):
     def restore_run(self, run_id):
         with self.ManagedSessionMaker() as session:
             run = self._get_run(run_uuid=run_id, session=session)
-            self._check_run_is_deleted(run)
             run.lifecycle_stage = LifecycleStage.ACTIVE
             run.deleted_time = None
             self._save_to_db(objs=run, session=session)
@@ -653,7 +642,6 @@ def restore_run(self, run_id):
     def delete_run(self, run_id):
         with self.ManagedSessionMaker() as session:
             run = self._get_run(run_uuid=run_id, session=session)
-            self._check_run_is_active(run)
             run.lifecycle_stage = LifecycleStage.DELETED
             run.deleted_time = int(time.time() * 1000)
             self._save_to_db(objs=run, session=session)


Test patch:
diff --git a/tests/store/tracking/test_file_store.py b/tests/store/tracking/test_file_store.py
--- a/tests/store/tracking/test_file_store.py
+++ b/tests/store/tracking/test_file_store.py
@@ -542,10 +542,14 @@ def test_delete_restore_run(self):
         _, run_dir = fs._find_run_root(run_id)
         # Should not throw.
         assert fs.get_run(run_id).info.lifecycle_stage == "active"
+        # Verify that run deletion is idempotent by deleting twice
+        fs.delete_run(run_id)
         fs.delete_run(run_id)
         assert fs.get_run(run_id).info.lifecycle_stage == "deleted"
         meta = read_yaml(run_dir, FileStore.META_DATA_FILE_NAME)
         assert "deleted_time" in meta and meta["deleted_time"] is not None
+        # Verify that run restoration is idempotent by restoring twice
+        fs.restore_run(run_id)
         fs.restore_run(run_id)
         assert fs.get_run(run_id).info.lifecycle_stage == "active"
         meta = read_yaml(run_dir, FileStore.META_DATA_FILE_NAME)
diff --git a/tests/store/tracking/test_sqlalchemy_store.py b/tests/store/tracking/test_sqlalchemy_store.py
--- a/tests/store/tracking/test_sqlalchemy_store.py
+++ b/tests/store/tracking/test_sqlalchemy_store.py
@@ -259,6 +259,7 @@ def test_delete_restore_experiment_with_runs(self):
         experiment_id = self._experiment_factory("test exp")
         run1 = self._run_factory(config=self._get_run_configs(experiment_id)).info.run_id
         run2 = self._run_factory(config=self._get_run_configs(experiment_id)).info.run_id
+        self.store.delete_run(run1)
         run_ids = [run1, run2]
 
         self.store.delete_experiment(experiment_id)
@@ -1246,21 +1247,21 @@ def test_delete_restore_run(self):
         run = self._run_factory()
         self.assertEqual(run.info.lifecycle_stage, entities.LifecycleStage.ACTIVE)
 
-        with pytest.raises(MlflowException, match=r"The run .+ must be in the 'deleted' state"):
-            self.store.restore_run(run.info.run_id)
+        # Verify that active runs can be restored (run restoration is idempotent)
+        self.store.restore_run(run.info.run_id)
 
+        # Verify that run deletion is idempotent
+        self.store.delete_run(run.info.run_id)
         self.store.delete_run(run.info.run_id)
-        with pytest.raises(MlflowException, match=r"The run .+ must be in the 'active' state"):
-            self.store.delete_run(run.info.run_id)
 
         deleted = self.store.get_run(run.info.run_id)
         self.assertEqual(deleted.info.run_id, run.info.run_id)
         self.assertEqual(deleted.info.lifecycle_stage, entities.LifecycleStage.DELETED)
         with self.store.ManagedSessionMaker() as session:
             assert self.store._get_run(session, deleted.info.run_id).deleted_time is not None
+        # Verify that restoration of a deleted run is idempotent
+        self.store.restore_run(run.info.run_id)
         self.store.restore_run(run.info.run_id)
-        with pytest.raises(MlflowException, match=r"The run .+ must be in the 'deleted' state"):
-            self.store.restore_run(run.info.run_id)
         restored = self.store.get_run(run.info.run_id)
         self.assertEqual(restored.info.run_id, run.info.run_id)
         self.assertEqual(restored.info.lifecycle_stage, entities.LifecycleStage.ACTIVE)