Repository: mlflow/mlflow

Issue:
[BUG] archive_existing_versions flag not working
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [x] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: NAME="Amazon Linux" VERSION="2" ID_LIKE="centos rhel fedora"
- **MLflow installed from (source or binary)**:
- **MLflow version (run ``mlflow --version``)**: mlflow, version 1.13
- **Python version**: Python 3.7.9
- **npm version, if running the dev UI**:
- **Exact command to reproduce**:

### Describe the problem
Describe the problem clearly here. Include descriptions of the expected behavior and the actual behavior.

The older version of the model **not** moving to the "Archived" stage when using the archive_existing_versions flag.

**Expected behavior:** The older versions of the model should move to the "Archived" stage when setting the archive_existing_versions  flag

**Actual behavior:** The older versions are not moving to the "Archived" stage

In the below screenshot, Version 1 did not move to the "Archived" stage while registering Version 2.


### Code to reproduce issue
```python
client = MlflowClient()
client.transition_model_version_stage(
    name="Model-HousingPricePrediction",
    version=2,
    stage="production",
    archive_existing_versions=True
)
```

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
<img width="1206" alt="Screen Shot 2020-12-30 at 1 35 53 PM" src="https://user-images.githubusercontent.com/64650227/103374024-7a302680-4aa4-11eb-9d29-97739847ae24.png">

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [x] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [x] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
@1pyroaqua Thanks for reporthing this. Could you run the code below and check what it prints out?

```python
from sklearn.linear_model import LinearRegression

import mlflow
from mlflow.tracking.client import MlflowClient

mlflow.set_tracking_uri("sqlite:///mlflow.db")
# this creates `mlflow.db` in the current working directory

model = LinearRegression()
artifact_path = "model"
registered_model_name = "test"
stage = "Production"
client = MlflowClient()

with mlflow.start_run() as run:
    mlflow.sklearn.log_model(model, artifact_path)

    model_uri = mlflow.get_artifact_uri(artifact_path)
    client.create_registered_model(registered_model_name)
    client.create_model_version(registered_model_name, model_uri, run.info.run_id)
    client.transition_model_version_stage(registered_model_name, 1, stage)

with mlflow.start_run() as run:
    mlflow.sklearn.log_model(model, artifact_path)

    model_uri = mlflow.get_artifact_uri(artifact_path)
    client.create_model_version(registered_model_name, model_uri, run.info.run_id)
    client.transition_model_version_stage(
        registered_model_name, 2, stage, archive_existing_versions=True
    )

print(client.get_model_version(registered_model_name, 1))
print(client.get_model_version(registered_model_name, 2))
```
> ```python
> from sklearn.linear_model import LinearRegression
> 
> import mlflow
> from mlflow.tracking.client import MlflowClient
> 
> mlflow.set_tracking_uri("sqlite:///mlflow.db")
> # this creates `mlflow.db` in the current working directory
> 
> model = LinearRegression()
> artifact_path = "model"
> registered_model_name = "test"
> stage = "Production"
> client = MlflowClient()
> 
> with mlflow.start_run() as run:
>     mlflow.sklearn.log_model(model, artifact_path)
> 
>     model_uri = mlflow.get_artifact_uri(artifact_path)
>     client.create_registered_model(registered_model_name)
>     client.create_model_version(registered_model_name, model_uri, run.info.run_id)
>     client.transition_model_version_stage(registered_model_name, 1, stage)
> 
> with mlflow.start_run() as run:
>     mlflow.sklearn.log_model(model, artifact_path)
> 
>     model_uri = mlflow.get_artifact_uri(artifact_path)
>     client.create_model_version(registered_model_name, model_uri, run.info.run_id)
>     client.transition_model_version_stage(
>         registered_model_name, 2, stage, archive_existing_versions=True
>     )
> 
> print(client.get_model_version(registered_model_name, 1))
> print(client.get_model_version(registered_model_name, 2))
> ```

```console
<ModelVersion: creation_timestamp=1609382457743, current_stage='Archived', description=None, last_updated_timestamp=1609382457924, name='test', run_id='9d99d4d61afc4c0381e7e3471aa9acf6', run_link=None, source='./mlruns/0/9d99d4d61afc4c0381e7e3471aa9acf6/artifacts/model', status='READY', status_message=None, tags={}, user_id=None, version=1>

<ModelVersion: creation_timestamp=1609382457915, current_stage='Production', description=None, last_updated_timestamp=1609382457924, name='test', run_id='24784c7801424218ad58a2214018a95a', run_link=None, source='./mlruns/0/24784c7801424218ad58a2214018a95a/artifacts/model', status='READY', status_message=None, tags={}, user_id=None, version=2>
```

```
<ModelVersion: creation_timestamp=1609382457743, current_stage='Archived', description=None, 
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^
<ModelVersion: creation_timestamp=1609382457915, current_stage='Production', description=None,
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Looks like this example works fine.
@1pyroaqua
Does the same error occur when you do stage transition on the MLflow UI?
@harupy 

I found the issue. It was a type in my code.  In my original code, I have used the word "production" without the uppercase "P". It works fine when I change the "p" to uppercase "P". I have edited my original comment to reflect my original code.

Changed the below...
```python
client = MlflowClient()
client.transition_model_version_stage(
    name="Model-HousingPricePrediction",
    version=2,
    stage="production", # Here is the typo
    archive_existing_versions=True
)
```
To.....
```python
client = MlflowClient()
client.transition_model_version_stage(
    name="Model-HousingPricePrediction",
    version=2,
    stage="Production",
    archive_existing_versions=True
)
```
I feel dumb :-(.. lol..
Did you get an error or something when you ran this?

```
client = MlflowClient()
client.transition_model_version_stage(
    name="Model-HousingPricePrediction",
    version=2,
    stage="production", # Here is the typo
    archive_existing_versions=True
)
```
> Did you get an error or something when you ran this?
> 
> ```
> client = MlflowClient()
> client.transition_model_version_stage(
>     name="Model-HousingPricePrediction",
>     version=2,
>     stage="production", # Here is the typo
>     archive_existing_versions=True
> )
> ```

Nope. I didn't notice an error. Please see the log below

**Code**
The one that you gave above. I just changed "Production" to "production" and reran it.

```console
2020/12/31 03:13:12 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2020/12/31 03:13:12 INFO mlflow.store.db.utils: Updating database tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 451aebb31d03, add metric step
INFO  [alembic.runtime.migration] Running upgrade 451aebb31d03 -> 90e64c465722, migrate user column to tags
INFO  [alembic.runtime.migration] Running upgrade 90e64c465722 -> 181f10493468, allow nulls for metric values
INFO  [alembic.runtime.migration] Running upgrade 181f10493468 -> df50e92ffc5e, Add Experiment Tags Table
INFO  [alembic.runtime.migration] Running upgrade df50e92ffc5e -> 7ac759974ad8, Update run tags with larger limit
INFO  [alembic.runtime.migration] Running upgrade 7ac759974ad8 -> 89d4b8295536, create latest metrics table
INFO  [89d4b8295536_create_latest_metrics_table_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 89d4b8295536 -> 2b4d017a5e9b, add model registry tables to db
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Adding registered_models and model_versions tables to database.
INFO  [2b4d017a5e9b_add_model_registry_tables_to_db_py] Migration complete!
INFO  [alembic.runtime.migration] Running upgrade 2b4d017a5e9b -> cfd24bdc0731, Update run status constraint with killed
INFO  [alembic.runtime.migration] Running upgrade cfd24bdc0731 -> 0a8213491aaa, drop_duplicate_killed_constraint
WARNI [0a8213491aaa_drop_duplicate_killed_constraint_py] Failed to drop check constraint. Dropping check constraints may not be supported by your SQL database. Exception content: No support for ALTER of constraints in SQLite dialectPlease refer to the batch mode feature which allows for SQLite migrations using a copy-and-move strategy.
INFO  [alembic.runtime.migration] Running upgrade 0a8213491aaa -> 728d730b5ebd, add registered model tags table
INFO  [alembic.runtime.migration] Running upgrade 728d730b5ebd -> 27a6a02d2cf1, add model version tags table
INFO  [alembic.runtime.migration] Running upgrade 27a6a02d2cf1 -> 84291f40a231, add run_link to model_version
INFO  [alembic.runtime.migration] Running upgrade 84291f40a231 -> a8c4a736bde6, allow nulls for run_id
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
2020/12/31 03:13:13 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: test, version 1
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
2020/12/31 03:13:13 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation.                     Model name: test, version 2
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
<ModelVersion: creation_timestamp=1609384393400, current_stage='Production', description=None, last_updated_timestamp=1609384393410, name='test', run_id='700eacf74ff047e0a0026631e8324c59', run_link=None, source='./mlruns/0/700eacf74ff047e0a0026631e8324c59/artifacts/model', status='READY', status_message=None, tags={}, user_id=None, version=1>
<ModelVersion: creation_timestamp=1609384393577, current_stage='Production', description=None, last_updated_timestamp=1609384393586, name='test', run_id='a12c99fab18b4be1a6f1d66adcb12970', run_link=None, source='./mlruns/0/a12c99fab18b4be1a6f1d66adcb12970/artifacts/model', status='READY', status_message=None, tags={}, user_id=None, version=2>
​
```

I was able to repro the issue.
https://github.com/mlflow/mlflow/blob/38be3432bac019194cbe56ae705f7c258ec42b86/mlflow/store/model_registry/sqlalchemy_store.py#L665

This line is the root cause of the issue. We need to apply [`get_canonical_stage`](https://github.com/mlflow/mlflow/blob/8e74cdd108ff324175e38d621b12d9b2aced05bd/mlflow/entities/model_registry/model_version_stages.py#L16) to `stage` to normalize casing.

Resolving patch:
diff --git a/mlflow/store/model_registry/sqlalchemy_store.py b/mlflow/store/model_registry/sqlalchemy_store.py
--- a/mlflow/store/model_registry/sqlalchemy_store.py
+++ b/mlflow/store/model_registry/sqlalchemy_store.py
@@ -662,7 +662,7 @@ def transition_model_version_stage(self, name, version, stage, archive_existing_
                 conditions = [
                     SqlModelVersion.name == name,
                     SqlModelVersion.version != version,
-                    SqlModelVersion.current_stage == stage,
+                    SqlModelVersion.current_stage == get_canonical_stage(stage),
                 ]
                 model_versions = session.query(SqlModelVersion).filter(*conditions).all()
                 for mv in model_versions:


Test patch:
diff --git a/tests/store/model_registry/test_sqlalchemy_store.py b/tests/store/model_registry/test_sqlalchemy_store.py
--- a/tests/store/model_registry/test_sqlalchemy_store.py
+++ b/tests/store/model_registry/test_sqlalchemy_store.py
@@ -617,6 +617,20 @@ def test_transition_model_version_stage_when_archive_existing_versions_is_true(s
         self.assertEqual(mvd3.current_stage, "Production")
         self.assertEqual(mvd2.last_updated_timestamp, mvd3.last_updated_timestamp)
 
+        for uncanonical_stage_name in ["STAGING", "staging", "StAgInG"]:
+            self.store.transition_model_version_stage(mv1.name, mv1.version, "Staging", False)
+            self.store.transition_model_version_stage(mv2.name, mv2.version, "None", False)
+
+            # stage names are case-insensitive and auto-corrected to system stage names
+            self.store.transition_model_version_stage(
+                mv2.name, mv2.version, uncanonical_stage_name, True
+            )
+
+            mvd1 = self.store.get_model_version(name=mv1.name, version=mv1.version)
+            mvd2 = self.store.get_model_version(name=mv2.name, version=mv2.version)
+            self.assertEqual(mvd1.current_stage, "Archived")
+            self.assertEqual(mvd2.current_stage, "Staging")
+
     def test_delete_model_version(self):
         name = "test_for_delete_MV"
         initial_tags = [