Repository: mlflow/mlflow

Issue:
[BUG] INTERNAL_SERVER_ERROR when sorting the runs table by metric or by date
### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: no
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Debian
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.7.0
- **Python version**: 3.7
- **npm version, if running the dev UI**: -
- **Exact command to reproduce**: run the mlflow server and log parameters and metrics

### Describe the problem
When I try to sort the runs by any metric or date (parameters, versions and users work) I get an `INTERNAL_SERVER_ERROR` and the whole table disappears. Then, if I sort by a parameter, for example, the table comes back.

**UPDATE:**
![V4-Model-MLflow-Experiment-Google-Chrome-2020-04-07-10-18-17-_online-video-cutter com_](https://user-images.githubusercontent.com/6637618/78646910-10d70a80-78ba-11ea-82ea-a4678a3337fe.gif)


### Code to reproduce issue
run the mlflow server and log parameters and metrics

### Other info / logs
This is the trace-back I got from the server:
```
2020/03/04 09:40:50 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/runs/search [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/db/utils.py", line 73, in make_managed_session
    yield session
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 651, in _search_runs
    .offset(offset).limit(max_results).all()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3233, in all
    return list(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)")
[SQL: SELECT DISTINCT TOP 100 runs.run_uuid AS runs_run_uuid, runs.name AS runs_name, runs.source_type AS runs_source_type, runs.source_name AS runs_source_name, runs.entry_point_name AS runs_entry_point_name, runs.user_id AS runs_user_id, runs.status AS runs_status, runs.start_time AS runs_start_time, runs.end_time AS runs_end_time, runs.source_version AS runs_source_version, runs.lifecycle_stage AS runs_lifecycle_stage, runs.artifact_uri AS runs_artifact_uri, runs.experiment_id AS runs_experiment_id, CASE WHEN (anon_1.is_nan IS 1) THEN ? WHEN (anon_1.value IS NULL) THEN ? ELSE ? END AS clause_1, anon_1.value AS anon_1_value 
FROM runs LEFT OUTER JOIN (SELECT latest_metrics.[key] AS [key], latest_metrics.value AS value, latest_metrics.timestamp AS timestamp, latest_metrics.step AS step, latest_metrics.is_nan AS is_nan, latest_metrics.run_uuid AS run_uuid 
FROM latest_metrics 
WHERE latest_metrics.[key] = ?) AS anon_1 ON runs.run_uuid = anon_1.run_uuid 
WHERE runs.experiment_id IN (?) AND runs.lifecycle_stage IN (?) ORDER BY clause_1, anon_1.value, runs.start_time DESC, runs.run_uuid]
[parameters: (1, 1, 0, 'auc', '4', 'active')]
(Background on this error at: http://sqlalche.me/e/f405)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mlflow/server/handlers.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mlflow/server/handlers.py", line 394, in _search_runs
    max_results, order_by, page_token)
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/tracking/abstract_store.py", line 230, in search_runs
    order_by, page_token)
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/tracking/sqlalchemy_store.py", line 654, in _search_runs
    next_page_token = compute_next_token(len(runs))
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/mlflow/store/db/utils.py", line 80, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '1'. (102) (SQLExecDirectW)")
[SQL: SELECT DISTINCT TOP 100 runs.run_uuid AS runs_run_uuid, runs.name AS runs_name, runs.source_type AS runs_source_type, runs.source_name AS runs_source_name, runs.entry_point_name AS runs_entry_point_name, runs.user_id AS runs_user_id, runs.status AS runs_status, runs.start_time AS runs_start_time, runs.end_time AS runs_end_time, runs.source_version AS runs_source_version, runs.lifecycle_stage AS runs_lifecycle_stage, runs.artifact_uri AS runs_artifact_uri, runs.experiment_id AS runs_experiment_id, CASE WHEN (anon_1.is_nan IS 1) THEN ? WHEN (anon_1.value IS NULL) THEN ? ELSE ? END AS clause_1, anon_1.value AS anon_1_value 
FROM runs LEFT OUTER JOIN (SELECT latest_metrics.[key] AS [key], latest_metrics.value AS value, latest_metrics.timestamp AS timestamp, latest_metrics.step AS step, latest_metrics.is_nan AS is_nan, latest_metrics.run_uuid AS run_uuid 
FROM latest_metrics 
WHERE latest_metrics.[key] = ?) AS anon_1 ON runs.run_uuid = anon_1.run_uuid 
WHERE runs.experiment_id IN (?) AND runs.lifecycle_stage IN (?) ORDER BY clause_1, anon_1.value, runs.start_time DESC, runs.run_uuid]
[parameters: (1, 1, 0, 'auc', '4', 'active')]
(Background on this error at: http://sqlalche.me/e/f405)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/mlflow/server/handlers.py", line 158, in wrapper
    response.set_data(e.serialize_as_json())
  File "/usr/local/lib/python3.7/site-packages/mlflow/exceptions.py", line 50, in serialize_as_json
    return json.dumps(exception_dict)
  File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ProgrammingError is not JSON serializable
```
### **UPDATE**
I've updated the version to v1.7.2 and the bug is still there.


Discussion:
Hey, 
I am not able to reproduce this error, could you please provide some addition details.
Thanks!
@arjundc-db I don't know what else I can provide.
I added an animated GIF showing the problem.

As the error comes from a SQL query, is it possible that my DB is corrupted? How can I verify if the schema of the tables is correct?
I've updated the version to v1.7.2 and the bug is still there.
Hey,
Are you using an SQL server?
@arjundc-db 
Yes, I'm using SQL server. 
That is where this issue is most likely coming from. Quoting the docs : https://mlflow.org/docs/latest/tracking.html#where-runs-are-recorded
"MLflow supports the dialects mysql, mssql, sqlite, and postgresql."
@arjundc-db
However, according to your quote, MLflow supports SQL server dialect = mssql.

That is where this issue is most likely coming from. Quoting the docs : https://mlflow.org/docs/latest/tracking.html#where-runs-are-recorded
"MLflow supports the dialects mysql, mssql, sqlite, and postgresql."

Is that an error on the documentation?
I have executed the query in the logs on the SQL server and I've found the problem.

The query appears like this in the error message:
```
[SQL: SELECT DISTINCT TOP 100 runs.run_uuid AS runs_run_uuid, runs.name AS runs_name, runs.source_type AS runs_source_type, runs.source_name AS runs_source_name, runs.entry_point_name AS runs_entry_point_name, runs.user_id AS runs_user_id, runs.status AS runs_status, runs.start_time AS runs_start_time, runs.end_time AS runs_end_time, runs.source_version AS runs_source_version, runs.lifecycle_stage AS runs_lifecycle_stage, runs.artifact_uri AS runs_artifact_uri, runs.experiment_id AS runs_experiment_id, CASE WHEN (anon_1.is_nan IS 1) THEN ? WHEN (anon_1.value IS NULL) THEN ? ELSE ? END AS clause_1, anon_1.value AS anon_1_value 
FROM runs LEFT OUTER JOIN (SELECT latest_metrics.[key] AS [key], latest_metrics.value AS value, latest_metrics.timestamp AS timestamp, latest_metrics.step AS step, latest_metrics.is_nan AS is_nan, latest_metrics.run_uuid AS run_uuid 
FROM latest_metrics 
WHERE latest_metrics.[key] = ?) AS anon_1 ON runs.run_uuid = anon_1.run_uuid 
WHERE runs.experiment_id IN (?) AND runs.lifecycle_stage IN (?) ORDER BY clause_1, anon_1.value, runs.start_time DESC, runs.run_uuid]
[parameters: (1, 1, 0, 'auc', '4', 'active')]
```

Performing the replacements... 
```
SELECT DISTINCT TOP 100 runs.run_uuid AS runs_run_uuid, runs.name AS runs_name, runs.source_type AS runs_source_type, runs.source_name AS runs_source_name, runs.entry_point_name AS runs_entry_point_name, runs.user_id AS runs_user_id, runs.status AS runs_status, runs.start_time AS runs_start_time, runs.end_time AS runs_end_time, runs.source_version AS runs_source_version, runs.lifecycle_stage AS runs_lifecycle_stage, runs.artifact_uri AS runs_artifact_uri, runs.experiment_id AS runs_experiment_id, CASE WHEN (anon_1.is_nan IS 1) THEN 1 WHEN (anon_1.value IS NULL) THEN 1 ELSE 0 END AS clause_1, anon_1.value AS anon_1_value 
FROM runs LEFT OUTER JOIN (SELECT latest_metrics.[key] AS [key], latest_metrics.value AS value, latest_metrics.timestamp AS timestamp, latest_metrics.step AS step, latest_metrics.is_nan AS is_nan, latest_metrics.run_uuid AS run_uuid 
FROM latest_metrics 
WHERE latest_metrics.[key] = 'auc') AS anon_1 ON runs.run_uuid = anon_1.run_uuid 
WHERE runs.experiment_id IN ('1') AND runs.lifecycle_stage IN ('active') ORDER BY clause_1, anon_1.value, runs.start_time DESC, runs.run_uuid
```

The problem comes from the statement `CASE WHEN (anon_1.is_nan IS 1) THEN ...` as you can see below.
![image](https://user-images.githubusercontent.com/6637618/79772410-5838ae80-8330-11ea-8e67-7be2a05e63ca.png)

By replacing `IS` to `=` in the statement (`CASE WHEN (anon_1.is_nan = 1) THEN ...`) the query runs normally.

### More information
Here is the table `latest_metrics` definition:
![image](https://user-images.githubusercontent.com/6637618/79773047-33910680-8331-11ea-8c85-e9b6f6c3bcd0.png)

```
CREATE TABLE [dbo].[latest_metrics](
	[key] [varchar](250) NOT NULL,
	[value] [float] NOT NULL,
	[timestamp] [bigint] NULL,
	[step] [bigint] NOT NULL,
	[is_nan] [bit] NOT NULL,
	[run_uuid] [varchar](32) NOT NULL,
 CONSTRAINT [latest_metric_pk] PRIMARY KEY CLUSTERED 
(
	[key] ASC,
	[run_uuid] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO

ALTER TABLE [dbo].[latest_metrics]  WITH CHECK ADD FOREIGN KEY([run_uuid])
REFERENCES [dbo].[runs] ([run_uuid])
GO
```
@arjundc-db 
Could you please take a second look at this?
The bug is still active. @arjundc-db Can you reopen it, please?
I apologize (did not see @lopezco comments after I closed the issue), I am reopening and issue and will look into @lopezco suggestion. 
Thank you @tsabata 
Hey @lopezco, your fix should work, thank you for investigating the bug and proposing a solution. Would you be able to raise a PR with this change. A good place to start would be here : https://github.com/mlflow/mlflow/blob/master/mlflow/store/tracking/sqlalchemy_store.py#L809.

For a complete PR, I would also request that you add tests to test_sqlalchemy_store.py that expose this error and show that your fix addresses them.

Thank you again!

Hello @arjundc-db, I'm currently packet at work. I might have time to work on it on September/October (not sure).
I hope someone could get to work on this PR before September.

I'm working with a mssql-server and I can reproduce this issue.

Resolving patch:
diff --git a/mlflow/store/tracking/sqlalchemy_store.py b/mlflow/store/tracking/sqlalchemy_store.py
--- a/mlflow/store/tracking/sqlalchemy_store.py
+++ b/mlflow/store/tracking/sqlalchemy_store.py
@@ -883,7 +883,16 @@ def _get_orderby_clauses(order_by_list, session):
             if SearchUtils.is_metric(key_type, "="):
                 clauses.append(
                     sql.case(
-                        [(subquery.c.is_nan.is_(True), 1), (order_value.is_(None), 1)], else_=0
+                        [
+                            # Ideally the use of "IS" is preferred here but owing to sqlalchemy
+                            # translation in MSSQL we are forced to use "=" instead.
+                            # These 2 options are functionally identical / unchanged because
+                            # the column (is_nan) is not nullable. However it could become an issue
+                            # if this precondition changes in the future.
+                            (subquery.c.is_nan == sqlalchemy.true(), 1),
+                            (order_value.is_(None), 1),
+                        ],
+                        else_=0,
                     ).label("clause_%s" % clause_id)
                 )
             else:  # other entities do not have an 'is_nan' field


Test patch:
diff --git a/tests/store/tracking/test_sqlalchemy_store.py b/tests/store/tracking/test_sqlalchemy_store.py
--- a/tests/store/tracking/test_sqlalchemy_store.py
+++ b/tests/store/tracking/test_sqlalchemy_store.py
@@ -1991,3 +1991,8 @@ def test_get_orderby_clauses():
 
         # test that an exception is NOT raised when key types are different
         _get_orderby_clauses(["param.a", "metric.a", "tag.a"], session)
+
+        # test that "=" is used rather than "is" when comparing to True
+        parsed = [str(x) for x in _get_orderby_clauses(["metric.a"], session)[0]]
+        assert "is_nan = true" in parsed[0]
+        assert "value IS NULL" in parsed[0]