Repository: mlflow/mlflow

Issue:
[BUG] If an experiment contains a run that has an empty or NaN metric, trying to sort the runs in the UI based on that metric fails
Thank you for submitting an issue. Please refer to our [issue policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md) for additional information about bug reports. For help with debugging your code, please refer to [Stack Overflow](https://stackoverflow.com/questions/tagged/mlflow).

**Please fill in this bug report template to ensure a timely and thorough response.**

### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ X] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 10.13.6
- **MLflow installed from (source or binary)**: binary (mlflow-1.11.0-py3-none-any.whl)
- **MLflow version (run ``mlflow --version``)**: 1.11.0
- **Python version**: 3.6
- **npm version, if running the dev UI**:
- **Exact command to reproduce**:

### Describe the problem
If an experiment contains a run that has an empty or NaN metric, trying to sort the runs in the UI based on that metric fails and an "INTERNAL_SERVER_ERROR" is displayed. 
An inspection of _search_utils.py_ reveals that a check for nulls and nans is done prior to sorting, to ensure runs with nulls/nans appear at the end of the sorted list. This does not prevent the sorting from failing. A quick fix can be done like this:

```
    def _get_value_for_sort(cls, run, key_type, key, ascending):
        """Returns a tuple suitable to be used as a sort key for runs."""
        sort_value = None
        if key_type == cls._METRIC_IDENTIFIER:
            sort_value = run.data.metrics.get(key)

            # NEW CODE: Check if the value is null or nan and replace with +/- inf
            if sort_value is None or (
                isinstance(sort_value, float) and math.isnan(sort_value) ):
                if ascending:
                    sort_value = math.inf
                else:
                    sort_value = -math.inf
           ...
```

Not sure if this problem also exists when sorting based on other parameter than metrics.
### Other info / logs
```
2020/09/10 12:15:10 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/runs/search [POST]
Traceback (most recent call last):
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/anaconda/envs/python36/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/anaconda/envs/python36/lib/python3.6/site-packages/mlflow/server/handlers.py", line 213, in wrapper
    return func(*args, **kwargs)
  File "/anaconda/envs/python36/lib/python3.6/site-packages/mlflow/server/handlers.py", line 464, in _search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/anaconda/envs/python36/lib/python3.6/site-packages/mlflow/store/tracking/abstract_store.py", line 235, in search_runs
    experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
  File "/anaconda/envs/python36/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py", line 734, in _search_runs
    sorted_runs = SearchUtils.sort(filtered, order_by)
  File "/anaconda/envs/python36/lib/python3.6/site-packages/mlflow/utils/search_utils.py", line 487, in sort
    reverse=not ascending,
TypeError: '<' not supported between instances of 'float' and 'NoneType'
```

### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [X ] `area/server-infra`: MLflow server, JavaScript dev server
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [X ] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
@humcasma Thanks for reporting this. I was able to reproduce the error.

![issue-3406](https://user-images.githubusercontent.com/17039389/92845762-4c09d680-f422-11ea-88e7-781cfbdf8269.gif)

Looks like the error occurs when metrics cotains both an empty and `NaN`.

![only-nan-or-empty](https://user-images.githubusercontent.com/17039389/92846866-7c9e4000-f423-11ea-9dca-5232a0395f77.gif)


Resolving patch:
diff --git a/mlflow/utils/search_utils.py b/mlflow/utils/search_utils.py
--- a/mlflow/utils/search_utils.py
+++ b/mlflow/utils/search_utils.py
@@ -491,12 +491,18 @@ def _get_value_for_sort(cls, run, key_type, key, ascending):
             )
 
         # Return a key such that None values are always at the end.
-        is_null_or_nan = sort_value is None or (
-            isinstance(sort_value, float) and math.isnan(sort_value)
-        )
-        if ascending:
-            return (is_null_or_nan, sort_value)
-        return (not is_null_or_nan, sort_value)
+        is_none = sort_value is None
+        is_nan = isinstance(sort_value, float) and math.isnan(sort_value)
+        fill_value = (1 if ascending else -1) * math.inf
+
+        if is_none:
+            sort_value = fill_value
+        elif is_nan:
+            sort_value = -fill_value
+
+        is_none_or_nan = is_none or is_nan
+
+        return (is_none_or_nan, sort_value) if ascending else (not is_none_or_nan, sort_value)
 
     @classmethod
     def sort(cls, runs, order_by_list):


Test patch:
diff --git a/tests/utils/test_search_utils.py b/tests/utils/test_search_utils.py
--- a/tests/utils/test_search_utils.py
+++ b/tests/utils/test_search_utils.py
@@ -382,8 +382,8 @@ def test_correct_sorting(order_bys, matching_runs):
     assert sorted_run_indices == matching_runs
 
 
-def test_order_by_metric_with_nans_and_infs():
-    metric_vals_str = ["nan", "inf", "-inf", "-1000", "0", "1000"]
+def test_order_by_metric_with_nans_infs_nones():
+    metric_vals_str = ["nan", "inf", "-inf", "-1000", "0", "1000", "None"]
     runs = [
         Run(
             run_info=RunInfo(
@@ -396,16 +396,16 @@ def test_order_by_metric_with_nans_and_infs():
                 end_time=1,
                 lifecycle_stage=LifecycleStage.ACTIVE,
             ),
-            run_data=RunData(metrics=[Metric("x", float(x), 1, 0)]),
+            run_data=RunData(metrics=[Metric("x", None if x == "None" else float(x), 1, 0)]),
         )
         for x in metric_vals_str
     ]
     sorted_runs_asc = [x.info.run_id for x in SearchUtils.sort(runs, ["metrics.x asc"])]
     sorted_runs_desc = [x.info.run_id for x in SearchUtils.sort(runs, ["metrics.x desc"])]
     # asc
-    assert ["-inf", "-1000", "0", "1000", "inf", "nan"] == sorted_runs_asc
+    assert ["-inf", "-1000", "0", "1000", "inf", "nan", "None"] == sorted_runs_asc
     # desc
-    assert ["inf", "1000", "0", "-1000", "-inf", "nan"] == sorted_runs_desc
+    assert ["inf", "1000", "0", "-1000", "-inf", "nan", "None"] == sorted_runs_desc
 
 
 @pytest.mark.parametrize(