Repository: mlflow/mlflow

Issue:
[BUG] - Logging a metric with a boolean value freezes the UI with no error when logging
### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [ ] Yes. I can contribute a fix for this bug independently.
- [X] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community. -> I can give a try to fix the bug, but I can't give a timeline.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: No
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux, CENTOS 7.5 
- **MLflow installed from (source or binary)**: source
- **MLflow version (run ``mlflow --version``)**: mlflow 1.11.0
- **Python version**: Python 3.6.8 :: Anaconda, Inc.
- **npm version, if running the dev UI**: N/A

### Describe the problem

When a **boolean** value is logged as a metric, no warning is raised and the metric is stored. 

When you want to do anything which involves this run you got a very bad error message:
- INTERNAL_SERVER_ERROR when opening the UI. I could not make it work until I edited manually the database to remove the faulty metric in the faulty run
- a `ValueError: could not convert string to float: 'True'` error  when trying to manipulate the run with the python API (e.g. retrieving the run, logging a model...)

This is really painful that this errors occurs at retrieving time and not at logging time: 
- if logging a boolean as a metric is forbidden, it should be checked at logging time. Currently, the error occur far too late in the process (and actually, may hurts someone else, e.g. someone using the same database fot the UI, that the person who logged the faulty metric). The UI suddenly does not work with a very bad INTERNAL_SERVER_ERROR  and no informative message, as well as no possibility to clean up easily the database.
- Actually, I don't really see why logging boolean should avoided absolutely. The use case I encountered was someone logging a metric to know whether the training of his DL model early_stopped. This is a state which varies across the model training, and this may be useful to decide whether the training was "sufficient enough" to trust the model, so it does not seem to be total nonsense. I understand that it may be much harder to support this use case and have much more side effects than checking at logtime whether the metric value is a bool or not.

### Code to reproduce issue
```python 
# exploratory.py
from mlflow.tracking import MlflowClient
from pathlib import Path

mlflow_tracking_uri=Path("mlruns").resolve().as_uri()
mlflow.set_tracking_uri(mlflow_tracking_uri)
with mlflow.start_run():
    run_id=mlflow.active_run().info.run_id
    mlflow.log_metric("metric", True) # logging is ok, no warning

# You can either open the UI and enjoy the "INTERNAL_SERVER_ERROR"
# or try to retrieve the run for instance:
MlflowClient(mlflow_tracking_uri).get_run(run_id) # an error is raised
```

### Other info / logs
```console
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<xxx>/exploratory.py in 
----> 266 MlflowClient(mlflow_tracking_uri).get_run(run_id)

<xxx>/lib/python3.6/site-packages/mlflow/tracking/client.py in get_run(self, run_id)
    108                  raises an exception.
    109         """
--> 110         return self._tracking_client.get_run(run_id)
    111 
    112     def get_metric_history(self, run_id, key):

<xxx>/lib/python3.6/site-packages/mlflow/tracking/_tracking_service/client.py in get_run(self, run_id)
     54         """
     55         _validate_run_id(run_id)
---> 56         return self.store.get_run(run_id)
     57 
     58     def get_metric_history(self, run_id, key):

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in get_run(self, run_id)
    502                 "Run '%s' metadata is in invalid state." % run_id, databricks_pb2.INVALID_STATE
    503             )
--> 504         return self._get_run_from_info(run_info)
    505 
    506     def _get_run_from_info(self, run_info):

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in _get_run_from_info(self, run_info)
    505 
    506     def _get_run_from_info(self, run_info):
--> 507         metrics = self._get_all_metrics(run_info)
    508         params = self._get_all_params(run_info)
    509         tags = self._get_all_tags(run_info)

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in _get_all_metrics(self, run_info)
    592         metrics = []
    593         for metric_file in metric_files:
--> 594             metrics.append(self._get_metric_from_file(parent_path, metric_file))
    595         return metrics
    596 

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in _get_metric_from_file(parent_path, metric_name)
    573         metric_objs = [
    574             FileStore._get_metric_from_line(metric_name, line)
--> 575             for line in read_file_lines(parent_path, metric_name)
    576         ]
    577         if len(metric_objs) == 0:

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in (.0)
    573         metric_objs = [
    574             FileStore._get_metric_from_line(metric_name, line)
--> 575             for line in read_file_lines(parent_path, metric_name)
    576         ]
    577         if len(metric_objs) == 0:

<xxx>/lib/python3.6/site-packages/mlflow/store/tracking/file_store.py in _get_metric_from_line(metric_name, metric_line)
    605             )
    606         ts = int(metric_parts[0])
--> 607         val = float(metric_parts[1])
    608         step = int(metric_parts[2]) if len(metric_parts) == 3 else 0
    609         return Metric(key=metric_name, value=val, timestamp=ts, step=step)

ValueError: could not convert string to float: 'True'
```

### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: Local serving, model deployment tools, spark UDFs
- [ ] `area/server-infra`: MLflow server, JavaScript dev server
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [] `area/uiux`: Front-end, user experience, JavaScript, plotting
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
@Galileo-Galilei  Thanks for filing this bug, along with code to reproduce it. I was able to reproduce this code with 

```
import mlflow
from mlflow.tracking import MlflowClient

if __name__ == '__main__':
    with mlflow.start_run() as run:
        run_id = run.info.run_id

        # logging is ok, no warning
        mlflow.log_metric("metric", True)

    MlflowClient().get_run(run_id) # an error is raised
```

![Screen Shot 2020-11-30 at 10 17 52 AM](https://user-images.githubusercontent.com/1117597/100648384-b6316980-32f5-11eb-8f0f-9301f126e490.png)

cc: @xanderwebs, it might be more of a tracking issue.

In Python, `bool` is a subclass of `int`. We need to update our metric validtion logic to reject `bool`.
Seems like a good first issue for a new contributor. 
@dmatrix I am interested in working on this. May I take it?
@HCoban Feel free to take it. Since this is your first issue, take a look at the code for the [validation logic](https://github.com/mlflow/mlflow/blob/b66dbac21b2d39ffaabd81e78a4edb23045b3486/mlflow/utils/validation.py#L66) and suggest how you would resolve it. It seems we do validate the metric value as an instance of a number and raise an exception.

As @harupy rightly pointed out `bool` is a subclass of `int`,  so  we need to reject the incoming `bool` value and raise the exception.

```
import numbers
isinstance(True, numbers.Number)
```

Will return `True`, hence the exception is never raised.
Hello @dmatrix,

Thank you for your reply.

In _validate_metric function, I would add a check for bool like 
```
if isinstance(value, bool):
        raise MlflowException(
            "Got invalid value %s for metric '%s' (timestamp=%s). Value must be a number, not bool"
            % (value, key, timestamp),
            INVALID_PARAMETER_VALUE,
        )
```

and add a test case for it in the `test_validate_batch_log_data`. 

What do you think?
Could we do something like this?

```diff
-    if not isinstance(value, numbers.Number):
+    # <comment on why we check a bool>
+    if isinstance(value, bool) or not isinstance(value, numbers.Number):
        raise MlflowException(
            "Got invalid value %s for metric '%s' (timestamp=%s). Please specify value as a valid "
            "double (64-bit floating point)" % (value, key, timestamp),
            INVALID_PARAMETER_VALUE,
        )
```
@HCoban You might want to go with @harupy suggestion as you need either of the conditions to be true to trigger an exception. 
So having that in one line makes sense.

Resolving patch:
diff --git a/mlflow/utils/validation.py b/mlflow/utils/validation.py
--- a/mlflow/utils/validation.py
+++ b/mlflow/utils/validation.py
@@ -69,7 +69,9 @@ def _validate_metric(key, value, timestamp, step):
     it isn't.
     """
     _validate_metric_name(key)
-    if not isinstance(value, numbers.Number):
+    # value must be a Number
+    # since bool is an instance of Number check for bool additionally
+    if isinstance(value, bool) or not isinstance(value, numbers.Number):
         raise MlflowException(
             "Got invalid value %s for metric '%s' (timestamp=%s). Please specify value as a valid "
             "double (64-bit floating point)" % (value, key, timestamp),


Test patch:
diff --git a/tests/utils/test_validation.py b/tests/utils/test_validation.py
--- a/tests/utils/test_validation.py
+++ b/tests/utils/test_validation.py
@@ -122,6 +122,7 @@ def test_validate_batch_log_data():
         Metric("super-long-bad-key" * 1000, 4.0, 0, 0),
     ]
     metrics_with_bad_val = [Metric("good-metric-key", "not-a-double-val", 0, 0)]
+    metrics_with_bool_val = [Metric("good-metric-key", True, 0, 0)]
     metrics_with_bad_ts = [Metric("good-metric-key", 1.0, "not-a-timestamp", 0)]
     metrics_with_neg_ts = [Metric("good-metric-key", 1.0, -123, 0)]
     metrics_with_bad_step = [Metric("good-metric-key", 1.0, 0, "not-a-step")]
@@ -145,6 +146,7 @@ def test_validate_batch_log_data():
         "metrics": [
             metrics_with_bad_key,
             metrics_with_bad_val,
+            metrics_with_bool_val,
             metrics_with_bad_ts,
             metrics_with_neg_ts,
             metrics_with_bad_step,