Repository: mlflow/mlflow

Issue:
[FR] Log HTML representation of estimator in scikit-learn autologging
### Willingness to contribute

No. I cannot contribute this feature at this time.

### Proposal Summary

Log an HTML representation of an estimator in scikit-learn autologging using [`sklearn.utils.estimator_html_repr`](https://scikit-learn.org/stable/modules/generated/sklearn.utils.estimator_html_repr.html#sklearn.utils.estimator_html_repr).

<img width="1166" alt="image" src="https://user-images.githubusercontent.com/17039389/189557559-d7c87334-d9a5-4b84-a05b-870c7bbb59fc.png">


### Motivation

> #### What is the use case for this feature?

To make it easier to understand the structure of a auto-logged estimator.

> #### Why is this use case valuable to support for MLflow users in general?

> #### Why is this use case valuable to support for your project(s) or organization?

> #### Why is it currently difficult to achieve this use case?


### Details

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [X] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/pipelines`: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
Hey  I want to work on the issue. Could you please elaborate more on this what exactly to be done?

@shraddhafalane Thanks! We can make a change like this:

```diff
diff --git a/mlflow/sklearn/utils.py b/mlflow/sklearn/utils.py
index 604407112..656faf092 100644
--- a/mlflow/sklearn/utils.py
+++ b/mlflow/sklearn/utils.py
@@ -553,6 +553,24 @@ def _log_specialized_estimator_content(
     return metrics
 
 
+def _log_estimator_html(run_id, estimator):
+    from sklearn.utils import estimator_html_repr
+
+    # Set charset to prevent the triangle toggle buttons from being garbled
+    estimator_html = f"""
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+  </head>
+  <body>
+    {estimator_html_repr.estimator_html_repr(estimator)}
+  </body>
+</html>
+"""
+    MlflowClient().log_text(run_id, estimator_html, artifact_file="estimator.html")
+
+
 def _log_estimator_content(
     autologging_client,
     estimator,
@@ -614,6 +632,8 @@ def _log_estimator_content(
             autologging_client.log_metrics(run_id=run_id, metrics={score_key: score})
             metrics[score_key] = score
 
+    _log_estimator_html(run_id, estimator)
     return metrics
```
We should add a test like this:

```diff
diff --git a/tests/sklearn/test_sklearn_autolog.py b/tests/sklearn/test_sklearn_autolog.py
index b71cb3a81..59b6cc101 100644
--- a/tests/sklearn/test_sklearn_autolog.py
+++ b/tests/sklearn/test_sklearn_autolog.py
@@ -2103,3 +2103,11 @@ def test_autolog_emits_warning_message_when_pos_label_used_for_multilabel():
             "Metric error: Target is multiclass but average='binary'. Please choose another "
             "average setting, one of [None, 'micro', 'macro', 'weighted']."
         )
+
+
+def test_estimator_html_is_logged():
+    mlflow.sklearn.autolog()
+    with mlflow.start_run() as run:
+        sklearn.svm.SVC().fit(*get_iris())
+        artifacts = get_run_data(run.info.run_id)[3]
+        assert "estimator.html" in artifacts

```
@RohanBha1 Is this issue something you'd be interested in working on as a replacement for #6757?
@harupy  if @RohanBha1  is not working on issue let me know. I'll work on it. Otherwise please assign me an another issue with little explanation if possible to you.
Sure that works for me, sorry on the late response, won't happen again
@RohanBha1 Assigned! Please let me know if you have any questions.
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
@RohanBha1 Any updates for this issue?
Yes, I'll be done by tonight, sorry haven't updated in a while.
Hey @harupy I had a few questions about what I'm supposed to do. Does this task involve making both the changes shown above, or just the lower one? In addition, what does "+    # Set charset to prevent the triangle toggle buttons from being garbled" mean, and what do I do about that?
@RohanBha1

> Does this task involve making both the changes shown above

Yes :)

> \# Set charset to prevent the triangle toggle buttons from being garbled" mean, and what do I do about that?

The generated HTML contains `▸`. This character may get garbled depending if charset is not specified.

@harupy Thank you so much for the clarification, it makes much more sense now. Since this project is likely in UTF-8, would the line 

`<meta charset="UTF-8" />  
`

do the necessary specification, or would I need to add anything else besides that?


Yes, it does.
@harupy got it, thank you so much. So I looked through some of the other files and I think I have a good understanding of the actual changes made in the first part, please correct me if any of it seems incorrect: The first line defines the new function, then the estimator_html_repr function is imported from sklearn.utils. Then estimator_html is set to be an f-string and since """ is used every new line in the code will be a new line in the f-string, the contents of the f-string are the normal html headings, and the charset must be specified as UTF-8 so that the triangle buttons don't get garbled, then in the body since it is an f-string, the stuff in {} are found before the string itself so this calls the estimator_html_repr function and the body for the estimator is added to the body of the f-string. Then the f-string ends, and the estimator_html string is written in the autologger.

I do have a few questions though, such as how does autologging this html representation of the estimator benefit users? I might just be misunderstanding what autologging is. If you could elaborate on this topic a little that would be amazing.

Finally, could you tell me a little more about the testing in the second code snippet above. I get the first line is starting the autologger and the upcoming run is defined as run, but the next line completely loses me. Finally, using the run_id of the estimator from the first snippet, an array called artifact is called, and it is checked if the estimator is in that array. I feel like my understanding is pretty flawed, so if you could explain the testing that would be amazing.

Sorry for the longer question, I just want to make sure that I understand the contribution I am making. Thanks for all your help, its greatly appreciated.
@RohanBha1 

> how does autologging this html representation of the estimator benefit users?

The HTML representation allows users to understand what the estimator consists of quickly.

> could you tell me a little more about the testing in the second code snippet above

Sure!

```python
def test_estimator_html_is_logged():
    mlflow.sklearn.autolog()  # -> monkey patch sklearn.svm.SVC.fit
    with mlflow.start_run() as run:  # -> start a run
        sklearn.svm.SVC().fit(*get_iris())  # -> log params, metrics, and artifacts in the run
        artifacts = get_run_data(run.info.run_id)[3]  # -> Fetch the logged artifacts
        assert "estimator.html" in artifacts  # Ensure 'estimator.html' is logged as an artifact
```
Hey @harupy sorry, I'm in Florida and we just got hit with a hurricane so my power is just coming back. I meant to ask the following a few days ago following our last conversation, but unfortunately given the hurricane I was unable.

Thank you so much for the continued explanation, I just had a few more questions about the estimator. I was looking through some of the scikit-learn documentation and found this:
<img width="992" alt="Screen Shot 2022-09-27 at 1 45 57 AM" src="https://user-images.githubusercontent.com/92875326/193399751-1285dfb4-e967-4875-a7e4-a822bf723bd6.png">

So from my understanding, the estimator is meant to be a model of some kind modeling the machine learning experiments that users perform using mlFlow. So is the thing labeled pipeline the estimator in the picture above? In addition, I read on the scikit-learn documentation that the HTML representation is automatically displayed, and I did not notice this being disabled anywhere. If the visualization is already displayed for the users to see, then how is copying the code for the visualization into the autologger beneficial for users? 

Again, I'm likely just misinterpreting something, I just want to fully wrap my head around the issue, and I really appreciate all your help in doing so.

As for the explanation of testing, thanks again for that it helped clarify everything.
@harupy hey, I just made the according changes, should I make a new branch and make a PR?

Resolving patch:
diff --git a/mlflow/sklearn/utils.py b/mlflow/sklearn/utils.py
--- a/mlflow/sklearn/utils.py
+++ b/mlflow/sklearn/utils.py
@@ -553,6 +553,34 @@ def _log_specialized_estimator_content(
     return metrics
 
 
+def _is_estimator_html_repr_supported():
+    import sklearn
+
+    # Only scikit-learn >= 0.23 supports `estimator_html_repr`
+    return Version(sklearn.__version__) >= Version("0.23.0")
+
+
+def _log_estimator_html(run_id, estimator):
+    if not _is_estimator_html_repr_supported():
+        return
+
+    from sklearn.utils import estimator_html_repr
+
+    # Specifies charset so triangle toggle buttons are not garbled
+    estimator_html_string = f"""
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8"/>
+  </head>
+  <body>
+    {estimator_html_repr(estimator)}
+  </body>
+</html>
+    """
+    MlflowClient().log_text(run_id, estimator_html_string, artifact_file="estimator.html")
+
+
 def _log_estimator_content(
     autologging_client,
     estimator,
@@ -613,7 +641,7 @@ def _log_estimator_content(
             score_key = prefix + "score"
             autologging_client.log_metrics(run_id=run_id, metrics={score_key: score})
             metrics[score_key] = score
-
+    _log_estimator_html(run_id, estimator)
     return metrics
 
 
diff --git a/mlflow/tracking/client.py b/mlflow/tracking/client.py
--- a/mlflow/tracking/client.py
+++ b/mlflow/tracking/client.py
@@ -1167,7 +1167,7 @@ def log_text(self, run_id: str, text: str, artifact_file: str) -> None:
             client.log_text(run.info.run_id, "<h1>header</h1>", "index.html")
         """
         with self._log_artifact_helper(run_id, artifact_file) as tmp_path:
-            with open(tmp_path, "w") as f:
+            with open(tmp_path, "w", encoding="utf-8") as f:
                 f.write(text)
 
     def log_dict(self, run_id: str, dictionary: Any, artifact_file: str) -> None:


Test patch:
diff --git a/tests/sklearn/test_sklearn_autolog.py b/tests/sklearn/test_sklearn_autolog.py
--- a/tests/sklearn/test_sklearn_autolog.py
+++ b/tests/sklearn/test_sklearn_autolog.py
@@ -30,6 +30,7 @@
     _is_plotting_supported,
     _get_arg_names,
     _log_child_runs_info,
+    _is_estimator_html_repr_supported,
 )
 from mlflow.utils import _truncate_dict
 from mlflow.utils.mlflow_tags import MLFLOW_AUTOLOGGING
@@ -1201,7 +1202,7 @@ def test_eval_and_log_metrics_for_regressor():
     run_id = run.info.run_id
     _, metrics, _, artifacts = get_run_data(run_id)
     assert metrics == eval_metrics
-    assert len(artifacts) == 0
+    assert artifacts == (["estimator.html"] if _is_estimator_html_repr_supported() else [])
 
 
 def test_eval_and_log_metrics_for_binary_classifier():
@@ -1247,7 +1248,7 @@ def test_eval_and_log_metrics_for_binary_classifier():
         )
     assert eval_metrics == expected_metrics
 
-    eval_artifacts = []
+    eval_artifacts = ["estimator.html"] if _is_estimator_html_repr_supported() else []
     if _is_plotting_supported():
         eval_artifacts.extend(
             [
@@ -1304,7 +1305,7 @@ def test_eval_and_log_metrics_for_binary_classifier_with_pos_label():
         )
     assert eval_metrics == pytest.approx(expected_metrics)
 
-    eval_artifacts = []
+    eval_artifacts = ["estimator.html"] if _is_estimator_html_repr_supported() else []
     if _is_plotting_supported():
         eval_artifacts.extend(
             [
@@ -1430,9 +1431,9 @@ def test_eval_and_log_metrics_for_classifier_multi_class():
 
     assert eval_metrics == expected_metrics
 
-    eval_artifacts = []
+    eval_artifacts = ["estimator.html"] if _is_estimator_html_repr_supported() else []
     if _is_plotting_supported():
-        eval_artifacts = ["{}.png".format("eval_confusion_matrix")]
+        eval_artifacts.append("{}.png".format("eval_confusion_matrix"))
 
     # Check that the logged metrics/artifacts are the same as the ones returned by the method.
     run_id = run.info.run_id
@@ -1459,14 +1460,14 @@ def test_eval_and_log_metrics_with_estimator(fit_func_name):
         )
 
     # Check contents of returned artifacts/metrics
-    assert eval_metrics == {"eval_score": model.score(X_eval, y_eval)}
+    np.testing.assert_allclose(eval_metrics["eval_score"], model.score(X_eval, y_eval))
 
     # Check that the logged metrics are the same as returned by the method.
     run_id = run.info.run_id
     _, metrics, _, artifacts = get_run_data(run_id)
 
     assert metrics == eval_metrics
-    assert len(artifacts) == 0
+    assert artifacts == (["estimator.html"] if _is_estimator_html_repr_supported() else [])
 
 
 def test_eval_and_log_metrics_with_meta_estimator():
@@ -1491,7 +1492,9 @@ def test_eval_and_log_metrics_with_meta_estimator():
             model=model, X=X_eval, y_true=y_eval, prefix="eval_"
         )
 
-    eval_artifacts = ["{}.png".format("eval_confusion_matrix")] if _is_plotting_supported() else []
+    eval_artifacts = ["estimator.html"] if _is_estimator_html_repr_supported() else []
+    if _is_plotting_supported():
+        eval_artifacts.append("{}.png".format("eval_confusion_matrix"))
 
     # Check that the logged metrics/artifacts for the run are exactly those returned by the call
     run_id = run.info.run_id
@@ -1543,7 +1546,7 @@ def test_eval_and_log_metrics_with_new_run():
     run_id = mlflow.active_run().info.run_id
     _, metrics, _, artifacts = get_run_data(run_id)
     assert eval_metrics == metrics
-    assert len(artifacts) == 0
+    assert artifacts == (["estimator.html"] if _is_estimator_html_repr_supported() else [])
     mlflow.end_run()
 
 
@@ -1572,7 +1575,7 @@ def predict(self, X):
     # No artifacts should be generated
     assert len(metrics) == 0
     assert len(eval_metrics) == 0
-    assert len(artifacts) == 0
+    assert artifacts == (["estimator.html"] if _is_estimator_html_repr_supported() else [])
 
 
 def test_eval_and_log_metrics_throws_with_invalid_args():
@@ -2057,7 +2060,10 @@ def test_autolog_print_warning_if_custom_estimator_pickling_raise_error():
 
     run_id = run.info.run_id
     params, metrics, tags, artifacts = get_run_data(run_id)
-    assert len(params) > 0 and len(metrics) > 0 and len(tags) > 0 and artifacts == []
+    assert len(params) > 0
+    assert len(metrics) > 0
+    assert len(tags) > 0
+    assert artifacts == (["estimator.html"] if _is_estimator_html_repr_supported() else [])
 
 
 def test_autolog_registering_model():