Repository: mlflow/mlflow
Pull request: Automatically update metric plots for in-progress runs #2099
Signed-off-by: Cedric Koffeto cedkoffeto@gmail.com

## What changes are proposed in this pull request?

setInterval function added in metricsPlotPanel component which retrieves execution status and metrics history from  API  every 8 seconds to update the plot and automatically stops when all executions are complete. closes #2099

## How is this patch tested?

- Unit tests
- Manually verified the proposed feature works correctly using this script:

```python
import time
import numpy as np
import mlflow
import multiprocessing as mp


def log(run_id, slope, repeat):
    sleep = 10
    with mlflow.start_run(run_id=run_id):
        for epoch in range(1, repeat + 1):
            print(epoch)
            mlflow.log_metric(key="metric1", value=slope * epoch * np.log(epoch), step=epoch)
            mlflow.log_metric(key="metric2", value=slope * (1 / epoch) * np.log(epoch), step=epoch)
            time.sleep(sleep)


client = mlflow.tracking.MlflowClient()
run_uuids = [client.create_run("0").info.run_id for _ in range(2)]
runs_param = "[" + ",".join(map(lambda s: f"%22{s}%22", run_uuids)) + "]"

print(
    "URL:",
    r"http://localhost:3000/#/metric/metric1?runs=<<< runs_param >>>&experiment=0&plot_metric_keys=[%22metric1%22]&plot_layout={%22autosize%22:true,%22xaxis%22:{},%22yaxis%22:{}}&x_axis=step&y_axis_scale=linear&line_smoothness=1&show_point=true&deselected_curves=[]&last_linear_y_axis_range=[]".replace(
        "<<< runs_param >>>", runs_param
    ),
)

args_list = [(run_uuid, idx + 1, 5 + idx * 3) for idx, run_uuid in enumerate(run_uuids)]

with mp.Pool() as pool:
    pool.starmap(log, args_list)
```

It can be tested with a unit test to check if the plot is actually updated

## Release Notes

### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [x] Yes. Give a description of this change to be included in the release notes for MLflow users.

Automatically update metric plots for in-progress runs

### What component(s), interfaces, languages, and integrations does this PR affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [x] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
https://github.com/mlflow/mlflow/pull/5017#user-content-ui
-->
<a name="UI"></a>
### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [x] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes


Problem statement:
[FR] Automatically update metric plots for in-progress runs
## MLflow Roadmap Item

This is an MLflow Roadmap item that has been prioritized by the MLflow maintainers. We're seeking help with the implementation of roadmap items tagged with the `help wanted` label. Items tagged with the `needs design` label are awaiting a design sketch from an MLflow maintainer: @harupy.

For requirements clarifications and implementation questions, or to request a PR review, please tag @harupy in your communications related to this issue.

## Describe the proposal
Automatically update MLflow metric plots for in-progress runs periodically, enabling users to track the performance of in-progress runs without having to perform periodic page refreshes and reset UI state.

### Motivation
It would be great to be able to monitor training progress via metric graphs for a particular set of runs.

### Proposed Changes
Currently when i plot a chart say for loss of an in-progress run, the MLflow UI creates a snapshot of the current status of the training. However the chart will not automatically update with new data. The browser must be manually refreshed in order to display updated data. However, updating the chart causes certain plot state & settings to be lost.

Hints from issue discussion:
would be much appreciated, prob better by a checkbox or similar, being disabled as default, if possible avoid refresh the whole page and keep all visual settings, filters,etc
Just come in to say that this feature is much appreciated! Will make mlflow a tensorboard killer! Now I am using some auto-refresh extension in firebox but obviously it is not ideal.
@miwojc thank you for the FR and this use case make sense to me. @gioa what do you think about providing an auto-update toggle for certain visualization pages?
Thank you for FR!
Stupid question: why don't we always auto-update visualization? Is there a use case that users don't want to see the updates?
Hi everyone I am new in the open-source world and I am interested in this feature so I  tried to do it by updating the MetricPlotPanel every 5 seconds with a new query of metric history and the result I think is quite acceptable for a beginning.

It looks like this...

![image](https://user-images.githubusercontent.com/38736306/140563584-e89302fc-98e8-428a-a34b-9ce7f7dea544.png)

What do you think about it  @harupy?

Commit message: bug fix

Signed-off-by: cedric koffeto <cedkoffeto@gmail.com>

Patch to review:
diff --git a/mlflow/server/js/src/experiment-tracking/components/MetricsPlotPanel.js b/mlflow/server/js/src/experiment-tracking/components/MetricsPlotPanel.js
index cb6d4d56681e1..11440128daf5b 100644
--- a/mlflow/server/js/src/experiment-tracking/components/MetricsPlotPanel.js
+++ b/mlflow/server/js/src/experiment-tracking/components/MetricsPlotPanel.js
@@ -2,11 +2,11 @@ import React from 'react';
 import { connect } from 'react-redux';
 import Utils from '../../common/utils/Utils';
 import RequestStateWrapper from '../../common/components/RequestStateWrapper';
-import { getMetricHistoryApi } from '../actions';
+import { getMetricHistoryApi, getRunApi } from '../actions';
 import PropTypes from 'prop-types';
 import _ from 'lodash';
 import { MetricsPlotView } from './MetricsPlotView';
-import { getRunTags } from '../reducers/Reducers';
+import { getRunTags, getRunInfo } from '../reducers/Reducers';
 import {
   MetricsPlotControls,
   X_AXIS_WALL,
@@ -37,6 +37,8 @@ export class MetricsPlotPanel extends React.Component {
     location: PropTypes.object.isRequired,
     history: PropTypes.object.isRequired,
     runDisplayNames: PropTypes.arrayOf(PropTypes.string).isRequired,
+    runs: PropTypes.arrayOf(PropTypes.object),
+    getRunApi: PropTypes.func,
   };
 
   // The fields below are exposed as instance attributes rather than component state so that they
@@ -63,6 +65,8 @@ export class MetricsPlotPanel extends React.Component {
   // a single-click event.
   SINGLE_CLICK_EVENT_DELAY_MS = this.MAX_DOUBLE_CLICK_INTERVAL_MS + 10;
 
+  DURATION_BETWEEN_HISTORY_UPDATES_MS = 8000;
+
   constructor(props) {
     super(props);
     this.state = {
@@ -71,11 +75,47 @@ export class MetricsPlotPanel extends React.Component {
       popoverX: 0,
       popoverY: 0,
       popoverRunItems: [],
+      timerId: null,
     };
     this.displayPopover = false;
     this.loadMetricHistory(this.props.runUuids, this.getUrlState().selectedMetricKeys);
   }
 
+  componentDidMount() {
+    const { runs } = this.props;
+
+    if (runs && this.checkOnRunUnfinished()) {
+      const timerId = setInterval(() => {
+        let requestIds = [];
+
+        if (this.checkOnRunUnfinished()) {
+          const getRequestIds = this.loadMetricHistory(
+            this.props.runUuids,
+            this.getUrlState().selectedMetricKeys,
+          );
+          requestIds = [...requestIds, ...getRequestIds];
+
+          this.props.runs.forEach((run) => {
+            const runUuid = run.getRunUuid();
+
+            const getRunRequestId = getUUID();
+            requestIds.push(getRunRequestId);
+            this.props.getRunApi(runUuid, getRunRequestId);
+          });
+
+          this.setState((prevState) => ({
+            historyRequestIds: [...prevState.historyRequestIds, ...requestIds],
+          }));
+        } else {
+          clearInterval(this.state.timerId);
+        }
+      }, this.DURATION_BETWEEN_HISTORY_UPDATES_MS);
+
+      // eslint-disable-next-line react/no-did-mount-set-state
+      this.setState({ timerId });
+    }
+  }
+
   getUrlState() {
     return Utils.getMetricPlotStateFromUrl(this.props.location.search);
   }
@@ -459,6 +499,18 @@ export class MetricsPlotPanel extends React.Component {
     }, 300);
   };
 
+  checkOnRunUnfinished() {
+    const { runs } = this.props;
+
+    let boolean = false;
+    runs &&
+      runs.forEach((run) => {
+        if (run.getStatus() === 'RUNNING') boolean = true;
+      });
+
+    return boolean;
+  }
+
   render() {
     const { experimentId, runUuids, runDisplayNames, distinctMetricKeys, location } = this.props;
     const { popoverVisible, popoverX, popoverY, popoverRunItems } = this.state;
@@ -536,6 +588,7 @@ const mapStateToProps = (state, ownProps) => {
   const distinctMetricKeys = [...new Set(metricKeys)].sort();
 
   const runDisplayNames = [];
+  const runs = runUuids.map((runUuid) => getRunInfo(runUuid, state));
 
   // Flat array of all metrics, with history and information of the run it belongs to
   // This is used for underlying MetricsPlotView & predicting chartType for MetricsPlotControls
@@ -561,9 +614,11 @@ const mapStateToProps = (state, ownProps) => {
     latestMetricsByRunUuid,
     distinctMetricKeys,
     metricsWithRunInfoAndHistory,
+    runs,
+    state,
   };
 };
 
-const mapDispatchToProps = { getMetricHistoryApi };
+const mapDispatchToProps = { getMetricHistoryApi, getRunApi };
 
 export default withRouter(connect(mapStateToProps, mapDispatchToProps)(MetricsPlotPanel));


Code review comments:
diff_hunk: @@ -63,6 +65,8 @@ export class MetricsPlotPanel extends React.Component {
   // a single-click event.
   SINGLE_CLICK_EVENT_DELAY_MS = this.MAX_DOUBLE_CLICK_INTERVAL_MS + 10;
 
+  DURATION_BETWEEN_HISTORY_UPDATES_MS = 8000;
path: mlflow/server/js/src/experiment-tracking/components/MetricsPlotPanel.js
text: @user1:
@author Is there any specific reason that you chose 8000?

@author:
Hi @user1  
Since the refresh of the plot was not as interesting to see, 5 or 3 seconds was a little too fast so I increased to 8 because 10 was also a little too long as the waiting time so I was trying to find a trade-off

but now with your version, the refresh seems to be way better so there is no need to have a high delay.

Hope it makes sense

@user1:
I think this does depend on how often you log metrics. For example, when you train a large deep learning model and logs metrics by epoch, 3 seconds might be too short.

One approach is to monitor how long it takes for the metric to update, and dynamically change the polling duration, but this does complicate the implementation. I'd prefer to start with a constant polling duration and do something fancy later (if necessary).

@author:
@user1   I do agree with you 
So we can keep 3 or increase to 5 to handle quick updates cases (3s is ok for me btw).