Repository: mlflow/mlflow
Pull request: Added diff-view switch
Signed-off-by: Marijn Valk <marijncv@hotmail.com>

## What changes are proposed in this pull request?

Added a checkbox for switching between diff-only view and regular view. closes #4819 

## How is this patch tested?

Create a couple of runs with params, metrics and tag of which some are different accross runs and some which are the same across runs. Mark/unmark the checkbox and see the columns disappear for which the value across all the runs is the same

## Release Notes

### Is this a user-facing change?

- [ ] No. You can skip the rest of this section.
- [x] Yes. Give a description of this change to be included in the release notes for MLflow users.

Added a checkbox that will filter out all columns for which every run has the same value

### What component(s), interfaces, languages, and integrations does this PR affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [x] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

<!--
Insert an empty named anchor here to allow jumping to this section with a fragment URL
(e.g. https://github.com/mlflow/mlflow/pull/123#user-content-release-note-category).
Note that GitHub prefixes anchor names in markdown with "user-content-".
-->
<a name="release-note-category"></a>
### How should the PR be classified in the release notes? Choose one:

- [ ] `rn/breaking-change` - The PR will be mentioned in the "Breaking Changes" section
- [ ] `rn/none` - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
- [x] `rn/feature` - A new user-facing feature worth mentioning in the release notes
- [ ] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes
- [ ] `rn/documentation` - A user-facing documentation change worth mentioning in the release notes


Problem statement:
[FR] Diff-only view for runs table on experiments page
## MLflow Roadmap Item

This is an MLflow Roadmap item that has been prioritized by the MLflow maintainers. We're seeking help with the implementation of roadmap items tagged with the `help wanted` label.

For requirements clarifications and implementation questions, or to request a PR review, please tag @harupy in your communications related to this issue.

## Proposal Summary

To make it easier for users to identify relevant parameters, tags, and metrics that differ between runs, we propose introducing a "diff-only" toggle to the Runs table. When selected, the "diff-only" toggle removes all columns from the Runs table that have a shared value (or are undefined) across all runs.

## Motivation
- What is the use case for this feature? Many MLflow runs share the same value for a particular param / tag / metric. This is particularly common for runs generated by autologging. When runs have many params, tags, and metrics, it can be difficult to view and compare them in the Runs table on the MLflow Experiment UI page.
- Why is this use case valuable to support for MLflow users in general? A diff-only view will make it easier for users to explore / compare runs and identify optimal runs via the MLflow UI.
- Why is this use case valuable to support for your project(s) or organization? Many Databricks customers use the MLflow UI to compare runs and identify optimal runs / models.
- Why is it currently difficult to achieve this use case? Presently, users must manually identify columns that are irrelevant to a comparison and manually hide them from the table.

### What component(s), interfaces, languages, and integrations does this feature affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interfaces
- [X] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Languages 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

Hints from issue discussion:
Hey @dbczumar & @harupy, I created a rough draft PR to address this FR. No unit tests and not much effort yet in the visual presentation yet but I was wondering if I'm heading in the right direction with this one. Any thoughts on where to place the checkbox in relation to the other buttons/dropdowns?

Commit message: Small comment update

Signed-off-by: Marijn Valk <marijncv@hotmail.com>

Patch to review:
diff --git a/mlflow/server/js/src/experiment-tracking/components/ExperimentView.js b/mlflow/server/js/src/experiment-tracking/components/ExperimentView.js
index bc4d3581deb86..9f4f0e065e394 100644
--- a/mlflow/server/js/src/experiment-tracking/components/ExperimentView.js
+++ b/mlflow/server/js/src/experiment-tracking/components/ExperimentView.js
@@ -5,7 +5,7 @@ import { connect } from 'react-redux';
 import { injectIntl, FormattedMessage } from 'react-intl';
 // eslint-disable-next-line no-unused-vars
 import { Link, withRouter } from 'react-router-dom';
-import { Alert, Badge, Descriptions, Icon, Menu, Popover, Select, Tooltip } from 'antd';
+import { Alert, Badge, Descriptions, Icon, Menu, Popover, Select, Tooltip, Switch } from 'antd';
 
 import './ExperimentView.css';
 import { getExperimentTags, getParams, getRunInfo, getRunTags } from '../reducers/Reducers';
@@ -85,6 +85,7 @@ export class ExperimentView extends Component {
     this.handleCancelEditNote = this.handleCancelEditNote.bind(this);
     this.getStartTimeColumnDisplayName = this.getStartTimeColumnDisplayName.bind(this);
     this.onHandleStartTimeDropdown = this.onHandleStartTimeDropdown.bind(this);
+    this.handleDiffSwitchChange = this.handleDiffSwitchChange.bind(this);
     const store = ExperimentView.getLocalStore(this.props.experiment.experiment_id);
     const persistedState = new ExperimentViewPersistedState(store.loadComponentState());
     const onboardingInformationStore = ExperimentView.getLocalStore(onboarding);
@@ -96,6 +97,7 @@ export class ExperimentView extends Component {
       showFilters: false,
       showOnboardingHelper: onboardingInformationStore.getItem('showTrackingHelper') === null,
       searchInput: props.searchInput,
+      diffSwitchSelected: false,
     };
   }
   static propTypes = {
@@ -783,6 +785,21 @@ export class ExperimentView extends Component {
                       categorizedUncheckedKeys={categorizedUncheckedKeys}
                       onCheck={this.handleColumnSelectionCheck}
                     />
+                    <Tooltip
+                      title={this.props.intl.formatMessage({
+                        defaultMessage: 'Turn on to select only columns with different values',
+                        description:
+                          'Switch to select only columns with different values across runs',
+                      })}
+                    >
+                      <Switch
+                        style={{ margin: '5px' }}
+                        dataTestId='diff-switch'
+                        defaultChecked={this.state.diffSwitchSelected}
+                        unCheckedChildren='Diff'
+                        onChange={this.handleDiffSwitchChange}
+                      />
+                    </Tooltip>
                   </Spacer>
                   <Spacer direction='horizontal' size='small'>
                     <Popover
@@ -1130,6 +1147,20 @@ export class ExperimentView extends Component {
     this.initiateSearch({ modelVersionFilterInput });
   }
 
+  handleDiffSwitchChange() {
+    this.setState({ diffSwitchSelected: !this.state.diffSwitchSelected }, () => {
+      const categorizedUncheckedKeys = this.state.diffSwitchSelected
+        ? ExperimentViewUtil.getCategorizedColumnsDiffView(this.props)
+        : {
+            [COLUMN_TYPES.ATTRIBUTES]: [],
+            [COLUMN_TYPES.PARAMS]: [],
+            [COLUMN_TYPES.METRICS]: [],
+            [COLUMN_TYPES.TAGS]: [],
+          };
+      this.handleColumnSelectionCheck(categorizedUncheckedKeys);
+    });
+  }
+
   onSearch(e, searchInput) {
     if (e !== undefined) {
       e.preventDefault();
diff --git a/mlflow/server/js/src/experiment-tracking/components/ExperimentView.test.js b/mlflow/server/js/src/experiment-tracking/components/ExperimentView.test.js
index 7ef2edf8d1f2e..309e3a83e1fcb 100644
--- a/mlflow/server/js/src/experiment-tracking/components/ExperimentView.test.js
+++ b/mlflow/server/js/src/experiment-tracking/components/ExperimentView.test.js
@@ -30,6 +30,7 @@ import {
   COLUMN_SORT_BY_ASC,
   COLUMN_SORT_BY_DESC,
 } from '../constants';
+import ExperimentViewUtil from './ExperimentViewUtil';
 
 let onSearchSpy;
 
@@ -93,6 +94,14 @@ const mountExperimentViewMock = (componentProps = {}) => {
   );
 };
 
+const createTags = (tags) => {
+  // Converts {key: value, ...} to {key: RunTag(key, value), ...}
+  return Object.entries(tags).reduce(
+    (acc, [key, value]) => ({ ...acc, [key]: RunTag.fromJs({ key, value }) }),
+    {},
+  );
+};
+
 test('Should render with minimal props without exploding', () => {
   const wrapper = getExperimentViewMock();
   expect(wrapper.length).toBe(1);
@@ -181,14 +190,6 @@ describe('Download CSV', () => {
     'mlflow.user': 'user',
   };
 
-  const createTags = (tags) => {
-    // Converts {key: value, ...} to {key: RunTag(key, value), ...}
-    return Object.entries(tags).reduce(
-      (acc, [key, value]) => ({ ...acc, [key]: RunTag.fromJs({ key, value }) }),
-      {},
-    );
-  };
-
   const blobOptionExpected = { type: 'application/csv;charset=utf-8' };
   const filenameExpected = 'runs.csv';
   const startTimeStringExpected = Utils.formatTimestamp(
@@ -440,3 +441,44 @@ describe('Start time dropdown', () => {
     );
   });
 });
+
+describe('Diff Switch', () => {
+  test('handleDiffSwitchChange changes state correctly', () => {
+    const getCategorizedColumnsDiffViewSpy = jest
+      .spyOn(ExperimentViewUtil, 'getCategorizedColumnsDiffView')
+      .mockImplementation(() => {
+        return {
+          [COLUMN_TYPES.ATTRIBUTES]: [],
+          [COLUMN_TYPES.PARAMS]: [],
+          [COLUMN_TYPES.METRICS]: [],
+          [COLUMN_TYPES.TAGS]: [],
+        };
+      });
+    const handleColumnSelectionCheckSpy = jest.fn();
+    const wrapper = getExperimentViewMock();
+    const instance = wrapper.instance();
+    instance.getCategorizedColumnsDiffView = getCategorizedColumnsDiffViewSpy;
+    instance.handleColumnSelectionCheck = handleColumnSelectionCheckSpy;
+
+    // Switch turned off by default
+    expect(wrapper.state().diffSwitchSelected).toBe(false);
+
+    // Switch turned on
+    instance.handleDiffSwitchChange();
+    expect(wrapper.state().diffSwitchSelected).toBe(true);
+    expect(getCategorizedColumnsDiffViewSpy).toHaveBeenCalledTimes(1);
+    expect(handleColumnSelectionCheckSpy).toHaveBeenCalledTimes(1);
+
+    // Switch tunred off
+    instance.handleDiffSwitchChange();
+    expect(wrapper.state().diffSwitchSelected).toBe(false);
+    expect(getCategorizedColumnsDiffViewSpy).toHaveBeenCalledTimes(1);
+    expect(handleColumnSelectionCheckSpy).toHaveBeenCalledTimes(2);
+    expect(handleColumnSelectionCheckSpy).toHaveBeenLastCalledWith({
+      [COLUMN_TYPES.ATTRIBUTES]: [],
+      [COLUMN_TYPES.PARAMS]: [],
+      [COLUMN_TYPES.METRICS]: [],
+      [COLUMN_TYPES.TAGS]: [],
+    });
+  });
+});
diff --git a/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.js b/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.js
index ab5c9f32f0ac4..f227661bfb227 100644
--- a/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.js
+++ b/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.js
@@ -14,6 +14,7 @@ import {
   ATTRIBUTE_COLUMN_LABELS,
   ATTRIBUTE_COLUMN_SORT_KEY,
   DEFAULT_EXPANDED_VALUE,
+  COLUMN_TYPES,
 } from '../constants';
 
 export default class ExperimentViewUtil {
@@ -556,6 +557,78 @@ export default class ExperimentViewUtil {
     }
     return numRunsFromLatestSearch < SEARCH_MAX_RESULTS;
   }
+
+  /**
+   * Obtain the categorized columns for which the values in them
+   * have only a single value (or are undefined)
+   */
+  static getCategorizedColumnsDiffView({
+    paramKeyList,
+    metricKeyList,
+    runInfos,
+    paramsList,
+    metricsList,
+    tagsList,
+  }) {
+    const tagKeyList = Utils.getVisibleTagKeyList(tagsList);
+    const attributeKeyList = [
+      ATTRIBUTE_COLUMN_LABELS.RUN_NAME,
+      ATTRIBUTE_COLUMN_LABELS.USER,
+      ATTRIBUTE_COLUMN_LABELS.VERSION,
+    ];
+    let attributes = [];
+    let params = [];
+    let metrics = [];
+    let tags = [];
+
+    for (let index = 0, n = runInfos.length; index < n; ++index) {
+      const paramsMap = ExperimentViewUtil.toParamsMap(paramsList[index]);
+      const metricsMap = ExperimentViewUtil.toMetricsMap(metricsList[index]);
+      const tagsMap = tagsList[index];
+
+      attributes.push([
+        Utils.getRunName(tagsList[index]),
+        Utils.getUser(runInfos[index], tagsList[index]),
+        Utils.getSourceVersion(tagsList[index]),
+      ]);
+      params.push(
+        paramKeyList.map((paramKey) => {
+          return paramsMap[paramKey] ? paramsMap[paramKey].getValue() : '';
+        }),
+      );
+      metrics.push(
+        metricKeyList.map((metricKey) => {
+          return metricsMap[metricKey] ? metricsMap[metricKey].getValue() : '';
+        }),
+      );
+      tags.push(
+        tagKeyList.map((tagKey) => {
+          return tagsMap[tagKey] ? tagsMap[tagKey].getValue() : '';
+        }),
+      );
+    }
+    // Transpose the matrices so that we can evaluate the values 'column-based'
+    attributes = _.unzip(attributes);
+    params = _.unzip(params);
+    metrics = _.unzip(metrics);
+    tags = _.unzip(tags);
+    const allEqual = (arr) => arr.every((val) => val === arr[0]);
+
+    return {
+      [COLUMN_TYPES.ATTRIBUTES]: attributeKeyList.filter((v, index) => {
+        return allEqual(attributes[index]);
+      }),
+      [COLUMN_TYPES.PARAMS]: paramKeyList.filter((v, index) => {
+        return allEqual(params[index]);
+      }),
+      [COLUMN_TYPES.METRICS]: metricKeyList.filter((v, index) => {
+        return allEqual(metrics[index]);
+      }),
+      [COLUMN_TYPES.TAGS]: tagKeyList.filter((v, index) => {
+        return allEqual(tags[index]);
+      }),
+    };
+  }
 }
 
 export class TreeNode {
diff --git a/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.test.js b/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.test.js
index 4bd6fa3603bf5..584157dc8e966 100644
--- a/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.test.js
+++ b/mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.test.js
@@ -4,7 +4,9 @@ import ExperimentViewUtil, { TreeNode } from './ExperimentViewUtil';
 import { getModelVersionPageRoute } from '../../model-registry/routes';
 import { BrowserRouter } from 'react-router-dom';
 import { SEARCH_MAX_RESULTS } from '../actions';
-import { ATTRIBUTE_COLUMN_LABELS } from '../constants';
+import { ATTRIBUTE_COLUMN_LABELS, COLUMN_TYPES } from '../constants';
+import { Metric, Param, RunTag, RunInfo } from '../sdk/MlflowMessages';
+import Utils from '../../common/utils/Utils';
 
 describe('ExperimentViewUtil', () => {
   test('getCheckboxForRow should render', () => {
@@ -210,4 +212,90 @@ describe('ExperimentViewUtil', () => {
 
     expect(grandchild.isCycle()).toBeTruthy();
   });
+
+  test('getCategorizedColumnsDiffView returns the correct column keys to uncheck', () => {
+    const runInfos = [
+      RunInfo.fromJs({
+        run_uuid: 'run-id1',
+        experiment_id: '3',
+        status: 'FINISHED',
+        start_time: 1,
+        end_time: 1,
+        artifact_uri: 'dummypath',
+        lifecycle_stage: 'active',
+      }),
+      RunInfo.fromJs({
+        run_uuid: 'run-id2',
+        experiment_id: '3',
+        status: 'FINISHED',
+        start_time: 2,
+        end_time: 2,
+        artifact_uri: 'dummypath',
+        lifecycle_stage: 'active',
+      }),
+    ];
+    const paramKeyList = ['param1', 'param2', 'param3', 'param4'];
+    const metricKeyList = ['metric1', 'metric2', 'metric3', 'metric4'];
+    const paramsList = [
+      [
+        Param.fromJs({ key: 'param1', value: '1' }),
+        Param.fromJs({ key: 'param2', value: '1' }),
+        Param.fromJs({ key: 'param3', value: '1' }),
+      ],
+      [Param.fromJs({ key: 'param1', value: '1' }), Param.fromJs({ key: 'param2', value: '2' })],
+    ];
+    const metricsList = [
+      [
+        Metric.fromJs({ key: 'metric1', value: '1' }),
+        Metric.fromJs({ key: 'metric2', value: '1' }),
+        Metric.fromJs({ key: 'metric3', value: '1' }),
+      ],
+      [
+        Metric.fromJs({ key: 'metric1', value: '1' }),
+        Metric.fromJs({ key: 'metric2', value: '2' }),
+      ],
+    ];
+
+    const createTags = (tags) => {
+      // Converts {key: value, ...} to {key: RunTag(key, value), ...}
+      return Object.entries(tags).reduce(
+        (acc, [key, value]) => ({ ...acc, [key]: RunTag.fromJs({ key, value }) }),
+        {},
+      );
+    };
+    const tagsList = [
+      createTags({
+        tag1: '1',
+        tag2: '1',
+        tag3: '1',
+        [Utils.runNameTag]: 'runname1',
+        [Utils.userTag]: 'usertag1',
+        [Utils.gitCommitTag]: 'gitcommit1',
+      }),
+      createTags({
+        tag1: '1',
+        tag2: '2',
+        [Utils.runNameTag]: 'runname1',
+        [Utils.userTag]: 'usertag2',
+        [Utils.gitCommitTag]: 'gitcommit2',
+      }),
+    ];
+    const expectedUncheckedKeys = {
+      [COLUMN_TYPES.ATTRIBUTES]: [ATTRIBUTE_COLUMN_LABELS.RUN_NAME],
+      [COLUMN_TYPES.PARAMS]: ['param1', 'param4'],
+      [COLUMN_TYPES.METRICS]: ['metric1', 'metric4'],
+      [COLUMN_TYPES.TAGS]: ['tag1'],
+    };
+
+    expect(
+      ExperimentViewUtil.getCategorizedColumnsDiffView({
+        runInfos,
+        paramKeyList,
+        metricKeyList,
+        paramsList,
+        metricsList,
+        tagsList,
+      }),
+    ).toEqual(expectedUncheckedKeys);
+  });
 });


Code review comments:
diff_hunk: @@ -556,6 +557,78 @@ export default class ExperimentViewUtil {
     }
     return numRunsFromLatestSearch < SEARCH_MAX_RESULTS;
   }
+
+  /**
+   * Obtain the categorized columns for which the values in them
+   * have only a single value (or are undefined)
+   */
+  static getCategorizedColumnsDiffView({
+    paramKeyList,
+    metricKeyList,
+    runInfos,
+    paramsList,
+    metricsList,
+    tagsList,
+  }) {
+    const tagKeyList = Utils.getVisibleTagKeyList(tagsList);
+    const attributeKeyList = [
+      ATTRIBUTE_COLUMN_LABELS.RUN_NAME,
+      ATTRIBUTE_COLUMN_LABELS.USER,
+      ATTRIBUTE_COLUMN_LABELS.VERSION,
+    ];
path: mlflow/server/js/src/experiment-tracking/components/ExperimentViewUtil.js
text: @user3:
@user1 @user2 Do you think it's ok to hide these attribute columns if they only contain a single value? I think it's ok since a constant run name column doesn't seem to give us any useful information and the same thing applies to the user and version columns.


@user1:
@user3 I think we can hide these columns if they are undefined. Otherwise, we should display them. In practice, this means that we'll hide `run_name` a lot of the time, which sounds like a good idea.

@user3:
> I think we can hide these columns if they are undefined. Otherwise, we should display them.

Sounds good!