Repository: iterative/dvc

Issue:
exp show: --show-md for experiment tables
We want to be able to display tables of our experiments in markdown documents similar to how we can do it with dvc params and metrics.

Our use case is that we are conducting hyperparameter tuning jobs in CML, and would like to easily format a table of the results in PR comments.

I appreciate that just doing a `dvc exp show --show-md` could by default be an overwhelming table but the conversion would be extremely handy along with the filtering flags to get a summary that is much more manageable.


Discussion:
Good idea, @mattlbeck!

I'm curious if you see benefits beyond ease of use for doing this over inserting the output directly as a code block like:

````
echo '```' >> report.md
dvc exp show >> report.md
echo '```' >> report.md
````
@dberenbaum Honestly hadn't thought of placing inside of a code block. Presumably this only works with `--no-pager`?

Without having properly tested this, the only additional benefit of a `--show-md` I can think of is that it would look a bit nicer.
Good to know. Would you be interested in either trying that workaround and letting us know how it works, or else contributing the `--show-md` option?
No problem, I will either submit a PR or close this ticket depending on the outcome.
Thank you! One more thought: the leftmost column condenses multiple items of info (compare to the csv output) that might be hard to show in the same way in markdown. 
Also, no need to close the ticket. This is at least a pattern we should explicitly support for cml and other ci needs. 
Raw markdown would be more likely to benefit from some features of the platform where the table is being rendered (i.e. in Jupyter you would get rows highlighted on hover).

Taking Github example below, I kind of like the `--show-md` format better.

Example "code block workaround" (this renders awful on VSCode markdown extension preview, btw):

```
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Experiment ┃ Created      ┃    loss ┃ accuracy ┃ train.batch_size ┃ train.hidden_units ┃ train.dropout ┃ train.num_epochs ┃ train.lr ┃ train.conv_activation ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ workspace  │ -            │ 0.26484 │   0.9038 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ main       │ Sep 14, 2021 │ 0.26484 │   0.9038 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ 5bcd44f    │ Sep 01, 2021 │ 0.25026 │   0.9095 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ b06a6ba    │ Aug 31, 2021 │ 0.25026 │   0.9095 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ d34fd8c    │ Aug 30, 2021 │ 0.30741 │   0.8929 │ 128              │ 64                 │ 0.4           │ 10               │ 0.01     │ relu                  │
│ 02b68b7    │ Aug 29, 2021 │ 0.44604 │   0.8483 │ 128              │ 64                 │ 0.4           │ 10               │ 0.01     │ relu                  │
│ 5337519    │ Aug 28, 2021 │       - │        - │ -                │ -                  │ -             │ -                │ -        │ -                     │
└────────────┴──────────────┴─────────┴──────────┴──────────────────┴────────────────────┴───────────────┴──────────────────┴──────────┴───────────────────────┘
```

Pure markdown:

| Experiment   | Created      | loss    | accuracy   | train.batch_size   | train.hidden_units   | train.dropout   | train.num_epochs   | train.lr   | train.conv_activation   |
|--------------|--------------|---------|------------|--------------------|----------------------|-----------------|--------------------|------------|-------------------------|
| workspace    | -            | 0.26484 | 0.9038     | 128                | 64                   | 0.4             | 10                 | 0.001      | relu                    |
| main         | Sep 14, 2021 | 0.26484 | 0.9038     | 128                | 64                   | 0.4             | 10                 | 0.001      | relu                    |
| 5bcd44f      | Sep 01, 2021 | 0.25026 | 0.9095     | 128                | 64                   | 0.4             | 10                 | 0.001      | relu                    |
| b06a6ba      | Aug 31, 2021 | 0.25026 | 0.9095     | 128                | 64                   | 0.4             | 10                 | 0.001      | relu                    |
| d34fd8c      | Aug 30, 2021 | 0.30741 | 0.8929     | 128                | 64                   | 0.4             | 10                 | 0.01       | relu                    |
| 02b68b7      | Aug 29, 2021 | 0.44604 | 0.8483     | 128                | 64                   | 0.4             | 10                 | 0.01       | relu                    |
| 5337519      | Aug 28, 2021 | -       | -          | -                  | -                    | -               | -                  | -          | -                       |


Resolving patch:
diff --git a/dvc/command/experiments.py b/dvc/command/experiments.py
--- a/dvc/command/experiments.py
+++ b/dvc/command/experiments.py
@@ -382,7 +382,12 @@ def baseline_styler(typ):
 
 
 def show_experiments(
-    all_experiments, pager=True, no_timestamp=False, csv=False, **kwargs
+    all_experiments,
+    pager=True,
+    no_timestamp=False,
+    csv=False,
+    markdown=False,
+    **kwargs,
 ):
     from funcy.seqs import flatten as flatten_list
 
@@ -473,6 +478,7 @@ def show_experiments(
         header_styles=styles,
         row_styles=row_styles,
         csv=csv,
+        markdown=markdown,
     )
 
 
@@ -530,6 +536,7 @@ def run(self):
                 iso=iso,
                 pager=not self.args.no_pager,
                 csv=self.args.csv,
+                markdown=self.args.markdown,
             )
         return 0
 
@@ -989,6 +996,14 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Print output in csv format instead of a human-readable table.",
     )
+    experiments_show_parser.add_argument(
+        "--md",
+        "--show-md",
+        action="store_true",
+        default=False,
+        dest="markdown",
+        help="Show tabulated output in the Markdown format (GFM).",
+    )
     experiments_show_parser.add_argument(
         "--precision",
         type=int,


Test patch:
diff --git a/tests/unit/command/test_compat_flag.py b/tests/unit/command/test_compat_flag.py
--- a/tests/unit/command/test_compat_flag.py
+++ b/tests/unit/command/test_compat_flag.py
@@ -20,6 +20,7 @@ def _id_gen(val) -> str:
         (["experiments", "diff", "--show-md"], "markdown"),
         (["experiments", "show", "--show-json"], "json"),
         (["experiments", "show", "--show-csv"], "csv"),
+        (["experiments", "show", "--show-md"], "markdown"),
         (["ls", "--show-json", "."], "json"),
         (["metrics", "diff", "--show-json"], "json"),
         (["metrics", "diff", "--show-md"], "markdown"),
diff --git a/tests/unit/command/test_experiments.py b/tests/unit/command/test_experiments.py
--- a/tests/unit/command/test_experiments.py
+++ b/tests/unit/command/test_experiments.py
@@ -1,4 +1,5 @@
 import csv
+import textwrap
 from datetime import datetime
 
 import pytest
@@ -283,104 +284,121 @@ def test_experiments_remove(dvc, scm, mocker, queue, clear_all, remote):
     )
 
 
-all_experiments = {
-    "workspace": {
-        "baseline": {
-            "data": {
-                "timestamp": None,
-                "params": {
-                    "params.yaml": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "parent": 20170428,
-                            "train": {
-                                "n_est": 100,
-                                "min_split": 36,
-                            },
+def test_show_experiments_csv(capsys):
+    all_experiments = {
+        "workspace": {
+            "baseline": {
+                "data": {
+                    "timestamp": None,
+                    "params": {
+                        "params.yaml": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "parent": 20170428,
+                                "train": {
+                                    "n_est": 100,
+                                    "min_split": 36,
+                                },
+                            }
                         }
-                    }
-                },
-                "queued": False,
-                "running": False,
-                "executor": None,
-                "metrics": {
-                    "scores.json": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "avg_prec": 0.5843640011189556,
-                            "roc_auc": 0.9544670443829399,
+                    },
+                    "queued": False,
+                    "running": False,
+                    "executor": None,
+                    "metrics": {
+                        "scores.json": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "avg_prec": 0.5843640011189556,
+                                "roc_auc": 0.9544670443829399,
+                            }
                         }
-                    }
-                },
+                    },
+                }
             }
-        }
-    },
-    "b05eecc666734e899f79af228ff49a7ae5a18cc0": {
-        "baseline": {
-            "data": {
-                "timestamp": datetime(2021, 8, 2, 16, 48, 14),
-                "params": {
-                    "params.yaml": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "parent": 20170428,
-                            "train": {
-                                "n_est": 100,
-                                "min_split": 2,
-                            },
+        },
+        "b05eecc666734e899f79af228ff49a7ae5a18cc0": {
+            "baseline": {
+                "data": {
+                    "timestamp": datetime(2021, 8, 2, 16, 48, 14),
+                    "params": {
+                        "params.yaml": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "parent": 20170428,
+                                "train": {
+                                    "n_est": 100,
+                                    "min_split": 2,
+                                },
+                            }
                         }
-                    }
-                },
-                "queued": False,
-                "running": False,
-                "executor": None,
-                "metrics": {
-                    "scores.json": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "avg_prec": 0.5325162867864254,
-                            "roc_auc": 0.9106964878520005,
+                    },
+                    "queued": False,
+                    "running": False,
+                    "executor": None,
+                    "metrics": {
+                        "scores.json": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "avg_prec": 0.5325162867864254,
+                                "roc_auc": 0.9106964878520005,
+                            }
                         }
-                    }
-                },
-                "name": "master",
-            }
-        },
-        "ae99936461d6c3092934160f8beafe66a294f98d": {
-            "data": {
-                "timestamp": datetime(2021, 8, 31, 14, 56, 55),
-                "params": {
-                    "params.yaml": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "parent": 20170428,
-                            "train": {
-                                "n_est": 100,
-                                "min_split": 36,
-                            },
+                    },
+                    "name": "master",
+                }
+            },
+            "ae99936461d6c3092934160f8beafe66a294f98d": {
+                "data": {
+                    "timestamp": datetime(2021, 8, 31, 14, 56, 55),
+                    "params": {
+                        "params.yaml": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "parent": 20170428,
+                                "train": {
+                                    "n_est": 100,
+                                    "min_split": 36,
+                                },
+                            }
                         }
-                    }
-                },
-                "queued": True,
-                "running": True,
-                "executor": None,
-                "metrics": {
-                    "scores.json": {
-                        "data": {
-                            "featurize": {"max_features": 3000, "ngrams": 1},
-                            "avg_prec": 0.5843640011189556,
-                            "roc_auc": 0.9544670443829399,
+                    },
+                    "queued": True,
+                    "running": True,
+                    "executor": None,
+                    "metrics": {
+                        "scores.json": {
+                            "data": {
+                                "featurize": {
+                                    "max_features": 3000,
+                                    "ngrams": 1,
+                                },
+                                "avg_prec": 0.5843640011189556,
+                                "roc_auc": 0.9544670443829399,
+                            }
                         }
-                    }
-                },
-                "name": "exp-44136",
-            }
+                    },
+                    "name": "exp-44136",
+                }
+            },
         },
-    },
-}
-
+    }
 
-def test_show_experiments(capsys):
     show_experiments(
         all_experiments, precision=None, fill_value="", iso=True, csv=True
     )
@@ -408,6 +426,37 @@ def test_show_experiments(capsys):
     )
 
 
+def test_show_experiments_md(capsys):
+    all_experiments = {
+        "workspace": {
+            "baseline": {
+                "data": {
+                    "timestamp": None,
+                    "params": {"params.yaml": {"data": {"foo": 1}}},
+                    "queued": False,
+                    "running": False,
+                    "executor": None,
+                    "metrics": {
+                        "scores.json": {"data": {"bar": 0.9544670443829399}}
+                    },
+                }
+            }
+        },
+    }
+    show_experiments(
+        all_experiments, precision=None, fill_value="", iso=True, markdown=True
+    )
+    cap = capsys.readouterr()
+
+    assert cap.out == textwrap.dedent(
+        """\
+        | Experiment   | Created   | bar                | foo   |
+        |--------------|-----------|--------------------|-------|
+        | workspace    |           | 0.9544670443829399 | 1     |\n
+    """
+    )
+
+
 @pytest.mark.parametrize("sort_order", ["asc", "desc"])
 def test_show_experiments_sort_by(capsys, sort_order):
     sort_experiments = {