Repository: iterative/dvc

Issue:
plots: return error messages for failed plots
## Description and Motivation

If plots can't be processed we log only basic message and skip those plots in the `json` output:

```
$ dvc plots diff main workspace -o .dvc/tmp/plots --split --show-json -v --targets missclassified.jpg
```

```
DVC failed to load some plots for following revisions: 'workspace, main'.
{
  "missclassified.jpg": []
}
```

We need to have better results, granular messages about failed plots so that we can show in VS Code properly instead of silently ignoring it, see 

> It's related to this issues - https://github.com/iterative/vscode-dvc/issues/2277 and https://github.com/iterative/vscode-dvc/issues/1649 in VS Code repo. Very high level - we need to distinguish absent plots from errors and show some signal to users vs silently ignoring things and/or showing misleading messages (refresh button when there is nothing to refresh in an experiment).

## Current Output

All examples are done for multiple revisions, `--json` + `--split` flags.

<details>
<summary>Single image</summary>

```json
"eval/importance.png": [
    {
      "type": "image",
      "revisions": [
        "workspace"
      ],
      "url": "/Users/ivan/Projects/example-repos-dev/example-get-started/build/example-get-started/dvc_plots/workspace_eval_importance.png"
    },
    {
      "type": "image",
      "revisions": [
        "c475deb7448319fab434d5650264dd2dd91bad43"
      ],
      "url": "/Users/ivan/Projects/example-repos-dev/example-get-started/build/example-get-started/dvc_plots/c475deb7448319fab434d5650264dd2dd91bad43_eval_importance.png"
    },
    {
      "type": "image",
      "revisions": [
        "7e4e86ca117f1bbef288f2abebfc7c97d0a9925d"
      ],
      "url": "/Users/ivan/Projects/example-repos-dev/example-get-started/build/example-get-started/dvc_plots/7e4e86ca117f1bbef288f2abebfc7c97d0a9925d_eval_importance.png"
    }
  ]
```
</details>

<details>
<summary>Flexible (top-level) plot</summary>

```json
"dvc.yaml::Precision-Recall": [
    {
      "type": "vega",
      "revisions": [
        "7e4e86ca117f1bbef288f2abebfc7c97d0a9925d",
        "c475deb7448319fab434d5650264dd2dd91bad43",
        "workspace"
      ],
      "content": {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "data": {
          "values": "<DVC_METRIC_DATA>"
        },
        "title": "dvc.yaml::Precision-Recall",
        "width": 300,
        "height": 300,
        "mark": {
          "type": "line",
          "point": true,
          "tooltip": {
            "content": "data"
          }
        },
        "encoding": {
          "x": {
            "field": "recall",
            "type": "quantitative",
            "title": "recall"
          },
          "y": {
            "field": "precision",
            "type": "quantitative",
            "title": "precision",
            "scale": {
              "zero": false
            }
          },
          "color": {
            "field": "rev",
            "type": "nominal"
          }
        }
      },
      "datapoints": {
        "workspace": [
          {
            "precision": 0.30321774445485783,
            "recall": 1.0,
            "threshold": 0.0,
            "dvc_data_version_info": {
              "revision": "workspace",
              "filename": "eval/prc/train.json",
              "field": "precision"
            }
          },
         {"...."},
         {
            "precision": 0.6694635900509439,
            "recall": 0.9359028068705488,
            "threshold": 0.20869278966952978,
            "dvc_data_version_info": {
              "revision": "workspace",
              "filename": "eval/prc/test.json",
              "field": "precision"
            }
          }
```

</details>

<details>
<summary>Multiple images</summary>

```json
"mispredicted/croissant/muffin-16115-13825-26827-1d8e67e0bffdfebcdb3b337787823ab6.jpeg": [
    {
      "type": "image",
      "revisions": [
        "workspace"
      ],
      "url": "/Users/ivan/Projects/hackathon/dvc_plots/workspace_mispredicted_croissant_muffin-16115-13825-26827-1d8e67e0bffdfebcdb3b337787823ab6.jpeg"
    }
  ],
  "mispredicted/muffin/croissant-0295ed7610487b3118febb5563bc58fd.jpg": [
    {
      "type": "image",
      "revisions": [
        "workspace"
      ],
      "url": "/Users/ivan/Projects/hackathon/dvc_plots/workspace_mispredicted_muffin_croissant-0295ed7610487b3118febb5563bc58fd.jpg"
    }
  ],
  "mispredicted/muffin/croissant-3f488b602f2a668e-3fd6af132b0dceafe014dfcf7809d2ff.jpg": [
    {
      "type": "image",
      "revisions": [
        "workspace"
      ],
      "url": "/Users/ivan/Projects/hackathon/dvc_plots/workspace_mispredicted_muffin_croissant-3f488b602f2a668e-3fd6af132b0dceafe014dfcf7809d2ff.jpg"
    }
  ],
  "mispredicted/muffin/dog-bec8602c36317744-1827c47f8ae15e6a3c4ee660035781a4.jpg": [
    {
      "type": "image",
      "revisions": [
        "workspace"
      ],
      "url": "/Users/ivan/Projects/hackathon/dvc_plots/workspace_mispredicted_muffin_dog-bec8602c36317744-1827c47f8ae15e6a3c4ee660035781a4.jpg"
    }
  ]
```
</details>

<details>
<summary>Stage linear plot</summary>

```json
"dvclive/scalars/eval/loss.tsv": [
    {
      "type": "vega",
      "revisions": [
        "d82452a"
      ],
      "content": {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "data": {
          "values": "<DVC_METRIC_DATA>"
        },
        "title": "dvclive/scalars/eval/loss.tsv",
        "width": 300,
        "height": 300,
        "mark": {
          "type": "line",
          "point": true,
          "tooltip": {
            "content": "data"
          }
        },
        "encoding": {
          "x": {
            "field": "step",
            "type": "quantitative",
            "title": "step"
          },
          "y": {
            "field": "eval/loss",
            "type": "quantitative",
            "title": "eval/loss",
            "scale": {
              "zero": false
            }
          },
          "color": {
            "field": "rev",
            "type": "nominal"
          }
        }
      },
      "datapoints": {
        "d82452a": [
          {
            "timestamp": "1660180711394",
            "step": "0",
            "eval/loss": "2.4602549076080322",
            "dvc_data_version_info": {
              "revision": "d82452a",
              "filename": "dvclive/scalars/eval/loss.tsv",
              "field": "eval/loss"
            }
          },
          {
            "timestamp": "1660180723400",
            "step": "1",
            "eval/loss": "1.3761318922042847",
            "dvc_data_version_info": {
              "revision": "d82452a",
              "filename": "dvclive/scalars/eval/loss.tsv",
              "field": "eval/loss"
            }
          }
        ]
      }
    }
  ],
```
</details>

## Unblocks, Related

https://github.com/iterative/vscode-dvc/issues/2277
https://github.com/iterative/vscode-dvc/issues/1649

## Next Steps

- [x] A bit of research. JSON structure looks extremely suboptimal (tons of duplication), since we are changing it, I'd like to have a bit better understanding of how it's being used. Entry point into VS Code is [here](https://github.com/iterative/vscode-dvc/blob/main/extension/src/plots/model/index.ts#L108).
- [ ] ⌛ Try to add an error for a single image
- [ ] Classify and suggest how to add errors in all other cases - including directories, regular plots (e.g. linear).



Discussion:
What will be the `error` data used for? Do we only want to pass the error message to the user? Or will there be some logic involved with processing the errors on `vs-code` side?
We'll be processing the errors before displaying anything to the user. Would be good to have a way to identify where certain revisions are missing data due to errors (as in the example provided in https://github.com/iterative/dvc/issues/7691).
@pared Any progress on this one?
No, but I believe we could include it as a part of implementing iterative/vscode-dvc#1757
@pared Any updates on looking into this one?
I consider it as a part of aforementioned issue on `vscode` - but the estimation for vscode depends on research on studio side. It is not yet finished.
Note to self: since returning errors will probably require data structure change, we need to remember to get rid of filling `rev` in datapoints - as vscode sometimes need to assign their own revision (eg 'main' vs short sha of main).
I didn't left any comment during research, so:
We were able to implement top level plots basing on old data format. In order to support errors we will need to change the data structure returned by `dvc plots ... --json`.
@mattseddon could you please share the location of the code that parses the `--json` result for plots on our end.

@dberenbaum do we know if anyone else besides vs code depends on `--json`?

- [ ] ⌛ A bit of research. JSON structure looks extremely suboptimal (tons of duplication), since we are changing it, I'd like to have a bit better understanding of how it's being used.
- [ ] Another question to answer and agree on how we process directories (if the whole plot dir can't be expanded and we don't know what files it has we can't send an error per file then, we'll have to send it per directory)
- [ ] Change the output

Use https://github.com/iterative/vscode-dvc/blob/main/extension/src/plots/model/index.ts#L108 as an entry point.
<img width="655" alt="Screen Shot 2023-01-15 at 6 23 00 PM" src="https://user-images.githubusercontent.com/3659196/212585125-0c931661-fe59-45d5-9467-1e582d2ec4be.png">

I see that data collection depends on the `datapoints` field and is not using `data` in the content: 

https://github.com/iterative/vscode-dvc/blob/main/extension/src/plots/model/collect.ts#L372
https://github.com/iterative/vscode-dvc/blob/main/extension/src/plots/model/collect.ts#L423

@mattseddon do you remember from the top of your head if we need `data` in the template, it looks identical (at least in the sample I have), do we need it in VS Code? And why did we decide to keep both (e.g. why don't we parse `plot.content.data` instead of `datapoints`).

Are there some proposals, tickets, PRs for the plots JSON format?

I do not think that we need it.
> Are there some proposals, tickets, PRs for the plots JSON format?

The original PR is here: https://github.com/iterative/dvc/pull/7367. From reading the description it looks like the data being duplicated is a bug for the `--split` flag.
> @dberenbaum do we know if anyone else besides vs code depends on `--json`?

No, I don't think so.

For the duplicated data, I'm missing something because I have different output from what @shcheklein shows above. I don't see all the data in `content.data.values`. My output for `dvc plots diff 504206e f586d67 workspace -o .dvc/tmp/plots --split --json` looks like this:

```yaml
{
  "dvc.yaml::Accuracy": [
    {
      "type": "vega",
      "revisions": [
        "504206e",
        "f586d67",
        "workspace"
      ],
      "content": {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "data": {
          "values": "<DVC_METRIC_DATA>" # Nothing else shows up in this field.
        },
...
```
@dberenbaum my bad, I didn't use `split` I think. I wonder what's the purpose of `datapoints` in the non-split mode then? (not critical I think at all, since JSON is not used anywhere now).
Updated the description - some examples of the current output. Next - try to add an error for an image plot (not directory with images_ case for now (`example-get-started`'s `importance.png`).
In general, I find returning errors to be a mistake. It increases a lot of maintenance burden, for which we are not ready internally.
@skshetry it's a bad user experience to not show anything at all in case something fails and since we. I think if it's done right it won't be a bigger burden at all and code doesn't have to be complicated. We already have this data we just need to propagate it (I think so at least, I can be wrong). And to clarify, we don't talk here about processing specific types of errors, we just need a signal that plot exists in a revision and that it can't be loaded for some reason.

On the maintenance side - I think the whole plots logic and related index part should be the first thing to improve. E.g. after the last refactoring we still have two plot accessors (`_plots` and `plots`), still some custom collect logic, a lot of logic with path manipulations, old code (like output plots, etc) - those are points that should be remove, refactored, etc to make it lighter and simpler.
>  We already have this data we just need to propagate it

That's where the complexity is, right? It's easy to log or suppress but extremely hard to propagate up. We need small sets of APIs at the high level where we do this. At the moment we are spreading this logic to deep layers which increases the burden.

I think there should be a symmetry between the product and the engineering side, and here I think the expectation on the product side is too high (or, was too high). :)
> That's where the complexity is, right? 

Doesn't have to be. Sometimes dropping some code (that removes and / or transforms things) instead of exposing them directly (which might be just fine in this case) can simplify it. We'll see how it goes. I definitely want to avoid adding tons of custom code for this.

> I think there should be a symmetry between the product and the engineering side, and here I think the expectation on the product side is too high (or, was too high). :)

I think it's a wrong dichotomy in this case. I'm not sure if it's possible to do it now w/o complicating things. It's definitely doesn't add much complexity to do this from scratch. If we had the standard in mind (it's not high at all) we would have spent some small additional percent of time.

Product expectation - we talk about VS Code, right (that's what I have in mind in the first place), not DVC? Just in case. I'm fine (more or less) for DVC to return a generic error (and write something in logs). In VS Code it leads to bad experience. It's not top priority (that's why I'm doing this in background), but it can and should be fixed. And we should have a higher standard for out products.
For visibility: got distracted by some other plots issues (broken smooth templates, new DVCLive release) and didn't have capacity for this hands on work (which is not a lot of time by default). I'll try to get back to this asap.

Some design decisions that are tricky here. If we have a plot directory we expand each file in that directory as its own plot when we return the result. It's fine. The problem is that we don't know the layout if we can't download the `.dir` in the first place. So, for these granular plots - we can't communicate errors at all- we don't know for sure if they exist or not in the directory in the failed revision. We'll have assume that they don't I guess + communicate that we were not able to process the whole directory.


@shcheklein Can you clarify the full scope of the issue? Is it only about plot directories, or is that merely one case you are trying to solve for?
Yes, @dberenbaum . It's related to this issues - https://github.com/iterative/vscode-dvc/issues/2277 and https://github.com/iterative/vscode-dvc/issues/1649 in VS Code repo. Very high level - we need to distinguish absent plots from errors and show some signal to users vs silently ignoring things and/or showing misleading messages (refresh button when there is nothing to refresh in an experiment).

> Can you clarify the full scope of the issue? Is it only about plot directories, or is that merely one case you are trying to solve for?

Thus: The full scope: show error message for all plot definitions, not only directories / images. 
@skshetry Can you follow up with questions you have, and @shcheklein and I can respond to define the scope better? By next week when you are finished with support duty, let's try to have a solid plan and estimate 🙏 .
I could not look into this during support duty, as some p0s/bugs came.
More related issues:
* https://github.com/iterative/vscode-dvc/issues/3222
* #7887
We do seem to preserve errors during `plots.collect()`. We transform internal representation to the JSON format, where we lose most of the information. We could start with exposing that, what would be a good json format for incorporating errors for vscode?
> We could start with exposing that, what would be a good json format for incorporating errors for vscode?

@skshetry 🤔 tbh I don't think VS Code requires anything specific here. We should come up with a decent general format for this data. We can adjust VS Code if needed. 
I think what we've learned is that it's helpful to share drafts early and often to get feedback as you go so we know mostly what works in both products by the time we are ready to merge.

Resolving patch:
diff --git a/dvc/commands/plots.py b/dvc/commands/plots.py
--- a/dvc/commands/plots.py
+++ b/dvc/commands/plots.py
@@ -1,8 +1,9 @@
 import argparse
 import logging
 import os
+from typing import TYPE_CHECKING, Dict, List
 
-from funcy import first
+from funcy import compact, first
 
 from dvc.cli import completion
 from dvc.cli.command import CmdBase
@@ -11,14 +12,43 @@
 from dvc.ui import ui
 from dvc.utils import format_link
 
+if TYPE_CHECKING:
+    from dvc.render.match import RendererWithErrors
+
+
 logger = logging.getLogger(__name__)
 
 
-def _show_json(renderers, split=False):
+def _show_json(renderers_with_errors: List["RendererWithErrors"], split=False):
     from dvc.render.convert import to_json
+    from dvc.utils.serialize import encode_exception
+
+    errors: List[Dict] = []
+    data = {}
+
+    for renderer, src_errors, def_errors in renderers_with_errors:
+        name = renderer.name
+        data[name] = to_json(renderer, split)
+        errors.extend(
+            {
+                "name": name,
+                "rev": rev,
+                "source": source,
+                **encode_exception(e),
+            }
+            for rev, per_rev_src_errors in src_errors.items()
+            for source, e in per_rev_src_errors.items()
+        )
+        errors.extend(
+            {
+                "name": name,
+                "rev": rev,
+                **encode_exception(e),
+            }
+            for rev, e in def_errors.items()
+        )
 
-    result = {renderer.name: to_json(renderer, split) for renderer in renderers}
-    ui.write_json(result, highlight=False)
+    ui.write_json(compact({"errors": errors, "data": data}), highlight=False)
 
 
 def _adjust_vega_renderers(renderers):
@@ -122,15 +152,16 @@ def run(self) -> int:  # noqa: C901, PLR0911, PLR0912
 
             renderers_out = out if self.args.json else os.path.join(out, "static")
 
-            renderers = match_defs_renderers(
+            renderers_with_errors = match_defs_renderers(
                 data=plots_data,
                 out=renderers_out,
                 templates_dir=self.repo.plots.templates_dir,
             )
             if self.args.json:
-                _show_json(renderers, self.args.split)
+                _show_json(renderers_with_errors, self.args.split)
                 return 0
 
+            renderers = [r.renderer for r in renderers_with_errors]
             _adjust_vega_renderers(renderers)
             if self.args.show_vega:
                 renderer = first(filter(lambda r: r.TYPE == "vega", renderers))
diff --git a/dvc/render/match.py b/dvc/render/match.py
--- a/dvc/render/match.py
+++ b/dvc/render/match.py
@@ -1,10 +1,10 @@
 import os
 from collections import defaultdict
-from typing import TYPE_CHECKING, Dict, List, Optional
+from typing import TYPE_CHECKING, DefaultDict, Dict, List, NamedTuple, Optional
 
 import dpath
 import dpath.options
-from funcy import last
+from funcy import get_in, last
 
 from dvc.repo.plots import _normpath, infer_data_sources
 from dvc.utils.plots import get_plot_id
@@ -13,6 +13,8 @@
 
 if TYPE_CHECKING:
     from dvc.types import StrPath
+    from dvc_render.base import Renderer
+
 
 dpath.options.ALLOW_EMPTY_STRING_KEYS = True
 
@@ -61,21 +63,31 @@ def get_definition_data(self, target_files, rev):
         return result
 
 
-def match_defs_renderers(
+class RendererWithErrors(NamedTuple):
+    renderer: "Renderer"
+    source_errors: Dict[str, Dict[str, Exception]]
+    definition_errors: Dict[str, Exception]
+
+
+def match_defs_renderers(  # noqa: C901, PLR0912
     data,
     out=None,
     templates_dir: Optional["StrPath"] = None,
-):
+) -> List[RendererWithErrors]:
     from dvc_render import ImageRenderer, VegaRenderer
 
     plots_data = PlotsData(data)
     renderers = []
     renderer_cls = None
+
     for plot_id, group in plots_data.group_definitions().items():
         plot_datapoints: List[Dict] = []
         props = _squash_plots_properties(group)
         final_props: Dict = {}
 
+        def_errors: Dict[str, Exception] = {}
+        src_errors: DefaultDict[str, Dict[str, Exception]] = defaultdict(dict)
+
         if out is not None:
             props["out"] = out
         if templates_dir is not None:
@@ -94,13 +106,24 @@ def match_defs_renderers(
 
             converter = _get_converter(renderer_cls, inner_id, props, definitions_data)
 
-            dps, rev_props = converter.flat_datapoints(rev)
+            for src in plot_sources:
+                if error := get_in(data, [rev, "sources", "data", src, "error"]):
+                    src_errors[rev][src] = error
+
+            try:
+                dps, rev_props = converter.flat_datapoints(rev)
+            except Exception as e:  # noqa: BLE001, pylint: disable=broad-except
+                def_errors[rev] = e
+                continue
+
             if not final_props and rev_props:
                 final_props = rev_props
             plot_datapoints.extend(dps)
 
         if "title" not in final_props:
             final_props["title"] = renderer_id
+
         if renderer_cls is not None:
-            renderers.append(renderer_cls(plot_datapoints, renderer_id, **final_props))
+            renderer = renderer_cls(plot_datapoints, renderer_id, **final_props)
+            renderers.append(RendererWithErrors(renderer, dict(src_errors), def_errors))
     return renderers
diff --git a/dvc/repo/plots/__init__.py b/dvc/repo/plots/__init__.py
--- a/dvc/repo/plots/__init__.py
+++ b/dvc/repo/plots/__init__.py
@@ -20,7 +20,7 @@
 
 import dpath
 import dpath.options
-from funcy import distinct, first, project, reraise
+from funcy import first, ldistinct, project, reraise
 
 from dvc.exceptions import DvcException
 from dvc.utils import error_handler, errored_revisions, onerror_collect
@@ -356,7 +356,7 @@ def infer_data_sources(plot_id, config=None):
     if isinstance(x, dict):
         sources.append(first(x.keys()))
 
-    return distinct(source for source in sources)
+    return ldistinct(source for source in sources)
 
 
 def _matches(targets, config_file, plot_id):


Test patch:
diff --git a/tests/func/plots/test_show.py b/tests/func/plots/test_show.py
--- a/tests/func/plots/test_show.py
+++ b/tests/func/plots/test_show.py
@@ -418,5 +418,8 @@ def test_show_plots_defined_with_native_os_path(tmp_dir, dvc, scm, capsys):
     assert main(["plots", "show", "--json"]) == 0
     out, _ = capsys.readouterr()
     json_out = json.loads(out)
-    assert json_out[f"dvc.yaml::{top_level_plot}"]
-    assert json_out[stage_plot]
+    assert "errors" not in json_out
+
+    json_data = json_out["data"]
+    assert json_data[f"dvc.yaml::{top_level_plot}"]
+    assert json_data[stage_plot]
diff --git a/tests/integration/plots/test_plots.py b/tests/integration/plots/test_plots.py
--- a/tests/integration/plots/test_plots.py
+++ b/tests/integration/plots/test_plots.py
@@ -182,7 +182,13 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
     html_path, json_result, split_json_result = call(capsys)
     html_result = extract_vega_specs(html_path, ["linear.json", "confusion.json"])
 
-    assert json_result["linear.json"][0]["content"]["data"][
+    assert "errors" not in json_result
+    assert "errors" not in split_json_result
+
+    json_data = json_result["data"]
+    split_json_data = split_json_result["data"]
+
+    assert json_data["linear.json"][0]["content"]["data"][
         "values"
     ] == _update_datapoints(
         linear_v1,
@@ -200,7 +206,7 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             REVISION_FIELD: "workspace",
         },
     )
-    assert json_result["confusion.json"][0]["content"]["data"][
+    assert json_data["confusion.json"][0]["content"]["data"][
         "values"
     ] == _update_datapoints(
         confusion_v1,
@@ -218,34 +224,40 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             REVISION_FIELD: "workspace",
         },
     )
-    verify_image(tmp_dir, "workspace", "image.png", image_v1, html_path, json_result)
+    verify_image(tmp_dir, "workspace", "image.png", image_v1, html_path, json_data)
 
     for plot in ["linear.json", "confusion.json"]:
         verify_vega(
             "workspace",
             html_result[plot],
-            json_result[plot],
-            split_json_result[plot],
+            json_data[plot],
+            split_json_data[plot],
         )
 
-    verify_vega_props("confusion.json", json_result, **confusion_props)
+    verify_vega_props("confusion.json", json_data, **confusion_props)
 
     image_v2, linear_v2, confusion_v2, confusion_props = next(repo_state)
 
     html_path, json_result, split_json_result = call(capsys, subcommand="diff")
     html_result = extract_vega_specs(html_path, ["linear.json", "confusion.json"])
 
-    verify_image(tmp_dir, "workspace", "image.png", image_v2, html_path, json_result)
-    verify_image(tmp_dir, "HEAD", "image.png", image_v1, html_path, json_result)
+    assert "errors" not in json_result
+    assert "errors" not in split_json_result
+
+    json_data = json_result["data"]
+    split_json_data = split_json_result["data"]
+
+    verify_image(tmp_dir, "workspace", "image.png", image_v2, html_path, json_data)
+    verify_image(tmp_dir, "HEAD", "image.png", image_v1, html_path, json_data)
 
     for plot in ["linear.json", "confusion.json"]:
         verify_vega(
             ["HEAD", "workspace"],
             html_result[plot],
-            json_result[plot],
-            split_json_result[plot],
+            json_data[plot],
+            split_json_data[plot],
         )
-    verify_vega_props("confusion.json", json_result, **confusion_props)
+    verify_vega_props("confusion.json", json_data, **confusion_props)
     path = tmp_dir / "subdir"
     path.mkdir()
     with path.chdir():
@@ -254,7 +266,13 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             html_path,
             ["../linear.json", "../confusion.json"],
         )
-        assert json_result["../linear.json"][0]["content"]["data"][
+
+        assert "errors" not in json_result
+        assert "errors" not in split_json_result
+
+        json_data = json_result["data"]
+        split_json_data = split_json_result["data"]
+        assert json_data["../linear.json"][0]["content"]["data"][
             "values"
         ] == _update_datapoints(
             linear_v2,
@@ -286,7 +304,7 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
                 REVISION_FIELD: "HEAD",
             },
         )
-        assert json_result["../confusion.json"][0]["content"]["data"][
+        assert json_data["../confusion.json"][0]["content"]["data"][
             "values"
         ] == _update_datapoints(
             confusion_v2,
@@ -326,8 +344,8 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             verify_vega(
                 ["HEAD", "workspace"],
                 html_result[plot],
-                json_result[plot],
-                split_json_result[plot],
+                json_data[plot],
+                split_json_data[plot],
             )
         verify_image(
             path,
@@ -335,7 +353,7 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             "../image.png",
             image_v2,
             html_path,
-            json_result,
+            json_data,
         )
         verify_image(
             path,
@@ -343,7 +361,7 @@ def test_repo_with_plots(tmp_dir, scm, dvc, capsys, run_copy_metrics, repo_with_
             "../image.png",
             image_v1,
             html_path,
-            json_result,
+            json_data,
         )
 
 
@@ -361,13 +379,30 @@ def test_repo_with_removed_plots(tmp_dir, capsys, repo_with_plots):
 
     for s in {"show", "diff"}:
         _, json_result, split_json_result = call(capsys, subcommand=s)
-        for p in {
-            "linear.json",
-            "confusion.json",
-            "image.png",
-        }:
-            assert json_result[p] == []
-            assert split_json_result[p] == []
+        errors = [
+            {
+                "name": p,
+                "source": p,
+                "rev": "workspace",
+                "type": "FileNotFoundError",
+                "msg": "",
+            }
+            for p in [
+                "linear.json",
+                "confusion.json",
+                "image.png",
+            ]
+        ]
+        expected_result = {
+            "errors": errors,
+            "data": {
+                "image.png": [],
+                "confusion.json": [],
+                "linear.json": [],
+            },
+        }
+        assert json_result == expected_result
+        assert split_json_result == expected_result
 
 
 def test_config_output_dir(tmp_dir, dvc, capsys):
diff --git a/tests/unit/command/test_plots.py b/tests/unit/command/test_plots.py
--- a/tests/unit/command/test_plots.py
+++ b/tests/unit/command/test_plots.py
@@ -7,6 +7,7 @@
 
 from dvc.cli import parse_args
 from dvc.commands.plots import CmdPlotsDiff, CmdPlotsShow, CmdPlotsTemplates
+from dvc.render.match import RendererWithErrors
 
 
 @pytest.fixture
@@ -268,8 +269,11 @@ def test_should_call_render(tmp_dir, mocker, capsys, plots_data, output):
 
     output = output or "dvc_plots"
     index_path = tmp_dir / output / "index.html"
-    renderers = mocker.MagicMock()
-    mocker.patch("dvc.render.match.match_defs_renderers", return_value=renderers)
+    renderer = mocker.MagicMock()
+    mocker.patch(
+        "dvc.render.match.match_defs_renderers",
+        return_value=[RendererWithErrors(renderer, {}, {})],
+    )
     render_mock = mocker.patch("dvc_render.render_html", return_value=index_path)
 
     assert cmd.run() == 0
@@ -278,7 +282,7 @@ def test_should_call_render(tmp_dir, mocker, capsys, plots_data, output):
     assert index_path.as_uri() in out
 
     render_mock.assert_called_once_with(
-        renderers=renderers,
+        renderers=[renderer],
         output_file=Path(tmp_dir / output / "index.html"),
         html_template=None,
     )
@@ -354,12 +358,13 @@ def test_show_json(split, mocker, capsys):
     import dvc.commands.plots
 
     renderer = mocker.MagicMock()
+    renderer_obj = RendererWithErrors(renderer, {}, {})
     renderer.name = "rname"
     to_json_mock = mocker.patch(
         "dvc.render.convert.to_json", return_value={"renderer": "json"}
     )
 
-    dvc.commands.plots._show_json([renderer], split)
+    dvc.commands.plots._show_json([renderer_obj], split)
 
     to_json_mock.assert_called_once_with(renderer, split)
 
diff --git a/tests/unit/render/test_match.py b/tests/unit/render/test_match.py
--- a/tests/unit/render/test_match.py
+++ b/tests/unit/render/test_match.py
@@ -1,4 +1,7 @@
+from funcy import set_in
+
 from dvc.render import VERSION_FIELD
+from dvc.render.converter.vega import VegaConverter
 from dvc.render.match import PlotsData, _squash_plots_properties, match_defs_renderers
 
 
@@ -35,7 +38,7 @@ def test_group_definitions():
     }
 
 
-def test_match_renderers(mocker):
+def test_match_renderers(M):
     data = {
         "v1": {
             "definitions": {
@@ -77,9 +80,9 @@ def test_match_renderers(mocker):
         },
     }
 
-    renderers = match_defs_renderers(data)
-    assert len(renderers) == 1
-    assert renderers[0].datapoints == [
+    (renderer_with_errors,) = match_defs_renderers(data)
+    renderer = renderer_with_errors[0]
+    assert renderer.datapoints == [
         {
             VERSION_FIELD: {
                 "revision": "v1",
@@ -99,13 +102,31 @@ def test_match_renderers(mocker):
             "y": 2,
         },
     ]
-    assert renderers[0].properties == {
+    assert renderer.properties == {
         "title": "config_file_1::plot_id_1",
         "x": "x",
         "y": "y",
         "x_label": "x",
         "y_label": "y",
     }
+    assert renderer_with_errors.source_errors == {
+        "revision_with_no_data": {"file.json": M.instance_of(FileNotFoundError)}
+    }
+    assert not renderer_with_errors.definition_errors
+
+
+def test_flat_datapoints_errors_are_caught(M, mocker):
+    d = {}
+    d = set_in(
+        d,
+        ["v1", "definitions", "data", "dvc.yaml", "data", "plot_id_1"],
+        {"x": "x", "y": {"file.json": "y"}},
+    )
+    d = set_in(d, ["v1", "sources", "data", "file.json", "data"], [{"x": 1, "y": 1}])
+    mocker.patch.object(VegaConverter, "flat_datapoints", side_effect=ValueError)
+    (renderer_with_errors,) = match_defs_renderers(d)
+    assert not renderer_with_errors.source_errors
+    assert renderer_with_errors.definition_errors == {"v1": M.instance_of(ValueError)}
 
 
 def test_squash_plots_properties():