Repository: iterative/dvc

Issue:
plots smoothing
Our default plot is not very informative when the number of data points is huge (1k):
![visualization_](https://user-images.githubusercontent.com/10081867/83245308-63fc6980-a155-11ea-91d5-830f0adf65e7.png)

We need to introduce some smoothing into the default plot or create a new smoothing template. The result should look like:
![visualization (2)](https://user-images.githubusercontent.com/10081867/83245327-6a8ae100-a155-11ea-9a31-7721d056052b.png)

PS: I've done the last plot by adding `loess` transformation to the template which breaks the revision. We need to fix the revision or find another solution. 
```
    "transform": [{
        "loess": "<DVC_METRIC_Y>",
        "on": "<DVC_METRIC_X>"
    }
    ],
```


Discussion:
So, I see a few ways how we can approach that:
1. Create new template, `default_smoot` or something like that - easy, but if we decide to go that way, soon we will have many plot templates, which is probably not desired.

2. Since we already support plot [definition in pipelines file](https://github.com/iterative/dvc/pull/3911), maybe we should consider introducing `update_dict` parameter, that would be a way to pass any desired modification, that user would like to made to existing plot. So in our case, entry for plot would probably look something like:
```
copy3:
    cmd: cat data >> result3
    deps:
    - data
    plots:
    - result3:
        update_dict:
        - transform: 
              - loess: <DVC_METRIC_Y>
              - on: <DVC_METRIC_X>
```
The cons:
- in that case the user needs to come up with JSON update and transform it to yaml (since we work with Vega, I would assume that this would be the default way to go -> find a solution in vega -> apply to dvc to preserve it). That is not very convenient. Maybe we could add another `plots` command that would make this easier.
- if we decide to go this way, I would reconsider using anchors. (DVC_METRIC_Y, DVC_METRIC_X, and others). In the proposed solution user needs to learn vega -> learn how to transform -> create transformation -> convert to yaml -> learn what anchors are -> replace x/y witch anchors -> write to yaml file.
Anchors were introduced to let the user create their own HTML templates, but since `diff` already supports plotting multiple targets, and we have `--show-vega` option, I am not sure we need them anymore. 

The pros:
- this approach lets user modify the template in any way he/she wants, so basing on default template, user can modify the title, width,height, whatever comes to mind. Assuming certain level of `vega.js` knowledge, of course.
The 2nd approach makes sense for many different transformations. However, we should not use it as a default approach for smoothing since smoothing is a super common use case and we don't want to complicate the pipeline every time.

If it possible I'd add the smoothing in the default one. If not possible - create a new template.
@dmpetrov Tried adding smoothing by default, but "normal" use cases can get ugly. So I guess we should go with new template.
@pared could you please share the result with code? Were you able to solve the revision issue?
@dmpetrov sure!
So, the thing that we had to do here was add `groupby` to our transform:
```
"transform": [
{
        "loess": "<DVC_METRIC_Y>",
        "on": "<DVC_METRIC_X>",
	"groupby": ["rev"]
}
]
```

And the results for "normal" use case looks as follows (example have 100 points, evenly spaced):


![visualization (1)](https://user-images.githubusercontent.com/10195618/84248297-899c5200-ab09-11ea-9cde-73b4c406c6ca.png)

and with loess applied:
![visualization](https://user-images.githubusercontent.com/10195618/84248340-98830480-ab09-11ea-80ef-cf5f43e9f2dc.png)

@dmpetrov also, regarding the `update_params`: I agree we should implement it at some point, I could totally imagine, user preserving the original plot AND applying some transformation, just like in [this](https://vega.github.io/vega-lite/docs/loess.html#example) example.
@dmpetrov 
I played around with smoothing values and here are results for different `bandwidth` values:
- 0.01
![001](https://user-images.githubusercontent.com/10195618/84659889-63ffb600-af18-11ea-8110-e22c60a02910.png)

- 0.05
![005](https://user-images.githubusercontent.com/10195618/84659964-7da0fd80-af18-11ea-8b76-85a0d7963c21.png)

- 0.1
![01](https://user-images.githubusercontent.com/10195618/84660008-8bef1980-af18-11ea-8166-b17a22d71451.png)

- 0.2
![02](https://user-images.githubusercontent.com/10195618/84660096-a5906100-af18-11ea-9a21-cffbb8d61418.png)


It seems to me that setting default smoothing to very small value is a reasonable decision for the default plot, `0.01` case seems to affect only plot with really big amount of points. It does not require too much additional work from us, and we can deal with `update_dict` later. If we agree on that, I would mention the smoothing in docs.

To make the comparison easier I created new plots out of `12500` point case, by extracting particular amount of points from the dataset, evenly spaced. Number of were 1250, 125, 25:

- 0.01
![new_001](https://user-images.githubusercontent.com/10195618/84666893-fa84a500-af21-11ea-9e19-dde96e2f7ff7.png)

- 0.05
![new_005](https://user-images.githubusercontent.com/10195618/84666936-01abb300-af22-11ea-9383-3afb35b6494c.png)

- 0.1
![new_01](https://user-images.githubusercontent.com/10195618/84666967-07a19400-af22-11ea-9b38-1149cf269c5a.png)

- 0.2
![new_02](https://user-images.githubusercontent.com/10195618/84666992-0d977500-af22-11ea-80fa-5609ea069e2b.png)




> 1. Create new template, default_smoot or something like that - easy, but if we decide to go that way, soon we will have many plot templates

A `smooth` template is not a bad idea TBH. Quick and easy, we just document the template (e.g. link [here](https://vega.github.io/vega/docs/transforms/)) and emphasize that it can be customized to change the parameters.

> 2. Since we already support plot definition in pipelines file, maybe we should consider introducing update_dict parameter, that would be a way to pass any desired modification

I don't understand why `update_dict` is needed in the YAML structure but anyway, if this means `dvc plots modify` will work more like `dvc config` accepting key:value pairs to set the display props (instead of having a specific -- flag per supported prop, which is a little confusing and limiting), I support this route.
agree with @jorgeorpinel about making `plots modify` more like `config`. It was a small experiment with `plots modify` in an attempt to possibly make it better than `metrics modify` that we used to have, but it feels more confusing indeed.
Per https://github.com/iterative/dvc/issues/3906#issuecomment-639735956 sounds like we're going with a template for this one, so should we extract the part about making plots modify more like config? Pawel suggested I commented here but I'd be happy to create a separate ticket.
@jorgeorpinel & @efiop yes, it has to be configurable (otherwise we need dozens of the templates :) ).

We already support `dvc plots modify`. The smoothing option should be part of this.

Resolving patch:
diff --git a/dvc/repo/plots/template.py b/dvc/repo/plots/template.py
--- a/dvc/repo/plots/template.py
+++ b/dvc/repo/plots/template.py
@@ -168,12 +168,46 @@ class DefaultScatterTemplate(Template):
     }
 
 
+class SmoothLinearTemplate(Template):
+    DEFAULT_NAME = "smooth"
+
+    DEFAULT_CONTENT = {
+        "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
+        "data": {"values": Template.METRIC_DATA_ANCHOR},
+        "title": Template.TITLE_ANCHOR,
+        "mark": {"type": "line"},
+        "encoding": {
+            "x": {
+                "field": Template.X_ANCHOR,
+                "type": "quantitative",
+                "title": Template.X_LABEL_ANCHOR,
+            },
+            "y": {
+                "field": Template.Y_ANCHOR,
+                "type": "quantitative",
+                "title": Template.Y_LABEL_ANCHOR,
+                "scale": {"zero": False},
+            },
+            "color": {"field": "rev", "type": "nominal"},
+        },
+        "transform": [
+            {
+                "loess": Template.Y_ANCHOR,
+                "on": Template.X_ANCHOR,
+                "groupby": ["rev"],
+                "bandwidth": 0.3,
+            }
+        ],
+    }
+
+
 class PlotTemplates:
     TEMPLATES_DIR = "plots"
     TEMPLATES = [
         DefaultLinearTemplate,
         DefaultConfusionTemplate,
         DefaultScatterTemplate,
+        SmoothLinearTemplate,
     ]
 
     @cached_property


Test patch: