Repository: VictoriaMetrics/VictoriaMetrics
Pull request #960 (2020-12-13 17:20:21)

vmalert: add function "query", "first" and "value" to alert templates…
… functions

The commit adds a support for template function `query`,
`first` and `value`. The function `query` executes
a MetricsQL query for active alerts. In vmalert we
update templates on every evaluation for active alerts
to keep them up to date. With `query` func it may become
a perf issue since it will fire a query on every execution.
We should keep it in mind for now.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=h1) Report
> Merging [#960](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=desc) (59411c5) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/83c0c241a75ae9e76f0d70da089cabab86e8b729?el=desc) (83c0c24) will **decrease** coverage by `0.10%`.
> The diff coverage is `62.50%`.

[![Impacted file tree graph](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/graphs/tree.svg?width=650&height=150&src=pr&token=YND8dYVcXn)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #960      +/-   ##
==========================================
- Coverage   60.90%   60.80%   -0.11%     
==========================================
  Files         201      201              
  Lines       22677    22738      +61     
==========================================
+ Hits        13811    13825      +14     
- Misses       7550     7598      +48     
+ Partials     1316     1315       -1     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [app/vmalert/datasource/datasource.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS9kYXRhc291cmNlLmdv) | `12.50% <0.00%> (-7.50%)` | :arrow_down: |
| [app/vmalert/alerting.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvYWxlcnRpbmcuZ28=) | `76.53% <62.50%> (-0.79%)` | :arrow_down: |
| [app/vmalert/notifier/alert.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvbm90aWZpZXIvYWxlcnQuZ28=) | `48.57% <62.50%> (+1.51%)` | :arrow_up: |
| [app/vmalert/notifier/template\_func.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvbm90aWZpZXIvdGVtcGxhdGVfZnVuYy5nbw==) | `21.69% <70.00%> (+10.46%)` | :arrow_up: |
| [app/vmalert/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `22.01% <100.00%> (-0.21%)` | :arrow_down: |
| [lib/mergeset/block\_stream\_reader.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-bGliL21lcmdlc2V0L2Jsb2NrX3N0cmVhbV9yZWFkZXIuZ28=) | `49.30% <0.00%> (-13.20%)` | :arrow_down: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `64.13% <0.00%> (-1.56%)` | :arrow_down: |
| [lib/promscrape/scrapework.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-bGliL3Byb21zY3JhcGUvc2NyYXBld29yay5nbw==) | `61.60% <0.00%> (-0.50%)` | :arrow_down: |
| [app/vmselect/promql/rollup.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9yb2xsdXAuZ28=) | `85.69% <0.00%> (-0.24%)` | :arrow_down: |
| ... and [11 more](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=footer). Last update [83c0c24...59411c5](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/960?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 669de3f20..16d3cdcef 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -22,9 +22,9 @@ jobs:
         env:
           GO111MODULE: on
         run: |
-          go get -u golang.org/x/lint/golint
-          go get -u github.com/kisielk/errcheck
-          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.27.0
+          go install golang.org/x/lint/golint
+          go install github.com/kisielk/errcheck
+          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.29.0
       - name: Code checkout
         uses: actions/checkout@master
       - name: Build
diff --git a/Makefile b/Makefile
index 3257e7777..b20c241fa 100644
--- a/Makefile
+++ b/Makefile
@@ -80,7 +80,7 @@ lint: install-golint
 	golint app/...
 
 install-golint:
-	which golint || GO111MODULE=off go get -u golang.org/x/lint/golint
+	which golint || GO111MODULE=off go install golang.org/x/lint/golint
 
 errcheck: install-errcheck
 	errcheck -exclude=errcheck_excludes.txt ./lib/...
@@ -94,7 +94,7 @@ errcheck: install-errcheck
 	errcheck -exclude=errcheck_excludes.txt ./app/vmrestore/...
 
 install-errcheck:
-	which errcheck || GO111MODULE=off go get -u github.com/kisielk/errcheck
+	which errcheck || GO111MODULE=off go install github.com/kisielk/errcheck
 
 check-all: fmt vet lint errcheck golangci-lint
 
@@ -140,14 +140,14 @@ quicktemplate-gen: install-qtc
 	qtc
 
 install-qtc:
-	which qtc || GO111MODULE=off go get -u github.com/valyala/quicktemplate/qtc
+	which qtc || GO111MODULE=off go install github.com/valyala/quicktemplate/qtc
 
 
 golangci-lint: install-golangci-lint
 	golangci-lint run --exclude '(SA4003|SA1019|SA5011):' -D errcheck -D structcheck --timeout 2m
 
 install-golangci-lint:
-	which golangci-lint || GO111MODULE=off go get -u github.com/golangci/golangci-lint/cmd/golangci-lint
+	which golangci-lint || curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(shell go env GOPATH)/bin v1.29.0
 
 docs-sync:
 	cp app/vmagent/README.md docs/vmagent.md
diff --git a/README.md b/README.md
index 045c3d000..cb81548c1 100644
--- a/README.md
+++ b/README.md
@@ -1332,7 +1332,8 @@ An alternative solution is to query `/internal/resetRollupResultCache` url after
 the query cache, which could contain incomplete data cached during the backfilling.
 
 Yet another solution is to increase `-search.cacheTimestampOffset` flag value in order to disable caching
-for data with timestamps close to the current time.
+for data with timestamps close to the current time. Single-node VictoriaMetrics automatically resets response
+cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to it.
 
 
 ## Data updates
diff --git a/app/victoria-metrics/main.go b/app/victoria-metrics/main.go
index c74d74768..517c10aea 100644
--- a/app/victoria-metrics/main.go
+++ b/app/victoria-metrics/main.go
@@ -3,12 +3,15 @@ package main
 import (
 	"flag"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
+	"path"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
@@ -50,7 +53,7 @@ func main() {
 	logger.Infof("starting VictoriaMetrics at %q...", *httpListenAddr)
 	startTime := time.Now()
 	storage.SetMinScrapeIntervalForDeduplication(*minScrapeInterval)
-	vmstorage.Init()
+	vmstorage.Init(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
 	startSelfScraper()
@@ -80,8 +83,16 @@ func main() {
 }
 
 func requestHandler(w http.ResponseWriter, r *http.Request) bool {
-	if r.RequestURI == "/" {
-		fmt.Fprintf(w, "Single-node VictoriaMetrics. See docs at https://victoriametrics.github.io/")
+	if r.URL.Path == "/" {
+		fmt.Fprintf(w, "<h2>Single-node VictoriaMetrics.</h2></br>")
+		fmt.Fprintf(w, "See docs at <a href='https://victoriametrics.github.io/'>https://victoriametrics.github.io/</a></br>")
+		fmt.Fprintf(w, "Useful endpoints: </br>")
+		writeAPIHelp(w, [][]string{
+			{"/targets", "discovered targets list"},
+			{"/api/v1/targets", "advanced information about discovered targets in JSON format"},
+			{"/metrics", "available service metrics"},
+			{"/api/v1/status/tsdb", "tsdb status page"},
+		})
 		return true
 	}
 	if vminsert.RequestHandler(w, r) {
@@ -95,3 +106,12 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 	}
 	return false
 }
+
+func writeAPIHelp(w io.Writer, pathList [][]string) {
+	pathPrefix := httpserver.GetPathPrefix()
+	for _, p := range pathList {
+		p, doc := p[0], p[1]
+		p = path.Join(pathPrefix, p)
+		fmt.Fprintf(w, "<a href='%s'>%q</a> - %s<br/>", p, p, doc)
+	}
+}
diff --git a/app/vmagent/main.go b/app/vmagent/main.go
index a66bbce22..865f03459 100644
--- a/app/vmagent/main.go
+++ b/app/vmagent/main.go
@@ -5,7 +5,6 @@ import (
 	"fmt"
 	"net/http"
 	"os"
-	"strconv"
 	"strings"
 	"sync/atomic"
 	"time"
@@ -144,7 +143,7 @@ func main() {
 }
 
 func requestHandler(w http.ResponseWriter, r *http.Request) bool {
-	if r.RequestURI == "/" {
+	if r.URL.Path == "/" {
 		fmt.Fprintf(w, "vmagent - see docs at https://victoriametrics.github.io/vmagent.html")
 		return true
 	}
@@ -212,9 +211,7 @@ func requestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/targets":
 		promscrapeTargetsRequests.Inc()
-		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
-		showOriginalLabels, _ := strconv.ParseBool(r.FormValue("show_original_labels"))
-		promscrape.WriteHumanReadableTargetsStatus(w, showOriginalLabels)
+		promscrape.WriteHumanReadableTargetsStatus(w, r)
 		return true
 	case "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 4c93979c3..773873039 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -21,7 +21,6 @@ may fail;
 * by default, rules execution is sequential within one group, but persisting of execution results to remote
 storage is asynchronous. Hence, user shouldn't rely on recording rules chaining when result of previous
 recording rule is reused in next one;
-* there is no `query` function support in templates yet;
 * `vmalert` has no UI, just an API for getting groups and rules statuses.
 
 ### QuickStart
diff --git a/app/vmalert/alerting.go b/app/vmalert/alerting.go
index 3b607eb4c..592d35226 100644
--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@@ -137,6 +137,7 @@ func (ar *AlertingRule) Exec(ctx context.Context, q datasource.Querier, series b
 		}
 	}
 
+	qFn := func(query string) ([]datasource.Metric, error) { return q.Query(ctx, query) }
 	updated := make(map[uint64]struct{})
 	// update list of active alerts
 	for _, m := range qMetrics {
@@ -158,14 +159,14 @@ func (ar *AlertingRule) Exec(ctx context.Context, q datasource.Querier, series b
 				a.Value = m.Value
 				// and re-exec template since Value can be used
 				// in templates
-				err = ar.template(a)
+				err = ar.template(a, qFn)
 				if err != nil {
 					return nil, err
 				}
 			}
 			continue
 		}
-		a, err := ar.newAlert(m, ar.lastExecTime)
+		a, err := ar.newAlert(m, ar.lastExecTime, qFn)
 		if err != nil {
 			ar.lastExecError = err
 			return nil, fmt.Errorf("failed to create alert: %w", err)
@@ -245,7 +246,7 @@ func hash(m datasource.Metric) uint64 {
 	return hash.Sum64()
 }
 
-func (ar *AlertingRule) newAlert(m datasource.Metric, start time.Time) (*notifier.Alert, error) {
+func (ar *AlertingRule) newAlert(m datasource.Metric, start time.Time, qFn notifier.QueryFn) (*notifier.Alert, error) {
 	a := &notifier.Alert{
 		GroupID: ar.GroupID,
 		Name:    ar.Name,
@@ -264,16 +265,16 @@ func (ar *AlertingRule) newAlert(m datasource.Metric, start time.Time) (*notifie
 		}
 		a.Labels[l.Name] = l.Value
 	}
-	return a, ar.template(a)
+	return a, ar.template(a, qFn)
 }
 
-func (ar *AlertingRule) template(a *notifier.Alert) error {
+func (ar *AlertingRule) template(a *notifier.Alert, qFn notifier.QueryFn) error {
 	var err error
-	a.Labels, err = a.ExecTemplate(a.Labels)
+	a.Labels, err = a.ExecTemplate(qFn, a.Labels)
 	if err != nil {
 		return err
 	}
-	a.Annotations, err = a.ExecTemplate(ar.Annotations)
+	a.Annotations, err = a.ExecTemplate(qFn, ar.Annotations)
 	return err
 }
 
@@ -393,6 +394,8 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, lookb
 		return fmt.Errorf("querier is nil")
 	}
 
+	qFn := func(query string) ([]datasource.Metric, error) { return q.Query(ctx, query) }
+
 	// account for external labels in filter
 	var labelsFilter string
 	for k, v := range labels {
@@ -421,7 +424,7 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, lookb
 			m.Labels = append(m.Labels, l)
 		}
 
-		a, err := ar.newAlert(m, time.Unix(int64(m.Value), 0))
+		a, err := ar.newAlert(m, time.Unix(int64(m.Value), 0), qFn)
 		if err != nil {
 			return fmt.Errorf("failed to create alert: %w", err)
 		}
diff --git a/app/vmalert/datasource/datasource.go b/app/vmalert/datasource/datasource.go
index de0265d24..5e6aa32d3 100644
--- a/app/vmalert/datasource/datasource.go
+++ b/app/vmalert/datasource/datasource.go
@@ -34,6 +34,17 @@ func (m *Metric) AddLabel(key, value string) {
 	m.Labels = append(m.Labels, Label{Name: key, Value: value})
 }
 
+// Label returns the given label value.
+// If label is missing empty string will be returned
+func (m *Metric) Label(key string) string {
+	for _, l := range m.Labels {
+		if l.Name == key {
+			return l.Value
+		}
+	}
+	return ""
+}
+
 // Label represents metric's label
 type Label struct {
 	Name  string
diff --git a/app/vmalert/main.go b/app/vmalert/main.go
index d90a3f31c..ea6922d86 100644
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@@ -206,7 +206,7 @@ func getAlertURLGenerator(externalURL *url.URL, externalAlertSource string, vali
 		"tpl": externalAlertSource,
 	}
 	return func(alert notifier.Alert) string {
-		templated, err := alert.ExecTemplate(m)
+		templated, err := alert.ExecTemplate(nil, m)
 		if err != nil {
 			logger.Errorf("can not exec source template %s", err)
 		}
diff --git a/app/vmalert/notifier/alert.go b/app/vmalert/notifier/alert.go
index 04d4d0a71..e71b549ea 100644
--- a/app/vmalert/notifier/alert.go
+++ b/app/vmalert/notifier/alert.go
@@ -62,21 +62,23 @@ const tplHeader = `{{ $value := .Value }}{{ $labels := .Labels }}{{ $expr := .Ex
 
 // ExecTemplate executes the Alert template for give
 // map of annotations.
-func (a *Alert) ExecTemplate(annotations map[string]string) (map[string]string, error) {
+// Every alert could have a different datasource, so function
+// requires a queryFunction as an argument.
+func (a *Alert) ExecTemplate(q QueryFn, annotations map[string]string) (map[string]string, error) {
 	tplData := alertTplData{Value: a.Value, Labels: a.Labels, Expr: a.Expr}
-	return templateAnnotations(annotations, tplHeader, tplData)
+	return templateAnnotations(annotations, tplData, funcsWithQuery(q))
 }
 
 // ValidateTemplates validate annotations for possible template error, uses empty data for template population
 func ValidateTemplates(annotations map[string]string) error {
-	_, err := templateAnnotations(annotations, tplHeader, alertTplData{
+	_, err := templateAnnotations(annotations, alertTplData{
 		Labels: map[string]string{},
 		Value:  0,
-	})
+	}, tmplFunc)
 	return err
 }
 
-func templateAnnotations(annotations map[string]string, header string, data alertTplData) (map[string]string, error) {
+func templateAnnotations(annotations map[string]string, data alertTplData, funcs template.FuncMap) (map[string]string, error) {
 	var builder strings.Builder
 	var buf bytes.Buffer
 	eg := new(utils.ErrGroup)
@@ -85,10 +87,10 @@ func templateAnnotations(annotations map[string]string, header string, data aler
 		r[key] = text
 		buf.Reset()
 		builder.Reset()
-		builder.Grow(len(header) + len(text))
-		builder.WriteString(header)
+		builder.Grow(len(tplHeader) + len(text))
+		builder.WriteString(tplHeader)
 		builder.WriteString(text)
-		if err := templateAnnotation(&buf, builder.String(), data); err != nil {
+		if err := templateAnnotation(&buf, builder.String(), data, funcs); err != nil {
 			eg.Add(fmt.Errorf("key %q, template %q: %w", key, text, err))
 			continue
 		}
@@ -97,8 +99,9 @@ func templateAnnotations(annotations map[string]string, header string, data aler
 	return r, eg.Err()
 }
 
-func templateAnnotation(dst io.Writer, text string, data alertTplData) error {
-	tpl, err := template.New("").Funcs(tmplFunc).Option("missingkey=zero").Parse(text)
+func templateAnnotation(dst io.Writer, text string, data alertTplData, funcs template.FuncMap) error {
+	t := template.New("").Funcs(funcs).Option("missingkey=zero")
+	tpl, err := t.Parse(text)
 	if err != nil {
 		return fmt.Errorf("error parsing annotation: %w", err)
 	}
diff --git a/app/vmalert/notifier/template_func.go b/app/vmalert/notifier/template_func.go
index abfd8fc91..e643d5880 100644
--- a/app/vmalert/notifier/template_func.go
+++ b/app/vmalert/notifier/template_func.go
@@ -14,21 +14,40 @@
 package notifier
 
 import (
+	"errors"
 	"fmt"
-	html_template "html/template"
 	"math"
 	"net/url"
 	"regexp"
 	"strings"
-	text_template "text/template"
 	"time"
+
+	htmlTpl "html/template"
+	textTpl "text/template"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 )
 
-var tmplFunc text_template.FuncMap
+// QueryFn is used to wrap a call to datasource into simple-to-use function
+// for templating functions.
+type QueryFn func(query string) ([]datasource.Metric, error)
+
+func funcsWithQuery(query QueryFn) textTpl.FuncMap {
+	fm := make(textTpl.FuncMap)
+	for k, fn := range tmplFunc {
+		fm[k] = fn
+	}
+	fm["query"] = func(q string) ([]datasource.Metric, error) {
+		return query(q)
+	}
+	return fm
+}
+
+var tmplFunc textTpl.FuncMap
 
-// InitTemplateFunc returns template helper functions
+// InitTemplateFunc initiates template helper functions
 func InitTemplateFunc(externalURL *url.URL) {
-	tmplFunc = text_template.FuncMap{
+	tmplFunc = textTpl.FuncMap{
 		"args": func(args ...interface{}) map[string]interface{} {
 			result := make(map[string]interface{})
 			for i, a := range args {
@@ -40,8 +59,8 @@ func InitTemplateFunc(externalURL *url.URL) {
 			re := regexp.MustCompile(pattern)
 			return re.ReplaceAllString(text, repl)
 		},
-		"safeHtml": func(text string) html_template.HTML {
-			return html_template.HTML(text)
+		"safeHtml": func(text string) htmlTpl.HTML {
+			return htmlTpl.HTML(text)
 		},
 		"match":   regexp.MatchString,
 		"title":   strings.Title,
@@ -151,6 +170,24 @@ func InitTemplateFunc(externalURL *url.URL) {
 		"quotesEscape": func(q string) string {
 			return strings.Replace(q, `"`, `\"`, -1)
 		},
+		// query function supposed to be substituted at funcsWithQuery().
+		// it is present here only for validation purposes, when there is no
+		// provided datasource.
+		"query": func(q string) ([]datasource.Metric, error) {
+			return nil, nil
+		},
+		"first": func(metrics []datasource.Metric) (datasource.Metric, error) {
+			if len(metrics) > 0 {
+				return metrics[0], nil
+			}
+			return datasource.Metric{}, errors.New("first() called on vector with no elements")
+		},
+		"label": func(label string, m datasource.Metric) string {
+			return m.Label(label)
+		},
+		"value": func(m datasource.Metric) float64 {
+			return m.Value
+		},
 	}
 }
 
diff --git a/app/vminsert/main.go b/app/vminsert/main.go
index 3bfd7f2c3..d58c0f86f 100644
--- a/app/vminsert/main.go
+++ b/app/vminsert/main.go
@@ -4,7 +4,6 @@ import (
 	"flag"
 	"fmt"
 	"net/http"
-	"strconv"
 	"strings"
 	"sync/atomic"
 
@@ -155,9 +154,7 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 		return true
 	case "/targets":
 		promscrapeTargetsRequests.Inc()
-		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
-		showOriginalLabels, _ := strconv.ParseBool(r.FormValue("show_original_labels"))
-		promscrape.WriteHumanReadableTargetsStatus(w, showOriginalLabels)
+		promscrape.WriteHumanReadableTargetsStatus(w, r)
 		return true
 	case "/api/v1/targets":
 		promscrapeAPIV1TargetsRequests.Inc()
diff --git a/app/vmselect/promql/rollup_result_cache.go b/app/vmselect/promql/rollup_result_cache.go
index db6f28931..8ef54398e 100644
--- a/app/vmselect/promql/rollup_result_cache.go
+++ b/app/vmselect/promql/rollup_result_cache.go
@@ -13,6 +13,7 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/storage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache"
 	"github.com/VictoriaMetrics/fastcache"
 	"github.com/VictoriaMetrics/metrics"
@@ -25,6 +26,39 @@ var (
 		"due to time synchronization issues between VictoriaMetrics and data sources")
 )
 
+// ResetRollupResultCacheIfNeeded resets rollup result cache if mrs contains timestamps outside `now - search.cacheTimestampOffset`.
+func ResetRollupResultCacheIfNeeded(mrs []storage.MetricRow) {
+	checkRollupResultCacheResetOnce.Do(func() {
+		go checkRollupResultCacheReset()
+	})
+	minTimestamp := int64(fasttime.UnixTimestamp()*1000) - cacheTimestampOffset.Milliseconds() + checkRollupResultCacheResetInterval.Milliseconds()
+	needCacheReset := false
+	for i := range mrs {
+		if mrs[i].Timestamp < minTimestamp {
+			needCacheReset = true
+			break
+		}
+	}
+	if needCacheReset {
+		// Do not call ResetRollupResultCache() here, since it may be heavy when frequently called.
+		atomic.StoreUint32(&needRollupResultCacheReset, 1)
+	}
+}
+
+func checkRollupResultCacheReset() {
+	for {
+		time.Sleep(checkRollupResultCacheResetInterval)
+		if atomic.SwapUint32(&needRollupResultCacheReset, 0) > 0 {
+			ResetRollupResultCache()
+		}
+	}
+}
+
+const checkRollupResultCacheResetInterval = 5 * time.Second
+
+var needRollupResultCacheReset uint32
+var checkRollupResultCacheResetOnce sync.Once
+
 var rollupResultCacheV = &rollupResultCache{
 	c: workingsetcache.New(1024*1024, time.Hour), // This is a cache for testing.
 }
diff --git a/app/vmstorage/main.go b/app/vmstorage/main.go
index cee1db78c..920708422 100644
--- a/app/vmstorage/main.go
+++ b/app/vmstorage/main.go
@@ -57,19 +57,20 @@ func CheckTimeRange(tr storage.TimeRange) error {
 }
 
 // Init initializes vmstorage.
-func Init() {
-	InitWithoutMetrics()
+func Init(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
+	InitWithoutMetrics(resetCacheIfNeeded)
 	registerStorageMetrics()
 }
 
 // InitWithoutMetrics must be called instead of Init inside tests.
 //
 // This allows multiple Init / Stop cycles.
-func InitWithoutMetrics() {
+func InitWithoutMetrics(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 	if err := encoding.CheckPrecisionBits(uint8(*precisionBits)); err != nil {
 		logger.Fatalf("invalid `-precisionBits`: %s", err)
 	}
 
+	resetResponseCacheIfNeeded = resetCacheIfNeeded
 	storage.SetFinalMergeDelay(*finalMergeDelay)
 	storage.SetBigMergeWorkersCount(*bigMergeConcurrency)
 	storage.SetSmallMergeWorkersCount(*smallMergeConcurrency)
@@ -105,8 +106,12 @@ var Storage *storage.Storage
 // Use syncwg instead of sync, since Add is called from concurrent goroutines.
 var WG syncwg.WaitGroup
 
+// resetResponseCacheIfNeeded is a callback for automatic resetting of response cache if needed.
+var resetResponseCacheIfNeeded func(mrs []storage.MetricRow)
+
 // AddRows adds mrs to the storage.
 func AddRows(mrs []storage.MetricRow) error {
+	resetResponseCacheIfNeeded(mrs)
 	WG.Add(1)
 	err := Storage.AddRows(mrs, uint8(*precisionBits))
 	WG.Done()
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 3c2afbff7..5a9a46dc8 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -2,6 +2,8 @@
 
 # tip
 
+* FEATURE: automatically reset response cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to VictoriaMetrics. This makes unnecessary disabling response cache during data backfilling or resetting it after backfilling is complete as described [in these docs](https://victoriametrics.github.io/#backfilling). This feature applies only to single-node VictoriaMetrics. It doesn't apply to cluster version of VictoriaMetrics because `vminsert` nodes don't know about `vmselect` nodes where the response cache must be reset.
+* FEATURE: vmagent: return user-friendly HTML page when requesting `/targets` page from web browser. The page is returned in the old plaintext format when requesting via curl or similar tool.
 * FEATURE: allow multiple whitespace chars between measurements, fields and timestamp when parsing InfluxDB line protocol.
   Though [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/) denies multiple whitespace chars between these entities,
   some apps improperly put multiple whitespace chars. This workaround allows accepting data from such apps.
@@ -16,6 +18,7 @@
 * BUGFIX: prevent from duplicate `name` tag returned from `/tags/autoComplete/tags` handler. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942
 * BUGFIX: do not enable strict parsing for `-promscrape.config` if `-promscrape.config.dryRun` comand-line flag is set. Strict parsing can be enabled with `-promscrape.config.strictParse` command-line flag. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/944
 * BUGFIX: vminsert: properly update `vm_rpc_rerouted_rows_processed_total` metric. Previously it wasn't updated. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/955
+* BUGFIX: vmagent: properly recover when opening incorrectly stored persistent queue. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/964
 
 
 # [v1.49.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.49.0)
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 045c3d000..cb81548c1 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -1332,7 +1332,8 @@ An alternative solution is to query `/internal/resetRollupResultCache` url after
 the query cache, which could contain incomplete data cached during the backfilling.
 
 Yet another solution is to increase `-search.cacheTimestampOffset` flag value in order to disable caching
-for data with timestamps close to the current time.
+for data with timestamps close to the current time. Single-node VictoriaMetrics automatically resets response
+cache when samples with timestamps older than `now - search.cacheTimestampOffset` are ingested to it.
 
 
 ## Data updates
diff --git a/lib/httpserver/httpserver.go b/lib/httpserver/httpserver.go
index daa8c8e59..6a760af5b 100644
--- a/lib/httpserver/httpserver.go
+++ b/lib/httpserver/httpserver.go
@@ -274,9 +274,12 @@ func handlerWrapper(s *server, w http.ResponseWriter, r *http.Request, rh Reques
 }
 
 func getCanonicalPath(path string) (string, error) {
-	if len(*pathPrefix) == 0 {
+	if len(*pathPrefix) == 0 || path == "/" {
 		return path, nil
 	}
+	if *pathPrefix == path {
+		return "/", nil
+	}
 	prefix := *pathPrefix
 	if !strings.HasSuffix(prefix, "/") {
 		prefix = prefix + "/"
@@ -573,3 +576,8 @@ func isTrivialNetworkError(err error) bool {
 func IsTLS() bool {
 	return *tlsEnable
 }
+
+// GetPathPrefix - returns http server path prefix.
+func GetPathPrefix() string {
+	return *pathPrefix
+}
diff --git a/lib/persistentqueue/persistentqueue.go b/lib/persistentqueue/persistentqueue.go
index 87e4b1a55..916276c3f 100644
--- a/lib/persistentqueue/persistentqueue.go
+++ b/lib/persistentqueue/persistentqueue.go
@@ -336,6 +336,10 @@ func tryOpeningQueue(path, name string, chunkFileSize, maxBlockSize, maxPendingB
 		cleanOnError()
 		return nil, fmt.Errorf("couldn't find chunk file for writing in %q", q.dir)
 	}
+	if q.readerOffset > q.writerOffset {
+		cleanOnError()
+		return nil, fmt.Errorf("readerOffset=%d cannot exceed writerOffset=%d", q.readerOffset, q.writerOffset)
+	}
 	mustCloseFlockF = false
 	return &q, nil
 }
diff --git a/lib/promscrape/targets_response.qtpl b/lib/promscrape/targets_response.qtpl
new file mode 100644
index 000000000..f1fa91f9b
--- /dev/null
+++ b/lib/promscrape/targets_response.qtpl
@@ -0,0 +1,104 @@
+{% import "github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+%}
+
+{% stripspace %}
+
+{% func TargetsResponsePlain (jts []jobTargetsStatuses,  showOriginLabels bool) -%}
+
+{% for _, js := range jts %}
+job={%q= js.job %}{% space %} ({%d js.upCount %}/{%d js.targetsTotal %} {% space %} up)
+{% newline %}
+{% for _, ts := range js.targetsStatus %}
+    {% code
+     labels :=  promLabelsString(ts.labels)
+     ol := promLabelsString(ts.originalLabels)
+    %}
+{%s= "\t" %}state={% if ts.up %}up{% else %}down{% endif %},
+    {% space %} endpoint={%s= ts.endpoint %},
+    {% space %} labels={%s= labels %}
+    {% if showOriginLabels %},{% space %} originalLabels={%s= ol %}{% endif %},
+    {% space %} last_scrape={%f.3 ts.lastScrapeTime.Seconds() %}s {% space %} ago,
+    {% space %} scrape_duration={%f.3 float64(ts.scrapeDuration.Seconds()) %}s,
+    {% space %} error={%q= ts.error %}
+    {% newline %}
+{% endfor %}
+{% endfor %}
+{% newline %}
+
+{% endfunc %}
+
+{% func TargetsResponseHTML(jts []jobTargetsStatuses, redirectPath string, onlyUnhealthy bool) %}
+<!DOCTYPE html>
+<style>
+    .border{
+        border-collapse: collapse;
+        border: 1px solid black;
+    }
+    .table-row:hover{
+        background-color: #f5f5f5;
+    }
+</style>
+<html lang="en">
+    <head>
+        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
+        <title>VictoriaMetrics Database</title>
+    </head>
+    <body>
+    <h1>Targets</h1>
+    <div id="showTargets" class="btn-group btn-group-toggle" data-toggle="buttons">
+      <label class="btn">
+        <input type="radio" name="targets" id="all-targets" autocomplete="off" onclick="location.href='{%s= redirectPath %}';" {% if !onlyUnhealthy %}checked {% endif %}> All
+      </label>
+      <label class="btn">
+        <input type="radio" name="targets" id="unhealthy-targets" autocomplete="off" onclick="location.href='{%s= redirectPath %}?show_only_unhealthy=true';" {% if onlyUnhealthy %}checked {% endif %}> Unhealthy
+      </label>
+      <br />
+  </div>
+  {% for _,js :=range jts %}
+  		    <div class="table-container">
+                <h2 class="job_header danger">
+                  <a id="job-{%q= js.job %}" >{%q= js.job %} ({%d js.upCount %}/{%d js.targetsTotal %} up)</a>
+                </h2>
+                <table class="table-bordered table-hover border">
+                  <thead class="job_details border">
+                    <tr class="table-row border">
+                      <th class="border">Endpoint</th>
+                      <th class="border">State</th>
+                      <th class="border">Labels</th>
+                      <th class="border">Last Scrape</th>
+                      <th class="border">Scrape Duration</th>
+                      <th class="border">Error</th>
+                    </tr>
+                  </thead>
+                  <tbody>
+                  {% for _, ts := range js.targetsStatus %}
+                    {% if onlyUnhealthy && ts.up %} {% continue %} {% endif %}
+                    <tr class="table-row border">
+                      <td class="endpoint border">
+                        <a href="{%s= ts.endpoint %}">{%s= ts.endpoint %}</a><br>
+                      </td>
+                      <td class="state border">
+                        <span class="state_indicator">{% if ts.up %}UP{% else %}DOWN{% endif %}</span>
+                      </td>
+                      <td class="labels border", title="Original {% space %} labels: {% space %} {%= formatLabel(ts.originalLabels) %}">
+                           {%= formatLabel(ts.labels) %}
+                      </td>
+                      <td class="last-scrape border">{%s ts.lastScrapeTime.String() %} {% space %} ago</td>
+                      <td class="scrape-duration border">{%s ts.scrapeDuration.String() %}</td>
+                      <td class="errors border"><span class="alert alert-danger state_indicator">{%s= ts.error %}</span></td>
+                    </tr>
+                  {% endfor %}
+                  </tbody>
+                </table>
+              </div>
+  {% endfor %}
+    </body>
+</html>
+{% endfunc %}
+
+{% func formatLabel(labels []prompbmarshal.Label) %}
+{% for _, label := range labels %}
+ {% space %} {%s label.Name %}={%q label.Value %} {% space %}
+{% endfor %}
+{% endfunc %}
+{% endstripspace %}
diff --git a/lib/promscrape/targets_response.qtpl.go b/lib/promscrape/targets_response.qtpl.go
new file mode 100644
index 000000000..b0472bddb
--- /dev/null
+++ b/lib/promscrape/targets_response.qtpl.go
@@ -0,0 +1,349 @@
+// Code generated by qtc from "targets_response.qtpl". DO NOT EDIT.
+// See https://github.com/valyala/quicktemplate for details.
+
+//line lib/promscrape/targets_response.qtpl:1
+package promscrape
+
+//line lib/promscrape/targets_response.qtpl:1
+import "github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
+
+//line lib/promscrape/targets_response.qtpl:6
+import (
+	qtio422016 "io"
+
+	qt422016 "github.com/valyala/quicktemplate"
+)
+
+//line lib/promscrape/targets_response.qtpl:6
+var (
+	_ = qtio422016.Copy
+	_ = qt422016.AcquireByteBuffer
+)
+
+//line lib/promscrape/targets_response.qtpl:6
+func StreamTargetsResponsePlain(qw422016 *qt422016.Writer, jts []jobTargetsStatuses, showOriginLabels bool) {
+//line lib/promscrape/targets_response.qtpl:8
+	for _, js := range jts {
+//line lib/promscrape/targets_response.qtpl:8
+		qw422016.N().S(`job=`)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().Q(js.job)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().S(`(`)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().D(js.upCount)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().S(`/`)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().D(js.targetsTotal)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:9
+		qw422016.N().S(`up)`)
+//line lib/promscrape/targets_response.qtpl:10
+		qw422016.N().S(`
+`)
+//line lib/promscrape/targets_response.qtpl:11
+		for _, ts := range js.targetsStatus {
+//line lib/promscrape/targets_response.qtpl:13
+			labels := promLabelsString(ts.labels)
+			ol := promLabelsString(ts.originalLabels)
+
+//line lib/promscrape/targets_response.qtpl:16
+			qw422016.N().S("\t")
+//line lib/promscrape/targets_response.qtpl:16
+			qw422016.N().S(`state=`)
+//line lib/promscrape/targets_response.qtpl:16
+			if ts.up {
+//line lib/promscrape/targets_response.qtpl:16
+				qw422016.N().S(`up`)
+//line lib/promscrape/targets_response.qtpl:16
+			} else {
+//line lib/promscrape/targets_response.qtpl:16
+				qw422016.N().S(`down`)
+//line lib/promscrape/targets_response.qtpl:16
+			}
+//line lib/promscrape/targets_response.qtpl:16
+			qw422016.N().S(`,`)
+//line lib/promscrape/targets_response.qtpl:17
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:17
+			qw422016.N().S(`endpoint=`)
+//line lib/promscrape/targets_response.qtpl:17
+			qw422016.N().S(ts.endpoint)
+//line lib/promscrape/targets_response.qtpl:17
+			qw422016.N().S(`,`)
+//line lib/promscrape/targets_response.qtpl:18
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:18
+			qw422016.N().S(`labels=`)
+//line lib/promscrape/targets_response.qtpl:18
+			qw422016.N().S(labels)
+//line lib/promscrape/targets_response.qtpl:19
+			if showOriginLabels {
+//line lib/promscrape/targets_response.qtpl:19
+				qw422016.N().S(`,`)
+//line lib/promscrape/targets_response.qtpl:19
+				qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:19
+				qw422016.N().S(`originalLabels=`)
+//line lib/promscrape/targets_response.qtpl:19
+				qw422016.N().S(ol)
+//line lib/promscrape/targets_response.qtpl:19
+			}
+//line lib/promscrape/targets_response.qtpl:19
+			qw422016.N().S(`,`)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().S(`last_scrape=`)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().FPrec(ts.lastScrapeTime.Seconds(), 3)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().S(`s`)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:20
+			qw422016.N().S(`ago,`)
+//line lib/promscrape/targets_response.qtpl:21
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:21
+			qw422016.N().S(`scrape_duration=`)
+//line lib/promscrape/targets_response.qtpl:21
+			qw422016.N().FPrec(float64(ts.scrapeDuration.Seconds()), 3)
+//line lib/promscrape/targets_response.qtpl:21
+			qw422016.N().S(`s,`)
+//line lib/promscrape/targets_response.qtpl:22
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:22
+			qw422016.N().S(`error=`)
+//line lib/promscrape/targets_response.qtpl:22
+			qw422016.N().Q(ts.error)
+//line lib/promscrape/targets_response.qtpl:23
+			qw422016.N().S(`
+`)
+//line lib/promscrape/targets_response.qtpl:24
+		}
+//line lib/promscrape/targets_response.qtpl:25
+	}
+//line lib/promscrape/targets_response.qtpl:26
+	qw422016.N().S(`
+`)
+//line lib/promscrape/targets_response.qtpl:28
+}
+
+//line lib/promscrape/targets_response.qtpl:28
+func WriteTargetsResponsePlain(qq422016 qtio422016.Writer, jts []jobTargetsStatuses, showOriginLabels bool) {
+//line lib/promscrape/targets_response.qtpl:28
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line lib/promscrape/targets_response.qtpl:28
+	StreamTargetsResponsePlain(qw422016, jts, showOriginLabels)
+//line lib/promscrape/targets_response.qtpl:28
+	qt422016.ReleaseWriter(qw422016)
+//line lib/promscrape/targets_response.qtpl:28
+}
+
+//line lib/promscrape/targets_response.qtpl:28
+func TargetsResponsePlain(jts []jobTargetsStatuses, showOriginLabels bool) string {
+//line lib/promscrape/targets_response.qtpl:28
+	qb422016 := qt422016.AcquireByteBuffer()
+//line lib/promscrape/targets_response.qtpl:28
+	WriteTargetsResponsePlain(qb422016, jts, showOriginLabels)
+//line lib/promscrape/targets_response.qtpl:28
+	qs422016 := string(qb422016.B)
+//line lib/promscrape/targets_response.qtpl:28
+	qt422016.ReleaseByteBuffer(qb422016)
+//line lib/promscrape/targets_response.qtpl:28
+	return qs422016
+//line lib/promscrape/targets_response.qtpl:28
+}
+
+//line lib/promscrape/targets_response.qtpl:30
+func StreamTargetsResponseHTML(qw422016 *qt422016.Writer, jts []jobTargetsStatuses, redirectPath string, onlyUnhealthy bool) {
+//line lib/promscrape/targets_response.qtpl:30
+	qw422016.N().S(`<!DOCTYPE html><style>.border{border-collapse: collapse;border: 1px solid black;}.table-row:hover{background-color: #f5f5f5;}</style><html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><title>VictoriaMetrics Database</title></head><body><h1>Targets</h1><div id="showTargets" class="btn-group btn-group-toggle" data-toggle="buttons"><label class="btn"><input type="radio" name="targets" id="all-targets" autocomplete="off" onclick="location.href='`)
+//line lib/promscrape/targets_response.qtpl:50
+	qw422016.N().S(redirectPath)
+//line lib/promscrape/targets_response.qtpl:50
+	qw422016.N().S(`';"`)
+//line lib/promscrape/targets_response.qtpl:50
+	if !onlyUnhealthy {
+//line lib/promscrape/targets_response.qtpl:50
+		qw422016.N().S(`checked`)
+//line lib/promscrape/targets_response.qtpl:50
+	}
+//line lib/promscrape/targets_response.qtpl:50
+	qw422016.N().S(`> All</label><label class="btn"><input type="radio" name="targets" id="unhealthy-targets" autocomplete="off" onclick="location.href='`)
+//line lib/promscrape/targets_response.qtpl:53
+	qw422016.N().S(redirectPath)
+//line lib/promscrape/targets_response.qtpl:53
+	qw422016.N().S(`?show_only_unhealthy=true';"`)
+//line lib/promscrape/targets_response.qtpl:53
+	if onlyUnhealthy {
+//line lib/promscrape/targets_response.qtpl:53
+		qw422016.N().S(`checked`)
+//line lib/promscrape/targets_response.qtpl:53
+	}
+//line lib/promscrape/targets_response.qtpl:53
+	qw422016.N().S(`> Unhealthy</label><br /></div>`)
+//line lib/promscrape/targets_response.qtpl:57
+	for _, js := range jts {
+//line lib/promscrape/targets_response.qtpl:57
+		qw422016.N().S(`<div class="table-container"><h2 class="job_header danger"><a id="job-`)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().Q(js.job)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().S(`" >`)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().Q(js.job)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().S(`(`)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().D(js.upCount)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().S(`/`)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().D(js.targetsTotal)
+//line lib/promscrape/targets_response.qtpl:60
+		qw422016.N().S(`up)</a></h2><table class="table-bordered table-hover border"><thead class="job_details border"><tr class="table-row border"><th class="border">Endpoint</th><th class="border">State</th><th class="border">Labels</th><th class="border">Last Scrape</th><th class="border">Scrape Duration</th><th class="border">Error</th></tr></thead><tbody>`)
+//line lib/promscrape/targets_response.qtpl:74
+		for _, ts := range js.targetsStatus {
+//line lib/promscrape/targets_response.qtpl:75
+			if onlyUnhealthy && ts.up {
+//line lib/promscrape/targets_response.qtpl:75
+				continue
+//line lib/promscrape/targets_response.qtpl:75
+			}
+//line lib/promscrape/targets_response.qtpl:75
+			qw422016.N().S(`<tr class="table-row border"><td class="endpoint border"><a href="`)
+//line lib/promscrape/targets_response.qtpl:78
+			qw422016.N().S(ts.endpoint)
+//line lib/promscrape/targets_response.qtpl:78
+			qw422016.N().S(`">`)
+//line lib/promscrape/targets_response.qtpl:78
+			qw422016.N().S(ts.endpoint)
+//line lib/promscrape/targets_response.qtpl:78
+			qw422016.N().S(`</a><br></td><td class="state border"><span class="state_indicator">`)
+//line lib/promscrape/targets_response.qtpl:81
+			if ts.up {
+//line lib/promscrape/targets_response.qtpl:81
+				qw422016.N().S(`UP`)
+//line lib/promscrape/targets_response.qtpl:81
+			} else {
+//line lib/promscrape/targets_response.qtpl:81
+				qw422016.N().S(`DOWN`)
+//line lib/promscrape/targets_response.qtpl:81
+			}
+//line lib/promscrape/targets_response.qtpl:81
+			qw422016.N().S(`</span></td><td class="labels border", title="Original`)
+//line lib/promscrape/targets_response.qtpl:83
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:83
+			qw422016.N().S(`labels:`)
+//line lib/promscrape/targets_response.qtpl:83
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:83
+			streamformatLabel(qw422016, ts.originalLabels)
+//line lib/promscrape/targets_response.qtpl:83
+			qw422016.N().S(`">`)
+//line lib/promscrape/targets_response.qtpl:84
+			streamformatLabel(qw422016, ts.labels)
+//line lib/promscrape/targets_response.qtpl:84
+			qw422016.N().S(`</td><td class="last-scrape border">`)
+//line lib/promscrape/targets_response.qtpl:86
+			qw422016.E().S(ts.lastScrapeTime.String())
+//line lib/promscrape/targets_response.qtpl:86
+			qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:86
+			qw422016.N().S(`ago</td><td class="scrape-duration border">`)
+//line lib/promscrape/targets_response.qtpl:87
+			qw422016.E().S(ts.scrapeDuration.String())
+//line lib/promscrape/targets_response.qtpl:87
+			qw422016.N().S(`</td><td class="errors border"><span class="alert alert-danger state_indicator">`)
+//line lib/promscrape/targets_response.qtpl:88
+			qw422016.N().S(ts.error)
+//line lib/promscrape/targets_response.qtpl:88
+			qw422016.N().S(`</span></td></tr>`)
+//line lib/promscrape/targets_response.qtpl:90
+		}
+//line lib/promscrape/targets_response.qtpl:90
+		qw422016.N().S(`</tbody></table></div>`)
+//line lib/promscrape/targets_response.qtpl:94
+	}
+//line lib/promscrape/targets_response.qtpl:94
+	qw422016.N().S(`</body></html>`)
+//line lib/promscrape/targets_response.qtpl:97
+}
+
+//line lib/promscrape/targets_response.qtpl:97
+func WriteTargetsResponseHTML(qq422016 qtio422016.Writer, jts []jobTargetsStatuses, redirectPath string, onlyUnhealthy bool) {
+//line lib/promscrape/targets_response.qtpl:97
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line lib/promscrape/targets_response.qtpl:97
+	StreamTargetsResponseHTML(qw422016, jts, redirectPath, onlyUnhealthy)
+//line lib/promscrape/targets_response.qtpl:97
+	qt422016.ReleaseWriter(qw422016)
+//line lib/promscrape/targets_response.qtpl:97
+}
+
+//line lib/promscrape/targets_response.qtpl:97
+func TargetsResponseHTML(jts []jobTargetsStatuses, redirectPath string, onlyUnhealthy bool) string {
+//line lib/promscrape/targets_response.qtpl:97
+	qb422016 := qt422016.AcquireByteBuffer()
+//line lib/promscrape/targets_response.qtpl:97
+	WriteTargetsResponseHTML(qb422016, jts, redirectPath, onlyUnhealthy)
+//line lib/promscrape/targets_response.qtpl:97
+	qs422016 := string(qb422016.B)
+//line lib/promscrape/targets_response.qtpl:97
+	qt422016.ReleaseByteBuffer(qb422016)
+//line lib/promscrape/targets_response.qtpl:97
+	return qs422016
+//line lib/promscrape/targets_response.qtpl:97
+}
+
+//line lib/promscrape/targets_response.qtpl:99
+func streamformatLabel(qw422016 *qt422016.Writer, labels []prompbmarshal.Label) {
+//line lib/promscrape/targets_response.qtpl:100
+	for _, label := range labels {
+//line lib/promscrape/targets_response.qtpl:101
+		qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:101
+		qw422016.E().S(label.Name)
+//line lib/promscrape/targets_response.qtpl:101
+		qw422016.N().S(`=`)
+//line lib/promscrape/targets_response.qtpl:101
+		qw422016.E().Q(label.Value)
+//line lib/promscrape/targets_response.qtpl:101
+		qw422016.N().S(` `)
+//line lib/promscrape/targets_response.qtpl:102
+	}
+//line lib/promscrape/targets_response.qtpl:103
+}
+
+//line lib/promscrape/targets_response.qtpl:103
+func writeformatLabel(qq422016 qtio422016.Writer, labels []prompbmarshal.Label) {
+//line lib/promscrape/targets_response.qtpl:103
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line lib/promscrape/targets_response.qtpl:103
+	streamformatLabel(qw422016, labels)
+//line lib/promscrape/targets_response.qtpl:103
+	qt422016.ReleaseWriter(qw422016)
+//line lib/promscrape/targets_response.qtpl:103
+}
+
+//line lib/promscrape/targets_response.qtpl:103
+func formatLabel(labels []prompbmarshal.Label) string {
+//line lib/promscrape/targets_response.qtpl:103
+	qb422016 := qt422016.AcquireByteBuffer()
+//line lib/promscrape/targets_response.qtpl:103
+	writeformatLabel(qb422016, labels)
+//line lib/promscrape/targets_response.qtpl:103
+	qs422016 := string(qb422016.B)
+//line lib/promscrape/targets_response.qtpl:103
+	qt422016.ReleaseByteBuffer(qb422016)
+//line lib/promscrape/targets_response.qtpl:103
+	return qs422016
+//line lib/promscrape/targets_response.qtpl:103
+}
diff --git a/lib/promscrape/targetstatus.go b/lib/promscrape/targetstatus.go
index 0a75f00e4..93ef68bcc 100644
--- a/lib/promscrape/targetstatus.go
+++ b/lib/promscrape/targetstatus.go
@@ -4,11 +4,16 @@ import (
 	"flag"
 	"fmt"
 	"io"
+	"net/http"
+	"path"
 	"sort"
+	"strconv"
+	"strings"
 	"sync"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
+	"github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promrelabel"
 )
@@ -19,9 +24,17 @@ var maxDroppedTargets = flag.Int("promscrape.maxDroppedTargets", 1000, "The maxi
 
 var tsmGlobal = newTargetStatusMap()
 
-// WriteHumanReadableTargetsStatus writes human-readable status for all the scrape targets to w.
-func WriteHumanReadableTargetsStatus(w io.Writer, showOriginalLabels bool) {
-	tsmGlobal.WriteHumanReadable(w, showOriginalLabels)
+// WriteHumanReadableTargetsStatus writes human-readable status for all the scrape targets to w according to r.
+func WriteHumanReadableTargetsStatus(w http.ResponseWriter, r *http.Request) {
+	showOriginalLabels, _ := strconv.ParseBool(r.FormValue("show_original_labels"))
+	showOnlyUnhealthy, _ := strconv.ParseBool(r.FormValue("show_only_unhealthy"))
+	if accept := r.Header.Get("Accept"); strings.Contains(accept, "text/html") {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		tsmGlobal.WriteTargetsHTML(w, showOnlyUnhealthy)
+	} else {
+		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+		tsmGlobal.WriteTargetsPlain(w, showOriginalLabels)
+	}
 }
 
 // WriteAPIV1Targets writes /api/v1/targets to w according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets
@@ -162,64 +175,6 @@ func writeLabelsJSON(w io.Writer, labels []prompbmarshal.Label) {
 	fmt.Fprintf(w, `}`)
 }
 
-func (tsm *targetStatusMap) WriteHumanReadable(w io.Writer, showOriginalLabels bool) {
-	byJob := make(map[string][]targetStatus)
-	tsm.mu.Lock()
-	for _, st := range tsm.m {
-		job := st.sw.Job()
-		byJob[job] = append(byJob[job], *st)
-	}
-	tsm.mu.Unlock()
-
-	var jss []jobStatus
-	for job, statuses := range byJob {
-		jss = append(jss, jobStatus{
-			job:      job,
-			statuses: statuses,
-		})
-	}
-	sort.Slice(jss, func(i, j int) bool {
-		return jss[i].job < jss[j].job
-	})
-
-	for _, js := range jss {
-		sts := js.statuses
-		sort.Slice(sts, func(i, j int) bool {
-			return sts[i].sw.ScrapeURL < sts[j].sw.ScrapeURL
-		})
-		ups := 0
-		for _, st := range sts {
-			if st.up {
-				ups++
-			}
-		}
-		fmt.Fprintf(w, "job=%q (%d/%d up)\n", js.job, ups, len(sts))
-		for _, st := range sts {
-			state := "up"
-			if !st.up {
-				state = "down"
-			}
-			labelsStr := st.sw.LabelsString()
-			if showOriginalLabels {
-				labelsStr += ", originalLabels=" + promLabelsString(st.sw.OriginalLabels)
-			}
-			lastScrape := st.getDurationFromLastScrape()
-			errMsg := ""
-			if st.err != nil {
-				errMsg = st.err.Error()
-			}
-			fmt.Fprintf(w, "\tstate=%s, endpoint=%s, labels=%s, last_scrape=%.3fs ago, scrape_duration=%.3fs, error=%q\n",
-				state, st.sw.ScrapeURL, labelsStr, lastScrape.Seconds(), float64(st.scrapeDuration)/1000, errMsg)
-		}
-	}
-	fmt.Fprintf(w, "\n")
-}
-
-type jobStatus struct {
-	job      string
-	statuses []targetStatus
-}
-
 type targetStatus struct {
 	sw             ScrapeWork
 	up             bool
@@ -303,3 +258,84 @@ func (dt *droppedTargets) WriteDroppedTargetsJSON(w io.Writer) {
 var droppedTargetsMap = &droppedTargets{
 	m: make(map[string]droppedTarget),
 }
+
+type jobTargetStatus struct {
+	up             bool
+	endpoint       string
+	labels         []prompbmarshal.Label
+	originalLabels []prompbmarshal.Label
+	lastScrapeTime time.Duration
+	scrapeDuration time.Duration
+	error          string
+}
+
+type jobTargetsStatuses struct {
+	job           string
+	upCount       int
+	targetsTotal  int
+	targetsStatus []jobTargetStatus
+}
+
+func (tsm *targetStatusMap) getTargetsStatusByJob() []jobTargetsStatuses {
+	byJob := make(map[string][]targetStatus)
+	tsm.mu.Lock()
+	for _, st := range tsm.m {
+		job := st.sw.Job()
+		byJob[job] = append(byJob[job], *st)
+	}
+	tsm.mu.Unlock()
+
+	var jts []jobTargetsStatuses
+	for job, statuses := range byJob {
+		sort.Slice(statuses, func(i, j int) bool {
+			return statuses[i].sw.ScrapeURL < statuses[j].sw.ScrapeURL
+		})
+		ups := 0
+		var targetsStatuses []jobTargetStatus
+		for _, ts := range statuses {
+			if ts.up {
+				ups++
+			}
+		}
+		for _, st := range statuses {
+			errMsg := ""
+			if st.err != nil {
+				errMsg = st.err.Error()
+			}
+			targetsStatuses = append(targetsStatuses, jobTargetStatus{
+				up:             st.up,
+				endpoint:       st.sw.ScrapeURL,
+				labels:         promrelabel.FinalizeLabels(nil, st.sw.Labels),
+				originalLabels: st.sw.OriginalLabels,
+				lastScrapeTime: st.getDurationFromLastScrape(),
+				scrapeDuration: time.Duration(st.scrapeDuration),
+				error:          errMsg,
+			})
+		}
+		jts = append(jts, jobTargetsStatuses{
+			job:           job,
+			upCount:       ups,
+			targetsTotal:  len(statuses),
+			targetsStatus: targetsStatuses,
+		})
+	}
+	sort.Slice(jts, func(i, j int) bool {
+		return jts[i].job < jts[j].job
+	})
+	return jts
+}
+
+// WriteTargetsHTML writes targets status grouped by job into writer w in html table,
+// accepts filter to show only unhealthy targets.
+func (tsm *targetStatusMap) WriteTargetsHTML(w io.Writer, showOnlyUnhealthy bool) {
+	jss := tsm.getTargetsStatusByJob()
+	targetsPath := path.Join(httpserver.GetPathPrefix(), "/targets")
+	WriteTargetsResponseHTML(w, jss, targetsPath, showOnlyUnhealthy)
+}
+
+// WriteTargetsPlain writes targets grouped by job into writer w in plain text,
+// accept filter to show original labels.
+func (tsm *targetStatusMap) WriteTargetsPlain(w io.Writer, showOriginalLabels bool) {
+	jss := tsm.getTargetsStatusByJob()
+	WriteTargetsResponsePlain(w, jss, showOriginalLabels)
+}


Test patch:
diff --git a/app/victoria-metrics/main_test.go b/app/victoria-metrics/main_test.go
index 6eb2c13c9..926157623 100644
--- a/app/victoria-metrics/main_test.go
+++ b/app/victoria-metrics/main_test.go
@@ -20,6 +20,7 @@ import (
 	testutil "github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-metrics/test"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vminsert"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect"
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql"
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
@@ -129,7 +130,7 @@ func setUp() {
 	storagePath = filepath.Join(os.TempDir(), testStorageSuffix)
 	processFlags()
 	logger.Init()
-	vmstorage.InitWithoutMetrics()
+	vmstorage.InitWithoutMetrics(promql.ResetRollupResultCacheIfNeeded)
 	vmselect.Init()
 	vminsert.Init()
 	go httpserver.Serve(*httpListenAddr, requestHandler)
@@ -192,7 +193,7 @@ func TestWriteRead(t *testing.T) {
 	time.Sleep(1 * time.Second)
 	vmstorage.Stop()
 	// open storage after stop in write
-	vmstorage.InitWithoutMetrics()
+	vmstorage.InitWithoutMetrics(promql.ResetRollupResultCacheIfNeeded)
 	t.Run("read", testRead)
 }
 
diff --git a/app/vmalert/config/config_test.go b/app/vmalert/config/config_test.go
index 0539986e3..13dd21731 100644
--- a/app/vmalert/config/config_test.go
+++ b/app/vmalert/config/config_test.go
@@ -7,8 +7,9 @@ import (
 	"testing"
 	"time"
 
-	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 	"gopkg.in/yaml.v2"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/notifier"
 )
 
 func TestMain(m *testing.M) {
@@ -42,7 +43,7 @@ func TestParseBad(t *testing.T) {
 		},
 		{
 			[]string{"testdata/dir/rules2-bad.rules"},
-			"function \"value\" not defined",
+			"function \"unknown\" not defined",
 		},
 		{
 			[]string{"testdata/dir/rules3-bad.rules"},
@@ -137,12 +138,14 @@ func TestGroup_Validate(t *testing.T) {
 						Alert: "alert",
 						Expr:  "up == 1",
 						Labels: map[string]string{
-							"summary": "{{ value|query }}",
+							"summary": `
+{{ with printf "node_memory_MemTotal{job='node',instance='%s'}" "localhost" | query }}
+  {{ . | first | value | humanize1024 }}B
+{{ end }}`,
 						},
 					},
 				},
 			},
-			expErr:              "error parsing annotation",
 			validateAnnotations: true,
 		},
 		{
diff --git a/app/vmalert/config/testdata/dir/rules2-bad.rules b/app/vmalert/config/testdata/dir/rules2-bad.rules
index 57ed21357..903921921 100644
--- a/app/vmalert/config/testdata/dir/rules2-bad.rules
+++ b/app/vmalert/config/testdata/dir/rules2-bad.rules
@@ -6,6 +6,6 @@ groups:
         expr: vm_rows > 0
         labels:
           label: bar
-          summary: "{{ value|query }}"
+          summary: "{{ unknown|query }}"
         annotations:
           description: "{{$labels}}"
diff --git a/app/vmalert/config/testdata/rules2-good.rules b/app/vmalert/config/testdata/rules2-good.rules
index 3a015d3eb..a3ce92342 100644
--- a/app/vmalert/config/testdata/rules2-good.rules
+++ b/app/vmalert/config/testdata/rules2-good.rules
@@ -7,11 +7,18 @@ groups:
         expr: sum(vm_tcplistener_conns) by(instance) > 1
         for: 3m
         annotations:
-          summary: "Too high connection number for {{$labels.instance}}"
+          summary: Too high connection number for {{$labels.instance}}
+            {{ with printf "sum(vm_tcplistener_conns{instance=%q})" .Labels.instance | query }}
+              {{ . | first | value }}
+            {{ end }}
           description: "It is {{ $value }} connections for {{$labels.instance}}"
       - alert: ExampleAlertAlwaysFiring
         expr: sum by(job)
           (up == 1)
+        annotations:
+          summary: Instances up {{ range query "up" }}
+            {{ . | label "instance" }}
+            {{ end }}
       - record: handler:requests:rate5m
         expr: sum(rate(prometheus_http_requests_total[5m])) by (handler)
         labels:
diff --git a/app/vmalert/group_test.go b/app/vmalert/group_test.go
index d8b98dcdd..093817abf 100644
--- a/app/vmalert/group_test.go
+++ b/app/vmalert/group_test.go
@@ -167,7 +167,7 @@ func TestGroupStart(t *testing.T) {
 	m2 := metricWithLabels(t, "instance", inst2, "job", job)
 
 	r := g.Rules[0].(*AlertingRule)
-	alert1, err := r.newAlert(m1, time.Now())
+	alert1, err := r.newAlert(m1, time.Now(), nil)
 	if err != nil {
 		t.Fatalf("faield to create alert: %s", err)
 	}
@@ -179,7 +179,7 @@ func TestGroupStart(t *testing.T) {
 	alert1.Labels["host"] = inst1
 	alert1.ID = hash(m1)
 
-	alert2, err := r.newAlert(m2, time.Now())
+	alert2, err := r.newAlert(m2, time.Now(), nil)
 	if err != nil {
 		t.Fatalf("faield to create alert: %s", err)
 	}
diff --git a/app/vmalert/notifier/alert_test.go b/app/vmalert/notifier/alert_test.go
index f663be1c0..bc4e7d2c3 100644
--- a/app/vmalert/notifier/alert_test.go
+++ b/app/vmalert/notifier/alert_test.go
@@ -2,6 +2,8 @@ package notifier
 
 import (
 	"testing"
+
+	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 )
 
 func TestAlert_ExecTemplate(t *testing.T) {
@@ -60,11 +62,41 @@ func TestAlert_ExecTemplate(t *testing.T) {
 				"exprEscapedPath":  "vm_rows%7B%5C%22label%5C%22=%5C%22bar%5C%22%7D%3E0",
 			},
 		},
+		{
+			name:  "query",
+			alert: &Alert{Expr: `vm_rows{"label"="bar"}>0`},
+			annotations: map[string]string{
+				"summary": `{{ query "foo" | first | value }}`,
+				"desc":    `{{ range query "bar" }}{{ . | label "foo" }} {{ . | value }};{{ end }}`,
+			},
+			expTpl: map[string]string{
+				"summary": "1",
+				"desc":    "bar 1;garply 2;",
+			},
+		},
 	}
 
+	qFn := func(q string) ([]datasource.Metric, error) {
+		return []datasource.Metric{
+			{
+				Labels: []datasource.Label{
+					{Name: "foo", Value: "bar"},
+					{Name: "baz", Value: "qux"},
+				},
+				Value: 1,
+			},
+			{
+				Labels: []datasource.Label{
+					{Name: "foo", Value: "garply"},
+					{Name: "baz", Value: "fred"},
+				},
+				Value: 2,
+			},
+		}, nil
+	}
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
-			tpl, err := tc.alert.ExecTemplate(tc.annotations)
+			tpl, err := tc.alert.ExecTemplate(qFn, tc.annotations)
 			if err != nil {
 				t.Fatal(err)
 			}