Repository: VictoriaMetrics/VictoriaMetrics
Pull request #4212 (2023-04-27 07:46:26)

vmalert: support reading rule from http url
fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3352

codecov[bot]: ## [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch coverage: **`24.00`**% and project coverage change: **`-0.06`** :warning:
> Comparison is base [(`b21a55f`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/b21a55febfad30070772563eaf725300299d2c6c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.59% compared to head [(`b88c1b6`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 61.54%.

> :exclamation: Current head b88c1b6 differs from pull request most recent head 9e4d551. Consider uploading reports for the commit 9e4d551 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #4212      +/-   ##
==========================================
- Coverage   61.59%   61.54%   -0.06%     
==========================================
  Files         325      325              
  Lines       58548    58567      +19     
==========================================
- Hits        36061    36043      -18     
- Misses      20515    20547      +32     
- Partials     1972     1977       +5     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `35.60% <ø> (ø)` | |
| [app/vmalert/config/fs.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvY29uZmlnL2ZzLmdv) | `51.31% <21.73%> (-14.76%)` | :arrow_down: |
| [app/vmalert/config/config.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvY29uZmlnL2NvbmZpZy5nbw==) | `82.97% <50.00%> (+0.43%)` | :arrow_up: |

... and [6 files with indirect coverage changes](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4212?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 974c3ef90..f3ff1488f 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -29,7 +29,8 @@ Use this feature for the following cases:
 * Recording and Alerting rules backfilling (aka `replay`). See [these docs](#rules-backfilling);
 * Lightweight and without extra dependencies.
 * Supports [reusable templates](#reusable-templates) for annotations;
-* Load of recording and alerting rules from local filesystem, GCS and S3.
+* Load of recording and alerting rules from local filesystem, GCS and S3;
+* Detect alerting rules which [don't match any series](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039).
 
 ## Limitations
 
@@ -812,6 +813,22 @@ and vmalert will start printing additional log messages:
 2022-09-15T13:36:56.153Z  DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
 ```
 
+### Never-firing alerts
+
+vmalert can detect if alert's expression doesn't match any time series in runtime. This problem usually happens
+when alerting expression selects time series which aren't present in the datasource (i.e. wrong `job` label)
+or there is a typo in the series selector (i.e. `env=rpod`). Such alerting rules will be marked with special icon in 
+vmalert's UI and exposed via `vmalert_alerting_rules_last_evaluation_series_fetched` metric. The metric's value will
+show how many time series were matched before the filtering by rule's expression. If metric's value is `-1`, then
+this feature is not supported by the datasource (old versions of VictoriaMetrics). The following expression can be
+used to detect rules matching no series:
+```
+max(vmalert_alerting_rules_last_evaluation_series_fetched) by(group, alertname) == 0
+```
+
+See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039).
+This feature is available only if vmalert is using VictoriaMetrics v1.90 or higher as a datasource.
+
 
 ## Profiling
 
diff --git a/app/vmalert/alerting.go b/app/vmalert/alerting.go
index 6d3970258..3d834673d 100644
--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@@ -47,10 +47,11 @@ type AlertingRule struct {
 }
 
 type alertingRuleMetrics struct {
-	errors  *utils.Gauge
-	pending *utils.Gauge
-	active  *utils.Gauge
-	samples *utils.Gauge
+	errors        *utils.Gauge
+	pending       *utils.Gauge
+	active        *utils.Gauge
+	samples       *utils.Gauge
+	seriesFetched *utils.Gauge
 }
 
 func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule) *AlertingRule {
@@ -121,6 +122,15 @@ func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule
 			e := ar.state.getLast()
 			return float64(e.samples)
 		})
+	ar.metrics.seriesFetched = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_alerting_rules_last_evaluation_series_fetched{%s}`, labels),
+		func() float64 {
+			e := ar.state.getLast()
+			if e.seriesFetched == nil {
+				// means seriesFetched is unsupported
+				return -1
+			}
+			return float64(*e.seriesFetched)
+		})
 	return ar
 }
 
@@ -130,6 +140,7 @@ func (ar *AlertingRule) Close() {
 	ar.metrics.pending.Unregister()
 	ar.metrics.errors.Unregister()
 	ar.metrics.samples.Unregister()
+	ar.metrics.seriesFetched.Unregister()
 }
 
 // String implements Stringer interface
@@ -234,7 +245,7 @@ func (ar *AlertingRule) toLabels(m datasource.Metric, qFn templates.QueryFn) (*l
 // to get time series for backfilling.
 // It returns ALERT and ALERT_FOR_STATE time series as result.
 func (ar *AlertingRule) ExecRange(ctx context.Context, start, end time.Time) ([]prompbmarshal.TimeSeries, error) {
-	series, err := ar.q.QueryRange(ctx, ar.Expr, start, end)
+	res, err := ar.q.QueryRange(ctx, ar.Expr, start, end)
 	if err != nil {
 		return nil, err
 	}
@@ -242,7 +253,7 @@ func (ar *AlertingRule) ExecRange(ctx context.Context, start, end time.Time) ([]
 	qFn := func(query string) ([]datasource.Metric, error) {
 		return nil, fmt.Errorf("`query` template isn't supported in replay mode")
 	}
-	for _, s := range series {
+	for _, s := range res.Data {
 		a, err := ar.newAlert(s, nil, time.Time{}, qFn) // initial alert
 		if err != nil {
 			return nil, fmt.Errorf("failed to create alert: %s", err)
@@ -282,14 +293,15 @@ const resolvedRetention = 15 * time.Minute
 // Based on the Querier results AlertingRule maintains notifier.Alerts
 func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
 	start := time.Now()
-	qMetrics, req, err := ar.q.Query(ctx, ar.Expr, ts)
+	res, req, err := ar.q.Query(ctx, ar.Expr, ts)
 	curState := ruleStateEntry{
-		time:     start,
-		at:       ts,
-		duration: time.Since(start),
-		samples:  len(qMetrics),
-		err:      err,
-		curl:     requestToCurl(req),
+		time:          start,
+		at:            ts,
+		duration:      time.Since(start),
+		samples:       len(res.Data),
+		seriesFetched: res.SeriesFetched,
+		err:           err,
+		curl:          requestToCurl(req),
 	}
 
 	defer func() {
@@ -315,11 +327,11 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 
 	qFn := func(query string) ([]datasource.Metric, error) {
 		res, _, err := ar.q.Query(ctx, query, ts)
-		return res, err
+		return res.Data, err
 	}
 	updated := make(map[uint64]struct{})
 	// update list of active alerts
-	for _, m := range qMetrics {
+	for _, m := range res.Data {
 		ls, err := ar.toLabels(m, qFn)
 		if err != nil {
 			curState.err = fmt.Errorf("failed to expand labels: %s", err)
@@ -485,22 +497,23 @@ func (ar *AlertingRule) AlertAPI(id uint64) *APIAlert {
 func (ar *AlertingRule) ToAPI() APIRule {
 	lastState := ar.state.getLast()
 	r := APIRule{
-		Type:           "alerting",
-		DatasourceType: ar.Type.String(),
-		Name:           ar.Name,
-		Query:          ar.Expr,
-		Duration:       ar.For.Seconds(),
-		Labels:         ar.Labels,
-		Annotations:    ar.Annotations,
-		LastEvaluation: lastState.time,
-		EvaluationTime: lastState.duration.Seconds(),
-		Health:         "ok",
-		State:          "inactive",
-		Alerts:         ar.AlertsToAPI(),
-		LastSamples:    lastState.samples,
-		MaxUpdates:     ar.state.size(),
-		Updates:        ar.state.getAll(),
-		Debug:          ar.Debug,
+		Type:              "alerting",
+		DatasourceType:    ar.Type.String(),
+		Name:              ar.Name,
+		Query:             ar.Expr,
+		Duration:          ar.For.Seconds(),
+		Labels:            ar.Labels,
+		Annotations:       ar.Annotations,
+		LastEvaluation:    lastState.time,
+		EvaluationTime:    lastState.duration.Seconds(),
+		Health:            "ok",
+		State:             "inactive",
+		Alerts:            ar.AlertsToAPI(),
+		LastSamples:       lastState.samples,
+		LastSeriesFetched: lastState.seriesFetched,
+		MaxUpdates:        ar.state.size(),
+		Updates:           ar.state.getAll(),
+		Debug:             ar.Debug,
 
 		// encode as strings to avoid rounding in JSON
 		ID:      fmt.Sprintf("%d", ar.ID()),
@@ -637,11 +650,12 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, ts ti
 
 		ar.logDebugf(ts, nil, "restoring alert state via query %q", expr)
 
-		qMetrics, _, err := q.Query(ctx, expr, ts)
+		res, _, err := q.Query(ctx, expr, ts)
 		if err != nil {
 			return err
 		}
 
+		qMetrics := res.Data
 		if len(qMetrics) < 1 {
 			ar.logDebugf(ts, nil, "no response was received from restore query")
 			continue
diff --git a/app/vmalert/config/config.go b/app/vmalert/config/config.go
index 0fc201ef9..438847d2e 100644
--- a/app/vmalert/config/config.go
+++ b/app/vmalert/config/config.go
@@ -209,8 +209,7 @@ var cLogger = &log.Logger{}
 func ParseSilent(pathPatterns []string, validateTplFn ValidateTplFn, validateExpressions bool) ([]Group, error) {
 	cLogger.Suppress(true)
 	defer cLogger.Suppress(false)
-
-	files, err := readFromFS(pathPatterns)
+	files, err := readFromFSOrHTTP(pathPatterns)
 	if err != nil {
 		return nil, fmt.Errorf("failed to read from the config: %s", err)
 	}
@@ -219,7 +218,7 @@ func ParseSilent(pathPatterns []string, validateTplFn ValidateTplFn, validateExp
 
 // Parse parses rule configs from given file patterns
 func Parse(pathPatterns []string, validateTplFn ValidateTplFn, validateExpressions bool) ([]Group, error) {
-	files, err := readFromFS(pathPatterns)
+	files, err := readFromFSOrHTTP(pathPatterns)
 	if err != nil {
 		return nil, fmt.Errorf("failed to read from the config: %s", err)
 	}
@@ -233,6 +232,34 @@ func Parse(pathPatterns []string, validateTplFn ValidateTplFn, validateExpressio
 	return groups, nil
 }
 
+// readFromFSOrHTTP reads path either from filesystem or from http if path starts with http or https.
+func readFromFSOrHTTP(paths []string) (map[string][]byte, error) {
+	var httpPaths []string
+	var fsPaths []string
+	for _, path := range paths {
+		if isHTTPURL(path) {
+			httpPaths = append(httpPaths, path)
+			continue
+		}
+		fsPaths = append(fsPaths, path)
+	}
+	result, err := readFromFS(fsPaths)
+	if err != nil {
+		return nil, err
+	}
+	httpResult, err := readFromHTTP(httpPaths)
+	if err != nil {
+		return nil, err
+	}
+	for k, v := range httpResult {
+		if _, ok := result[k]; ok {
+			return nil, fmt.Errorf("duplicate found for config name %q: config names must be unique", k)
+		}
+		result[k] = v
+	}
+	return result, nil
+}
+
 func parse(files map[string][]byte, validateTplFn ValidateTplFn, validateExpressions bool) ([]Group, error) {
 	errGroup := new(utils.ErrGroup)
 	var groups []Group
diff --git a/app/vmalert/config/fs.go b/app/vmalert/config/fs.go
index 372d8066f..609d6b8a6 100644
--- a/app/vmalert/config/fs.go
+++ b/app/vmalert/config/fs.go
@@ -2,6 +2,7 @@ package config
 
 import (
 	"fmt"
+	"net/url"
 	"strings"
 	"sync"
 	"time"
@@ -32,17 +33,16 @@ var (
 )
 
 // readFromFS parses the given path list and inits FS for each item.
-// Once inited, readFromFS will try to read and return files from each FS.
+// Once initialed, readFromFS will try to read and return files from each FS.
 // readFromFS returns an error if at least one FS failed to init.
 // The function can be called multiple times but each unique path
-// will be inited only once.
+// will be initialed only once.
 //
 // It is allowed to mix different FS types in path list.
 func readFromFS(paths []string) (map[string][]byte, error) {
 	var err error
 	result := make(map[string][]byte)
 	for _, path := range paths {
-
 		fsRegistryMu.Lock()
 		fs, ok := fsRegistry[path]
 		if !ok {
@@ -106,3 +106,9 @@ func newFS(path string) (FS, error) {
 		return nil, fmt.Errorf("unsupported scheme %q", scheme)
 	}
 }
+
+// isHTTPURL checks if a given targetURL is valid and contains a valid http scheme
+func isHTTPURL(targetURL string) bool {
+	parsed, err := url.Parse(targetURL)
+	return err == nil && (parsed.Scheme == "http" || parsed.Scheme == "https") && parsed.Host != ""
+}
diff --git a/app/vmalert/config/url.go b/app/vmalert/config/url.go
new file mode 100644
index 000000000..a89fdf313
--- /dev/null
+++ b/app/vmalert/config/url.go
@@ -0,0 +1,34 @@
+package config
+
+import (
+	"fmt"
+	"io"
+	"net/http"
+)
+
+// readFromHTTP reads config from http path.
+func readFromHTTP(paths []string) (map[string][]byte, error) {
+	result := make(map[string][]byte)
+	for _, path := range paths {
+		if _, ok := result[path]; ok {
+			return nil, fmt.Errorf("duplicate found for url path %q: url path must be unique", path)
+		}
+		resp, err := http.Get(path)
+		if err != nil {
+			return nil, fmt.Errorf("cannot fetch %q: %w", path, err)
+		}
+		data, err := io.ReadAll(resp.Body)
+		_ = resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			if len(data) > 4*1024 {
+				data = data[:4*1024]
+			}
+			return nil, fmt.Errorf("unexpected status code when fetching %q: %d, expecting %d; response: %q", path, resp.StatusCode, http.StatusOK, data)
+		}
+		if err != nil {
+			return nil, fmt.Errorf("cannot read %q: %s", path, err)
+		}
+		result[path] = data
+	}
+	return result, nil
+}
diff --git a/app/vmalert/datasource/datasource.go b/app/vmalert/datasource/datasource.go
index 3134d1c31..bae112d14 100644
--- a/app/vmalert/datasource/datasource.go
+++ b/app/vmalert/datasource/datasource.go
@@ -13,11 +13,22 @@ type Querier interface {
 	// It returns list of Metric in response, the http.Request used for sending query
 	// and error if any. Returned http.Request can't be reused and its body is already read.
 	// Query should stop once ctx is cancelled.
-	Query(ctx context.Context, query string, ts time.Time) ([]Metric, *http.Request, error)
+	Query(ctx context.Context, query string, ts time.Time) (Result, *http.Request, error)
 	// QueryRange executes range request with the given query on the given time range.
 	// It returns list of Metric in response and error if any.
 	// QueryRange should stop once ctx is cancelled.
-	QueryRange(ctx context.Context, query string, from, to time.Time) ([]Metric, error)
+	QueryRange(ctx context.Context, query string, from, to time.Time) (Result, error)
+}
+
+// Result represents expected response from the datasource
+type Result struct {
+	// Data contains list of received Metric
+	Data []Metric
+	// SeriesFetched contains amount of time series processed by datasource
+	// during query evaluation.
+	// If nil, then this feature is not supported by the datasource.
+	// SeriesFetched is supported by VictoriaMetrics since v1.90.
+	SeriesFetched *int
 }
 
 // QuerierBuilder builds Querier with given params.
diff --git a/app/vmalert/datasource/vm.go b/app/vmalert/datasource/vm.go
index fc538266f..2210b8e18 100644
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@@ -99,10 +99,10 @@ func NewVMStorage(baseURL string, authCfg *promauth.Config, lookBack time.Durati
 }
 
 // Query executes the given query and returns parsed response
-func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Metric, *http.Request, error) {
+func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) (Result, *http.Request, error) {
 	req, err := s.newRequestPOST()
 	if err != nil {
-		return nil, nil, err
+		return Result{}, nil, err
 	}
 
 	switch s.dataSourceType {
@@ -111,12 +111,12 @@ func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Me
 	case datasourceGraphite:
 		s.setGraphiteReqParams(req, query, ts)
 	default:
-		return nil, nil, fmt.Errorf("engine not found: %q", s.dataSourceType)
+		return Result{}, nil, fmt.Errorf("engine not found: %q", s.dataSourceType)
 	}
 
 	resp, err := s.do(ctx, req)
 	if err != nil {
-		return nil, req, err
+		return Result{}, req, err
 	}
 	defer func() {
 		_ = resp.Body.Close()
@@ -133,24 +133,24 @@ func (s *VMStorage) Query(ctx context.Context, query string, ts time.Time) ([]Me
 // QueryRange executes the given query on the given time range.
 // For Prometheus type see https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries
 // Graphite type isn't supported.
-func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end time.Time) ([]Metric, error) {
+func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end time.Time) (res Result, err error) {
 	if s.dataSourceType != datasourcePrometheus {
-		return nil, fmt.Errorf("%q is not supported for QueryRange", s.dataSourceType)
+		return res, fmt.Errorf("%q is not supported for QueryRange", s.dataSourceType)
 	}
 	req, err := s.newRequestPOST()
 	if err != nil {
-		return nil, err
+		return res, err
 	}
 	if start.IsZero() {
-		return nil, fmt.Errorf("start param is missing")
+		return res, fmt.Errorf("start param is missing")
 	}
 	if end.IsZero() {
-		return nil, fmt.Errorf("end param is missing")
+		return res, fmt.Errorf("end param is missing")
 	}
 	s.setPrometheusRangeReqParams(req, query, start, end)
 	resp, err := s.do(ctx, req)
 	if err != nil {
-		return nil, err
+		return res, err
 	}
 	defer func() {
 		_ = resp.Body.Close()
diff --git a/app/vmalert/datasource/vm_graphite_api.go b/app/vmalert/datasource/vm_graphite_api.go
index e4576f7a4..4ad5c2a6c 100644
--- a/app/vmalert/datasource/vm_graphite_api.go
+++ b/app/vmalert/datasource/vm_graphite_api.go
@@ -35,12 +35,12 @@ func (r graphiteResponse) metrics() []Metric {
 	return ms
 }
 
-func parseGraphiteResponse(req *http.Request, resp *http.Response) ([]Metric, error) {
+func parseGraphiteResponse(req *http.Request, resp *http.Response) (Result, error) {
 	r := &graphiteResponse{}
 	if err := json.NewDecoder(resp.Body).Decode(r); err != nil {
-		return nil, fmt.Errorf("error parsing graphite metrics for %s: %w", req.URL.Redacted(), err)
+		return Result{}, fmt.Errorf("error parsing graphite metrics for %s: %w", req.URL.Redacted(), err)
 	}
-	return r.metrics(), nil
+	return Result{Data: r.metrics()}, nil
 }
 
 const (
diff --git a/app/vmalert/datasource/vm_prom_api.go b/app/vmalert/datasource/vm_prom_api.go
index 7689626fb..950165f6a 100644
--- a/app/vmalert/datasource/vm_prom_api.go
+++ b/app/vmalert/datasource/vm_prom_api.go
@@ -22,6 +22,10 @@ type promResponse struct {
 		ResultType string          `json:"resultType"`
 		Result     json.RawMessage `json:"result"`
 	} `json:"data"`
+	// Stats supported by VictoriaMetrics since v1.90
+	Stats struct {
+		SeriesFetched *string `json:"seriesFetched,omitempty"`
+	} `json:"stats,omitempty"`
 }
 
 type promInstant struct {
@@ -96,39 +100,54 @@ const (
 	rtVector, rtMatrix, rScalar = "vector", "matrix", "scalar"
 )
 
-func parsePrometheusResponse(req *http.Request, resp *http.Response) ([]Metric, error) {
+func parsePrometheusResponse(req *http.Request, resp *http.Response) (res Result, err error) {
 	r := &promResponse{}
-	if err := json.NewDecoder(resp.Body).Decode(r); err != nil {
-		return nil, fmt.Errorf("error parsing prometheus metrics for %s: %w", req.URL.Redacted(), err)
+	if err = json.NewDecoder(resp.Body).Decode(r); err != nil {
+		return res, fmt.Errorf("error parsing prometheus metrics for %s: %w", req.URL.Redacted(), err)
 	}
 	if r.Status == statusError {
-		return nil, fmt.Errorf("response error, query: %s, errorType: %s, error: %s", req.URL.Redacted(), r.ErrorType, r.Error)
+		return res, fmt.Errorf("response error, query: %s, errorType: %s, error: %s", req.URL.Redacted(), r.ErrorType, r.Error)
 	}
 	if r.Status != statusSuccess {
-		return nil, fmt.Errorf("unknown status: %s, Expected success or error ", r.Status)
+		return res, fmt.Errorf("unknown status: %s, Expected success or error ", r.Status)
 	}
+	var parseFn func() ([]Metric, error)
 	switch r.Data.ResultType {
 	case rtVector:
 		var pi promInstant
 		if err := json.Unmarshal(r.Data.Result, &pi.Result); err != nil {
-			return nil, fmt.Errorf("umarshal err %s; \n %#v", err, string(r.Data.Result))
+			return res, fmt.Errorf("umarshal err %s; \n %#v", err, string(r.Data.Result))
 		}
-		return pi.metrics()
+		parseFn = pi.metrics
 	case rtMatrix:
 		var pr promRange
 		if err := json.Unmarshal(r.Data.Result, &pr.Result); err != nil {
-			return nil, err
+			return res, err
 		}
-		return pr.metrics()
+		parseFn = pr.metrics
 	case rScalar:
 		var ps promScalar
 		if err := json.Unmarshal(r.Data.Result, &ps); err != nil {
-			return nil, err
+			return res, err
 		}
-		return ps.metrics()
+		parseFn = ps.metrics
 	default:
-		return nil, fmt.Errorf("unknown result type %q", r.Data.ResultType)
+		return res, fmt.Errorf("unknown result type %q", r.Data.ResultType)
 	}
+
+	ms, err := parseFn()
+	if err != nil {
+		return res, err
+	}
+	res = Result{Data: ms}
+	if r.Stats.SeriesFetched != nil {
+		intV, err := strconv.Atoi(*r.Stats.SeriesFetched)
+		if err != nil {
+			return res, fmt.Errorf("failed to convert stats.seriesFetched to int: %w", err)
+		}
+		res.SeriesFetched = &intV
+	}
+	return res, nil
 }
 
 func (s *VMStorage) setPrometheusInstantReqParams(r *http.Request, query string, timestamp time.Time) {
diff --git a/app/vmalert/main.go b/app/vmalert/main.go
index 8b49029e2..cea9be80f 100644
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@@ -29,10 +29,11 @@ import (
 )
 
 var (
-	rulePath = flagutil.NewArrayString("rule", `Path to the files with alerting and/or recording rules.
+	rulePath = flagutil.NewArrayString("rule", `Path to the files or http url with alerting and/or recording rules.
 Supports hierarchical patterns and regexpes.
 Examples:
  -rule="/path/to/file". Path to a single file with alerting rules
+ -rule="http://<some-server-addr>:path/to/rules". HTTP URL to alerting rules.
  -rule="dir/*.yaml" -rule="/*.yaml" -rule="gcs://vmalert-rules/tenant_%{TENANT_ID}/prod". 
  -rule="dir/**/*.yaml". Includes to all .yaml files in "dir" folder and it's subfolders recursively. 
 Rule files may contain %{ENV_VAR} placeholders, which are substituted by the corresponding env vars.
diff --git a/app/vmalert/recording.go b/app/vmalert/recording.go
index 7cf0b2a33..fd137d05e 100644
--- a/app/vmalert/recording.go
+++ b/app/vmalert/recording.go
@@ -99,13 +99,13 @@ func (rr *RecordingRule) Close() {
 // It doesn't update internal states of the Rule and meant to be used just
 // to get time series for backfilling.
 func (rr *RecordingRule) ExecRange(ctx context.Context, start, end time.Time) ([]prompbmarshal.TimeSeries, error) {
-	series, err := rr.q.QueryRange(ctx, rr.Expr, start, end)
+	res, err := rr.q.QueryRange(ctx, rr.Expr, start, end)
 	if err != nil {
 		return nil, err
 	}
-	duplicates := make(map[string]struct{}, len(series))
+	duplicates := make(map[string]struct{}, len(res.Data))
 	var tss []prompbmarshal.TimeSeries
-	for _, s := range series {
+	for _, s := range res.Data {
 		ts := rr.toTimeSeries(s)
 		key := stringifyLabels(ts)
 		if _, ok := duplicates[key]; ok {
@@ -120,13 +120,14 @@ func (rr *RecordingRule) ExecRange(ctx context.Context, start, end time.Time) ([
 // Exec executes RecordingRule expression via the given Querier.
 func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
 	start := time.Now()
-	qMetrics, req, err := rr.q.Query(ctx, rr.Expr, ts)
+	res, req, err := rr.q.Query(ctx, rr.Expr, ts)
 	curState := ruleStateEntry{
-		time:     start,
-		at:       ts,
-		duration: time.Since(start),
-		samples:  len(qMetrics),
-		curl:     requestToCurl(req),
+		time:          start,
+		at:            ts,
+		duration:      time.Since(start),
+		samples:       len(res.Data),
+		seriesFetched: res.SeriesFetched,
+		curl:          requestToCurl(req),
 	}
 
 	defer func() {
@@ -138,6 +139,7 @@ func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]p
 		return nil, curState.err
 	}
 
+	qMetrics := res.Data
 	numSeries := len(qMetrics)
 	if limit > 0 && numSeries > limit {
 		curState.err = fmt.Errorf("exec exceeded limit of %d with %d series", limit, numSeries)
@@ -208,17 +210,18 @@ func (rr *RecordingRule) UpdateWith(r Rule) error {
 func (rr *RecordingRule) ToAPI() APIRule {
 	lastState := rr.state.getLast()
 	r := APIRule{
-		Type:           "recording",
-		DatasourceType: rr.Type.String(),
-		Name:           rr.Name,
-		Query:          rr.Expr,
-		Labels:         rr.Labels,
-		LastEvaluation: lastState.time,
-		EvaluationTime: lastState.duration.Seconds(),
-		Health:         "ok",
-		LastSamples:    lastState.samples,
-		MaxUpdates:     rr.state.size(),
-		Updates:        rr.state.getAll(),
+		Type:              "recording",
+		DatasourceType:    rr.Type.String(),
+		Name:              rr.Name,
+		Query:             rr.Expr,
+		Labels:            rr.Labels,
+		LastEvaluation:    lastState.time,
+		EvaluationTime:    lastState.duration.Seconds(),
+		Health:            "ok",
+		LastSamples:       lastState.samples,
+		LastSeriesFetched: lastState.seriesFetched,
+		MaxUpdates:        rr.state.size(),
+		Updates:           rr.state.getAll(),
 
 		// encode as strings to avoid rounding
 		ID:      fmt.Sprintf("%d", rr.ID()),
diff --git a/app/vmalert/rule.go b/app/vmalert/rule.go
index a46f44b15..d701e2de1 100644
--- a/app/vmalert/rule.go
+++ b/app/vmalert/rule.go
@@ -53,6 +53,12 @@ type ruleStateEntry struct {
 	// stores the number of samples returned during
 	// the last evaluation
 	samples int
+	// stores the number of time series fetched during
+	// the last evaluation.
+	// Is supported by VictoriaMetrics only, starting from v1.90.0
+	// If seriesFetched == nil, then this attribute was missing in
+	// datasource response (unsupported).
+	seriesFetched *int
 	// stores the curl command reflecting the HTTP request used during rule.Exec
 	curl string
 }
diff --git a/app/vmalert/tpl/footer.qtpl b/app/vmalert/tpl/footer.qtpl
index df1e3a4ba..4a5b0c319 100644
--- a/app/vmalert/tpl/footer.qtpl
+++ b/app/vmalert/tpl/footer.qtpl
@@ -38,6 +38,10 @@
                 group.click();
               }
             });
+
+            $(document).ready(function() {
+                $('[data-bs-toggle="tooltip"]').tooltip();
+            });
         </script>
     </body>
 </html>
diff --git a/app/vmalert/tpl/footer.qtpl.go b/app/vmalert/tpl/footer.qtpl.go
index 3f8b09801..bb9d3d831 100644
--- a/app/vmalert/tpl/footer.qtpl.go
+++ b/app/vmalert/tpl/footer.qtpl.go
@@ -73,35 +73,39 @@ func StreamFooter(qw422016 *qt422016.Writer, r *http.Request) {
                 group.click();
               }
             });
+
+            $(document).ready(function() {
+                $('[data-bs-toggle="tooltip"]').tooltip();
+            });
         </script>
     </body>
 </html>
 `)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 }
 
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 func WriteFooter(qq422016 qtio422016.Writer, r *http.Request) {
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	StreamFooter(qw422016, r)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 }
 
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 func Footer(r *http.Request) string {
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	WriteFooter(qb422016, r)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	qs422016 := string(qb422016.B)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 	return qs422016
-//line app/vmalert/tpl/footer.qtpl:44
+//line app/vmalert/tpl/footer.qtpl:48
 }
diff --git a/app/vmalert/web.qtpl b/app/vmalert/web.qtpl
index cd41ce487..f027459ba 100644
--- a/app/vmalert/web.qtpl
+++ b/app/vmalert/web.qtpl
@@ -91,7 +91,9 @@
                                         {% else %}
                                         <b>record:</b> {%s r.Name %}
                                         {% endif %}
-                                        | <span><a target="_blank" href="{%s prefix+r.WebLink() %}">Details</a></span>
+                                        |
+                                        {%= seriesFetchedWarn(r) %}
+                                        <span><a target="_blank" href="{%s prefix+r.WebLink() %}">Details</a></span>
                                     </div>
                                     <div class="col-12">
                                         <code><pre>{%s r.Query %}</pre></code>
@@ -377,8 +379,20 @@
             annotationKeys = append(annotationKeys, k)
         }
         sort.Strings(annotationKeys)
+
+        var seriesFetchedEnabled bool
+        var seriesFetchedWarning bool
+        for _, u := range rule.Updates {
+          if u.seriesFetched != nil {
+            seriesFetchedEnabled = true
+            if *u.seriesFetched == 0 && u.samples == 0{
+                seriesFetchedWarning = true
+            }
+          }
+        }
+
     %}
-    <div class="display-6 pb-3 mb-3">Rule: {%s rule.Name %}<span class="ms-2 badge {% if rule.Health!="ok" %}bg-danger{% else %} bg-warning text-dark{% endif %}">{%s rule.Health %}</span></div>
+    <div class="display-6 pb-3 mb-3">Rule: {%s rule.Name %}<span class="ms-2 badge {% if rule.Health!="ok" %}bg-danger{% else %} bg-success text-dark{% endif %}">{%s rule.Health %}</span></div>
     <div class="container border-bottom p-2">
       <div class="row">
         <div class="col-2">
@@ -450,12 +464,26 @@
     </div>
 
     <br>
+    {% if seriesFetchedWarning %}
+    <div class="alert alert-warning" role="alert">
+       <strong>Warning:</strong> some of updates have "Series fetched" equal to 0.<br>
+       It might be that either this data is missing in the datasource or there is a typo in rule's expression.
+       For example, <strong>foo{label="bar"} > 0</strong> could never trigger because <strong>foo{label="bar"}</strong>
+       metric doesn't exist.
+       <br>
+       Rule's expressions without time series selector, like <strong>expr: 42</strong> or <strong>expr: time()</strong>
+       aren't fetching time series from datasource, so they could have "Series fetched" equal to 0 and this won't be a problem.
+       <br>
+       See more details about this detection <a target="_blank" href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039">here</a>.
+    </div>
+    {% endif %}
     <div class="display-6 pb-3">Last {%d len(rule.Updates) %}/{%d rule.MaxUpdates %} updates</span>:</div>
         <table class="table table-striped table-hover table-sm">
             <thead>
                 <tr>
                     <th scope="col" title="The time when event was created">Updated at</th>
                     <th scope="col" style="width: 10%" class="text-center" title="How many samples were returned">Samples</th>
+                    {% if seriesFetchedEnabled %}<th scope="col" style="width: 10%" class="text-center" title="How many series were scanned by datasource during the evaluation">Series fetched</th>{% endif %}
                     <th scope="col" style="width: 10%" class="text-center" title="How many seconds request took">Duration</th>
                     <th scope="col" class="text-center" title="Time used for rule execution">Executed at</th>
                     <th scope="col" class="text-center" title="cURL command with request example">cURL</th>
@@ -468,7 +496,8 @@
                  <td>
                     <span class="badge bg-primary rounded-pill me-3" title="Updated at">{%s u.time.Format(time.RFC3339) %}</span>
                  </td>
-                 <td class="text-center" wi>{%d u.samples %}</td>
+                 <td class="text-center">{%d u.samples %}</td>
+                 {% if seriesFetchedEnabled %}<td class="text-center">{% if u.seriesFetched != nil %}{%d *u.seriesFetched %}{% endif %}</td>{% endif %}
                  <td class="text-center">{%f.3 u.duration.Seconds() %}s</td>
                  <td class="text-center">{%s u.at.Format(time.RFC3339) %}</td>
                  <td>
@@ -478,7 +507,7 @@
           </li>
           {% if u.err != nil %}
              <tr{% if u.err != nil %} class="alert-danger"{% endif %}>
-               <td colspan="5">
+               <td colspan="{% if seriesFetchedEnabled %}6{%else%}5{%endif%}">
                    <span class="alert-danger">{%v u.err %}</span>
                </td>
              </tr>
@@ -503,3 +532,16 @@
 {% func badgeRestored() %}
 <span class="badge bg-warning text-dark" title="Alert state was restored after the service restart from remote storage">restored</span>
 {% endfunc %}
+
+{% func seriesFetchedWarn(r APIRule) %}
+{% if r.LastSamples == 0 && r.LastSeriesFetched != nil && *r.LastSeriesFetched == 0 %}
+<svg xmlns="http://www.w3.org/2000/svg"
+    data-bs-toggle="tooltip"
+    title="This rule last evaluation hasn't selected any time series from the datasource.
+    It might be that either this data is missing in the datasource or there is a typo in rule's expression.
+    See more in Details."
+    width="18" height="18" fill="currentColor" class="bi bi-exclamation-triangle-fill flex-shrink-0 me-2" viewBox="0 0 16 16" role="img" aria-label="Warning:">
+       <path d="M8 16A8 8 0 1 0 8 0a8 8 0 0 0 0 16zm.93-9.412-1 4.705c-.07.34.029.533.304.533.194 0 .487-.07.686-.246l-.088.416c-.287.346-.92.598-1.465.598-.703 0-1.002-.422-.808-1.319l.738-3.468c.064-.293.006-.399-.287-.47l-.451-.081.082-.381 2.29-.287zM8 5.5a1 1 0 1 1 0-2 1 1 0 0 1 0 2z"/>
+</svg>
+{% endif %}
+{% endfunc %}
\ No newline at end of file
diff --git a/app/vmalert/web.qtpl.go b/app/vmalert/web.qtpl.go
index 0b584a94d..e170eb6d8 100644
--- a/app/vmalert/web.qtpl.go
+++ b/app/vmalert/web.qtpl.go
@@ -366,222 +366,228 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []APIGr
 				}
 //line app/vmalert/web.qtpl:93
 				qw422016.N().S(`
-                                        | <span><a target="_blank" href="`)
-//line app/vmalert/web.qtpl:94
+                                        |
+                                        `)
+//line app/vmalert/web.qtpl:95
+				streamseriesFetchedWarn(qw422016, r)
+//line app/vmalert/web.qtpl:95
+				qw422016.N().S(`
+                                        <span><a target="_blank" href="`)
+//line app/vmalert/web.qtpl:96
 				qw422016.E().S(prefix + r.WebLink())
-//line app/vmalert/web.qtpl:94
+//line app/vmalert/web.qtpl:96
 				qw422016.N().S(`">Details</a></span>
                                     </div>
                                     <div class="col-12">
                                         <code><pre>`)
-//line app/vmalert/web.qtpl:97
+//line app/vmalert/web.qtpl:99
 				qw422016.E().S(r.Query)
-//line app/vmalert/web.qtpl:97
+//line app/vmalert/web.qtpl:99
 				qw422016.N().S(`</pre></code>
                                     </div>
                                     <div class="col-12 mb-2">
                                         `)
-//line app/vmalert/web.qtpl:100
+//line app/vmalert/web.qtpl:102
 				if len(r.Labels) > 0 {
-//line app/vmalert/web.qtpl:100
+//line app/vmalert/web.qtpl:102
 					qw422016.N().S(` <b>Labels:</b>`)
-//line app/vmalert/web.qtpl:100
+//line app/vmalert/web.qtpl:102
 				}
-//line app/vmalert/web.qtpl:100
+//line app/vmalert/web.qtpl:102
 				qw422016.N().S(`
                                         `)
-//line app/vmalert/web.qtpl:101
+//line app/vmalert/web.qtpl:103
 				for k, v := range r.Labels {
-//line app/vmalert/web.qtpl:101
+//line app/vmalert/web.qtpl:103
 					qw422016.N().S(`
                                                 <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.E().S(k)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.E().S(v)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.N().S(`</span>
                                         `)
-//line app/vmalert/web.qtpl:103
+//line app/vmalert/web.qtpl:105
 				}
-//line app/vmalert/web.qtpl:103
+//line app/vmalert/web.qtpl:105
 				qw422016.N().S(`
                                     </div>
                                     `)
-//line app/vmalert/web.qtpl:105
+//line app/vmalert/web.qtpl:107
 				if r.LastError != "" {
-//line app/vmalert/web.qtpl:105
+//line app/vmalert/web.qtpl:107
 					qw422016.N().S(`
                                     <div class="col-12">
                                         <b>Error:</b>
                                         <div class="error-cell">
                                         `)
-//line app/vmalert/web.qtpl:109
+//line app/vmalert/web.qtpl:111
 					qw422016.E().S(r.LastError)
-//line app/vmalert/web.qtpl:109
+//line app/vmalert/web.qtpl:111
 					qw422016.N().S(`
                                         </div>
                                     </div>
                                     `)
-//line app/vmalert/web.qtpl:112
+//line app/vmalert/web.qtpl:114
 				}
-//line app/vmalert/web.qtpl:112
+//line app/vmalert/web.qtpl:114
 				qw422016.N().S(`
                                 </div>
                             </td>
                             <td class="text-center">`)
-//line app/vmalert/web.qtpl:115
+//line app/vmalert/web.qtpl:117
 				qw422016.N().D(r.LastSamples)
-//line app/vmalert/web.qtpl:115
+//line app/vmalert/web.qtpl:117
 				qw422016.N().S(`</td>
                             <td class="text-center">`)
-//line app/vmalert/web.qtpl:116
+//line app/vmalert/web.qtpl:118
 				qw422016.N().FPrec(time.Since(r.LastEvaluation).Seconds(), 3)
-//line app/vmalert/web.qtpl:116
+//line app/vmalert/web.qtpl:118
 				qw422016.N().S(`s ago</td>
                         </tr>
                     `)
-//line app/vmalert/web.qtpl:118
+//line app/vmalert/web.qtpl:120
 			}
-//line app/vmalert/web.qtpl:118
+//line app/vmalert/web.qtpl:120
 			qw422016.N().S(`
                  </tbody>
                 </table>
             </div>
         `)
-//line app/vmalert/web.qtpl:122
+//line app/vmalert/web.qtpl:124
 		}
-//line app/vmalert/web.qtpl:122
+//line app/vmalert/web.qtpl:124
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:124
+//line app/vmalert/web.qtpl:126
 	} else {
-//line app/vmalert/web.qtpl:124
+//line app/vmalert/web.qtpl:126
 		qw422016.N().S(`
         <div>
             <p>No groups...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:128
+//line app/vmalert/web.qtpl:130
 	}
-//line app/vmalert/web.qtpl:128
+//line app/vmalert/web.qtpl:130
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:130
+//line app/vmalert/web.qtpl:132
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:130
+//line app/vmalert/web.qtpl:132
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 }
 
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 func WriteListGroups(qq422016 qtio422016.Writer, r *http.Request, groups []APIGroup) {
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	StreamListGroups(qw422016, r, groups)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 }
 
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 func ListGroups(r *http.Request, groups []APIGroup) string {
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	WriteListGroups(qb422016, r, groups)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 	return qs422016
-//line app/vmalert/web.qtpl:132
+//line app/vmalert/web.qtpl:134
 }
 
-//line app/vmalert/web.qtpl:135
+//line app/vmalert/web.qtpl:137
 func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []GroupAlerts) {
-//line app/vmalert/web.qtpl:135
+//line app/vmalert/web.qtpl:137
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:136
+//line app/vmalert/web.qtpl:138
 	prefix := utils.Prefix(r.URL.Path)
 
-//line app/vmalert/web.qtpl:136
+//line app/vmalert/web.qtpl:138
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:137
+//line app/vmalert/web.qtpl:139
 	tpl.StreamHeader(qw422016, r, navItems, "Alerts")
-//line app/vmalert/web.qtpl:137
+//line app/vmalert/web.qtpl:139
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 	if len(groupAlerts) > 0 {
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 		qw422016.N().S(`
          <a class="btn btn-primary" role="button" onclick="collapseAll()">Collapse All</a>
          <a class="btn btn-primary" role="button" onclick="expandAll()">Expand All</a>
          `)
-//line app/vmalert/web.qtpl:141
+//line app/vmalert/web.qtpl:143
 		for _, ga := range groupAlerts {
-//line app/vmalert/web.qtpl:141
+//line app/vmalert/web.qtpl:143
 			qw422016.N().S(`
             `)
-//line app/vmalert/web.qtpl:142
+//line app/vmalert/web.qtpl:144
 			g := ga.Group
 
-//line app/vmalert/web.qtpl:142
+//line app/vmalert/web.qtpl:144
 			qw422016.N().S(`
             <div class="group-heading alert-danger" data-bs-target="rules-`)
-//line app/vmalert/web.qtpl:143
+//line app/vmalert/web.qtpl:145
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:143
+//line app/vmalert/web.qtpl:145
 			qw422016.N().S(`">
                 <span class="anchor" id="group-`)
-//line app/vmalert/web.qtpl:144
+//line app/vmalert/web.qtpl:146
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:144
+//line app/vmalert/web.qtpl:146
 			qw422016.N().S(`"></span>
                 <a href="#group-`)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			qw422016.E().S(g.Name)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			if g.Type != "prometheus" {
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 				qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 				qw422016.E().S(g.Type)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 				qw422016.N().S(`)`)
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			}
-//line app/vmalert/web.qtpl:145
+//line app/vmalert/web.qtpl:147
 			qw422016.N().S(`</a>
                 <span class="badge bg-danger" title="Number of active alerts">`)
-//line app/vmalert/web.qtpl:146
+//line app/vmalert/web.qtpl:148
 			qw422016.N().D(len(ga.Alerts))
-//line app/vmalert/web.qtpl:146
+//line app/vmalert/web.qtpl:148
 			qw422016.N().S(`</span>
                 <br>
                 <p class="fs-6 fw-lighter">`)
-//line app/vmalert/web.qtpl:148
+//line app/vmalert/web.qtpl:150
 			qw422016.E().S(g.File)
-//line app/vmalert/web.qtpl:148
+//line app/vmalert/web.qtpl:150
 			qw422016.N().S(`</p>
             </div>
             `)
-//line app/vmalert/web.qtpl:151
+//line app/vmalert/web.qtpl:153
 			var keys []string
 			alertsByRule := make(map[string][]*APIAlert)
 			for _, alert := range ga.Alerts {
@@ -592,20 +598,20 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
 			}
 			sort.Strings(keys)
 
-//line app/vmalert/web.qtpl:160
+//line app/vmalert/web.qtpl:162
 			qw422016.N().S(`
             <div class="collapse" id="rules-`)
-//line app/vmalert/web.qtpl:161
+//line app/vmalert/web.qtpl:163
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:161
+//line app/vmalert/web.qtpl:163
 			qw422016.N().S(`">
                 `)
-//line app/vmalert/web.qtpl:162
+//line app/vmalert/web.qtpl:164
 			for _, ruleID := range keys {
-//line app/vmalert/web.qtpl:162
+//line app/vmalert/web.qtpl:164
 				qw422016.N().S(`
                     `)
-//line app/vmalert/web.qtpl:164
+//line app/vmalert/web.qtpl:166
 				defaultAR := alertsByRule[ruleID][0]
 				var labelKeys []string
 				for k := range defaultAR.Labels {
@@ -613,28 +619,28 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
 				}
 				sort.Strings(labelKeys)
 
-//line app/vmalert/web.qtpl:170
+//line app/vmalert/web.qtpl:172
 				qw422016.N().S(`
                     <br>
                     <b>alert:</b> `)
-//line app/vmalert/web.qtpl:172
+//line app/vmalert/web.qtpl:174
 				qw422016.E().S(defaultAR.Name)
-//line app/vmalert/web.qtpl:172
+//line app/vmalert/web.qtpl:174
 				qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:172
+//line app/vmalert/web.qtpl:174
 				qw422016.N().D(len(alertsByRule[ruleID]))
-//line app/vmalert/web.qtpl:172
+//line app/vmalert/web.qtpl:174
 				qw422016.N().S(`)
                      | <span><a target="_blank" href="`)
-//line app/vmalert/web.qtpl:173
+//line app/vmalert/web.qtpl:175
 				qw422016.E().S(defaultAR.SourceLink)
-//line app/vmalert/web.qtpl:173
+//line app/vmalert/web.qtpl:175
 				qw422016.N().S(`">Source</a></span>
                     <br>
                     <b>expr:</b><code><pre>`)
-//line app/vmalert/web.qtpl:175
+//line app/vmalert/web.qtpl:177
 				qw422016.E().S(defaultAR.Expression)
-//line app/vmalert/web.qtpl:175
+//line app/vmalert/web.qtpl:177
 				qw422016.N().S(`</pre></code>
                     <table class="table table-striped table-hover table-sm">
                         <thead>
@@ -648,204 +654,204 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
                         </thead>
                         <tbody>
                         `)
-//line app/vmalert/web.qtpl:187
+//line app/vmalert/web.qtpl:189
 				for _, ar := range alertsByRule[ruleID] {
-//line app/vmalert/web.qtpl:187
+//line app/vmalert/web.qtpl:189
 					qw422016.N().S(`
                             <tr>
                                 <td>
                                     `)
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 					for _, k := range labelKeys {
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 						qw422016.N().S(`
                                         <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:191
+//line app/vmalert/web.qtpl:193
 						qw422016.E().S(k)
-//line app/vmalert/web.qtpl:191
+//line app/vmalert/web.qtpl:193
 						qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:191
+//line app/vmalert/web.qtpl:193
 						qw422016.E().S(ar.Labels[k])
-//line app/vmalert/web.qtpl:191
+//line app/vmalert/web.qtpl:193
 						qw422016.N().S(`</span>
                                     `)
-//line app/vmalert/web.qtpl:192
+//line app/vmalert/web.qtpl:194
 					}
-//line app/vmalert/web.qtpl:192
+//line app/vmalert/web.qtpl:194
 					qw422016.N().S(`
                                 </td>
                                 <td>`)
-//line app/vmalert/web.qtpl:194
+//line app/vmalert/web.qtpl:196
 					streambadgeState(qw422016, ar.State)
-//line app/vmalert/web.qtpl:194
+//line app/vmalert/web.qtpl:196
 					qw422016.N().S(`</td>
                                 <td>
                                     `)
-//line app/vmalert/web.qtpl:196
+//line app/vmalert/web.qtpl:198
 					qw422016.E().S(ar.ActiveAt.Format("2006-01-02T15:04:05Z07:00"))
-//line app/vmalert/web.qtpl:196
+//line app/vmalert/web.qtpl:198
 					qw422016.N().S(`
                                     `)
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 					if ar.Restored {
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 						streambadgeRestored(qw422016)
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 					}
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 					qw422016.N().S(`
                                 </td>
                                 <td>`)
-//line app/vmalert/web.qtpl:199
+//line app/vmalert/web.qtpl:201
 					qw422016.E().S(ar.Value)
-//line app/vmalert/web.qtpl:199
+//line app/vmalert/web.qtpl:201
 					qw422016.N().S(`</td>
                                 <td>
                                     <a href="`)
-//line app/vmalert/web.qtpl:201
+//line app/vmalert/web.qtpl:203
 					qw422016.E().S(prefix + ar.WebLink())
-//line app/vmalert/web.qtpl:201
+//line app/vmalert/web.qtpl:203
 					qw422016.N().S(`">Details</a>
                                 </td>
                             </tr>
                         `)
-//line app/vmalert/web.qtpl:204
+//line app/vmalert/web.qtpl:206
 				}
-//line app/vmalert/web.qtpl:204
+//line app/vmalert/web.qtpl:206
 				qw422016.N().S(`
                      </tbody>
                     </table>
                 `)
-//line app/vmalert/web.qtpl:207
+//line app/vmalert/web.qtpl:209
 			}
-//line app/vmalert/web.qtpl:207
+//line app/vmalert/web.qtpl:209
 			qw422016.N().S(`
             </div>
             <br>
         `)
-//line app/vmalert/web.qtpl:210
+//line app/vmalert/web.qtpl:212
 		}
-//line app/vmalert/web.qtpl:210
+//line app/vmalert/web.qtpl:212
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:212
+//line app/vmalert/web.qtpl:214
 	} else {
-//line app/vmalert/web.qtpl:212
+//line app/vmalert/web.qtpl:214
 		qw422016.N().S(`
         <div>
             <p>No active alerts...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:216
+//line app/vmalert/web.qtpl:218
 	}
-//line app/vmalert/web.qtpl:216
+//line app/vmalert/web.qtpl:218
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:218
+//line app/vmalert/web.qtpl:220
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:218
+//line app/vmalert/web.qtpl:220
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 }
 
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 func WriteListAlerts(qq422016 qtio422016.Writer, r *http.Request, groupAlerts []GroupAlerts) {
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	StreamListAlerts(qw422016, r, groupAlerts)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 }
 
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 func ListAlerts(r *http.Request, groupAlerts []GroupAlerts) string {
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	WriteListAlerts(qb422016, r, groupAlerts)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 	return qs422016
-//line app/vmalert/web.qtpl:220
+//line app/vmalert/web.qtpl:222
 }
 
-//line app/vmalert/web.qtpl:222
+//line app/vmalert/web.qtpl:224
 func StreamListTargets(qw422016 *qt422016.Writer, r *http.Request, targets map[notifier.TargetType][]notifier.Target) {
-//line app/vmalert/web.qtpl:222
+//line app/vmalert/web.qtpl:224
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:223
+//line app/vmalert/web.qtpl:225
 	tpl.StreamHeader(qw422016, r, navItems, "Notifiers")
-//line app/vmalert/web.qtpl:223
+//line app/vmalert/web.qtpl:225
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:224
+//line app/vmalert/web.qtpl:226
 	if len(targets) > 0 {
-//line app/vmalert/web.qtpl:224
+//line app/vmalert/web.qtpl:226
 		qw422016.N().S(`
          <a class="btn btn-primary" role="button" onclick="collapseAll()">Collapse All</a>
          <a class="btn btn-primary" role="button" onclick="expandAll()">Expand All</a>
 
          `)
-//line app/vmalert/web.qtpl:229
+//line app/vmalert/web.qtpl:231
 		var keys []string
 		for key := range targets {
 			keys = append(keys, string(key))
 		}
 		sort.Strings(keys)
 
-//line app/vmalert/web.qtpl:234
+//line app/vmalert/web.qtpl:236
 		qw422016.N().S(`
 
          `)
-//line app/vmalert/web.qtpl:236
+//line app/vmalert/web.qtpl:238
 		for i := range keys {
-//line app/vmalert/web.qtpl:236
+//line app/vmalert/web.qtpl:238
 			qw422016.N().S(`
            `)
-//line app/vmalert/web.qtpl:237
+//line app/vmalert/web.qtpl:239
 			typeK, ns := keys[i], targets[notifier.TargetType(keys[i])]
 			count := len(ns)
 
-//line app/vmalert/web.qtpl:239
+//line app/vmalert/web.qtpl:241
 			qw422016.N().S(`
            <div class="group-heading data-bs-target="rules-`)
-//line app/vmalert/web.qtpl:240
+//line app/vmalert/web.qtpl:242
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:240
+//line app/vmalert/web.qtpl:242
 			qw422016.N().S(`">
              <span class="anchor" id="notifiers-`)
-//line app/vmalert/web.qtpl:241
+//line app/vmalert/web.qtpl:243
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:241
+//line app/vmalert/web.qtpl:243
 			qw422016.N().S(`"></span>
              <a href="#notifiers-`)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.N().D(count)
-//line app/vmalert/web.qtpl:242
+//line app/vmalert/web.qtpl:244
 			qw422016.N().S(`)</a>
          </div>
          <div class="collapse show" id="notifiers-`)
-//line app/vmalert/web.qtpl:244
+//line app/vmalert/web.qtpl:246
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:244
+//line app/vmalert/web.qtpl:246
 			qw422016.N().S(`">
              <table class="table table-striped table-hover table-sm">
                  <thead>
@@ -856,119 +862,119 @@ func StreamListTargets(qw422016 *qt422016.Writer, r *http.Request, targets map[n
                  </thead>
                  <tbody>
                  `)
-//line app/vmalert/web.qtpl:253
+//line app/vmalert/web.qtpl:255
 			for _, n := range ns {
-//line app/vmalert/web.qtpl:253
+//line app/vmalert/web.qtpl:255
 				qw422016.N().S(`
                      <tr>
                          <td>
                               `)
-//line app/vmalert/web.qtpl:256
+//line app/vmalert/web.qtpl:258
 				for _, l := range n.Labels.GetLabels() {
-//line app/vmalert/web.qtpl:256
+//line app/vmalert/web.qtpl:258
 					qw422016.N().S(`
                                       <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:257
+//line app/vmalert/web.qtpl:259
 					qw422016.E().S(l.Name)
-//line app/vmalert/web.qtpl:257
+//line app/vmalert/web.qtpl:259
 					qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:257
+//line app/vmalert/web.qtpl:259
 					qw422016.E().S(l.Value)
-//line app/vmalert/web.qtpl:257
+//line app/vmalert/web.qtpl:259
 					qw422016.N().S(`</span>
                               `)
-//line app/vmalert/web.qtpl:258
+//line app/vmalert/web.qtpl:260
 				}
-//line app/vmalert/web.qtpl:258
+//line app/vmalert/web.qtpl:260
 				qw422016.N().S(`
                           </td>
                          <td>`)
-//line app/vmalert/web.qtpl:260
+//line app/vmalert/web.qtpl:262
 				qw422016.E().S(n.Notifier.Addr())
-//line app/vmalert/web.qtpl:260
+//line app/vmalert/web.qtpl:262
 				qw422016.N().S(`</td>
                      </tr>
                  `)
-//line app/vmalert/web.qtpl:262
+//line app/vmalert/web.qtpl:264
 			}
-//line app/vmalert/web.qtpl:262
+//line app/vmalert/web.qtpl:264
 			qw422016.N().S(`
               </tbody>
              </table>
          </div>
      `)
-//line app/vmalert/web.qtpl:266
+//line app/vmalert/web.qtpl:268
 		}
-//line app/vmalert/web.qtpl:266
+//line app/vmalert/web.qtpl:268
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:268
+//line app/vmalert/web.qtpl:270
 	} else {
-//line app/vmalert/web.qtpl:268
+//line app/vmalert/web.qtpl:270
 		qw422016.N().S(`
         <div>
             <p>No targets...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:272
+//line app/vmalert/web.qtpl:274
 	}
-//line app/vmalert/web.qtpl:272
+//line app/vmalert/web.qtpl:274
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:274
+//line app/vmalert/web.qtpl:276
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:274
+//line app/vmalert/web.qtpl:276
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 }
 
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 func WriteListTargets(qq422016 qtio422016.Writer, r *http.Request, targets map[notifier.TargetType][]notifier.Target) {
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	StreamListTargets(qw422016, r, targets)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 }
 
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 func ListTargets(r *http.Request, targets map[notifier.TargetType][]notifier.Target) string {
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	WriteListTargets(qb422016, r, targets)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 	return qs422016
-//line app/vmalert/web.qtpl:276
+//line app/vmalert/web.qtpl:278
 }
 
-//line app/vmalert/web.qtpl:278
+//line app/vmalert/web.qtpl:280
 func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
-//line app/vmalert/web.qtpl:278
+//line app/vmalert/web.qtpl:280
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:279
+//line app/vmalert/web.qtpl:281
 	prefix := utils.Prefix(r.URL.Path)
 
-//line app/vmalert/web.qtpl:279
+//line app/vmalert/web.qtpl:281
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:280
+//line app/vmalert/web.qtpl:282
 	tpl.StreamHeader(qw422016, r, navItems, "")
-//line app/vmalert/web.qtpl:280
+//line app/vmalert/web.qtpl:282
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:282
+//line app/vmalert/web.qtpl:284
 	var labelKeys []string
 	for k := range alert.Labels {
 		labelKeys = append(labelKeys, k)
@@ -981,28 +987,28 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
 	}
 	sort.Strings(annotationKeys)
 
-//line app/vmalert/web.qtpl:293
+//line app/vmalert/web.qtpl:295
 	qw422016.N().S(`
     <div class="display-6 pb-3 mb-3">Alert: `)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.E().S(alert.Name)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.N().S(`<span class="ms-2 badge `)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	if alert.State == "firing" {
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 		qw422016.N().S(`bg-danger`)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	} else {
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 		qw422016.N().S(` bg-warning text-dark`)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	}
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.E().S(alert.State)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.N().S(`</span></div>
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1011,9 +1017,9 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
           `)
-//line app/vmalert/web.qtpl:301
+//line app/vmalert/web.qtpl:303
 	qw422016.E().S(alert.ActiveAt.Format("2006-01-02T15:04:05Z07:00"))
-//line app/vmalert/web.qtpl:301
+//line app/vmalert/web.qtpl:303
 	qw422016.N().S(`
         </div>
       </div>
@@ -1025,9 +1031,9 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
           <code><pre>`)
-//line app/vmalert/web.qtpl:311
+//line app/vmalert/web.qtpl:313
 	qw422016.E().S(alert.Expression)
-//line app/vmalert/web.qtpl:311
+//line app/vmalert/web.qtpl:313
 	qw422016.N().S(`</pre></code>
         </div>
       </div>
@@ -1039,23 +1045,23 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            `)
-//line app/vmalert/web.qtpl:321
+//line app/vmalert/web.qtpl:323
 	for _, k := range labelKeys {
-//line app/vmalert/web.qtpl:321
+//line app/vmalert/web.qtpl:323
 		qw422016.N().S(`
                 <span class="m-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:322
+//line app/vmalert/web.qtpl:324
 		qw422016.E().S(k)
-//line app/vmalert/web.qtpl:322
+//line app/vmalert/web.qtpl:324
 		qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:322
+//line app/vmalert/web.qtpl:324
 		qw422016.E().S(alert.Labels[k])
-//line app/vmalert/web.qtpl:322
+//line app/vmalert/web.qtpl:324
 		qw422016.N().S(`</span>
           `)
-//line app/vmalert/web.qtpl:323
+//line app/vmalert/web.qtpl:325
 	}
-//line app/vmalert/web.qtpl:323
+//line app/vmalert/web.qtpl:325
 	qw422016.N().S(`
         </div>
       </div>
@@ -1067,24 +1073,24 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            `)
-//line app/vmalert/web.qtpl:333
+//line app/vmalert/web.qtpl:335
 	for _, k := range annotationKeys {
-//line app/vmalert/web.qtpl:333
+//line app/vmalert/web.qtpl:335
 		qw422016.N().S(`
                 <b>`)
-//line app/vmalert/web.qtpl:334
+//line app/vmalert/web.qtpl:336
 		qw422016.E().S(k)
-//line app/vmalert/web.qtpl:334
+//line app/vmalert/web.qtpl:336
 		qw422016.N().S(`:</b><br>
                 <p>`)
-//line app/vmalert/web.qtpl:335
+//line app/vmalert/web.qtpl:337
 		qw422016.E().S(alert.Annotations[k])
-//line app/vmalert/web.qtpl:335
+//line app/vmalert/web.qtpl:337
 		qw422016.N().S(`</p>
           `)
-//line app/vmalert/web.qtpl:336
+//line app/vmalert/web.qtpl:338
 	}
-//line app/vmalert/web.qtpl:336
+//line app/vmalert/web.qtpl:338
 	qw422016.N().S(`
         </div>
       </div>
@@ -1096,17 +1102,17 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            <a target="_blank" href="`)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.E().S(prefix)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.N().S(`groups#group-`)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.E().S(alert.GroupID)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.E().S(alert.GroupID)
-//line app/vmalert/web.qtpl:346
+//line app/vmalert/web.qtpl:348
 	qw422016.N().S(`</a>
         </div>
       </div>
@@ -1118,66 +1124,66 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            <a target="_blank" href="`)
-//line app/vmalert/web.qtpl:356
+//line app/vmalert/web.qtpl:358
 	qw422016.E().S(alert.SourceLink)
-//line app/vmalert/web.qtpl:356
+//line app/vmalert/web.qtpl:358
 	qw422016.N().S(`">Link</a>
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:360
+//line app/vmalert/web.qtpl:362
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:360
+//line app/vmalert/web.qtpl:362
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 }
 
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 func WriteAlert(qq422016 qtio422016.Writer, r *http.Request, alert *APIAlert) {
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	StreamAlert(qw422016, r, alert)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 }
 
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 func Alert(r *http.Request, alert *APIAlert) string {
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	WriteAlert(qb422016, r, alert)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 	return qs422016
-//line app/vmalert/web.qtpl:362
+//line app/vmalert/web.qtpl:364
 }
 
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:367
 func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule) {
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:367
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:366
+//line app/vmalert/web.qtpl:368
 	prefix := utils.Prefix(r.URL.Path)
 
-//line app/vmalert/web.qtpl:366
+//line app/vmalert/web.qtpl:368
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:367
+//line app/vmalert/web.qtpl:369
 	tpl.StreamHeader(qw422016, r, navItems, "")
-//line app/vmalert/web.qtpl:367
+//line app/vmalert/web.qtpl:369
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:371
 	var labelKeys []string
 	for k := range rule.Labels {
 		labelKeys = append(labelKeys, k)
@@ -1190,28 +1196,39 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
 	}
 	sort.Strings(annotationKeys)
 
-//line app/vmalert/web.qtpl:380
+	var seriesFetchedEnabled bool
+	var seriesFetchedWarning bool
+	for _, u := range rule.Updates {
+		if u.seriesFetched != nil {
+			seriesFetchedEnabled = true
+			if *u.seriesFetched == 0 && u.samples == 0 {
+				seriesFetchedWarning = true
+			}
+		}
+	}
+
+//line app/vmalert/web.qtpl:394
 	qw422016.N().S(`
     <div class="display-6 pb-3 mb-3">Rule: `)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	qw422016.E().S(rule.Name)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	qw422016.N().S(`<span class="ms-2 badge `)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	if rule.Health != "ok" {
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 		qw422016.N().S(`bg-danger`)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	} else {
-//line app/vmalert/web.qtpl:381
-		qw422016.N().S(` bg-warning text-dark`)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
+		qw422016.N().S(` bg-success text-dark`)
+//line app/vmalert/web.qtpl:395
 	}
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	qw422016.E().S(rule.Health)
-//line app/vmalert/web.qtpl:381
+//line app/vmalert/web.qtpl:395
 	qw422016.N().S(`</span></div>
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1220,17 +1237,17 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
           <code><pre>`)
-//line app/vmalert/web.qtpl:388
+//line app/vmalert/web.qtpl:402
 	qw422016.E().S(rule.Query)
-//line app/vmalert/web.qtpl:388
+//line app/vmalert/web.qtpl:402
 	qw422016.N().S(`</pre></code>
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:392
+//line app/vmalert/web.qtpl:406
 	if rule.Type == "alerting" {
-//line app/vmalert/web.qtpl:392
+//line app/vmalert/web.qtpl:406
 		qw422016.N().S(`
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1239,17 +1256,17 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
          `)
-//line app/vmalert/web.qtpl:399
+//line app/vmalert/web.qtpl:413
 		qw422016.E().V(rule.Duration)
-//line app/vmalert/web.qtpl:399
+//line app/vmalert/web.qtpl:413
 		qw422016.N().S(` seconds
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:403
+//line app/vmalert/web.qtpl:417
 	}
-//line app/vmalert/web.qtpl:403
+//line app/vmalert/web.qtpl:417
 	qw422016.N().S(`
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1258,31 +1275,31 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
           `)
-//line app/vmalert/web.qtpl:410
+//line app/vmalert/web.qtpl:424
 	for _, k := range labelKeys {
-//line app/vmalert/web.qtpl:410
+//line app/vmalert/web.qtpl:424
 		qw422016.N().S(`
                 <span class="m-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:411
+//line app/vmalert/web.qtpl:425
 		qw422016.E().S(k)
-//line app/vmalert/web.qtpl:411
+//line app/vmalert/web.qtpl:425
 		qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:411
+//line app/vmalert/web.qtpl:425
 		qw422016.E().S(rule.Labels[k])
-//line app/vmalert/web.qtpl:411
+//line app/vmalert/web.qtpl:425
 		qw422016.N().S(`</span>
           `)
-//line app/vmalert/web.qtpl:412
+//line app/vmalert/web.qtpl:426
 	}
-//line app/vmalert/web.qtpl:412
+//line app/vmalert/web.qtpl:426
 	qw422016.N().S(`
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:416
+//line app/vmalert/web.qtpl:430
 	if rule.Type == "alerting" {
-//line app/vmalert/web.qtpl:416
+//line app/vmalert/web.qtpl:430
 		qw422016.N().S(`
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1291,24 +1308,24 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
           `)
-//line app/vmalert/web.qtpl:423
+//line app/vmalert/web.qtpl:437
 		for _, k := range annotationKeys {
-//line app/vmalert/web.qtpl:423
+//line app/vmalert/web.qtpl:437
 			qw422016.N().S(`
                 <b>`)
-//line app/vmalert/web.qtpl:424
+//line app/vmalert/web.qtpl:438
 			qw422016.E().S(k)
-//line app/vmalert/web.qtpl:424
+//line app/vmalert/web.qtpl:438
 			qw422016.N().S(`:</b><br>
                 <p>`)
-//line app/vmalert/web.qtpl:425
+//line app/vmalert/web.qtpl:439
 			qw422016.E().S(rule.Annotations[k])
-//line app/vmalert/web.qtpl:425
+//line app/vmalert/web.qtpl:439
 			qw422016.N().S(`</p>
           `)
-//line app/vmalert/web.qtpl:426
+//line app/vmalert/web.qtpl:440
 		}
-//line app/vmalert/web.qtpl:426
+//line app/vmalert/web.qtpl:440
 		qw422016.N().S(`
         </div>
       </div>
@@ -1320,17 +1337,17 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
            `)
-//line app/vmalert/web.qtpl:436
+//line app/vmalert/web.qtpl:450
 		qw422016.E().V(rule.Debug)
-//line app/vmalert/web.qtpl:436
+//line app/vmalert/web.qtpl:450
 		qw422016.N().S(`
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:440
+//line app/vmalert/web.qtpl:454
 	}
-//line app/vmalert/web.qtpl:440
+//line app/vmalert/web.qtpl:454
 	qw422016.N().S(`
     <div class="container border-bottom p-2">
       <div class="row">
@@ -1339,37 +1356,67 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
         </div>
         <div class="col">
            <a target="_blank" href="`)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.E().S(prefix)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.N().S(`groups#group-`)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.E().S(rule.GroupID)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.E().S(rule.GroupID)
-//line app/vmalert/web.qtpl:447
+//line app/vmalert/web.qtpl:461
 	qw422016.N().S(`</a>
         </div>
       </div>
     </div>
 
     <br>
+    `)
+//line app/vmalert/web.qtpl:467
+	if seriesFetchedWarning {
+//line app/vmalert/web.qtpl:467
+		qw422016.N().S(`
+    <div class="alert alert-warning" role="alert">
+       <strong>Warning:</strong> some of updates have "Series fetched" equal to 0.<br>
+       It might be that either this data is missing in the datasource or there is a typo in rule's expression.
+       For example, <strong>foo{label="bar"} > 0</strong> could never trigger because <strong>foo{label="bar"}</strong>
+       metric doesn't exist.
+       <br>
+       Rule's expressions without time series selector, like <strong>expr: 42</strong> or <strong>expr: time()</strong>
+       aren't fetching time series from datasource, so they could have "Series fetched" equal to 0 and this won't be a problem.
+       <br>
+       See more details about this detection <a target="_blank" href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039">here</a>.
+    </div>
+    `)
+//line app/vmalert/web.qtpl:479
+	}
+//line app/vmalert/web.qtpl:479
+	qw422016.N().S(`
     <div class="display-6 pb-3">Last `)
-//line app/vmalert/web.qtpl:453
+//line app/vmalert/web.qtpl:480
 	qw422016.N().D(len(rule.Updates))
-//line app/vmalert/web.qtpl:453
+//line app/vmalert/web.qtpl:480
 	qw422016.N().S(`/`)
-//line app/vmalert/web.qtpl:453
+//line app/vmalert/web.qtpl:480
 	qw422016.N().D(rule.MaxUpdates)
-//line app/vmalert/web.qtpl:453
+//line app/vmalert/web.qtpl:480
 	qw422016.N().S(` updates</span>:</div>
         <table class="table table-striped table-hover table-sm">
             <thead>
                 <tr>
                     <th scope="col" title="The time when event was created">Updated at</th>
                     <th scope="col" style="width: 10%" class="text-center" title="How many samples were returned">Samples</th>
+                    `)
+//line app/vmalert/web.qtpl:486
+	if seriesFetchedEnabled {
+//line app/vmalert/web.qtpl:486
+		qw422016.N().S(`<th scope="col" style="width: 10%" class="text-center" title="How many series were scanned by datasource during the evaluation">Series fetched</th>`)
+//line app/vmalert/web.qtpl:486
+	}
+//line app/vmalert/web.qtpl:486
+	qw422016.N().S(`
                     <th scope="col" style="width: 10%" class="text-center" title="How many seconds request took">Duration</th>
                     <th scope="col" class="text-center" title="Time used for rule execution">Executed at</th>
                     <th scope="col" class="text-center" title="cURL command with request example">cURL</th>
@@ -1378,201 +1425,282 @@ func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule)
             <tbody>
 
      `)
-//line app/vmalert/web.qtpl:466
+//line app/vmalert/web.qtpl:494
 	for _, u := range rule.Updates {
-//line app/vmalert/web.qtpl:466
+//line app/vmalert/web.qtpl:494
 		qw422016.N().S(`
              <tr`)
-//line app/vmalert/web.qtpl:467
+//line app/vmalert/web.qtpl:495
 		if u.err != nil {
-//line app/vmalert/web.qtpl:467
+//line app/vmalert/web.qtpl:495
 			qw422016.N().S(` class="alert-danger"`)
-//line app/vmalert/web.qtpl:467
+//line app/vmalert/web.qtpl:495
 		}
-//line app/vmalert/web.qtpl:467
+//line app/vmalert/web.qtpl:495
 		qw422016.N().S(`>
                  <td>
                     <span class="badge bg-primary rounded-pill me-3" title="Updated at">`)
-//line app/vmalert/web.qtpl:469
+//line app/vmalert/web.qtpl:497
 		qw422016.E().S(u.time.Format(time.RFC3339))
-//line app/vmalert/web.qtpl:469
+//line app/vmalert/web.qtpl:497
 		qw422016.N().S(`</span>
                  </td>
-                 <td class="text-center" wi>`)
-//line app/vmalert/web.qtpl:471
+                 <td class="text-center">`)
+//line app/vmalert/web.qtpl:499
 		qw422016.N().D(u.samples)
-//line app/vmalert/web.qtpl:471
+//line app/vmalert/web.qtpl:499
 		qw422016.N().S(`</td>
+                 `)
+//line app/vmalert/web.qtpl:500
+		if seriesFetchedEnabled {
+//line app/vmalert/web.qtpl:500
+			qw422016.N().S(`<td class="text-center">`)
+//line app/vmalert/web.qtpl:500
+			if u.seriesFetched != nil {
+//line app/vmalert/web.qtpl:500
+				qw422016.N().D(*u.seriesFetched)
+//line app/vmalert/web.qtpl:500
+			}
+//line app/vmalert/web.qtpl:500
+			qw422016.N().S(`</td>`)
+//line app/vmalert/web.qtpl:500
+		}
+//line app/vmalert/web.qtpl:500
+		qw422016.N().S(`
                  <td class="text-center">`)
-//line app/vmalert/web.qtpl:472
+//line app/vmalert/web.qtpl:501
 		qw422016.N().FPrec(u.duration.Seconds(), 3)
-//line app/vmalert/web.qtpl:472
+//line app/vmalert/web.qtpl:501
 		qw422016.N().S(`s</td>
                  <td class="text-center">`)
-//line app/vmalert/web.qtpl:473
+//line app/vmalert/web.qtpl:502
 		qw422016.E().S(u.at.Format(time.RFC3339))
-//line app/vmalert/web.qtpl:473
+//line app/vmalert/web.qtpl:502
 		qw422016.N().S(`</td>
                  <td>
                     <textarea class="curl-area" rows="1" onclick="this.focus();this.select()">`)
-//line app/vmalert/web.qtpl:475
+//line app/vmalert/web.qtpl:504
 		qw422016.E().S(u.curl)
-//line app/vmalert/web.qtpl:475
+//line app/vmalert/web.qtpl:504
 		qw422016.N().S(`</textarea>
                 </td>
              </tr>
           </li>
           `)
-//line app/vmalert/web.qtpl:479
+//line app/vmalert/web.qtpl:508
 		if u.err != nil {
-//line app/vmalert/web.qtpl:479
+//line app/vmalert/web.qtpl:508
 			qw422016.N().S(`
              <tr`)
-//line app/vmalert/web.qtpl:480
+//line app/vmalert/web.qtpl:509
 			if u.err != nil {
-//line app/vmalert/web.qtpl:480
+//line app/vmalert/web.qtpl:509
 				qw422016.N().S(` class="alert-danger"`)
-//line app/vmalert/web.qtpl:480
+//line app/vmalert/web.qtpl:509
 			}
-//line app/vmalert/web.qtpl:480
+//line app/vmalert/web.qtpl:509
 			qw422016.N().S(`>
-               <td colspan="5">
+               <td colspan="`)
+//line app/vmalert/web.qtpl:510
+			if seriesFetchedEnabled {
+//line app/vmalert/web.qtpl:510
+				qw422016.N().S(`6`)
+//line app/vmalert/web.qtpl:510
+			} else {
+//line app/vmalert/web.qtpl:510
+				qw422016.N().S(`5`)
+//line app/vmalert/web.qtpl:510
+			}
+//line app/vmalert/web.qtpl:510
+			qw422016.N().S(`">
                    <span class="alert-danger">`)
-//line app/vmalert/web.qtpl:482
+//line app/vmalert/web.qtpl:511
 			qw422016.E().V(u.err)
-//line app/vmalert/web.qtpl:482
+//line app/vmalert/web.qtpl:511
 			qw422016.N().S(`</span>
                </td>
              </tr>
           `)
-//line app/vmalert/web.qtpl:485
+//line app/vmalert/web.qtpl:514
 		}
-//line app/vmalert/web.qtpl:485
+//line app/vmalert/web.qtpl:514
 		qw422016.N().S(`
      `)
-//line app/vmalert/web.qtpl:486
+//line app/vmalert/web.qtpl:515
 	}
-//line app/vmalert/web.qtpl:486
+//line app/vmalert/web.qtpl:515
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:488
+//line app/vmalert/web.qtpl:517
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:488
+//line app/vmalert/web.qtpl:517
 	qw422016.N().S(`
 `)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 }
 
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 func WriteRuleDetails(qq422016 qtio422016.Writer, r *http.Request, rule APIRule) {
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	StreamRuleDetails(qw422016, r, rule)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 }
 
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 func RuleDetails(r *http.Request, rule APIRule) string {
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	WriteRuleDetails(qb422016, r, rule)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 	return qs422016
-//line app/vmalert/web.qtpl:489
+//line app/vmalert/web.qtpl:518
 }
 
-//line app/vmalert/web.qtpl:493
+//line app/vmalert/web.qtpl:522
 func streambadgeState(qw422016 *qt422016.Writer, state string) {
-//line app/vmalert/web.qtpl:493
+//line app/vmalert/web.qtpl:522
 	qw422016.N().S(`
 `)
-//line app/vmalert/web.qtpl:495
+//line app/vmalert/web.qtpl:524
 	badgeClass := "bg-warning text-dark"
 	if state == "firing" {
 		badgeClass = "bg-danger"
 	}
 
-//line app/vmalert/web.qtpl:499
+//line app/vmalert/web.qtpl:528
 	qw422016.N().S(`
 <span class="badge `)
-//line app/vmalert/web.qtpl:500
+//line app/vmalert/web.qtpl:529
 	qw422016.E().S(badgeClass)
-//line app/vmalert/web.qtpl:500
+//line app/vmalert/web.qtpl:529
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:500
+//line app/vmalert/web.qtpl:529
 	qw422016.E().S(state)
-//line app/vmalert/web.qtpl:500
+//line app/vmalert/web.qtpl:529
 	qw422016.N().S(`</span>
 `)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 }
 
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 func writebadgeState(qq422016 qtio422016.Writer, state string) {
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	streambadgeState(qw422016, state)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 }
 
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 func badgeState(state string) string {
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	writebadgeState(qb422016, state)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 	return qs422016
-//line app/vmalert/web.qtpl:501
+//line app/vmalert/web.qtpl:530
 }
 
-//line app/vmalert/web.qtpl:503
+//line app/vmalert/web.qtpl:532
 func streambadgeRestored(qw422016 *qt422016.Writer) {
-//line app/vmalert/web.qtpl:503
+//line app/vmalert/web.qtpl:532
 	qw422016.N().S(`
 <span class="badge bg-warning text-dark" title="Alert state was restored after the service restart from remote storage">restored</span>
 `)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 }
 
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 func writebadgeRestored(qq422016 qtio422016.Writer) {
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 	streambadgeRestored(qw422016)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 }
 
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 func badgeRestored() string {
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
 	writebadgeRestored(qb422016)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:534
+	qs422016 := string(qb422016.B)
+//line app/vmalert/web.qtpl:534
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/web.qtpl:534
+	return qs422016
+//line app/vmalert/web.qtpl:534
+}
+
+//line app/vmalert/web.qtpl:536
+func streamseriesFetchedWarn(qw422016 *qt422016.Writer, r APIRule) {
+//line app/vmalert/web.qtpl:536
+	qw422016.N().S(`
+`)
+//line app/vmalert/web.qtpl:537
+	if r.LastSamples == 0 && r.LastSeriesFetched != nil && *r.LastSeriesFetched == 0 {
+//line app/vmalert/web.qtpl:537
+		qw422016.N().S(`
+<svg xmlns="http://www.w3.org/2000/svg"
+    data-bs-toggle="tooltip"
+    title="This rule last evaluation hasn't selected any time series from the datasource.
+    It might be that either this data is missing in the datasource or there is a typo in rule's expression.
+    See more in Details."
+    width="18" height="18" fill="currentColor" class="bi bi-exclamation-triangle-fill flex-shrink-0 me-2" viewBox="0 0 16 16" role="img" aria-label="Warning:">
+       <path d="M8 16A8 8 0 1 0 8 0a8 8 0 0 0 0 16zm.93-9.412-1 4.705c-.07.34.029.533.304.533.194 0 .487-.07.686-.246l-.088.416c-.287.346-.92.598-1.465.598-.703 0-1.002-.422-.808-1.319l.738-3.468c.064-.293.006-.399-.287-.47l-.451-.081.082-.381 2.29-.287zM8 5.5a1 1 0 1 1 0-2 1 1 0 0 1 0 2z"/>
+</svg>
+`)
+//line app/vmalert/web.qtpl:546
+	}
+//line app/vmalert/web.qtpl:546
+	qw422016.N().S(`
+`)
+//line app/vmalert/web.qtpl:547
+}
+
+//line app/vmalert/web.qtpl:547
+func writeseriesFetchedWarn(qq422016 qtio422016.Writer, r APIRule) {
+//line app/vmalert/web.qtpl:547
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/web.qtpl:547
+	streamseriesFetchedWarn(qw422016, r)
+//line app/vmalert/web.qtpl:547
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/web.qtpl:547
+}
+
+//line app/vmalert/web.qtpl:547
+func seriesFetchedWarn(r APIRule) string {
+//line app/vmalert/web.qtpl:547
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/web.qtpl:547
+	writeseriesFetchedWarn(qb422016, r)
+//line app/vmalert/web.qtpl:547
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:547
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:547
 	return qs422016
-//line app/vmalert/web.qtpl:505
+//line app/vmalert/web.qtpl:547
 }
diff --git a/app/vmalert/web_types.go b/app/vmalert/web_types.go
index a23163216..f9a162769 100644
--- a/app/vmalert/web_types.go
+++ b/app/vmalert/web_types.go
@@ -117,7 +117,12 @@ type APIRule struct {
 
 	// DatasourceType of the rule: prometheus or graphite
 	DatasourceType string `json:"datasourceType"`
-	LastSamples    int    `json:"lastSamples"`
+	// LastSamples stores the amount of data samples received on last evaluation
+	LastSamples int `json:"lastSamples"`
+	// LastSeriesFetched stores the amount of time series fetched by datasource
+	// during the last evaluation
+	LastSeriesFetched *int `json:"lastSeriesFetched,omitempty"`
+
 	// ID is a unique Alert's ID within a group
 	ID string `json:"id"`
 	// GroupID is an unique Group's ID
diff --git a/deployment/docker/Makefile b/deployment/docker/Makefile
index f082cbbeb..6a0e4077f 100644
--- a/deployment/docker/Makefile
+++ b/deployment/docker/Makefile
@@ -1,6 +1,6 @@
 # All these commands must run from repository root.
 
-DOCKER_NAMESPACE := victoriametrics
+DOCKER_NAMESPACE ?= victoriametrics
 
 ROOT_IMAGE ?= alpine:3.17.3
 CERTS_IMAGE := alpine:3.17.3
diff --git a/deployment/docker/alerts-health.yml b/deployment/docker/alerts-health.yml
index 63f66c8c2..0a58cc853 100644
--- a/deployment/docker/alerts-health.yml
+++ b/deployment/docker/alerts-health.yml
@@ -34,12 +34,12 @@ groups:
           Consider to increase the limit as fast as possible."
 
       - alert: TooHighMemoryUsage
-        expr: (process_resident_memory_anon_bytes / vm_available_memory_bytes) > 0.9
+        expr: (process_resident_memory_anon_bytes / vm_available_memory_bytes) > 0.8
         for: 5m
         labels:
           severity: critical
         annotations:
-          summary: "It is more than 90% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m"
+          summary: "It is more than 80% of memory used by \"{{ $labels.job }}\"(\"{{ $labels.instance }}\") during the last 5m"
           description: "Too high memory usage may result into multiple issues such as OOMs or degraded performance.
            Consider to either increase available memory or decrease the load on the process."
 
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 029d66090..068b7b1a9 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -40,6 +40,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add ability to proxy requests for unauthorized users. See [this doc](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4083).
 * FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add ability to specify default route (`default_url`) for processing non-matched requests. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084). 
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): support configuring of custom HTTP headers sent to notifiers on the Group level. See  [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260).
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): detect alerting rules which don't match any series. See this [issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039) for details.
 * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): add `-s3StorageClass` command-line flag for setting the storage class for AWS S3 backups. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4164). Thanks to @justcompile for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4166).
 
 * BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores.
@@ -52,9 +53,10 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix issue where vmui would freeze when adding a query that returned regular rows alongside a heatmap query.
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): properly display an error when using `query` function for templating value of `-external.alert.source` flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4181).
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221).
-* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details.
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `__meta_kubernetes_endpoints_name` label for all ports discovered from endpoint. Previously, ports not matched by `Service` did not have this label. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154) for details. Thanks to @thunderbird86 for discovering and [fixing](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4253) the issue.
 * BUGFIX: fix indexdb rotation getting in infinite loop when using `retentionTimezoneOffset` and local timezone is not UTC. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207) for details. Thanks to @faceair for the fix.
 * BUGFIX: max value for `memory.allowedPercent` changed from 200 to 100. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4171).
+* BUGFIX: [alerts-health](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/alerts-health.yml): update threshold for TooHighMemoryUsage from 90% to 80%. 
 
 ## [v1.90.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.90.0)
 
diff --git a/docs/vmalert.md b/docs/vmalert.md
index 751e6dd0e..c8ac6ef10 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -33,7 +33,8 @@ Use this feature for the following cases:
 * Recording and Alerting rules backfilling (aka `replay`). See [these docs](#rules-backfilling);
 * Lightweight and without extra dependencies.
 * Supports [reusable templates](#reusable-templates) for annotations;
-* Load of recording and alerting rules from local filesystem, GCS and S3.
+* Load of recording and alerting rules from local filesystem, GCS and S3;
+* Detect alerting rules which [don't match any series](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039).
 
 ## Limitations
 
@@ -816,6 +817,22 @@ and vmalert will start printing additional log messages:
 2022-09-15T13:36:56.153Z  DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
 ```
 
+### Never-firing alerts
+
+vmalert can detect if alert's expression doesn't match any time series in runtime. This problem usually happens
+when alerting expression selects time series which aren't present in the datasource (i.e. wrong `job` label)
+or there is a typo in the series selector (i.e. `env=rpod`). Such alerting rules will be marked with special icon in 
+vmalert's UI and exposed via `vmalert_alerting_rules_last_evaluation_series_fetched` metric. The metric's value will
+show how many time series were matched before the filtering by rule's expression. If metric's value is `-1`, then
+this feature is not supported by the datasource (old versions of VictoriaMetrics). The following expression can be
+used to detect rules matching no series:
+```
+max(vmalert_alerting_rules_last_evaluation_series_fetched) by(group, alertname) == 0
+```
+
+See more details [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039).
+This feature is available only if vmalert is using VictoriaMetrics v1.90 or higher as a datasource.
+
 
 ## Profiling
 
diff --git a/lib/promscrape/discovery/kubernetes/endpoints.go b/lib/promscrape/discovery/kubernetes/endpoints.go
index 25497f4f0..7ab1a3591 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints.go
@@ -114,10 +114,6 @@ func (eps *Endpoints) getTargetLabels(gw *groupWatcher) []*promutils.Labels {
 		logger.Warnf(`the number of targets for "role: endpoints" %q exceeds 1000 and will be truncated in the next k8s releases; please use "role: endpointslice" instead`, eps.Metadata.key())
 	}
 
-	// Prometheus sets endpoints_name and namespace labels for all endpoints
-	// Even if port is not matching service port.
-	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154
-	commonEpLabels := eps.getCommonLabels()
 	// Append labels for skipped ports on seen pods.
 	portSeen := func(port int, ports []int) bool {
 		for _, p := range ports {
@@ -138,7 +134,11 @@ func (eps *Endpoints) getTargetLabels(gw *groupWatcher) []*promutils.Labels {
 				m.Add("__address__", addr)
 				p.appendCommonLabels(m, gw)
 				p.appendContainerLabels(m, c, &cp)
-				m.AddFrom(commonEpLabels)
+
+				// Prometheus sets endpoints_name and namespace labels for all endpoints
+				// Even if port is not matching service port.
+				// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154
+				p.appendEndpointLabels(m, eps)
 				if svc != nil {
 					svc.appendCommonLabels(m)
 				}
@@ -151,13 +151,6 @@ func (eps *Endpoints) getTargetLabels(gw *groupWatcher) []*promutils.Labels {
 	return ms
 }
 
-func (eps *Endpoints) getCommonLabels() *promutils.Labels {
-	m := promutils.GetLabels()
-	m.Add("__meta_kubernetes_namespace", eps.Metadata.Namespace)
-	m.Add("__meta_kubernetes_endpoints_name", eps.Metadata.Name)
-	return m
-}
-
 func appendEndpointLabelsForAddresses(ms []*promutils.Labels, gw *groupWatcher, podPortsSeen map[*Pod][]int, eps *Endpoints,
 	eas []EndpointAddress, epp EndpointPort, svc *Service, ready string) []*promutils.Labels {
 	for _, ea := range eas {
diff --git a/lib/promscrape/discovery/kubernetes/endpointslice.go b/lib/promscrape/discovery/kubernetes/endpointslice.go
index 3d66c2264..cfd1b5e55 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslice.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslice.go
@@ -84,6 +84,11 @@ func (eps *EndpointSlice) getTargetLabels(gw *groupWatcher) []*promutils.Labels
 				m.Add("__address__", addr)
 				p.appendCommonLabels(m, gw)
 				p.appendContainerLabels(m, c, &cp)
+
+				// Prometheus sets endpoints_name and namespace labels for all endpoints
+				// Even if port is not matching service port.
+				// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4154
+				p.appendEndpointSliceLabels(m, eps)
 				if svc != nil {
 					svc.appendCommonLabels(m)
 				}
diff --git a/lib/promscrape/discovery/kubernetes/pod.go b/lib/promscrape/discovery/kubernetes/pod.go
index 7256c8b57..5c89a5a59 100644
--- a/lib/promscrape/discovery/kubernetes/pod.go
+++ b/lib/promscrape/discovery/kubernetes/pod.go
@@ -178,6 +178,21 @@ func (p *Pod) appendContainerLabels(m *promutils.Labels, c Container, cp *Contai
 	}
 }
 
+func (p *Pod) appendEndpointLabels(m *promutils.Labels, eps *Endpoints) {
+	m.Add("__meta_kubernetes_endpoints_name", eps.Metadata.Name)
+	m.Add("__meta_kubernetes_endpoint_address_target_kind", "Pod")
+	m.Add("__meta_kubernetes_endpoint_address_target_name", p.Metadata.Name)
+	eps.Metadata.registerLabelsAndAnnotations("__meta_kubernetes_endpoints", m)
+}
+
+func (p *Pod) appendEndpointSliceLabels(m *promutils.Labels, eps *EndpointSlice) {
+	m.Add("__meta_kubernetes_endpointslice_name", eps.Metadata.Name)
+	m.Add("__meta_kubernetes_endpointslice_address_target_kind", "Pod")
+	m.Add("__meta_kubernetes_endpointslice_address_target_name", p.Metadata.Name)
+	m.Add("__meta_kubernetes_endpointslice_address_type", eps.AddressType)
+	eps.Metadata.registerLabelsAndAnnotations("__meta_kubernetes_endpointslice", m)
+}
+
 func (p *Pod) appendCommonLabels(m *promutils.Labels, gw *groupWatcher) {
 	if gw.attachNodeMetadata {
 		m.Add("__meta_kubernetes_node_name", p.Spec.NodeName)
diff --git a/lib/storage/storage.go b/lib/storage/storage.go
index 85f9bed5f..0c2f068d7 100644
--- a/lib/storage/storage.go
+++ b/lib/storage/storage.go
@@ -2162,10 +2162,14 @@ func (dmc *dateMetricIDCache) syncLocked() {
 		}
 		v = v.Clone()
 		v.Union(&e.v)
-		byDateMutable.m[date] = &byDateMetricIDEntry{
+		dme := &byDateMetricIDEntry{
 			date: date,
 			v:    *v,
 		}
+		if date == byDateMutable.hotEntry.Load().(*byDateMetricIDEntry).date {
+			byDateMutable.hotEntry.Store(dme)
+		}
+		byDateMutable.m[date] = dme
 	}
 	for date, e := range byDate.m {
 		v := byDateMutable.get(date)


Test patch:
diff --git a/app/vmalert/config/config_test.go b/app/vmalert/config/config_test.go
index 71eee2822..4d7ed3ec4 100644
--- a/app/vmalert/config/config_test.go
+++ b/app/vmalert/config/config_test.go
@@ -64,6 +64,10 @@ func TestParseBad(t *testing.T) {
 			[]string{"testdata/dir/rules6-bad.rules"},
 			"missing ':' in header",
 		},
+		{
+			[]string{"http://unreachable-url"},
+			"failed to read from the config: cannot fetch \"http://unreachable-url\"",
+		},
 	}
 	for _, tc := range testCases {
 		_, err := Parse(tc.path, notifier.ValidateTemplates, true)
@@ -102,7 +106,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "group name must be set",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{
 						Record: "record",
@@ -113,7 +118,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{
 						Record: "record",
@@ -125,7 +131,8 @@ func TestGroup_Validate(t *testing.T) {
 			validateExpressions: true,
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{
 						Alert: "alert",
@@ -139,7 +146,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{
 						Alert: "alert",
@@ -156,7 +164,8 @@ func TestGroup_Validate(t *testing.T) {
 			validateAnnotations: true,
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{
 						Alert: "alert",
@@ -171,7 +180,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "duplicate",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
 						"summary": "{{ value|query }}",
@@ -184,7 +194,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "duplicate",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{Record: "record", Expr: "up == 1", Labels: map[string]string{
 						"summary": "{{ value|query }}",
@@ -197,7 +208,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "duplicate",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
 						"summary": "{{ value|query }}",
@@ -210,7 +222,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "",
 		},
 		{
-			group: &Group{Name: "test",
+			group: &Group{
+				Name: "test",
 				Rules: []Rule{
 					{Record: "alert", Expr: "up == 1", Labels: map[string]string{
 						"summary": "{{ value|query }}",
@@ -223,7 +236,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr: "",
 		},
 		{
-			group: &Group{Name: "test thanos",
+			group: &Group{
+				Name: "test thanos",
 				Type: NewRawType("thanos"),
 				Rules: []Rule{
 					{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
@@ -235,7 +249,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr:              "unknown datasource type",
 		},
 		{
-			group: &Group{Name: "test graphite",
+			group: &Group{
+				Name: "test graphite",
 				Type: NewGraphiteType(),
 				Rules: []Rule{
 					{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
@@ -247,7 +262,8 @@ func TestGroup_Validate(t *testing.T) {
 			expErr:              "",
 		},
 		{
-			group: &Group{Name: "test prometheus",
+			group: &Group{
+				Name: "test prometheus",
 				Type: NewPrometheusType(),
 				Rules: []Rule{
 					{Alert: "alert", Expr: "up == 1", Labels: map[string]string{
diff --git a/app/vmalert/datasource/vm_test.go b/app/vmalert/datasource/vm_test.go
index d6413ab6e..3ae1bf173 100644
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@@ -35,13 +35,6 @@ func TestVMInstantQuery(t *testing.T) {
 		t.Errorf("should not be called")
 	})
 	c := -1
-	mux.HandleFunc("/render", func(w http.ResponseWriter, request *http.Request) {
-		c++
-		switch c {
-		case 7:
-			w.Write([]byte(`[{"target":"constantLine(10)","tags":{"name":"constantLine(10)"},"datapoints":[[10,1611758343],[10,1611758373],[10,1611758403]]}]`))
-		}
-	})
 	mux.HandleFunc("/api/v1/query", func(w http.ResponseWriter, r *http.Request) {
 		c++
 		if r.Method != http.MethodPost {
@@ -75,6 +68,15 @@ func TestVMInstantQuery(t *testing.T) {
 			w.Write([]byte(`{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"vm_rows","foo":"bar"},"value":[1583786142,"13763"]},{"metric":{"__name__":"vm_requests","foo":"baz"},"value":[1583786140,"2000"]}]}}`))
 		case 6:
 			w.Write([]byte(`{"status":"success","data":{"resultType":"scalar","result":[1583786142, "1"]}}`))
+		case 7:
+			w.Write([]byte(`{"status":"success","data":{"resultType":"scalar","result":[1583786142, "1"]},"stats":{"seriesFetched": "42"}}`))
+		}
+	})
+	mux.HandleFunc("/render", func(w http.ResponseWriter, request *http.Request) {
+		c++
+		switch c {
+		case 8:
+			w.Write([]byte(`[{"target":"constantLine(10)","tags":{"name":"constantLine(10)"},"datapoints":[[10,1611758343],[10,1611758373],[10,1611758403]]}]`))
 		}
 	})
 
@@ -107,12 +109,12 @@ func TestVMInstantQuery(t *testing.T) {
 	expErr("unknown status")                   // 3
 	expErr("unexpected end of JSON input")     // 4
 
-	m, _, err := pq.Query(ctx, query, ts) // 6 - vector
+	res, _, err := pq.Query(ctx, query, ts) // 5 - vector
 	if err != nil {
 		t.Fatalf("unexpected %s", err)
 	}
-	if len(m) != 2 {
-		t.Fatalf("expected 2 metrics got %d in %+v", len(m), m)
+	if len(res.Data) != 2 {
+		t.Fatalf("expected 2 metrics got %d in %+v", len(res.Data), res.Data)
 	}
 	expected := []Metric{
 		{
@@ -126,17 +128,39 @@ func TestVMInstantQuery(t *testing.T) {
 			Values:     []float64{2000},
 		},
 	}
-	metricsEqual(t, m, expected)
+	metricsEqual(t, res.Data, expected)
 
-	m, req, err := pq.Query(ctx, query, ts) // 7 - scalar
+	res, req, err := pq.Query(ctx, query, ts) // 6 - scalar
 	if err != nil {
 		t.Fatalf("unexpected %s", err)
 	}
 	if req == nil {
 		t.Fatalf("expected request to be non-nil")
 	}
-	if len(m) != 1 {
-		t.Fatalf("expected 1 metrics got %d in %+v", len(m), m)
+	if len(res.Data) != 1 {
+		t.Fatalf("expected 1 metrics got %d in %+v", len(res.Data), res.Data)
+	}
+	expected = []Metric{
+		{
+			Timestamps: []int64{1583786142},
+			Values:     []float64{1},
+		},
+	}
+	if !reflect.DeepEqual(res.Data, expected) {
+		t.Fatalf("unexpected metric %+v want %+v", res.Data, expected)
+	}
+
+	if res.SeriesFetched != nil {
+		t.Fatalf("expected `seriesFetched` field to be nil when it is missing in datasource response; got %v instead",
+			res.SeriesFetched)
+	}
+
+	res, _, err = pq.Query(ctx, query, ts) // 7 - scalar with stats
+	if err != nil {
+		t.Fatalf("unexpected %s", err)
+	}
+	if len(res.Data) != 1 {
+		t.Fatalf("expected 1 metrics got %d in %+v", len(res.Data), res)
 	}
 	expected = []Metric{
 		{
@@ -144,18 +168,22 @@ func TestVMInstantQuery(t *testing.T) {
 			Values:     []float64{1},
 		},
 	}
-	if !reflect.DeepEqual(m, expected) {
-		t.Fatalf("unexpected metric %+v want %+v", m, expected)
+	if !reflect.DeepEqual(res.Data, expected) {
+		t.Fatalf("unexpected metric %+v want %+v", res.Data, expected)
+	}
+	if *res.SeriesFetched != 42 {
+		t.Fatalf("expected `seriesFetched` field to be 42; got %d instead",
+			*res.SeriesFetched)
 	}
 
 	gq := s.BuildWithParams(QuerierParams{DataSourceType: string(datasourceGraphite)})
 
-	m, _, err = gq.Query(ctx, queryRender, ts) // 8 - graphite
+	res, _, err = gq.Query(ctx, queryRender, ts) // 8 - graphite
 	if err != nil {
 		t.Fatalf("unexpected %s", err)
 	}
-	if len(m) != 1 {
-		t.Fatalf("expected 1 metric  got %d in %+v", len(m), m)
+	if len(res.Data) != 1 {
+		t.Fatalf("expected 1 metric  got %d in %+v", len(res.Data), res.Data)
 	}
 	exp := []Metric{
 		{
@@ -164,7 +192,7 @@ func TestVMInstantQuery(t *testing.T) {
 			Values:     []float64{10},
 		},
 	}
-	metricsEqual(t, m, exp)
+	metricsEqual(t, res.Data, exp)
 
 }
 
@@ -213,10 +241,11 @@ func TestVMInstantQueryWithRetry(t *testing.T) {
 	}
 
 	expValue := func(v float64) {
-		m, _, err := pq.Query(ctx, query, time.Now())
+		res, _, err := pq.Query(ctx, query, time.Now())
 		if err != nil {
 			t.Fatalf("unexpected %s", err)
 		}
+		m := res.Data
 		if len(m) != 1 {
 			t.Fatalf("expected 1 metrics got %d in %+v", len(m), m)
 		}
@@ -319,10 +348,11 @@ func TestVMRangeQuery(t *testing.T) {
 
 	start, end := time.Now().Add(-time.Minute), time.Now()
 
-	m, err := pq.QueryRange(ctx, query, start, end)
+	res, err := pq.QueryRange(ctx, query, start, end)
 	if err != nil {
 		t.Fatalf("unexpected %s", err)
 	}
+	m := res.Data
 	if len(m) != 1 {
 		t.Fatalf("expected 1 metric  got %d in %+v", len(m), m)
 	}
diff --git a/app/vmalert/helpers_test.go b/app/vmalert/helpers_test.go
index 19684e52b..de0db3462 100644
--- a/app/vmalert/helpers_test.go
+++ b/app/vmalert/helpers_test.go
@@ -44,21 +44,21 @@ func (fq *fakeQuerier) BuildWithParams(_ datasource.QuerierParams) datasource.Qu
 	return fq
 }
 
-func (fq *fakeQuerier) QueryRange(ctx context.Context, q string, _, _ time.Time) ([]datasource.Metric, error) {
+func (fq *fakeQuerier) QueryRange(ctx context.Context, q string, _, _ time.Time) (datasource.Result, error) {
 	req, _, err := fq.Query(ctx, q, time.Now())
 	return req, err
 }
 
-func (fq *fakeQuerier) Query(_ context.Context, _ string, _ time.Time) ([]datasource.Metric, *http.Request, error) {
+func (fq *fakeQuerier) Query(_ context.Context, _ string, _ time.Time) (datasource.Result, *http.Request, error) {
 	fq.Lock()
 	defer fq.Unlock()
 	if fq.err != nil {
-		return nil, nil, fq.err
+		return datasource.Result{}, nil, fq.err
 	}
 	cp := make([]datasource.Metric, len(fq.metrics))
 	copy(cp, fq.metrics)
 	req, _ := http.NewRequest(http.MethodPost, "foo.com", nil)
-	return cp, req, nil
+	return datasource.Result{Data: cp}, req, nil
 }
 
 type fakeQuerierWithRegistry struct {
@@ -85,23 +85,23 @@ func (fqr *fakeQuerierWithRegistry) BuildWithParams(_ datasource.QuerierParams)
 	return fqr
 }
 
-func (fqr *fakeQuerierWithRegistry) QueryRange(ctx context.Context, q string, _, _ time.Time) ([]datasource.Metric, error) {
+func (fqr *fakeQuerierWithRegistry) QueryRange(ctx context.Context, q string, _, _ time.Time) (datasource.Result, error) {
 	req, _, err := fqr.Query(ctx, q, time.Now())
 	return req, err
 }
 
-func (fqr *fakeQuerierWithRegistry) Query(_ context.Context, expr string, _ time.Time) ([]datasource.Metric, *http.Request, error) {
+func (fqr *fakeQuerierWithRegistry) Query(_ context.Context, expr string, _ time.Time) (datasource.Result, *http.Request, error) {
 	fqr.Lock()
 	defer fqr.Unlock()
 
 	req, _ := http.NewRequest(http.MethodPost, "foo.com", nil)
 	metrics, ok := fqr.registry[expr]
 	if !ok {
-		return nil, req, nil
+		return datasource.Result{}, req, nil
 	}
 	cp := make([]datasource.Metric, len(metrics))
 	copy(cp, metrics)
-	return cp, req, nil
+	return datasource.Result{Data: cp}, req, nil
 }
 
 type fakeQuerierWithDelay struct {
@@ -109,7 +109,7 @@ type fakeQuerierWithDelay struct {
 	delay time.Duration
 }
 
-func (fqd *fakeQuerierWithDelay) Query(ctx context.Context, expr string, ts time.Time) ([]datasource.Metric, *http.Request, error) {
+func (fqd *fakeQuerierWithDelay) Query(ctx context.Context, expr string, ts time.Time) (datasource.Result, *http.Request, error) {
 	timer := time.NewTimer(fqd.delay)
 	select {
 	case <-ctx.Done():
diff --git a/app/vmalert/replay_test.go b/app/vmalert/replay_test.go
index 003e916a9..bce1856a8 100644
--- a/app/vmalert/replay_test.go
+++ b/app/vmalert/replay_test.go
@@ -20,21 +20,21 @@ func (fr *fakeReplayQuerier) BuildWithParams(_ datasource.QuerierParams) datasou
 	return fr
 }
 
-func (fr *fakeReplayQuerier) QueryRange(_ context.Context, q string, from, to time.Time) ([]datasource.Metric, error) {
+func (fr *fakeReplayQuerier) QueryRange(_ context.Context, q string, from, to time.Time) (res datasource.Result, err error) {
 	key := fmt.Sprintf("%s+%s", from.Format("15:04:05"), to.Format("15:04:05"))
 	dps, ok := fr.registry[q]
 	if !ok {
-		return nil, fmt.Errorf("unexpected query received: %q", q)
+		return res, fmt.Errorf("unexpected query received: %q", q)
 	}
 	_, ok = dps[key]
 	if !ok {
-		return nil, fmt.Errorf("unexpected time range received: %q", key)
+		return res, fmt.Errorf("unexpected time range received: %q", key)
 	}
 	delete(dps, key)
 	if len(fr.registry[q]) < 1 {
 		delete(fr.registry, q)
 	}
-	return nil, nil
+	return res, nil
 }
 
 func TestReplay(t *testing.T) {
diff --git a/lib/promscrape/discovery/kubernetes/endpoints_test.go b/lib/promscrape/discovery/kubernetes/endpoints_test.go
index bb6b9e8ac..36b4c5745 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints_test.go
@@ -292,7 +292,9 @@ func TestGetEndpointsLabels(t *testing.T) {
 				"__meta_kubernetes_service_type":                 "service-type",
 			}),
 			promutils.NewLabelsFromMap(map[string]string{
-				"__address__":                                    "192.168.15.1:8428",
+				"__address__": "192.168.15.1:8428",
+				"__meta_kubernetes_endpoint_address_target_kind": "Pod",
+				"__meta_kubernetes_endpoint_address_target_name": "test-pod",
 				"__meta_kubernetes_endpoints_name":               "test-eps",
 				"__meta_kubernetes_namespace":                    "default",
 				"__meta_kubernetes_node_label_node_label":        "xyz",
diff --git a/lib/promscrape/discovery/kubernetes/endpointslice_test.go b/lib/promscrape/discovery/kubernetes/endpointslice_test.go
index ef5c51e6a..7a15c0b3b 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslice_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslice_test.go
@@ -426,26 +426,32 @@ func TestGetEndpointsliceLabels(t *testing.T) {
 				"__meta_kubernetes_service_type":                                          "service-type",
 			}),
 			promutils.NewLabelsFromMap(map[string]string{
-				"__address__":                                    "192.168.15.1:8428",
-				"__meta_kubernetes_namespace":                    "default",
-				"__meta_kubernetes_node_label_node_label":        "xyz",
-				"__meta_kubernetes_node_labelpresent_node_label": "true",
-				"__meta_kubernetes_node_name":                    "test-node",
-				"__meta_kubernetes_pod_container_image":          "test-image",
-				"__meta_kubernetes_pod_container_name":           "metrics",
-				"__meta_kubernetes_pod_container_port_name":      "http-metrics",
-				"__meta_kubernetes_pod_container_port_number":    "8428",
-				"__meta_kubernetes_pod_container_port_protocol":  "foobar",
-				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
-				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
-				"__meta_kubernetes_pod_name":                     "test-pod",
-				"__meta_kubernetes_pod_node_name":                "test-node",
-				"__meta_kubernetes_pod_phase":                    "abc",
-				"__meta_kubernetes_pod_ready":                    "unknown",
-				"__meta_kubernetes_pod_uid":                      "pod-uid",
-				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
-				"__meta_kubernetes_service_name":                 "test-svc",
-				"__meta_kubernetes_service_type":                 "service-type",
+				"__address__": "192.168.15.1:8428",
+				"__meta_kubernetes_endpointslice_address_target_kind":                     "Pod",
+				"__meta_kubernetes_endpointslice_address_target_name":                     "test-pod",
+				"__meta_kubernetes_endpointslice_address_type":                            "foobar",
+				"__meta_kubernetes_endpointslice_label_kubernetes_io_service_name":        "test-svc",
+				"__meta_kubernetes_endpointslice_labelpresent_kubernetes_io_service_name": "true",
+				"__meta_kubernetes_endpointslice_name":                                    "test-eps",
+				"__meta_kubernetes_namespace":                                             "default",
+				"__meta_kubernetes_node_label_node_label":                                 "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label":                          "true",
+				"__meta_kubernetes_node_name":                                             "test-node",
+				"__meta_kubernetes_pod_container_image":                                   "test-image",
+				"__meta_kubernetes_pod_container_name":                                    "metrics",
+				"__meta_kubernetes_pod_container_port_name":                               "http-metrics",
+				"__meta_kubernetes_pod_container_port_number":                             "8428",
+				"__meta_kubernetes_pod_container_port_protocol":                           "foobar",
+				"__meta_kubernetes_pod_host_ip":                                           "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                                                "192.168.15.1",
+				"__meta_kubernetes_pod_name":                                              "test-pod",
+				"__meta_kubernetes_pod_node_name":                                         "test-node",
+				"__meta_kubernetes_pod_phase":                                             "abc",
+				"__meta_kubernetes_pod_ready":                                             "unknown",
+				"__meta_kubernetes_pod_uid":                                               "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":                                    "1.2.3.4",
+				"__meta_kubernetes_service_name":                                          "test-svc",
+				"__meta_kubernetes_service_type":                                          "service-type",
 			}),
 		})
 	})