Repository: VictoriaMetrics/VictoriaMetrics
Pull request #3106 (2022-09-13 10:53:55)

vmalert: add experimental feature of storing Rule's evaluation state
The new feature keeps last 20 state changes of each Rule in memory. The state are available for view on the Rule's view page. The page can be opened by clicking on `Details` link next to Rule's name on the `/groups` page.

States change suppose to help in investigating cases when Rule doesn't generate alerts or records.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

--------------------------

<img width="1110" alt="image" src="https://user-images.githubusercontent.com/2902918/189883298-373bb068-dc02-4352-b8e7-32594fccb4da.png">


codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Base: **57.94**% // Head: **58.43**% // Increases project coverage by **`+0.49%`** :tada:
> Coverage data is based on head [(`c90c05d`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) compared to base [(`99bc187`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/99bc18774c4373f01a55cf3606e1c6c09e767f9f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> Patch coverage: 63.82% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #3106      +/-   ##
==========================================
+ Coverage   57.94%   58.43%   +0.49%     
==========================================
  Files         276      277       +1     
  Lines       46736    46935     +199     
==========================================
+ Hits        27080    27427     +347     
+ Misses      17878    17724     -154     
- Partials     1778     1784       +6     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/web.qtpl.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvd2ViLnF0cGwuZ28=) | `55.60% <49.61%> (+52.06%)` | :arrow_up: |
| [app/vmalert/recording.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVjb3JkaW5nLmdv) | `67.34% <60.00%> (+1.63%)` | :arrow_up: |
| [app/vmalert/alerting.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvYWxlcnRpbmcuZ28=) | `75.70% <64.44%> (+0.59%)` | :arrow_up: |
| [app/vmalert/web.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvd2ViLmdv) | `65.49% <72.72%> (+13.98%)` | :arrow_up: |
| [app/vmalert/manager.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFuYWdlci5nbw==) | `74.69% <86.66%> (+2.14%)` | :arrow_up: |
| [app/vmalert/rule.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcnVsZS5nbw==) | `100.00% <100.00%> (ø)` | |
| [app/vmalert/web\_types.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvd2ViX3R5cGVzLmdv) | `100.00% <100.00%> (ø)` | |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `65.10% <0.00%> (-1.35%)` | :arrow_down: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `86.44% <0.00%> (-1.21%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3106?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index d8b5c8e86..8253c2d71 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -511,6 +511,7 @@ or time series modification via [relabeling](https://docs.victoriametrics.com/vm
 * `http://<vmalert-addr>/vmalert/api/v1/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in JSON format.
   Used as alert source in AlertManager.
 * `http://<vmalert-addr>/vmalert/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in web UI.
+* `http://<vmalert-addr>/vmalert/rule?group_id=<group_id>&rule_id=<rule_id>` - get rule status in web UI.
 * `http://<vmalert-addr>/metrics` - application metrics.
 * `http://<vmalert-addr>/-/reload` - hot configuration reload.
 
diff --git a/app/vmalert/alerting.go b/app/vmalert/alerting.go
index 6c4e5a5b6..8c9d98060 100644
--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@@ -35,21 +35,13 @@ type AlertingRule struct {
 
 	q datasource.Querier
 
-	// guard status fields
-	mu sync.RWMutex
+	alertsMu sync.RWMutex
 	// stores list of active alerts
 	alerts map[uint64]*notifier.Alert
-	// stores last moment of time Exec was called
-	lastExecTime time.Time
-	// stores the duration of the last Exec call
-	lastExecDuration time.Duration
-	// stores last error that happened in Exec func
-	// resets on every successful Exec
-	// may be used as Health state
-	lastExecError error
-	// stores the number of samples returned during
-	// the last evaluation
-	lastExecSamples int
+
+	// state stores recent state changes
+	// during evaluations
+	state *ruleState
 
 	metrics *alertingRuleMetrics
 }
@@ -82,14 +74,15 @@ func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule
 			Debug:              cfg.Debug,
 		}),
 		alerts:  make(map[uint64]*notifier.Alert),
+		state:   newRuleState(),
 		metrics: &alertingRuleMetrics{},
 	}
 
 	labels := fmt.Sprintf(`alertname=%q, group=%q, id="%d"`, ar.Name, group.Name, ar.ID())
 	ar.metrics.pending = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_alerts_pending{%s}`, labels),
 		func() float64 {
-			ar.mu.RLock()
-			defer ar.mu.RUnlock()
+			ar.alertsMu.RLock()
+			defer ar.alertsMu.RUnlock()
 			var num int
 			for _, a := range ar.alerts {
 				if a.State == notifier.StatePending {
@@ -100,8 +93,8 @@ func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule
 		})
 	ar.metrics.active = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_alerts_firing{%s}`, labels),
 		func() float64 {
-			ar.mu.RLock()
-			defer ar.mu.RUnlock()
+			ar.alertsMu.RLock()
+			defer ar.alertsMu.RUnlock()
 			var num int
 			for _, a := range ar.alerts {
 				if a.State == notifier.StateFiring {
@@ -112,18 +105,16 @@ func newAlertingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rule
 		})
 	ar.metrics.errors = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_alerting_rules_error{%s}`, labels),
 		func() float64 {
-			ar.mu.RLock()
-			defer ar.mu.RUnlock()
-			if ar.lastExecError == nil {
+			e := ar.state.getLast()
+			if e.err == nil {
 				return 0
 			}
 			return 1
 		})
 	ar.metrics.samples = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_alerting_rules_last_evaluation_samples{%s}`, labels),
 		func() float64 {
-			ar.mu.RLock()
-			defer ar.mu.RUnlock()
-			return float64(ar.lastExecSamples)
+			e := ar.state.getLast()
+			return float64(e.samples)
 		})
 	return ar
 }
@@ -274,18 +265,26 @@ const resolvedRetention = 15 * time.Minute
 func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
 	start := time.Now()
 	qMetrics, err := ar.q.Query(ctx, ar.Expr, ts)
-	ar.mu.Lock()
-	defer ar.mu.Unlock()
+	curState := ruleStateEntry{
+		time:     start,
+		at:       ts,
+		duration: time.Since(start),
+		samples:  len(qMetrics),
+		err:      err,
+	}
+
+	defer func() {
+		ar.state.add(curState)
+	}()
+
+	ar.alertsMu.Lock()
+	defer ar.alertsMu.Unlock()
 
-	ar.lastExecTime = start
-	ar.lastExecDuration = time.Since(start)
-	ar.lastExecError = err
-	ar.lastExecSamples = len(qMetrics)
 	if err != nil {
 		return nil, fmt.Errorf("failed to execute query %q: %w", ar.Expr, err)
 	}
 
-	ar.logDebugf(ts, nil, "query returned %d samples (elapsed: %s)", ar.lastExecSamples, ar.lastExecDuration)
+	ar.logDebugf(ts, nil, "query returned %d samples (elapsed: %s)", curState.samples, curState.duration)
 
 	for h, a := range ar.alerts {
 		// cleanup inactive alerts from previous Exec
@@ -301,14 +300,15 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 	for _, m := range qMetrics {
 		ls, err := ar.toLabels(m, qFn)
 		if err != nil {
-			return nil, fmt.Errorf("failed to expand labels: %s", err)
+			curState.err = fmt.Errorf("failed to expand labels: %s", err)
+			return nil, curState.err
 		}
 		h := hash(ls.processed)
 		if _, ok := updated[h]; ok {
 			// duplicate may be caused by extra labels
 			// conflicting with the metric labels
-			ar.lastExecError = fmt.Errorf("labels %v: %w", ls.processed, errDuplicate)
-			return nil, ar.lastExecError
+			curState.err = fmt.Errorf("labels %v: %w", ls.processed, errDuplicate)
+			return nil, curState.err
 		}
 		updated[h] = struct{}{}
 		if a, ok := ar.alerts[h]; ok {
@@ -327,15 +327,16 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 				// in annotations
 				a.Annotations, err = a.ExecTemplate(qFn, ls.origin, ar.Annotations)
 				if err != nil {
-					return nil, err
+					curState.err = err
+					return nil, curState.err
 				}
 			}
 			continue
 		}
-		a, err := ar.newAlert(m, ls, ar.lastExecTime, qFn)
+		a, err := ar.newAlert(m, ls, start, qFn)
 		if err != nil {
-			ar.lastExecError = err
-			return nil, fmt.Errorf("failed to create alert: %w", err)
+			curState.err = fmt.Errorf("failed to create alert: %w", err)
+			return nil, curState.err
 		}
 		a.ID = h
 		a.State = notifier.StatePending
@@ -372,7 +373,8 @@ func (ar *AlertingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]pr
 	}
 	if limit > 0 && numActivePending > limit {
 		ar.alerts = map[uint64]*notifier.Alert{}
-		return nil, fmt.Errorf("exec exceeded limit of %d with %d alerts", limit, numActivePending)
+		curState.err = fmt.Errorf("exec exceeded limit of %d with %d alerts", limit, numActivePending)
+		return nil, curState.err
 	}
 	return ar.toTimeSeries(ts.Unix()), nil
 }
@@ -449,8 +451,8 @@ func (ar *AlertingRule) newAlert(m datasource.Metric, ls *labelSet, start time.T
 
 // AlertAPI generates APIAlert object from alert by its id(hash)
 func (ar *AlertingRule) AlertAPI(id uint64) *APIAlert {
-	ar.mu.RLock()
-	defer ar.mu.RUnlock()
+	ar.alertsMu.RLock()
+	defer ar.alertsMu.RUnlock()
 	a, ok := ar.alerts[id]
 	if !ok {
 		return nil
@@ -458,9 +460,10 @@ func (ar *AlertingRule) AlertAPI(id uint64) *APIAlert {
 	return ar.newAlertAPI(*a)
 }
 
-// ToAPI returns Rule representation in form
-// of APIRule
+// ToAPI returns Rule representation in form of APIRule
+// Isn't thread-safe. Call must be protected by AlertingRule mutex.
 func (ar *AlertingRule) ToAPI() APIRule {
+	lastState := ar.state.getLast()
 	r := APIRule{
 		Type:           "alerting",
 		DatasourceType: ar.Type.String(),
@@ -469,19 +472,20 @@ func (ar *AlertingRule) ToAPI() APIRule {
 		Duration:       ar.For.Seconds(),
 		Labels:         ar.Labels,
 		Annotations:    ar.Annotations,
-		LastEvaluation: ar.lastExecTime,
-		EvaluationTime: ar.lastExecDuration.Seconds(),
+		LastEvaluation: lastState.time,
+		EvaluationTime: lastState.duration.Seconds(),
 		Health:         "ok",
 		State:          "inactive",
 		Alerts:         ar.AlertsToAPI(),
-		LastSamples:    ar.lastExecSamples,
+		LastSamples:    lastState.samples,
+		Updates:        ar.state.getAll(),
 
 		// encode as strings to avoid rounding in JSON
 		ID:      fmt.Sprintf("%d", ar.ID()),
 		GroupID: fmt.Sprintf("%d", ar.GroupID),
 	}
-	if ar.lastExecError != nil {
-		r.LastError = ar.lastExecError.Error()
+	if lastState.err != nil {
+		r.LastError = lastState.err.Error()
 		r.Health = "err"
 	}
 	// satisfy APIRule.State logic
@@ -501,14 +505,14 @@ func (ar *AlertingRule) ToAPI() APIRule {
 // AlertsToAPI generates list of APIAlert objects from existing alerts
 func (ar *AlertingRule) AlertsToAPI() []*APIAlert {
 	var alerts []*APIAlert
-	ar.mu.RLock()
+	ar.alertsMu.RLock()
 	for _, a := range ar.alerts {
 		if a.State == notifier.StateInactive {
 			continue
 		}
 		alerts = append(alerts, ar.newAlertAPI(*a))
 	}
-	ar.mu.RUnlock()
+	ar.alertsMu.RUnlock()
 	return alerts
 }
 
diff --git a/app/vmalert/manager.go b/app/vmalert/manager.go
index b1fbb9ec8..98b41dafb 100644
--- a/app/vmalert/manager.go
+++ b/app/vmalert/manager.go
@@ -30,6 +30,23 @@ type manager struct {
 	groups   map[uint64]*Group
 }
 
+// RuleAPI generates APIRule object from alert by its ID(hash)
+func (m *manager) RuleAPI(gID, rID uint64) (APIRule, error) {
+	m.groupsMu.RLock()
+	defer m.groupsMu.RUnlock()
+
+	g, ok := m.groups[gID]
+	if !ok {
+		return APIRule{}, fmt.Errorf("can't find group with id %d", gID)
+	}
+	for _, rule := range g.Rules {
+		if rule.ID() == rID {
+			return rule.ToAPI(), nil
+		}
+	}
+	return APIRule{}, fmt.Errorf("can't find rule with id %d in group %q", rID, g.Name)
+}
+
 // AlertAPI generates APIAlert object from alert by its ID(hash)
 func (m *manager) AlertAPI(gID, aID uint64) (*APIAlert, error) {
 	m.groupsMu.RLock()
@@ -70,9 +87,9 @@ func (m *manager) startGroup(ctx context.Context, group *Group, restore bool) er
 		err := group.Restore(ctx, m.rr, *remoteReadLookBack, m.labels)
 		if err != nil {
 			if !*remoteReadIgnoreRestoreErrors {
-				return fmt.Errorf("failed to restore state for group %q: %w", group.Name, err)
+				return fmt.Errorf("failed to restore ruleState for group %q: %w", group.Name, err)
 			}
-			logger.Errorf("error while restoring state for group %q: %s", group.Name, err)
+			logger.Errorf("error while restoring ruleState for group %q: %s", group.Name, err)
 		}
 	}
 
diff --git a/app/vmalert/recording.go b/app/vmalert/recording.go
index 50013f2f9..b15059395 100644
--- a/app/vmalert/recording.go
+++ b/app/vmalert/recording.go
@@ -5,7 +5,6 @@ import (
 	"fmt"
 	"sort"
 	"strings"
-	"sync"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/config"
@@ -27,19 +26,9 @@ type RecordingRule struct {
 
 	q datasource.Querier
 
-	// guard status fields
-	mu sync.RWMutex
-	// stores last moment of time Exec was called
-	lastExecTime time.Time
-	// stores the duration of the last Exec call
-	lastExecDuration time.Duration
-	// stores last error that happened in Exec func
-	// resets on every successful Exec
-	// may be used as Health state
-	lastExecError error
-	// stores the number of samples returned during
-	// the last evaluation
-	lastExecSamples int
+	// state stores recent state changes
+	// during evaluations
+	state *ruleState
 
 	metrics *recordingRuleMetrics
 }
@@ -69,6 +58,7 @@ func newRecordingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rul
 		Labels:  cfg.Labels,
 		GroupID: group.ID(),
 		metrics: &recordingRuleMetrics{},
+		state:   newRuleState(),
 		q: qb.BuildWithParams(datasource.QuerierParams{
 			DataSourceType:     group.Type.String(),
 			EvaluationInterval: group.Interval,
@@ -80,18 +70,16 @@ func newRecordingRule(qb datasource.QuerierBuilder, group *Group, cfg config.Rul
 	labels := fmt.Sprintf(`recording=%q, group=%q, id="%d"`, rr.Name, group.Name, rr.ID())
 	rr.metrics.errors = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_recording_rules_error{%s}`, labels),
 		func() float64 {
-			rr.mu.RLock()
-			defer rr.mu.RUnlock()
-			if rr.lastExecError == nil {
+			e := rr.state.getLast()
+			if e.err == nil {
 				return 0
 			}
 			return 1
 		})
 	rr.metrics.samples = utils.GetOrCreateGauge(fmt.Sprintf(`vmalert_recording_rules_last_evaluation_samples{%s}`, labels),
 		func() float64 {
-			rr.mu.RLock()
-			defer rr.mu.RUnlock()
-			return float64(rr.lastExecSamples)
+			e := rr.state.getLast()
+			return float64(e.samples)
 		})
 	return rr
 }
@@ -126,21 +114,28 @@ func (rr *RecordingRule) ExecRange(ctx context.Context, start, end time.Time) ([
 
 // Exec executes RecordingRule expression via the given Querier.
 func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]prompbmarshal.TimeSeries, error) {
+	start := time.Now()
 	qMetrics, err := rr.q.Query(ctx, rr.Expr, ts)
-	rr.mu.Lock()
-	defer rr.mu.Unlock()
+	curState := ruleStateEntry{
+		time:     start,
+		at:       ts,
+		duration: time.Since(start),
+		samples:  len(qMetrics),
+	}
+
+	defer func() {
+		rr.state.add(curState)
+	}()
 
-	rr.lastExecTime = ts
-	rr.lastExecDuration = time.Since(ts)
-	rr.lastExecError = err
-	rr.lastExecSamples = len(qMetrics)
 	if err != nil {
-		return nil, fmt.Errorf("failed to execute query %q: %w", rr.Expr, err)
+		curState.err = fmt.Errorf("failed to execute query %q: %w", rr.Expr, err)
+		return nil, curState.err
 	}
 
 	numSeries := len(qMetrics)
 	if limit > 0 && numSeries > limit {
-		return nil, fmt.Errorf("exec exceeded limit of %d with %d series", limit, numSeries)
+		curState.err = fmt.Errorf("exec exceeded limit of %d with %d series", limit, numSeries)
+		return nil, curState.err
 	}
 
 	duplicates := make(map[string]struct{}, len(qMetrics))
@@ -149,8 +144,8 @@ func (rr *RecordingRule) Exec(ctx context.Context, ts time.Time, limit int) ([]p
 		ts := rr.toTimeSeries(r)
 		key := stringifyLabels(ts)
 		if _, ok := duplicates[key]; ok {
-			rr.lastExecError = errDuplicate
-			return nil, fmt.Errorf("original metric %v; resulting labels %q: %w", r, key, errDuplicate)
+			curState.err = fmt.Errorf("original metric %v; resulting labels %q: %w", r, key, errDuplicate)
+			return nil, curState.err
 		}
 		duplicates[key] = struct{}{}
 		tss = append(tss, ts)
@@ -205,23 +200,25 @@ func (rr *RecordingRule) UpdateWith(r Rule) error {
 // ToAPI returns Rule's representation in form
 // of APIRule
 func (rr *RecordingRule) ToAPI() APIRule {
+	lastState := rr.state.getLast()
 	r := APIRule{
 		Type:           "recording",
 		DatasourceType: rr.Type.String(),
 		Name:           rr.Name,
 		Query:          rr.Expr,
 		Labels:         rr.Labels,
-		LastEvaluation: rr.lastExecTime,
-		EvaluationTime: rr.lastExecDuration.Seconds(),
+		LastEvaluation: lastState.time,
+		EvaluationTime: lastState.duration.Seconds(),
 		Health:         "ok",
-		LastSamples:    rr.lastExecSamples,
+		LastSamples:    lastState.samples,
+		Updates:        rr.state.getAll(),
+
 		// encode as strings to avoid rounding
 		ID:      fmt.Sprintf("%d", rr.ID()),
 		GroupID: fmt.Sprintf("%d", rr.GroupID),
 	}
-
-	if rr.lastExecError != nil {
-		r.LastError = rr.lastExecError.Error()
+	if lastState.err != nil {
+		r.LastError = lastState.err.Error()
 		r.Health = "err"
 	}
 	return r
diff --git a/app/vmalert/rule.go b/app/vmalert/rule.go
index 630947060..55e6f7070 100644
--- a/app/vmalert/rule.go
+++ b/app/vmalert/rule.go
@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"errors"
+	"sync"
 	"time"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
@@ -31,3 +32,72 @@ type Rule interface {
 }
 
 var errDuplicate = errors.New("result contains metrics with the same labelset after applying rule labels")
+
+type ruleState struct {
+	sync.RWMutex
+	entries []ruleStateEntry
+	cur     int
+}
+
+type ruleStateEntry struct {
+	// stores last moment of time rule.Exec was called
+	time time.Time
+	// stores the timesteamp with which rule.Exec was called
+	at time.Time
+	// stores the duration of the last rule.Exec call
+	duration time.Duration
+	// stores last error that happened in Exec func
+	// resets on every successful Exec
+	// may be used as Health ruleState
+	err error
+	// stores the number of samples returned during
+	// the last evaluation
+	samples int
+}
+
+const defaultStateEntriesLimit = 20
+
+func newRuleState() *ruleState {
+	return &ruleState{
+		entries: make([]ruleStateEntry, defaultStateEntriesLimit),
+	}
+}
+
+func (s *ruleState) getLast() ruleStateEntry {
+	s.RLock()
+	defer s.RUnlock()
+	return s.entries[s.cur]
+}
+
+func (s *ruleState) getAll() []ruleStateEntry {
+	entries := make([]ruleStateEntry, 0)
+
+	s.RLock()
+	defer s.RUnlock()
+
+	cur := s.cur
+	for {
+		e := s.entries[cur]
+		if !e.time.IsZero() || !e.at.IsZero() {
+			entries = append(entries, e)
+		}
+		cur--
+		if cur < 0 {
+			cur = cap(s.entries) - 1
+		}
+		if cur == s.cur {
+			return entries
+		}
+	}
+}
+
+func (s *ruleState) add(e ruleStateEntry) {
+	s.Lock()
+	defer s.Unlock()
+
+	s.cur++
+	if s.cur > cap(s.entries)-1 {
+		s.cur = 0
+	}
+	s.entries[s.cur] = e
+}
diff --git a/app/vmalert/web.go b/app/vmalert/web.go
index 17f937081..0f7567509 100644
--- a/app/vmalert/web.go
+++ b/app/vmalert/web.go
@@ -85,6 +85,14 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 		}
 		WriteAlert(w, r, alert)
 		return true
+	case "/vmalert/rule":
+		rule, err := rh.getRule(r)
+		if err != nil {
+			httpserver.Errorf(w, r, "%s", err)
+			return true
+		}
+		WriteRuleDetails(w, r, rule)
+		return true
 	case "/vmalert/groups":
 		WriteListGroups(w, r, rh.groups())
 		return true
@@ -168,8 +176,25 @@ func (rh *requestHandler) handler(w http.ResponseWriter, r *http.Request) bool {
 const (
 	paramGroupID = "group_id"
 	paramAlertID = "alert_id"
+	paramRuleID  = "rule_id"
 )
 
+func (rh *requestHandler) getRule(r *http.Request) (APIRule, error) {
+	groupID, err := strconv.ParseUint(r.FormValue(paramGroupID), 10, 0)
+	if err != nil {
+		return APIRule{}, fmt.Errorf("failed to read %q param: %s", paramGroupID, err)
+	}
+	ruleID, err := strconv.ParseUint(r.FormValue(paramRuleID), 10, 0)
+	if err != nil {
+		return APIRule{}, fmt.Errorf("failed to read %q param: %s", paramRuleID, err)
+	}
+	rule, err := rh.m.RuleAPI(groupID, ruleID)
+	if err != nil {
+		return APIRule{}, errResponse(err, http.StatusNotFound)
+	}
+	return rule, nil
+}
+
 func (rh *requestHandler) getAlert(r *http.Request) (*APIAlert, error) {
 	groupID, err := strconv.ParseUint(r.FormValue(paramGroupID), 10, 0)
 	if err != nil {
diff --git a/app/vmalert/web.qtpl b/app/vmalert/web.qtpl
index 693186550..b743a36b5 100644
--- a/app/vmalert/web.qtpl
+++ b/app/vmalert/web.qtpl
@@ -26,6 +26,7 @@
 {% endfunc %}
 
 {% func ListGroups(r *http.Request, groups []APIGroup) %}
+    {%code prefix := utils.Prefix(r.URL.Path) %}
     {%= tpl.Header(r, navItems, "Groups") %}
     {%  if len(groups) > 0 %}
         {%code
@@ -85,6 +86,7 @@
                                         {% else %}
                                         <b>record:</b> {%s r.Name %}
                                         {% endif %}
+                                        | <span><a target="_blank" href="{%s prefix+r.WebLink() %}">Details</a></span>
                                     </div>
                                     <div class="col-12">
                                         <code><pre>{%s r.Query %}</pre></code>
@@ -116,7 +118,7 @@
 
     {% else %}
         <div>
-            <p>No items...</p>
+            <p>No groups...</p>
         </div>
     {% endif %}
 
@@ -204,7 +206,7 @@
 
     {% else %}
         <div>
-            <p>No items...</p>
+            <p>No active alerts...</p>
         </div>
     {% endif %}
 
@@ -260,7 +262,7 @@
 
     {% else %}
         <div>
-            <p>No items...</p>
+            <p>No targets...</p>
         </div>
     {% endif %}
 
@@ -284,7 +286,7 @@
         }
         sort.Strings(annotationKeys)
     %}
-    <div class="display-6 pb-3 mb-3">{%s alert.Name %}<span class="ms-2 badge {% if alert.State=="firing" %}bg-danger{% else %} bg-warning text-dark{% endif %}">{%s alert.State %}</span></div>
+    <div class="display-6 pb-3 mb-3">Alert: {%s alert.Name %}<span class="ms-2 badge {% if alert.State=="firing" %}bg-danger{% else %} bg-warning text-dark{% endif %}">{%s alert.State %}</span></div>
     <div class="container border-bottom p-2">
       <div class="row">
         <div class="col-2">
@@ -354,6 +356,117 @@
 
 {% endfunc %}
 
+
+{% func RuleDetails(r *http.Request, rule APIRule) %}
+    {%code prefix := utils.Prefix(r.URL.Path) %}
+    {%= tpl.Header(r, navItems, "") %}
+    {%code
+        var labelKeys []string
+        for k := range rule.Labels {
+            labelKeys = append(labelKeys, k)
+        }
+        sort.Strings(labelKeys)
+
+        var annotationKeys []string
+        for k := range rule.Annotations {
+            annotationKeys = append(annotationKeys, k)
+        }
+        sort.Strings(annotationKeys)
+    %}
+    <div class="display-6 pb-3 mb-3">Rule: {%s rule.Name %}<span class="ms-2 badge {% if rule.Health!="ok" %}bg-danger{% else %} bg-warning text-dark{% endif %}">{%s rule.Health %}</span></div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Expr
+        </div>
+        <div class="col">
+          <code><pre>{%s rule.Query %}</pre></code>
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          For
+        </div>
+        <div class="col">
+         {%v rule.Duration %} seconds
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Labels
+        </div>
+        <div class="col">
+          {% for _, k := range labelKeys %}
+                <span class="m-1 badge bg-primary">{%s k %}={%s rule.Labels[k] %}</span>
+          {% endfor %}
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Annotations
+        </div>
+        <div class="col">
+          {% for _, k := range annotationKeys %}
+                <b>{%s k %}:</b><br>
+                <p>{%s rule.Annotations[k] %}</p>
+          {% endfor %}
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Group
+        </div>
+        <div class="col">
+           <a target="_blank" href="{%s prefix %}groups#group-{%s rule.GroupID %}">{%s rule.GroupID %}</a>
+        </div>
+      </div>
+    </div>
+
+    <br>
+    <div class="display-6 pb-3">Last {%d len(rule.Updates) %} updates</span>:</div>
+        <table class="table table-striped table-hover table-sm">
+            <thead>
+                <tr>
+                    <th scope="col" style="width: 20%" title="The time when event was created">Updated at</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="How many samples were returned">Samples</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="How many seconds request took">Duration</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="Time used for rule execution">Executed at</th>
+                </tr>
+            </thead>
+            <tbody>
+
+     {% for _, u := range rule.Updates %}
+             <tr{% if u.err != nil %} class="alert-danger"{% endif %}>
+                 <td>
+                    <span class="badge bg-primary rounded-pill me-3" title="Updated at">{%s u.time.Format(time.RFC3339) %}</span>
+                 </td>
+                 <td class="text-center">{%d u.samples %}</td>
+                 <td class="text-center">{%f.3 u.duration.Seconds() %}s</td>
+                 <td class="text-center">{%s u.at.Format(time.RFC3339) %}</td>
+             </tr>
+          </li>
+          {% if u.err != nil %}
+             <tr{% if u.err != nil %} class="alert-danger"{% endif %}>
+               <td colspan="4">
+                   <span class="alert-danger">{%v u.err %}</span>
+               </td>
+             </tr>
+          {% endif %}
+     {% endfor %}
+
+    {%= tpl.Footer(r) %}
+{% endfunc %}
+
+
+
 {% func badgeState(state string) %}
 {%code
     badgeClass := "bg-warning text-dark"
diff --git a/app/vmalert/web.qtpl.go b/app/vmalert/web.qtpl.go
index 6f67cd551..29f490065 100644
--- a/app/vmalert/web.qtpl.go
+++ b/app/vmalert/web.qtpl.go
@@ -110,16 +110,22 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []APIGr
 	qw422016.N().S(`
     `)
 //line app/vmalert/web.qtpl:29
-	tpl.StreamHeader(qw422016, r, navItems, "Groups")
+	prefix := utils.Prefix(r.URL.Path)
+
 //line app/vmalert/web.qtpl:29
 	qw422016.N().S(`
     `)
 //line app/vmalert/web.qtpl:30
-	if len(groups) > 0 {
+	tpl.StreamHeader(qw422016, r, navItems, "Groups")
 //line app/vmalert/web.qtpl:30
+	qw422016.N().S(`
+    `)
+//line app/vmalert/web.qtpl:31
+	if len(groups) > 0 {
+//line app/vmalert/web.qtpl:31
 		qw422016.N().S(`
         `)
-//line app/vmalert/web.qtpl:32
+//line app/vmalert/web.qtpl:33
 		rOk := make(map[string]int)
 		rNotOk := make(map[string]int)
 		for _, g := range groups {
@@ -132,138 +138,138 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []APIGr
 			}
 		}
 
-//line app/vmalert/web.qtpl:43
+//line app/vmalert/web.qtpl:44
 		qw422016.N().S(`
          <a class="btn btn-primary" role="button" onclick="collapseAll()">Collapse All</a>
          <a class="btn btn-primary" role="button" onclick="expandAll()">Expand All</a>
         `)
-//line app/vmalert/web.qtpl:46
+//line app/vmalert/web.qtpl:47
 		for _, g := range groups {
-//line app/vmalert/web.qtpl:46
+//line app/vmalert/web.qtpl:47
 			qw422016.N().S(`
               <div class="group-heading`)
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 			if rNotOk[g.Name] > 0 {
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 				qw422016.N().S(` alert-danger`)
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 			}
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 			qw422016.N().S(`"  data-bs-target="rules-`)
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:47
+//line app/vmalert/web.qtpl:48
 			qw422016.N().S(`">
                 <span class="anchor" id="group-`)
-//line app/vmalert/web.qtpl:48
+//line app/vmalert/web.qtpl:49
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:48
+//line app/vmalert/web.qtpl:49
 			qw422016.N().S(`"></span>
                 <a href="#group-`)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.E().S(g.Name)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			if g.Type != "prometheus" {
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 				qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 				qw422016.E().S(g.Type)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 				qw422016.N().S(`)`)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			}
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.N().S(` (every `)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.N().FPrec(g.Interval, 0)
-//line app/vmalert/web.qtpl:49
+//line app/vmalert/web.qtpl:50
 			qw422016.N().S(`s)</a>
                  `)
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 			if rNotOk[g.Name] > 0 {
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 				qw422016.N().S(`<span class="badge bg-danger" title="Number of rules with status Error">`)
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 				qw422016.N().D(rNotOk[g.Name])
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 				qw422016.N().S(`</span> `)
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 			}
-//line app/vmalert/web.qtpl:50
+//line app/vmalert/web.qtpl:51
 			qw422016.N().S(`
                 <span class="badge bg-success" title="Number of rules withs status Ok">`)
-//line app/vmalert/web.qtpl:51
+//line app/vmalert/web.qtpl:52
 			qw422016.N().D(rOk[g.Name])
-//line app/vmalert/web.qtpl:51
+//line app/vmalert/web.qtpl:52
 			qw422016.N().S(`</span>
                 <p class="fs-6 fw-lighter">`)
-//line app/vmalert/web.qtpl:52
+//line app/vmalert/web.qtpl:53
 			qw422016.E().S(g.File)
-//line app/vmalert/web.qtpl:52
+//line app/vmalert/web.qtpl:53
 			qw422016.N().S(`</p>
                 `)
-//line app/vmalert/web.qtpl:53
+//line app/vmalert/web.qtpl:54
 			if len(g.Params) > 0 {
-//line app/vmalert/web.qtpl:53
+//line app/vmalert/web.qtpl:54
 				qw422016.N().S(`
                     <div class="fs-6 fw-lighter">Extra params
                     `)
-//line app/vmalert/web.qtpl:55
+//line app/vmalert/web.qtpl:56
 				for _, param := range g.Params {
-//line app/vmalert/web.qtpl:55
+//line app/vmalert/web.qtpl:56
 					qw422016.N().S(`
                             <span class="float-left badge bg-primary">`)
-//line app/vmalert/web.qtpl:56
+//line app/vmalert/web.qtpl:57
 					qw422016.E().S(param)
-//line app/vmalert/web.qtpl:56
+//line app/vmalert/web.qtpl:57
 					qw422016.N().S(`</span>
                     `)
-//line app/vmalert/web.qtpl:57
+//line app/vmalert/web.qtpl:58
 				}
-//line app/vmalert/web.qtpl:57
+//line app/vmalert/web.qtpl:58
 				qw422016.N().S(`
                     </div>
                 `)
-//line app/vmalert/web.qtpl:59
+//line app/vmalert/web.qtpl:60
 			}
-//line app/vmalert/web.qtpl:59
+//line app/vmalert/web.qtpl:60
 			qw422016.N().S(`
                 `)
-//line app/vmalert/web.qtpl:60
+//line app/vmalert/web.qtpl:61
 			if len(g.Headers) > 0 {
-//line app/vmalert/web.qtpl:60
+//line app/vmalert/web.qtpl:61
 				qw422016.N().S(`
                     <div class="fs-6 fw-lighter">Extra headers
                     `)
-//line app/vmalert/web.qtpl:62
+//line app/vmalert/web.qtpl:63
 				for _, header := range g.Headers {
-//line app/vmalert/web.qtpl:62
+//line app/vmalert/web.qtpl:63
 					qw422016.N().S(`
                             <span class="float-left badge bg-primary">`)
-//line app/vmalert/web.qtpl:63
+//line app/vmalert/web.qtpl:64
 					qw422016.E().S(header)
-//line app/vmalert/web.qtpl:63
+//line app/vmalert/web.qtpl:64
 					qw422016.N().S(`</span>
                     `)
-//line app/vmalert/web.qtpl:64
+//line app/vmalert/web.qtpl:65
 				}
-//line app/vmalert/web.qtpl:64
+//line app/vmalert/web.qtpl:65
 				qw422016.N().S(`
                     </div>
                 `)
-//line app/vmalert/web.qtpl:66
+//line app/vmalert/web.qtpl:67
 			}
-//line app/vmalert/web.qtpl:66
+//line app/vmalert/web.qtpl:67
 			qw422016.N().S(`
             </div>
             <div class="collapse" id="rules-`)
-//line app/vmalert/web.qtpl:68
+//line app/vmalert/web.qtpl:69
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:68
+//line app/vmalert/web.qtpl:69
 			qw422016.N().S(`">
                 <table class="table table-striped table-hover table-sm">
                     <thead>
@@ -275,262 +281,267 @@ func StreamListGroups(qw422016 *qt422016.Writer, r *http.Request, groups []APIGr
                     </thead>
                     <tbody>
                     `)
-//line app/vmalert/web.qtpl:78
+//line app/vmalert/web.qtpl:79
 			for _, r := range g.Rules {
-//line app/vmalert/web.qtpl:78
+//line app/vmalert/web.qtpl:79
 				qw422016.N().S(`
                         <tr`)
-//line app/vmalert/web.qtpl:79
+//line app/vmalert/web.qtpl:80
 				if r.LastError != "" {
-//line app/vmalert/web.qtpl:79
+//line app/vmalert/web.qtpl:80
 					qw422016.N().S(` class="alert-danger"`)
-//line app/vmalert/web.qtpl:79
+//line app/vmalert/web.qtpl:80
 				}
-//line app/vmalert/web.qtpl:79
+//line app/vmalert/web.qtpl:80
 				qw422016.N().S(`>
                             <td>
                                 <div class="row">
                                     <div class="col-12 mb-2">
                                         `)
-//line app/vmalert/web.qtpl:83
+//line app/vmalert/web.qtpl:84
 				if r.Type == "alerting" {
-//line app/vmalert/web.qtpl:83
+//line app/vmalert/web.qtpl:84
 					qw422016.N().S(`
                                         <b>alert:</b> `)
-//line app/vmalert/web.qtpl:84
+//line app/vmalert/web.qtpl:85
 					qw422016.E().S(r.Name)
-//line app/vmalert/web.qtpl:84
+//line app/vmalert/web.qtpl:85
 					qw422016.N().S(` (for: `)
-//line app/vmalert/web.qtpl:84
+//line app/vmalert/web.qtpl:85
 					qw422016.E().V(r.Duration)
-//line app/vmalert/web.qtpl:84
+//line app/vmalert/web.qtpl:85
 					qw422016.N().S(` seconds)
                                         `)
-//line app/vmalert/web.qtpl:85
+//line app/vmalert/web.qtpl:86
 				} else {
-//line app/vmalert/web.qtpl:85
+//line app/vmalert/web.qtpl:86
 					qw422016.N().S(`
                                         <b>record:</b> `)
-//line app/vmalert/web.qtpl:86
+//line app/vmalert/web.qtpl:87
 					qw422016.E().S(r.Name)
-//line app/vmalert/web.qtpl:86
+//line app/vmalert/web.qtpl:87
 					qw422016.N().S(`
                                         `)
-//line app/vmalert/web.qtpl:87
+//line app/vmalert/web.qtpl:88
 				}
-//line app/vmalert/web.qtpl:87
+//line app/vmalert/web.qtpl:88
 				qw422016.N().S(`
+                                        | <span><a target="_blank" href="`)
+//line app/vmalert/web.qtpl:89
+				qw422016.E().S(prefix + r.WebLink())
+//line app/vmalert/web.qtpl:89
+				qw422016.N().S(`">Details</a></span>
                                     </div>
                                     <div class="col-12">
                                         <code><pre>`)
-//line app/vmalert/web.qtpl:90
+//line app/vmalert/web.qtpl:92
 				qw422016.E().S(r.Query)
-//line app/vmalert/web.qtpl:90
+//line app/vmalert/web.qtpl:92
 				qw422016.N().S(`</pre></code>
                                     </div>
                                     <div class="col-12 mb-2">
                                         `)
-//line app/vmalert/web.qtpl:93
+//line app/vmalert/web.qtpl:95
 				if len(r.Labels) > 0 {
-//line app/vmalert/web.qtpl:93
+//line app/vmalert/web.qtpl:95
 					qw422016.N().S(` <b>Labels:</b>`)
-//line app/vmalert/web.qtpl:93
+//line app/vmalert/web.qtpl:95
 				}
-//line app/vmalert/web.qtpl:93
+//line app/vmalert/web.qtpl:95
 				qw422016.N().S(`
                                         `)
-//line app/vmalert/web.qtpl:94
+//line app/vmalert/web.qtpl:96
 				for k, v := range r.Labels {
-//line app/vmalert/web.qtpl:94
+//line app/vmalert/web.qtpl:96
 					qw422016.N().S(`
                                                 <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:95
+//line app/vmalert/web.qtpl:97
 					qw422016.E().S(k)
-//line app/vmalert/web.qtpl:95
+//line app/vmalert/web.qtpl:97
 					qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:95
+//line app/vmalert/web.qtpl:97
 					qw422016.E().S(v)
-//line app/vmalert/web.qtpl:95
+//line app/vmalert/web.qtpl:97
 					qw422016.N().S(`</span>
                                         `)
-//line app/vmalert/web.qtpl:96
+//line app/vmalert/web.qtpl:98
 				}
-//line app/vmalert/web.qtpl:96
+//line app/vmalert/web.qtpl:98
 				qw422016.N().S(`
                                     </div>
                                     `)
-//line app/vmalert/web.qtpl:98
+//line app/vmalert/web.qtpl:100
 				if r.LastError != "" {
-//line app/vmalert/web.qtpl:98
+//line app/vmalert/web.qtpl:100
 					qw422016.N().S(`
                                     <div class="col-12">
                                         <b>Error:</b>
                                         <div class="error-cell">
                                         `)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.E().S(r.LastError)
-//line app/vmalert/web.qtpl:102
+//line app/vmalert/web.qtpl:104
 					qw422016.N().S(`
                                         </div>
                                     </div>
                                     `)
-//line app/vmalert/web.qtpl:105
+//line app/vmalert/web.qtpl:107
 				}
-//line app/vmalert/web.qtpl:105
+//line app/vmalert/web.qtpl:107
 				qw422016.N().S(`
                                 </div>
                             </td>
                             <td class="text-center">`)
-//line app/vmalert/web.qtpl:108
+//line app/vmalert/web.qtpl:110
 				qw422016.N().D(r.LastSamples)
-//line app/vmalert/web.qtpl:108
+//line app/vmalert/web.qtpl:110
 				qw422016.N().S(`</td>
                             <td class="text-center">`)
-//line app/vmalert/web.qtpl:109
+//line app/vmalert/web.qtpl:111
 				qw422016.N().FPrec(time.Since(r.LastEvaluation).Seconds(), 3)
-//line app/vmalert/web.qtpl:109
+//line app/vmalert/web.qtpl:111
 				qw422016.N().S(`s ago</td>
                         </tr>
                     `)
-//line app/vmalert/web.qtpl:111
+//line app/vmalert/web.qtpl:113
 			}
-//line app/vmalert/web.qtpl:111
+//line app/vmalert/web.qtpl:113
 			qw422016.N().S(`
                  </tbody>
                 </table>
             </div>
         `)
-//line app/vmalert/web.qtpl:115
+//line app/vmalert/web.qtpl:117
 		}
-//line app/vmalert/web.qtpl:115
+//line app/vmalert/web.qtpl:117
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:117
+//line app/vmalert/web.qtpl:119
 	} else {
-//line app/vmalert/web.qtpl:117
+//line app/vmalert/web.qtpl:119
 		qw422016.N().S(`
         <div>
-            <p>No items...</p>
+            <p>No groups...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:121
+//line app/vmalert/web.qtpl:123
 	}
-//line app/vmalert/web.qtpl:121
+//line app/vmalert/web.qtpl:123
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:123
+//line app/vmalert/web.qtpl:125
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:123
+//line app/vmalert/web.qtpl:125
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 }
 
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 func WriteListGroups(qq422016 qtio422016.Writer, r *http.Request, groups []APIGroup) {
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	StreamListGroups(qw422016, r, groups)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 }
 
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 func ListGroups(r *http.Request, groups []APIGroup) string {
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	WriteListGroups(qb422016, r, groups)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 	return qs422016
-//line app/vmalert/web.qtpl:125
+//line app/vmalert/web.qtpl:127
 }
 
-//line app/vmalert/web.qtpl:128
+//line app/vmalert/web.qtpl:130
 func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []GroupAlerts) {
-//line app/vmalert/web.qtpl:128
+//line app/vmalert/web.qtpl:130
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:129
+//line app/vmalert/web.qtpl:131
 	prefix := utils.Prefix(r.URL.Path)
 
-//line app/vmalert/web.qtpl:129
+//line app/vmalert/web.qtpl:131
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:130
+//line app/vmalert/web.qtpl:132
 	tpl.StreamHeader(qw422016, r, navItems, "Alerts")
-//line app/vmalert/web.qtpl:130
+//line app/vmalert/web.qtpl:132
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:131
+//line app/vmalert/web.qtpl:133
 	if len(groupAlerts) > 0 {
-//line app/vmalert/web.qtpl:131
+//line app/vmalert/web.qtpl:133
 		qw422016.N().S(`
          <a class="btn btn-primary" role="button" onclick="collapseAll()">Collapse All</a>
          <a class="btn btn-primary" role="button" onclick="expandAll()">Expand All</a>
          `)
-//line app/vmalert/web.qtpl:134
+//line app/vmalert/web.qtpl:136
 		for _, ga := range groupAlerts {
-//line app/vmalert/web.qtpl:134
+//line app/vmalert/web.qtpl:136
 			qw422016.N().S(`
             `)
-//line app/vmalert/web.qtpl:135
+//line app/vmalert/web.qtpl:137
 			g := ga.Group
 
-//line app/vmalert/web.qtpl:135
+//line app/vmalert/web.qtpl:137
 			qw422016.N().S(`
             <div class="group-heading alert-danger" data-bs-target="rules-`)
-//line app/vmalert/web.qtpl:136
+//line app/vmalert/web.qtpl:138
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:136
+//line app/vmalert/web.qtpl:138
 			qw422016.N().S(`">
                 <span class="anchor" id="group-`)
-//line app/vmalert/web.qtpl:137
+//line app/vmalert/web.qtpl:139
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:137
+//line app/vmalert/web.qtpl:139
 			qw422016.N().S(`"></span>
                 <a href="#group-`)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			qw422016.E().S(g.Name)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			if g.Type != "prometheus" {
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 				qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 				qw422016.E().S(g.Type)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 				qw422016.N().S(`)`)
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			}
-//line app/vmalert/web.qtpl:138
+//line app/vmalert/web.qtpl:140
 			qw422016.N().S(`</a>
                 <span class="badge bg-danger" title="Number of active alerts">`)
-//line app/vmalert/web.qtpl:139
+//line app/vmalert/web.qtpl:141
 			qw422016.N().D(len(ga.Alerts))
-//line app/vmalert/web.qtpl:139
+//line app/vmalert/web.qtpl:141
 			qw422016.N().S(`</span>
                 <br>
                 <p class="fs-6 fw-lighter">`)
-//line app/vmalert/web.qtpl:141
+//line app/vmalert/web.qtpl:143
 			qw422016.E().S(g.File)
-//line app/vmalert/web.qtpl:141
+//line app/vmalert/web.qtpl:143
 			qw422016.N().S(`</p>
             </div>
             `)
-//line app/vmalert/web.qtpl:144
+//line app/vmalert/web.qtpl:146
 			var keys []string
 			alertsByRule := make(map[string][]*APIAlert)
 			for _, alert := range ga.Alerts {
@@ -541,20 +552,20 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
 			}
 			sort.Strings(keys)
 
-//line app/vmalert/web.qtpl:153
+//line app/vmalert/web.qtpl:155
 			qw422016.N().S(`
             <div class="collapse" id="rules-`)
-//line app/vmalert/web.qtpl:154
+//line app/vmalert/web.qtpl:156
 			qw422016.E().S(g.ID)
-//line app/vmalert/web.qtpl:154
+//line app/vmalert/web.qtpl:156
 			qw422016.N().S(`">
                 `)
-//line app/vmalert/web.qtpl:155
+//line app/vmalert/web.qtpl:157
 			for _, ruleID := range keys {
-//line app/vmalert/web.qtpl:155
+//line app/vmalert/web.qtpl:157
 				qw422016.N().S(`
                     `)
-//line app/vmalert/web.qtpl:157
+//line app/vmalert/web.qtpl:159
 				defaultAR := alertsByRule[ruleID][0]
 				var labelKeys []string
 				for k := range defaultAR.Labels {
@@ -562,28 +573,28 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
 				}
 				sort.Strings(labelKeys)
 
-//line app/vmalert/web.qtpl:163
+//line app/vmalert/web.qtpl:165
 				qw422016.N().S(`
                     <br>
                     <b>alert:</b> `)
-//line app/vmalert/web.qtpl:165
+//line app/vmalert/web.qtpl:167
 				qw422016.E().S(defaultAR.Name)
-//line app/vmalert/web.qtpl:165
+//line app/vmalert/web.qtpl:167
 				qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:165
+//line app/vmalert/web.qtpl:167
 				qw422016.N().D(len(alertsByRule[ruleID]))
-//line app/vmalert/web.qtpl:165
+//line app/vmalert/web.qtpl:167
 				qw422016.N().S(`)
                      | <span><a target="_blank" href="`)
-//line app/vmalert/web.qtpl:166
+//line app/vmalert/web.qtpl:168
 				qw422016.E().S(defaultAR.SourceLink)
-//line app/vmalert/web.qtpl:166
+//line app/vmalert/web.qtpl:168
 				qw422016.N().S(`">Source</a></span>
                     <br>
                     <b>expr:</b><code><pre>`)
-//line app/vmalert/web.qtpl:168
+//line app/vmalert/web.qtpl:170
 				qw422016.E().S(defaultAR.Expression)
-//line app/vmalert/web.qtpl:168
+//line app/vmalert/web.qtpl:170
 				qw422016.N().S(`</pre></code>
                     <table class="table table-striped table-hover table-sm">
                         <thead>
@@ -597,204 +608,204 @@ func StreamListAlerts(qw422016 *qt422016.Writer, r *http.Request, groupAlerts []
                         </thead>
                         <tbody>
                         `)
-//line app/vmalert/web.qtpl:180
+//line app/vmalert/web.qtpl:182
 				for _, ar := range alertsByRule[ruleID] {
-//line app/vmalert/web.qtpl:180
+//line app/vmalert/web.qtpl:182
 					qw422016.N().S(`
                             <tr>
                                 <td>
                                     `)
-//line app/vmalert/web.qtpl:183
+//line app/vmalert/web.qtpl:185
 					for _, k := range labelKeys {
-//line app/vmalert/web.qtpl:183
+//line app/vmalert/web.qtpl:185
 						qw422016.N().S(`
                                         <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:184
+//line app/vmalert/web.qtpl:186
 						qw422016.E().S(k)
-//line app/vmalert/web.qtpl:184
+//line app/vmalert/web.qtpl:186
 						qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:184
+//line app/vmalert/web.qtpl:186
 						qw422016.E().S(ar.Labels[k])
-//line app/vmalert/web.qtpl:184
+//line app/vmalert/web.qtpl:186
 						qw422016.N().S(`</span>
                                     `)
-//line app/vmalert/web.qtpl:185
+//line app/vmalert/web.qtpl:187
 					}
-//line app/vmalert/web.qtpl:185
+//line app/vmalert/web.qtpl:187
 					qw422016.N().S(`
                                 </td>
                                 <td>`)
-//line app/vmalert/web.qtpl:187
+//line app/vmalert/web.qtpl:189
 					streambadgeState(qw422016, ar.State)
-//line app/vmalert/web.qtpl:187
+//line app/vmalert/web.qtpl:189
 					qw422016.N().S(`</td>
                                 <td>
                                     `)
-//line app/vmalert/web.qtpl:189
+//line app/vmalert/web.qtpl:191
 					qw422016.E().S(ar.ActiveAt.Format("2006-01-02T15:04:05Z07:00"))
-//line app/vmalert/web.qtpl:189
+//line app/vmalert/web.qtpl:191
 					qw422016.N().S(`
                                     `)
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 					if ar.Restored {
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 						streambadgeRestored(qw422016)
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 					}
-//line app/vmalert/web.qtpl:190
+//line app/vmalert/web.qtpl:192
 					qw422016.N().S(`
                                 </td>
                                 <td>`)
-//line app/vmalert/web.qtpl:192
+//line app/vmalert/web.qtpl:194
 					qw422016.E().S(ar.Value)
-//line app/vmalert/web.qtpl:192
+//line app/vmalert/web.qtpl:194
 					qw422016.N().S(`</td>
                                 <td>
                                     <a href="`)
-//line app/vmalert/web.qtpl:194
+//line app/vmalert/web.qtpl:196
 					qw422016.E().S(prefix + ar.WebLink())
-//line app/vmalert/web.qtpl:194
+//line app/vmalert/web.qtpl:196
 					qw422016.N().S(`">Details</a>
                                 </td>
                             </tr>
                         `)
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 				}
-//line app/vmalert/web.qtpl:197
+//line app/vmalert/web.qtpl:199
 				qw422016.N().S(`
                      </tbody>
                     </table>
                 `)
-//line app/vmalert/web.qtpl:200
+//line app/vmalert/web.qtpl:202
 			}
-//line app/vmalert/web.qtpl:200
+//line app/vmalert/web.qtpl:202
 			qw422016.N().S(`
             </div>
             <br>
         `)
-//line app/vmalert/web.qtpl:203
+//line app/vmalert/web.qtpl:205
 		}
-//line app/vmalert/web.qtpl:203
+//line app/vmalert/web.qtpl:205
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:205
+//line app/vmalert/web.qtpl:207
 	} else {
-//line app/vmalert/web.qtpl:205
+//line app/vmalert/web.qtpl:207
 		qw422016.N().S(`
         <div>
-            <p>No items...</p>
+            <p>No active alerts...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:209
+//line app/vmalert/web.qtpl:211
 	}
-//line app/vmalert/web.qtpl:209
+//line app/vmalert/web.qtpl:211
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:211
+//line app/vmalert/web.qtpl:213
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:211
+//line app/vmalert/web.qtpl:213
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 }
 
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 func WriteListAlerts(qq422016 qtio422016.Writer, r *http.Request, groupAlerts []GroupAlerts) {
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	StreamListAlerts(qw422016, r, groupAlerts)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 }
 
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 func ListAlerts(r *http.Request, groupAlerts []GroupAlerts) string {
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	WriteListAlerts(qb422016, r, groupAlerts)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 	return qs422016
-//line app/vmalert/web.qtpl:213
+//line app/vmalert/web.qtpl:215
 }
 
-//line app/vmalert/web.qtpl:215
+//line app/vmalert/web.qtpl:217
 func StreamListTargets(qw422016 *qt422016.Writer, r *http.Request, targets map[notifier.TargetType][]notifier.Target) {
-//line app/vmalert/web.qtpl:215
+//line app/vmalert/web.qtpl:217
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:216
+//line app/vmalert/web.qtpl:218
 	tpl.StreamHeader(qw422016, r, navItems, "Notifiers")
-//line app/vmalert/web.qtpl:216
+//line app/vmalert/web.qtpl:218
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:217
+//line app/vmalert/web.qtpl:219
 	if len(targets) > 0 {
-//line app/vmalert/web.qtpl:217
+//line app/vmalert/web.qtpl:219
 		qw422016.N().S(`
          <a class="btn btn-primary" role="button" onclick="collapseAll()">Collapse All</a>
          <a class="btn btn-primary" role="button" onclick="expandAll()">Expand All</a>
 
          `)
-//line app/vmalert/web.qtpl:222
+//line app/vmalert/web.qtpl:224
 		var keys []string
 		for key := range targets {
 			keys = append(keys, string(key))
 		}
 		sort.Strings(keys)
 
-//line app/vmalert/web.qtpl:227
+//line app/vmalert/web.qtpl:229
 		qw422016.N().S(`
 
          `)
-//line app/vmalert/web.qtpl:229
+//line app/vmalert/web.qtpl:231
 		for i := range keys {
-//line app/vmalert/web.qtpl:229
+//line app/vmalert/web.qtpl:231
 			qw422016.N().S(`
            `)
-//line app/vmalert/web.qtpl:230
+//line app/vmalert/web.qtpl:232
 			typeK, ns := keys[i], targets[notifier.TargetType(keys[i])]
 			count := len(ns)
 
-//line app/vmalert/web.qtpl:232
+//line app/vmalert/web.qtpl:234
 			qw422016.N().S(`
            <div class="group-heading data-bs-target="rules-`)
-//line app/vmalert/web.qtpl:233
+//line app/vmalert/web.qtpl:235
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:233
+//line app/vmalert/web.qtpl:235
 			qw422016.N().S(`">
              <span class="anchor" id="notifiers-`)
-//line app/vmalert/web.qtpl:234
+//line app/vmalert/web.qtpl:236
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:234
+//line app/vmalert/web.qtpl:236
 			qw422016.N().S(`"></span>
              <a href="#notifiers-`)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.N().S(` (`)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.N().D(count)
-//line app/vmalert/web.qtpl:235
+//line app/vmalert/web.qtpl:237
 			qw422016.N().S(`)</a>
          </div>
          <div class="collapse show" id="notifiers-`)
-//line app/vmalert/web.qtpl:237
+//line app/vmalert/web.qtpl:239
 			qw422016.E().S(typeK)
-//line app/vmalert/web.qtpl:237
+//line app/vmalert/web.qtpl:239
 			qw422016.N().S(`">
              <table class="table table-striped table-hover table-sm">
                  <thead>
@@ -805,119 +816,119 @@ func StreamListTargets(qw422016 *qt422016.Writer, r *http.Request, targets map[n
                  </thead>
                  <tbody>
                  `)
-//line app/vmalert/web.qtpl:246
+//line app/vmalert/web.qtpl:248
 			for _, n := range ns {
-//line app/vmalert/web.qtpl:246
+//line app/vmalert/web.qtpl:248
 				qw422016.N().S(`
                      <tr>
                          <td>
                               `)
-//line app/vmalert/web.qtpl:249
+//line app/vmalert/web.qtpl:251
 				for _, l := range n.Labels {
-//line app/vmalert/web.qtpl:249
+//line app/vmalert/web.qtpl:251
 					qw422016.N().S(`
                                       <span class="ms-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:250
+//line app/vmalert/web.qtpl:252
 					qw422016.E().S(l.Name)
-//line app/vmalert/web.qtpl:250
+//line app/vmalert/web.qtpl:252
 					qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:250
+//line app/vmalert/web.qtpl:252
 					qw422016.E().S(l.Value)
-//line app/vmalert/web.qtpl:250
+//line app/vmalert/web.qtpl:252
 					qw422016.N().S(`</span>
                               `)
-//line app/vmalert/web.qtpl:251
+//line app/vmalert/web.qtpl:253
 				}
-//line app/vmalert/web.qtpl:251
+//line app/vmalert/web.qtpl:253
 				qw422016.N().S(`
                           </td>
                          <td>`)
-//line app/vmalert/web.qtpl:253
+//line app/vmalert/web.qtpl:255
 				qw422016.E().S(n.Notifier.Addr())
-//line app/vmalert/web.qtpl:253
+//line app/vmalert/web.qtpl:255
 				qw422016.N().S(`</td>
                      </tr>
                  `)
-//line app/vmalert/web.qtpl:255
+//line app/vmalert/web.qtpl:257
 			}
-//line app/vmalert/web.qtpl:255
+//line app/vmalert/web.qtpl:257
 			qw422016.N().S(`
               </tbody>
              </table>
          </div>
      `)
-//line app/vmalert/web.qtpl:259
+//line app/vmalert/web.qtpl:261
 		}
-//line app/vmalert/web.qtpl:259
+//line app/vmalert/web.qtpl:261
 		qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:261
+//line app/vmalert/web.qtpl:263
 	} else {
-//line app/vmalert/web.qtpl:261
+//line app/vmalert/web.qtpl:263
 		qw422016.N().S(`
         <div>
-            <p>No items...</p>
+            <p>No targets...</p>
         </div>
     `)
-//line app/vmalert/web.qtpl:265
+//line app/vmalert/web.qtpl:267
 	}
-//line app/vmalert/web.qtpl:265
+//line app/vmalert/web.qtpl:267
 	qw422016.N().S(`
 
     `)
-//line app/vmalert/web.qtpl:267
+//line app/vmalert/web.qtpl:269
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:267
+//line app/vmalert/web.qtpl:269
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 }
 
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 func WriteListTargets(qq422016 qtio422016.Writer, r *http.Request, targets map[notifier.TargetType][]notifier.Target) {
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	StreamListTargets(qw422016, r, targets)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 }
 
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 func ListTargets(r *http.Request, targets map[notifier.TargetType][]notifier.Target) string {
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	WriteListTargets(qb422016, r, targets)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 	return qs422016
-//line app/vmalert/web.qtpl:269
+//line app/vmalert/web.qtpl:271
 }
 
-//line app/vmalert/web.qtpl:271
+//line app/vmalert/web.qtpl:273
 func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
-//line app/vmalert/web.qtpl:271
+//line app/vmalert/web.qtpl:273
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:272
+//line app/vmalert/web.qtpl:274
 	prefix := utils.Prefix(r.URL.Path)
 
-//line app/vmalert/web.qtpl:272
+//line app/vmalert/web.qtpl:274
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:273
+//line app/vmalert/web.qtpl:275
 	tpl.StreamHeader(qw422016, r, navItems, "")
-//line app/vmalert/web.qtpl:273
+//line app/vmalert/web.qtpl:275
 	qw422016.N().S(`
     `)
-//line app/vmalert/web.qtpl:275
+//line app/vmalert/web.qtpl:277
 	var labelKeys []string
 	for k := range alert.Labels {
 		labelKeys = append(labelKeys, k)
@@ -930,28 +941,28 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
 	}
 	sort.Strings(annotationKeys)
 
-//line app/vmalert/web.qtpl:286
+//line app/vmalert/web.qtpl:288
 	qw422016.N().S(`
-    <div class="display-6 pb-3 mb-3">`)
-//line app/vmalert/web.qtpl:287
+    <div class="display-6 pb-3 mb-3">Alert: `)
+//line app/vmalert/web.qtpl:289
 	qw422016.E().S(alert.Name)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	qw422016.N().S(`<span class="ms-2 badge `)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	if alert.State == "firing" {
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 		qw422016.N().S(`bg-danger`)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	} else {
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 		qw422016.N().S(` bg-warning text-dark`)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	}
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	qw422016.E().S(alert.State)
-//line app/vmalert/web.qtpl:287
+//line app/vmalert/web.qtpl:289
 	qw422016.N().S(`</span></div>
     <div class="container border-bottom p-2">
       <div class="row">
@@ -960,9 +971,9 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
           `)
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.E().S(alert.ActiveAt.Format("2006-01-02T15:04:05Z07:00"))
-//line app/vmalert/web.qtpl:294
+//line app/vmalert/web.qtpl:296
 	qw422016.N().S(`
         </div>
       </div>
@@ -974,9 +985,9 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
           <code><pre>`)
-//line app/vmalert/web.qtpl:304
+//line app/vmalert/web.qtpl:306
 	qw422016.E().S(alert.Expression)
-//line app/vmalert/web.qtpl:304
+//line app/vmalert/web.qtpl:306
 	qw422016.N().S(`</pre></code>
         </div>
       </div>
@@ -988,23 +999,23 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            `)
-//line app/vmalert/web.qtpl:314
+//line app/vmalert/web.qtpl:316
 	for _, k := range labelKeys {
-//line app/vmalert/web.qtpl:314
+//line app/vmalert/web.qtpl:316
 		qw422016.N().S(`
                 <span class="m-1 badge bg-primary">`)
-//line app/vmalert/web.qtpl:315
+//line app/vmalert/web.qtpl:317
 		qw422016.E().S(k)
-//line app/vmalert/web.qtpl:315
+//line app/vmalert/web.qtpl:317
 		qw422016.N().S(`=`)
-//line app/vmalert/web.qtpl:315
+//line app/vmalert/web.qtpl:317
 		qw422016.E().S(alert.Labels[k])
-//line app/vmalert/web.qtpl:315
+//line app/vmalert/web.qtpl:317
 		qw422016.N().S(`</span>
           `)
-//line app/vmalert/web.qtpl:316
+//line app/vmalert/web.qtpl:318
 	}
-//line app/vmalert/web.qtpl:316
+//line app/vmalert/web.qtpl:318
 	qw422016.N().S(`
         </div>
       </div>
@@ -1016,24 +1027,24 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            `)
-//line app/vmalert/web.qtpl:326
+//line app/vmalert/web.qtpl:328
 	for _, k := range annotationKeys {
-//line app/vmalert/web.qtpl:326
+//line app/vmalert/web.qtpl:328
 		qw422016.N().S(`
                 <b>`)
-//line app/vmalert/web.qtpl:327
+//line app/vmalert/web.qtpl:329
 		qw422016.E().S(k)
-//line app/vmalert/web.qtpl:327
+//line app/vmalert/web.qtpl:329
 		qw422016.N().S(`:</b><br>
                 <p>`)
-//line app/vmalert/web.qtpl:328
+//line app/vmalert/web.qtpl:330
 		qw422016.E().S(alert.Annotations[k])
-//line app/vmalert/web.qtpl:328
+//line app/vmalert/web.qtpl:330
 		qw422016.N().S(`</p>
           `)
-//line app/vmalert/web.qtpl:329
+//line app/vmalert/web.qtpl:331
 	}
-//line app/vmalert/web.qtpl:329
+//line app/vmalert/web.qtpl:331
 	qw422016.N().S(`
         </div>
       </div>
@@ -1045,17 +1056,17 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            <a target="_blank" href="`)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.E().S(prefix)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.N().S(`groups#group-`)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.E().S(alert.GroupID)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.E().S(alert.GroupID)
-//line app/vmalert/web.qtpl:339
+//line app/vmalert/web.qtpl:341
 	qw422016.N().S(`</a>
         </div>
       </div>
@@ -1067,132 +1078,415 @@ func StreamAlert(qw422016 *qt422016.Writer, r *http.Request, alert *APIAlert) {
         </div>
         <div class="col">
            <a target="_blank" href="`)
-//line app/vmalert/web.qtpl:349
+//line app/vmalert/web.qtpl:351
 	qw422016.E().S(alert.SourceLink)
-//line app/vmalert/web.qtpl:349
+//line app/vmalert/web.qtpl:351
 	qw422016.N().S(`">Link</a>
         </div>
       </div>
     </div>
     `)
-//line app/vmalert/web.qtpl:353
+//line app/vmalert/web.qtpl:355
 	tpl.StreamFooter(qw422016, r)
-//line app/vmalert/web.qtpl:353
+//line app/vmalert/web.qtpl:355
 	qw422016.N().S(`
 
 `)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 }
 
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 func WriteAlert(qq422016 qtio422016.Writer, r *http.Request, alert *APIAlert) {
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	StreamAlert(qw422016, r, alert)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 }
 
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 func Alert(r *http.Request, alert *APIAlert) string {
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	WriteAlert(qb422016, r, alert)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 	return qs422016
-//line app/vmalert/web.qtpl:355
+//line app/vmalert/web.qtpl:357
 }
 
-//line app/vmalert/web.qtpl:357
+//line app/vmalert/web.qtpl:360
+func StreamRuleDetails(qw422016 *qt422016.Writer, r *http.Request, rule APIRule) {
+//line app/vmalert/web.qtpl:360
+	qw422016.N().S(`
+    `)
+//line app/vmalert/web.qtpl:361
+	prefix := utils.Prefix(r.URL.Path)
+
+//line app/vmalert/web.qtpl:361
+	qw422016.N().S(`
+    `)
+//line app/vmalert/web.qtpl:362
+	tpl.StreamHeader(qw422016, r, navItems, "")
+//line app/vmalert/web.qtpl:362
+	qw422016.N().S(`
+    `)
+//line app/vmalert/web.qtpl:364
+	var labelKeys []string
+	for k := range rule.Labels {
+		labelKeys = append(labelKeys, k)
+	}
+	sort.Strings(labelKeys)
+
+	var annotationKeys []string
+	for k := range rule.Annotations {
+		annotationKeys = append(annotationKeys, k)
+	}
+	sort.Strings(annotationKeys)
+
+//line app/vmalert/web.qtpl:375
+	qw422016.N().S(`
+    <div class="display-6 pb-3 mb-3">Rule: `)
+//line app/vmalert/web.qtpl:376
+	qw422016.E().S(rule.Name)
+//line app/vmalert/web.qtpl:376
+	qw422016.N().S(`<span class="ms-2 badge `)
+//line app/vmalert/web.qtpl:376
+	if rule.Health != "ok" {
+//line app/vmalert/web.qtpl:376
+		qw422016.N().S(`bg-danger`)
+//line app/vmalert/web.qtpl:376
+	} else {
+//line app/vmalert/web.qtpl:376
+		qw422016.N().S(` bg-warning text-dark`)
+//line app/vmalert/web.qtpl:376
+	}
+//line app/vmalert/web.qtpl:376
+	qw422016.N().S(`">`)
+//line app/vmalert/web.qtpl:376
+	qw422016.E().S(rule.Health)
+//line app/vmalert/web.qtpl:376
+	qw422016.N().S(`</span></div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Expr
+        </div>
+        <div class="col">
+          <code><pre>`)
+//line app/vmalert/web.qtpl:383
+	qw422016.E().S(rule.Query)
+//line app/vmalert/web.qtpl:383
+	qw422016.N().S(`</pre></code>
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          For
+        </div>
+        <div class="col">
+         `)
+//line app/vmalert/web.qtpl:393
+	qw422016.E().V(rule.Duration)
+//line app/vmalert/web.qtpl:393
+	qw422016.N().S(` seconds
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Labels
+        </div>
+        <div class="col">
+          `)
+//line app/vmalert/web.qtpl:403
+	for _, k := range labelKeys {
+//line app/vmalert/web.qtpl:403
+		qw422016.N().S(`
+                <span class="m-1 badge bg-primary">`)
+//line app/vmalert/web.qtpl:404
+		qw422016.E().S(k)
+//line app/vmalert/web.qtpl:404
+		qw422016.N().S(`=`)
+//line app/vmalert/web.qtpl:404
+		qw422016.E().S(rule.Labels[k])
+//line app/vmalert/web.qtpl:404
+		qw422016.N().S(`</span>
+          `)
+//line app/vmalert/web.qtpl:405
+	}
+//line app/vmalert/web.qtpl:405
+	qw422016.N().S(`
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Annotations
+        </div>
+        <div class="col">
+          `)
+//line app/vmalert/web.qtpl:415
+	for _, k := range annotationKeys {
+//line app/vmalert/web.qtpl:415
+		qw422016.N().S(`
+                <b>`)
+//line app/vmalert/web.qtpl:416
+		qw422016.E().S(k)
+//line app/vmalert/web.qtpl:416
+		qw422016.N().S(`:</b><br>
+                <p>`)
+//line app/vmalert/web.qtpl:417
+		qw422016.E().S(rule.Annotations[k])
+//line app/vmalert/web.qtpl:417
+		qw422016.N().S(`</p>
+          `)
+//line app/vmalert/web.qtpl:418
+	}
+//line app/vmalert/web.qtpl:418
+	qw422016.N().S(`
+        </div>
+      </div>
+    </div>
+    <div class="container border-bottom p-2">
+      <div class="row">
+        <div class="col-2">
+          Group
+        </div>
+        <div class="col">
+           <a target="_blank" href="`)
+//line app/vmalert/web.qtpl:428
+	qw422016.E().S(prefix)
+//line app/vmalert/web.qtpl:428
+	qw422016.N().S(`groups#group-`)
+//line app/vmalert/web.qtpl:428
+	qw422016.E().S(rule.GroupID)
+//line app/vmalert/web.qtpl:428
+	qw422016.N().S(`">`)
+//line app/vmalert/web.qtpl:428
+	qw422016.E().S(rule.GroupID)
+//line app/vmalert/web.qtpl:428
+	qw422016.N().S(`</a>
+        </div>
+      </div>
+    </div>
+
+    <br>
+    <div class="display-6 pb-3">Last `)
+//line app/vmalert/web.qtpl:434
+	qw422016.N().D(len(rule.Updates))
+//line app/vmalert/web.qtpl:434
+	qw422016.N().S(` updates</span>:</div>
+        <table class="table table-striped table-hover table-sm">
+            <thead>
+                <tr>
+                    <th scope="col" style="width: 20%" title="The time when event was created">Updated at</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="How many samples were returned">Samples</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="How many seconds request took">Duration</th>
+                    <th scope="col" style="width: 20%" class="text-center" title="Time used for rule execution">Executed at</th>
+                </tr>
+            </thead>
+            <tbody>
+
+     `)
+//line app/vmalert/web.qtpl:446
+	for _, u := range rule.Updates {
+//line app/vmalert/web.qtpl:446
+		qw422016.N().S(`
+             <tr`)
+//line app/vmalert/web.qtpl:447
+		if u.err != nil {
+//line app/vmalert/web.qtpl:447
+			qw422016.N().S(` class="alert-danger"`)
+//line app/vmalert/web.qtpl:447
+		}
+//line app/vmalert/web.qtpl:447
+		qw422016.N().S(`>
+                 <td>
+                    <span class="badge bg-primary rounded-pill me-3" title="Updated at">`)
+//line app/vmalert/web.qtpl:449
+		qw422016.E().S(u.time.Format(time.RFC3339))
+//line app/vmalert/web.qtpl:449
+		qw422016.N().S(`</span>
+                 </td>
+                 <td class="text-center">`)
+//line app/vmalert/web.qtpl:451
+		qw422016.N().D(u.samples)
+//line app/vmalert/web.qtpl:451
+		qw422016.N().S(`</td>
+                 <td class="text-center">`)
+//line app/vmalert/web.qtpl:452
+		qw422016.N().FPrec(u.duration.Seconds(), 3)
+//line app/vmalert/web.qtpl:452
+		qw422016.N().S(`s</td>
+                 <td class="text-center">`)
+//line app/vmalert/web.qtpl:453
+		qw422016.E().S(u.at.Format(time.RFC3339))
+//line app/vmalert/web.qtpl:453
+		qw422016.N().S(`</td>
+             </tr>
+          </li>
+          `)
+//line app/vmalert/web.qtpl:456
+		if u.err != nil {
+//line app/vmalert/web.qtpl:456
+			qw422016.N().S(`
+             <tr`)
+//line app/vmalert/web.qtpl:457
+			if u.err != nil {
+//line app/vmalert/web.qtpl:457
+				qw422016.N().S(` class="alert-danger"`)
+//line app/vmalert/web.qtpl:457
+			}
+//line app/vmalert/web.qtpl:457
+			qw422016.N().S(`>
+               <td colspan="4">
+                   <span class="alert-danger">`)
+//line app/vmalert/web.qtpl:459
+			qw422016.E().V(u.err)
+//line app/vmalert/web.qtpl:459
+			qw422016.N().S(`</span>
+               </td>
+             </tr>
+          `)
+//line app/vmalert/web.qtpl:462
+		}
+//line app/vmalert/web.qtpl:462
+		qw422016.N().S(`
+     `)
+//line app/vmalert/web.qtpl:463
+	}
+//line app/vmalert/web.qtpl:463
+	qw422016.N().S(`
+
+    `)
+//line app/vmalert/web.qtpl:465
+	tpl.StreamFooter(qw422016, r)
+//line app/vmalert/web.qtpl:465
+	qw422016.N().S(`
+`)
+//line app/vmalert/web.qtpl:466
+}
+
+//line app/vmalert/web.qtpl:466
+func WriteRuleDetails(qq422016 qtio422016.Writer, r *http.Request, rule APIRule) {
+//line app/vmalert/web.qtpl:466
+	qw422016 := qt422016.AcquireWriter(qq422016)
+//line app/vmalert/web.qtpl:466
+	StreamRuleDetails(qw422016, r, rule)
+//line app/vmalert/web.qtpl:466
+	qt422016.ReleaseWriter(qw422016)
+//line app/vmalert/web.qtpl:466
+}
+
+//line app/vmalert/web.qtpl:466
+func RuleDetails(r *http.Request, rule APIRule) string {
+//line app/vmalert/web.qtpl:466
+	qb422016 := qt422016.AcquireByteBuffer()
+//line app/vmalert/web.qtpl:466
+	WriteRuleDetails(qb422016, r, rule)
+//line app/vmalert/web.qtpl:466
+	qs422016 := string(qb422016.B)
+//line app/vmalert/web.qtpl:466
+	qt422016.ReleaseByteBuffer(qb422016)
+//line app/vmalert/web.qtpl:466
+	return qs422016
+//line app/vmalert/web.qtpl:466
+}
+
+//line app/vmalert/web.qtpl:470
 func streambadgeState(qw422016 *qt422016.Writer, state string) {
-//line app/vmalert/web.qtpl:357
+//line app/vmalert/web.qtpl:470
 	qw422016.N().S(`
 `)
-//line app/vmalert/web.qtpl:359
+//line app/vmalert/web.qtpl:472
 	badgeClass := "bg-warning text-dark"
 	if state == "firing" {
 		badgeClass = "bg-danger"
 	}
 
-//line app/vmalert/web.qtpl:363
+//line app/vmalert/web.qtpl:476
 	qw422016.N().S(`
 <span class="badge `)
-//line app/vmalert/web.qtpl:364
+//line app/vmalert/web.qtpl:477
 	qw422016.E().S(badgeClass)
-//line app/vmalert/web.qtpl:364
+//line app/vmalert/web.qtpl:477
 	qw422016.N().S(`">`)
-//line app/vmalert/web.qtpl:364
+//line app/vmalert/web.qtpl:477
 	qw422016.E().S(state)
-//line app/vmalert/web.qtpl:364
+//line app/vmalert/web.qtpl:477
 	qw422016.N().S(`</span>
 `)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 }
 
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 func writebadgeState(qq422016 qtio422016.Writer, state string) {
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	streambadgeState(qw422016, state)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 }
 
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 func badgeState(state string) string {
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	writebadgeState(qb422016, state)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 	return qs422016
-//line app/vmalert/web.qtpl:365
+//line app/vmalert/web.qtpl:478
 }
 
-//line app/vmalert/web.qtpl:367
+//line app/vmalert/web.qtpl:480
 func streambadgeRestored(qw422016 *qt422016.Writer) {
-//line app/vmalert/web.qtpl:367
+//line app/vmalert/web.qtpl:480
 	qw422016.N().S(`
 <span class="badge bg-warning text-dark" title="Alert state was restored after the service restart from remote storage">restored</span>
 `)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 }
 
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 func writebadgeRestored(qq422016 qtio422016.Writer) {
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	qw422016 := qt422016.AcquireWriter(qq422016)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	streambadgeRestored(qw422016)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	qt422016.ReleaseWriter(qw422016)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 }
 
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 func badgeRestored() string {
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	qb422016 := qt422016.AcquireByteBuffer()
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	writebadgeRestored(qb422016)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	qs422016 := string(qb422016.B)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	qt422016.ReleaseByteBuffer(qb422016)
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 	return qs422016
-//line app/vmalert/web.qtpl:369
+//line app/vmalert/web.qtpl:482
 }
diff --git a/app/vmalert/web_types.go b/app/vmalert/web_types.go
index 27789c897..c61aacdbe 100644
--- a/app/vmalert/web_types.go
+++ b/app/vmalert/web_types.go
@@ -5,11 +5,11 @@ import (
 	"time"
 )
 
-// APIAlert represents a notifier.AlertingRule state
+// APIAlert represents a notifier.AlertingRule ruleState
 // for WEB view
 // https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIAlert struct {
-	State       string            `json:"state"`
+	State       string            `json:"ruleState"`
 	Name        string            `json:"name"`
 	Value       string            `json:"value"`
 	Labels      map[string]string `json:"labels,omitempty"`
@@ -30,7 +30,7 @@ type APIAlert struct {
 	// SourceLink contains a link to a system which should show
 	// why Alert was generated
 	SourceLink string `json:"source"`
-	// Restored shows whether Alert's state was restored on restart
+	// Restored shows whether Alert's ruleState was restored on restart
 	Restored bool `json:"restored"`
 }
 
@@ -86,10 +86,10 @@ type GroupAlerts struct {
 // see https://github.com/prometheus/compliance/blob/main/alert_generator/specification.md#get-apiv1rules
 type APIRule struct {
 	// State must be one of these under following scenarios
-	//  "pending": at least 1 alert in the rule in pending state and no other alert in firing state.
-	//  "firing": at least 1 alert in the rule in firing state.
-	//  "inactive": no alert in the rule in firing or pending state.
-	State string `json:"state"`
+	//  "pending": at least 1 alert in the rule in pending ruleState and no other alert in firing ruleState.
+	//  "firing": at least 1 alert in the rule in firing ruleState.
+	//  "inactive": no alert in the rule in firing or pending ruleState.
+	State string `json:"ruleState"`
 	Name  string `json:"name"`
 	// Query represents Rule's `expression` field
 	Query string `json:"query"`
@@ -116,8 +116,17 @@ type APIRule struct {
 	// Type of the rule: recording or alerting
 	DatasourceType string `json:"datasourceType"`
 	LastSamples    int    `json:"lastSamples"`
-	// ID is an unique Alert's ID within a group
+	// ID is a unique Alert's ID within a group
 	ID string `json:"id"`
 	// GroupID is an unique Group's ID
 	GroupID string `json:"group_id"`
+
+	// TODO:
+	Updates []ruleStateEntry `json:"updates"`
+}
+
+// WebLink returns a link to the alert which can be used in UI.
+func (ar APIRule) WebLink() string {
+	return fmt.Sprintf("rule?%s=%s&%s=%s",
+		paramGroupID, ar.GroupID, paramRuleID, ar.ID)
 }
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 6ea658710..66d515108 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -24,6 +24,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `top queries` tab, which shows various stats for recently executed queries. See [these docs](https://docs.victoriametrics.com/#top-queries) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2707).
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `debug` mode to the alerting rule settings for printing additional information into logs during evaluation. See `debug` param in [alerting rule config](https://docs.victoriametrics.com/vmalert.html#alerting-rules).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): minimize the time needed for reading large responses from scrape targets in [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode). This should reduce scrape durations for such targets as [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) running in a big Kubernetes cluster.
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add experimental feature for displaying last 10 states of the rule (recording or alerting) evaluation. The state is available on the Rule page, which can be opened by clicking on `Details` link next to Rule's name on the `/groups` page.
 
 * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly calculate `rate_over_sum(m[d])` as `sum_over_time(m[d])/d`. Previously the `sum_over_time(m[d])` could be improperly divided by smaller than `d` time range. See [rate_over_sum() docs](https://docs.victoriametrics.com/MetricsQL.html#rate_over_sum) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3045).
 * BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): properly calculate query results at `vmselect`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3067). The issue has been introduced in [v1.81.0](https://docs.victoriametrics.com/CHANGELOG.html#v1810).
diff --git a/docs/vmalert.md b/docs/vmalert.md
index b11855f1d..6603209e9 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -515,6 +515,7 @@ or time series modification via [relabeling](https://docs.victoriametrics.com/vm
 * `http://<vmalert-addr>/vmalert/api/v1/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in JSON format.
   Used as alert source in AlertManager.
 * `http://<vmalert-addr>/vmalert/alert?group_id=<group_id>&alert_id=<alert_id>` - get alert status in web UI.
+* `http://<vmalert-addr>/vmalert/rule?group_id=<group_id>&rule_id=<rule_id>` - get rule status in web UI.
 * `http://<vmalert-addr>/metrics` - application metrics.
 * `http://<vmalert-addr>/-/reload` - hot configuration reload.
 


Test patch:
diff --git a/app/vmalert/alerting_test.go b/app/vmalert/alerting_test.go
index 2e4c231ed..a222b824e 100644
--- a/app/vmalert/alerting_test.go
+++ b/app/vmalert/alerting_test.go
@@ -735,6 +735,7 @@ func TestAlertingRule_Template(t *testing.T) {
 					"description": `{{ $labels.alertname}}: It is {{ $value }} connections for "{{ $labels.instance }}"`,
 				},
 				alerts: make(map[uint64]*notifier.Alert),
+				state:  newRuleState(),
 			},
 			[]datasource.Metric{
 				metricWithValueAndLabels(t, 2, "__name__", "first", "instance", "foo", alertNameLabel, "override"),
@@ -774,6 +775,7 @@ func TestAlertingRule_Template(t *testing.T) {
 					"summary": `Alert "{{ $labels.alertname }}({{ $labels.alertgroup }})" for instance {{ $labels.instance }}`,
 				},
 				alerts: make(map[uint64]*notifier.Alert),
+				state:  newRuleState(),
 			},
 			[]datasource.Metric{
 				metricWithValueAndLabels(t, 1,
@@ -915,5 +917,11 @@ func newTestRuleWithLabels(name string, labels ...string) *AlertingRule {
 }
 
 func newTestAlertingRule(name string, waitFor time.Duration) *AlertingRule {
-	return &AlertingRule{Name: name, alerts: make(map[uint64]*notifier.Alert), For: waitFor, EvalInterval: waitFor}
+	return &AlertingRule{
+		Name:         name,
+		For:          waitFor,
+		EvalInterval: waitFor,
+		alerts:       make(map[uint64]*notifier.Alert),
+		state:        newRuleState(),
+	}
 }
diff --git a/app/vmalert/recording_test.go b/app/vmalert/recording_test.go
index 5594c11ec..a03f356ca 100644
--- a/app/vmalert/recording_test.go
+++ b/app/vmalert/recording_test.go
@@ -19,7 +19,7 @@ func TestRecordingRule_Exec(t *testing.T) {
 		expTS   []prompbmarshal.TimeSeries
 	}{
 		{
-			&RecordingRule{Name: "foo"},
+			&RecordingRule{Name: "foo", state: newRuleState()},
 			[]datasource.Metric{metricWithValueAndLabels(t, 10,
 				"__name__", "bar",
 			)},
@@ -30,7 +30,7 @@ func TestRecordingRule_Exec(t *testing.T) {
 			},
 		},
 		{
-			&RecordingRule{Name: "foobarbaz"},
+			&RecordingRule{Name: "foobarbaz", state: newRuleState()},
 			[]datasource.Metric{
 				metricWithValueAndLabels(t, 1, "__name__", "foo", "job", "foo"),
 				metricWithValueAndLabels(t, 2, "__name__", "bar", "job", "bar"),
@@ -52,9 +52,12 @@ func TestRecordingRule_Exec(t *testing.T) {
 			},
 		},
 		{
-			&RecordingRule{Name: "job:foo", Labels: map[string]string{
-				"source": "test",
-			}},
+			&RecordingRule{
+				Name:  "job:foo",
+				state: newRuleState(),
+				Labels: map[string]string{
+					"source": "test",
+				}},
 			[]datasource.Metric{
 				metricWithValueAndLabels(t, 2, "__name__", "foo", "job", "foo"),
 				metricWithValueAndLabels(t, 1, "__name__", "bar", "job", "bar")},
@@ -195,7 +198,7 @@ func TestRecordingRuleLimit(t *testing.T) {
 		metricWithValuesAndLabels(t, []float64{2, 3}, "__name__", "bar", "job", "bar"),
 		metricWithValuesAndLabels(t, []float64{4, 5, 6}, "__name__", "baz", "job", "baz"),
 	}
-	rule := &RecordingRule{Name: "job:foo", Labels: map[string]string{
+	rule := &RecordingRule{Name: "job:foo", state: newRuleState(), Labels: map[string]string{
 		"source": "test_limit",
 	}}
 	var err error
@@ -211,9 +214,13 @@ func TestRecordingRuleLimit(t *testing.T) {
 }
 
 func TestRecordingRule_ExecNegative(t *testing.T) {
-	rr := &RecordingRule{Name: "job:foo", Labels: map[string]string{
-		"job": "test",
-	}}
+	rr := &RecordingRule{
+		Name:  "job:foo",
+		state: newRuleState(),
+		Labels: map[string]string{
+			"job": "test",
+		},
+	}
 
 	fq := &fakeQuerier{}
 	expErr := "connection reset by peer"
diff --git a/app/vmalert/rule_test.go b/app/vmalert/rule_test.go
new file mode 100644
index 000000000..5af5f3c8c
--- /dev/null
+++ b/app/vmalert/rule_test.go
@@ -0,0 +1,81 @@
+package main
+
+import (
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestRule_state(t *testing.T) {
+	state := newRuleState()
+	e := state.getLast()
+	if !e.at.IsZero() {
+		t.Fatalf("expected entry to be zero")
+	}
+
+	now := time.Now()
+	state.add(ruleStateEntry{at: now})
+
+	e = state.getLast()
+	if e.at != now {
+		t.Fatalf("expected entry at %v to be equal to %v",
+			e.at, now)
+	}
+
+	time.Sleep(time.Millisecond)
+	now2 := time.Now()
+	state.add(ruleStateEntry{at: now2})
+
+	e = state.getLast()
+	if e.at != now2 {
+		t.Fatalf("expected entry at %v to be equal to %v",
+			e.at, now2)
+	}
+
+	if len(state.getAll()) != 2 {
+		t.Fatalf("expected for state to have 2 entries only; got %d",
+			len(state.getAll()),
+		)
+	}
+
+	var last time.Time
+	for i := 0; i < defaultStateEntriesLimit*2; i++ {
+		last = time.Now()
+		state.add(ruleStateEntry{at: last})
+	}
+
+	e = state.getLast()
+	if e.at != last {
+		t.Fatalf("expected entry at %v to be equal to %v",
+			e.at, last)
+	}
+
+	if len(state.getAll()) != defaultStateEntriesLimit {
+		t.Fatalf("expected for state to have %d entries only; got %d",
+			defaultStateEntriesLimit, len(state.getAll()),
+		)
+	}
+}
+
+// TestRule_stateConcurrent supposed to test concurrent
+// execution of state updates.
+// Should be executed with -race flag
+func TestRule_stateConcurrent(t *testing.T) {
+	state := newRuleState()
+
+	const workers = 50
+	const iterations = 100
+	wg := sync.WaitGroup{}
+	wg.Add(workers)
+	for i := 0; i < workers; i++ {
+		go func() {
+			defer wg.Done()
+			for i := 0; i < iterations; i++ {
+				state.add(ruleStateEntry{at: time.Now()})
+				state.getAll()
+				state.getLast()
+			}
+		}()
+	}
+	wg.Wait()
+}
diff --git a/app/vmalert/web_test.go b/app/vmalert/web_test.go
index 6271c2a64..52bd91f63 100644
--- a/app/vmalert/web_test.go
+++ b/app/vmalert/web_test.go
@@ -17,6 +17,7 @@ func TestHandler(t *testing.T) {
 		alerts: map[uint64]*notifier.Alert{
 			0: {State: notifier.StateFiring},
 		},
+		state: newRuleState(),
 	}
 	g := &Group{
 		Name:  "group",
@@ -52,6 +53,22 @@ func TestHandler(t *testing.T) {
 	t.Run("/", func(t *testing.T) {
 		getResp(ts.URL, nil, 200)
 		getResp(ts.URL+"/vmalert", nil, 200)
+		getResp(ts.URL+"/vmalert/alerts", nil, 200)
+		getResp(ts.URL+"/vmalert/groups", nil, 200)
+		getResp(ts.URL+"/vmalert/notifiers", nil, 200)
+		getResp(ts.URL+"/rules", nil, 200)
+	})
+
+	t.Run("/vmalert/rule", func(t *testing.T) {
+		a := ar.ToAPI()
+		getResp(ts.URL+"/vmalert/"+a.WebLink(), nil, 200)
+	})
+	t.Run("/vmalert/rule?badParam", func(t *testing.T) {
+		params := fmt.Sprintf("?%s=0&%s=1", paramGroupID, paramRuleID)
+		getResp(ts.URL+"/vmalert/rule"+params, nil, 404)
+
+		params = fmt.Sprintf("?%s=1&%s=0", paramGroupID, paramRuleID)
+		getResp(ts.URL+"/vmalert/rule"+params, nil, 404)
 	})
 
 	t.Run("/api/v1/alerts", func(t *testing.T) {