Repository: VictoriaMetrics/VictoriaMetrics
Pull request #2195 (2022-02-15 13:59:59)

vmalert: support `$externalLabels` and `$externalURL` in templates
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2193
Signed-off-by: hagen1778 <roman@victoriametrics.com>

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
> Merging [#2195](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (74b1b1d) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/0d3e00e512d809d34d24403a6dff014ebaa6e1e5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (0d3e00e) will **decrease** coverage by `0.13%`.
> The diff coverage is `37.50%`.

> :exclamation: Current head 74b1b1d differs from pull request most recent head 1a070c9. Consider uploading reports for the commit 1a070c9 to get more accurate results

[![Impacted file tree graph](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/graphs/tree.svg?width=650&height=150&src=pr&token=YND8dYVcXn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

```diff
@@            Coverage Diff             @@
##           master    #2195      +/-   ##
==========================================
- Coverage   60.67%   60.54%   -0.14%     
==========================================
  Files         240      240              
  Lines       40665    40689      +24     
==========================================
- Hits        24674    24635      -39     
- Misses      14351    14416      +65     
+ Partials     1640     1638       -2     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `35.04% <0.00%> (-0.17%)` | :arrow_down: |
| [app/vmalert/notifier/init.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbm90aWZpZXIvaW5pdC5nbw==) | `9.58% <57.14%> (+9.58%)` | :arrow_up: |
| [app/vmalert/notifier/alert.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbm90aWZpZXIvYWxlcnQuZ28=) | `46.80% <83.33%> (+2.36%)` | :arrow_up: |
| [lib/mergeset/block\_stream\_reader.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L2Jsb2NrX3N0cmVhbV9yZWFkZXIuZ28=) | `50.24% <0.00%> (-11.95%)` | :arrow_down: |
| [app/vmalert/config/config.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvY29uZmlnL2NvbmZpZy5nbw==) | `75.88% <0.00%> (-3.53%)` | :arrow_down: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `65.68% <0.00%> (-2.35%)` | :arrow_down: |
| [lib/promscrape/targetstatus.qtpl.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb21zY3JhcGUvdGFyZ2V0c3RhdHVzLnF0cGwuZ28=) | `0.00% <0.00%> (ø)` | |
| [lib/storage/storage.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2Uvc3RvcmFnZS5nbw==) | `67.58% <0.00%> (+0.05%)` | :arrow_up: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `87.65% <0.00%> (+1.20%)` | :arrow_up: |
| [lib/bloomfilter/filter.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2Jsb29tZmlsdGVyL2ZpbHRlci5nbw==) | `100.00% <0.00%> (+2.00%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Last update [0d3e00e...1a070c9](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/2195?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/app/vmalert/main.go b/app/vmalert/main.go
index 16cd5e39b..0d23d56b6 100644
--- a/app/vmalert/main.go
+++ b/app/vmalert/main.go
@@ -167,7 +167,20 @@ func newManager(ctx context.Context) (*manager, error) {
 	if err != nil {
 		return nil, fmt.Errorf("failed to init datasource: %w", err)
 	}
-	nts, err := notifier.Init(alertURLGeneratorFn)
+
+	labels := make(map[string]string, 0)
+	for _, s := range *externalLabels {
+		if len(s) == 0 {
+			continue
+		}
+		n := strings.IndexByte(s, '=')
+		if n < 0 {
+			return nil, fmt.Errorf("missing '=' in `-label`. It must contain label in the form `Name=value`; got %q", s)
+		}
+		labels[s[:n]] = s[n+1:]
+	}
+
+	nts, err := notifier.Init(alertURLGeneratorFn, labels, *externalURL)
 	if err != nil {
 		return nil, fmt.Errorf("failed to init notifier: %w", err)
 	}
@@ -175,7 +188,7 @@ func newManager(ctx context.Context) (*manager, error) {
 		groups:         make(map[uint64]*Group),
 		querierBuilder: q,
 		notifiers:      nts,
-		labels:         map[string]string{},
+		labels:         labels,
 	}
 	rw, err := remotewrite.Init(ctx)
 	if err != nil {
@@ -189,16 +202,6 @@ func newManager(ctx context.Context) (*manager, error) {
 	}
 	manager.rr = rr
 
-	for _, s := range *externalLabels {
-		if len(s) == 0 {
-			continue
-		}
-		n := strings.IndexByte(s, '=')
-		if n < 0 {
-			return nil, fmt.Errorf("missing '=' in `-label`. It must contain label in the form `Name=value`; got %q", s)
-		}
-		manager.labels[s[:n]] = s[n+1:]
-	}
 	return manager, nil
 }
 
diff --git a/app/vmalert/notifier/alert.go b/app/vmalert/notifier/alert.go
index c2889b94f..4aa661d24 100644
--- a/app/vmalert/notifier/alert.go
+++ b/app/vmalert/notifier/alert.go
@@ -70,7 +70,13 @@ type AlertTplData struct {
 	Expr   string
 }
 
-const tplHeader = `{{ $value := .Value }}{{ $labels := .Labels }}{{ $expr := .Expr }}`
+var tplHeaders = []string{
+	"{{ $value := .Value }}",
+	"{{ $labels := .Labels }}",
+	"{{ $expr := .Expr }}",
+	"{{ $externalLabels := .ExternalLabels }}",
+	"{{ $externalURL := .ExternalURL }}",
+}
 
 // ExecTemplate executes the Alert template for given
 // map of annotations.
@@ -100,13 +106,15 @@ func templateAnnotations(annotations map[string]string, data AlertTplData, funcs
 	var buf bytes.Buffer
 	eg := new(utils.ErrGroup)
 	r := make(map[string]string, len(annotations))
+	tData := tplData{data, externalLabels, externalURL}
+	header := strings.Join(tplHeaders, "")
 	for key, text := range annotations {
 		buf.Reset()
 		builder.Reset()
-		builder.Grow(len(tplHeader) + len(text))
-		builder.WriteString(tplHeader)
+		builder.Grow(len(header) + len(text))
+		builder.WriteString(header)
 		builder.WriteString(text)
-		if err := templateAnnotation(&buf, builder.String(), data, funcs); err != nil {
+		if err := templateAnnotation(&buf, builder.String(), tData, funcs); err != nil {
 			r[key] = text
 			eg.Add(fmt.Errorf("key %q, template %q: %w", key, text, err))
 			continue
@@ -116,7 +124,13 @@ func templateAnnotations(annotations map[string]string, data AlertTplData, funcs
 	return r, eg.Err()
 }
 
-func templateAnnotation(dst io.Writer, text string, data AlertTplData, funcs template.FuncMap) error {
+type tplData struct {
+	AlertTplData
+	ExternalLabels map[string]string
+	ExternalURL    string
+}
+
+func templateAnnotation(dst io.Writer, text string, data tplData, funcs template.FuncMap) error {
 	t := template.New("").Funcs(funcs).Option("missingkey=zero")
 	tpl, err := t.Parse(text)
 	if err != nil {
diff --git a/app/vmalert/notifier/init.go b/app/vmalert/notifier/init.go
index d550de2fc..784751d02 100644
--- a/app/vmalert/notifier/init.go
+++ b/app/vmalert/notifier/init.go
@@ -46,13 +46,29 @@ func Reload() error {
 
 var staticNotifiersFn func() []Notifier
 
+var (
+	// externalLabels is a global variable for holding external labels configured via flags
+	// It is supposed to be inited via Init function only.
+	externalLabels map[string]string
+	// externalURL is a global variable for holding external URL value configured via flag
+	// It is supposed to be inited via Init function only.
+	externalURL string
+)
+
 // Init returns a function for retrieving actual list of Notifier objects.
 // Init works in two mods:
 //   * configuration via flags (for backward compatibility). Is always static
 //     and don't support live reloads.
 //   * configuration via file. Supports live reloads and service discovery.
 // Init returns an error if both mods are used.
-func Init(gen AlertURLGenerator) (func() []Notifier, error) {
+func Init(gen AlertURLGenerator, extLabels map[string]string, extURL string) (func() []Notifier, error) {
+	if externalLabels != nil || externalURL != "" {
+		return nil, fmt.Errorf("BUG: notifier.Init was called multiple times")
+	}
+
+	externalURL = extURL
+	externalLabels = extLabels
+
 	if *configPath == "" && len(*addrs) == 0 {
 		return nil, nil
 	}
diff --git a/docs/url-examples.md b/docs/url-examples.md
new file mode 100644
index 000000000..79d7bbb7a
--- /dev/null
+++ b/docs/url-examples.md
@@ -0,0 +1,265 @@
+---
+sort: 20
+---
+
+# URL Examples
+
+
+
+## /api/v1/admin/tsdb/delete_series
+
+**Delete time series**
+ 
+Single:
+```bash
+curl 'http://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=vm_http_request_errors_total'
+```
+
+Cluster:
+```bash
+curl 'http://<vmselect>:8481/delete/0/prometheus/api/v1/admin/tsdb/delete_series?match[]=vm_http_request_errors_total'
+```
+
+Additional information:
+* [How to delete time series](https://docs.victoriametrics.com/#how-to-delete-time-series)
+
+
+
+## /api/v1/export/csv
+
+**Exports CSV data from VictoriaMetrics**
+ 
+Single:
+```bash
+curl 'http://<victoriametrics-addr>:8428/api/v1/export/csv?format=__name__,__value__,__timestamp__:unix_s&match=vm_http_request_errors_total' > filename.txt
+```
+ 
+Cluster:
+```bash
+curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/export/csv?format=__name__,__value__,__timestamp__:unix_s&match=vm_http_request_errors_total' > filename.txt
+```
+
+Additional information: 
+* [How to export time series](https://docs.victoriametrics.com/#how-to-export-csv-data)
+* [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
+
+If run from docker, <vminsert> address can be found via: “docker ps” command
+The assigned port might be different
+
+
+## /api/v1/export/native
+  
+**Exporting in native format**
+
+Single:
+```bash
+curl -G 'http://<victoriametrics-addr>:8428/api/v1/export/native?match[]=vm_http_request_errors_total' > filename.txt
+```
+
+Cluster:
+```bash
+curl -G 'http://<vmselect>:8481/select/0/prometheus/api/v1/export/native?match=vm_http_request_errors_total' > filename.txt
+```
+
+More information:
+* [How to export data in native format](https://docs.victoriametrics.com/#how-to-export-data-in-native-format) 
+
+
+
+## /api/v1/import/csv 
+
+**Imports CSV data to VictoriaMetrics**
+ 
+Single:
+```bash
+curl --data-binary "@import.txt" -X POST 'http://localhost:8428/api/v1/import/prometheus'
+curl -d "GOOG,1.23,4.56,NYSE" 'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
+```
+
+Cluster:
+```bash
+curl --data-binary "@import.txt" -X POST  'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/csv'
+curl -d "GOOG,1.23,4.56,NYSE" 'http://<vminsert>:8480/insert/0/prometheus/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'
+```
+
+Additional information: 
+* [URL format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format) 
+* [How to import CSV data](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-csv-data) 
+
+
+
+## /datadog/api/v1/series
+
+**Sends data from DataDog agent to VM**
+ 
+Single:
+```bash
+echo '
+{
+  "series": [
+    {
+      "host": "test.example.com",
+      "interval": 20,
+      "metric": "system.load.1",
+      "points": [[
+        0,
+        0.5
+      ]],
+      "tags": [
+        "environment:test"
+      ],
+      "type": "rate"
+    }
+  ]
+}
+' | curl -X POST --data-binary @- http://localhost:8428/datadog/api/v1/series
+```
+
+Cluster:
+```bash
+echo '
+{
+  "series": [
+    {
+      "host": "test.example.com",
+      "interval": 20,
+      "metric": "system.load.1",
+      "points": [[
+        0,
+        0.5
+      ]],
+      "tags": [
+        "environment:test"
+      ],
+      "type": "rate"
+    }
+  ]
+}
+' | curl -X POST --data-binary @- 'http://<vminsert>:8480/insert/0/datadog/api/v1/series'
+```
+
+Additional information:
+* [How to send data from datadog agent](https://docs.victoriametrics.com/?highlight=post#how-to-send-data-from-datadog-agent) 
+
+
+
+## /graphite/metrics/find
+
+**Searches Graphite metrics**
+
+Single:
+```bash
+curl -G 'http://localhost:8428/graphite/metrics/find?query=vm_http_request_errors_total'
+```
+ 
+Cluster:
+```bash
+curl -G 'http://<vmselect>:8481/select/0/graphite/metrics/find?query=vm_http_request_errors_total'
+```
+ 
+Additional information:
+* [Metrics find](https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find)
+* [How to send data from graphite compatible agents such as statsd](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
+* [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html?highlight=url%20format#url-format)
+
+
+
+## /influx/write
+
+**Writes data with InfluxDB line protocol to local VictoriaMetrics**
+
+Single:
+```bash
+curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://localhost:8428/write'
+```
+ 
+Cluster:
+```bash
+curl -d 'measurement,tag1=value1,tag2=value2 field1=123,field2=1.23' -X POST 'http://<vminsert>:8480/insert/0/influx/write'
+```
+ 
+Additional information:
+* [How to send data from influxdb compatible agents such as telegraf](https://docs.victoriametrics.com/?highlight=post#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) 
+
+
+
+
+## /prometheus/api/v1/import
+
+**Importing data obtained via api/v1/export at vmselect**
+
+Single:
+```bash
+curl --data-binary "@import.txt" -X POST 'http://destination-victoriametrics:8428/api/v1/import'
+```
+ 
+Cluster:
+```bash
+curl --data-binary "@import.txt" -X POST 'http://<vminsert>:8480/insert/prometheus/api/v1/import'
+```
+ 
+Additional information:
+* [How to import time series data](https://docs.victoriametrics.com/?highlight=echo#how-to-import-time-series-data)
+
+
+
+## TCP and UDP
+
+**Sends data from OpenTSDB-compatible agents**
+
+Turned off by default.  
+Enable OpenTSDB receiver in VictoriaMetrics by setting -opentsdbListenAddr command line flag.
+*If run from docker, '-opentsdbListenAddr' port should be exposed*
+
+Single:
+```bash
+echo "put foo.bar.baz `date +%s` 123 tag1=value1 tag2=value2" | nc -N localhost 4242
+```
+ 
+Cluster:
+```bash
+echo "put foo.bar.baz `date +%s` 123  tag1=value1 tag2=value2 VictoriaMetrics_AccountID=0" | nc -N http://<vminsert> 4242
+```
+ 
+
+**Enable HTTP server for OpenTSDB /api/put requests by setting -opentsdbHTTPListenAddr**
+ 
+Single:
+```bash
+curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]' http://localhost:4242/api/put
+```
+ 
+Cluster:
+```bash
+curl -H 'Content-Type: application/json' -d '[{"metric":"foo","value":45.34},{"metric":"bar","value":43}]'
+ 'http://<vminsert>:8480/insert/42/opentsdb/api/put'
+```
+ 
+Additional information:
+* [Api http put](http://opentsdb.net/docs/build/html/api_http/put.html)
+* [How to send data from opentsdb compatible agents](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-opentsdb-compatible-agents)
+
+
+
+**Writes data with Graphite plaintext protocol to local VictoriaMetrics using nc**
+
+Enable Graphite receiver in VictoriaMetrics by setting -graphiteListenAddr command line flag
+ 
+Single:
+```bash
+echo "foo.bar.baz;tag1=value1;tag2=value2 123 `date +%s`" |
+ nc -N localhost 2003
+```
+ 
+Cluster:
+```bash
+echo "foo.bar.baz;tag1=value1;tag2=value2;VictoriaMetrics_AccountID=42 123 `date +%s`" | nc -N http://<vminsert> 2003
+```
+
+Additional information:
+
+VictoriaMetrics_AccountID=42 - tag that indicated tennant ID.
+* [Request handler](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/a3eafd2e7fc75776dfc19d3c68c85589454d9dce/app/vminsert/opentsdb/request_handler.go#L47)
+* [How to send data from graphite compatible agents such as statsd](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
+
+


Test patch:
diff --git a/app/vmalert/config/testdata/rules2-good.rules b/app/vmalert/config/testdata/rules2-good.rules
index c307375ac..1f431b32f 100644
--- a/app/vmalert/config/testdata/rules2-good.rules
+++ b/app/vmalert/config/testdata/rules2-good.rules
@@ -25,6 +25,7 @@ groups:
           dynamic: '{{ $x := query "up" | first | value }}{{ if eq 1.0 $x }}one{{ else }}unknown{{ end }}'
         annotations:
           description: Job {{ $labels.job }} is up!
+          external: cluster-{{ $externalLabels.cluster }}; replica-{{ $externalLabels.replica }}
           summary: All instances up {{ range query "up" }}
             {{ . | label "instance" }}
             {{ end }}
diff --git a/app/vmalert/notifier/alert_test.go b/app/vmalert/notifier/alert_test.go
index 769116979..f8e0c77ee 100644
--- a/app/vmalert/notifier/alert_test.go
+++ b/app/vmalert/notifier/alert_test.go
@@ -1,12 +1,24 @@
 package notifier
 
 import (
+	"fmt"
 	"testing"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
 )
 
 func TestAlert_ExecTemplate(t *testing.T) {
+	extLabels := make(map[string]string, 0)
+	const (
+		extCluster = "prod"
+		extDC      = "east"
+		extURL     = "https://foo.bar"
+	)
+	extLabels["cluster"] = extCluster
+	extLabels["dc"] = extDC
+	_, err := Init(nil, extLabels, extURL)
+	checkErr(t, err)
+
 	testCases := []struct {
 		name        string
 		alert       *Alert
@@ -74,6 +86,26 @@ func TestAlert_ExecTemplate(t *testing.T) {
 				"desc":    "bar 1;garply 2;",
 			},
 		},
+		{
+			name: "external",
+			alert: &Alert{
+				Value: 1e4,
+				Labels: map[string]string{
+					"job":      "staging",
+					"instance": "localhost",
+				},
+			},
+			annotations: map[string]string{
+				"url":         "{{ $externalURL }}",
+				"summary":     "Issues with {{$labels.instance}} (dc-{{$externalLabels.dc}}) for job {{$labels.job}}",
+				"description": "It is {{ $value }} connections for {{$labels.instance}} (cluster-{{$externalLabels.cluster}})",
+			},
+			expTpl: map[string]string{
+				"url":         extURL,
+				"summary":     fmt.Sprintf("Issues with localhost (dc-%s) for job staging", extDC),
+				"description": fmt.Sprintf("It is 10000 connections for localhost (cluster-%s)", extCluster),
+			},
+		},
 	}
 
 	qFn := func(q string) ([]datasource.Metric, error) {