Repository: VictoriaMetrics/VictoriaMetrics
Pull request #5775 (2024-02-07 10:17:48)

added ddsketch permalink


Patch:
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 3439d4182..8e583210f 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -31,6 +31,7 @@ The sandbox cluster installation is running under the constant load generated by
 * SECURITY: upgrade Go builder from Go1.21.6 to Go1.21.7. See [the list of issues addressed in Go1.21.7](https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved).
 
 * FEATURE: all VictoriaMetrics components: add support for TLS client certificate verification at `-httpListenAddr` (aka [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication)). See [these docs](https://docs.victoriametrics.com/#mtls-protection). See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5458).
+* FEATURE: all VictoriaMetrics components: add support for empty command flag values in short array notation. For example, `-remoteWrite.sendTimeout=',20s,'` specifies three `-remoteWrite.sendTimeout` values - the first and the last ones are default values (`30s` in this case), while the second one is `20s`.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html) and [single-node VictoriaMetrics](https://docs.victoriametrics.com): add support for data ingestion via [DataDog lambda extension](https://docs.datadoghq.com/serverless/libraries_integrations/extension/) aka `/api/beta/sketches` endpoint. See [these docs](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3091). Thanks to @AndrewChubatiuk for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5584).
 * FEATURE: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): add `-disableReroutingOnUnavailable` command-line flag, which can be used for reducing resource usage spikes at `vmstorage` nodes during rolling restart. Thanks to @Muxa1L for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5713).
 * FEATURE: [dashboards/vmagent](https://grafana.com/grafana/dashboards/12683): add `Targets scraped/s` stat panel showing the number of targets scraped by the vmagent per-second.
diff --git a/docs/relabeling.md b/docs/relabeling.md
index 112501bf6..7e4034ada 100644
--- a/docs/relabeling.md
+++ b/docs/relabeling.md
@@ -28,6 +28,9 @@ The relabeling is mostly used for the following tasks:
 * Removing prefixes from target label names. See [how to remove prefixes from target label names](#how-to-remove-prefixes-from-target-label-names).
 * Removing some labels from discovered targets. See [how to remove labels from targets](#how-to-remove-labels-from-targets).
 * Dropping some metrics during scape. See [how to drop metrics during scrape](#how-to-drop-metrics-during-scrape).
+* Renaming scraped metrics. See [how to rename scraped metrics](#how-to-rename-scraped-metrics).
+* Adding labels to scraped metrics. See [how to add labels to scraped metrics](#how-to-add-labels-to-scraped-metrics).
+* Changing label values in scraped metrics. See [how to change label values in scraped metrics](#how-to-change-label-values-in-scraped-metrics).
 * Removing some labels from scraped metrics. See [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics).
 * Removing some labels from metrics matching some [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors).
   See [how to remove labels from metrics subset](#how-to-remove-labels-from-metrics-subset).
@@ -54,7 +57,125 @@ scrape_configs:
     regex: "foo_.*"
 ```
 
-See also [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics).
+See also:
+
+- [how to remove labels from scraped metrics](#how-to-remove-labels-from-scraped-metrics)
+- [useful tips for metric relabeling](#useful-tips-for-metric-relabeling)
+
+
+## How to rename scraped metrics
+
+Metric name is a regular label with special name - `__name__` (see [these docs](https://docs.victoriametrics.com/keyconcepts/#labels)).
+So renaming of metric name is performed in the same way as changing label value.
+
+Let's look at a few examples.
+
+The following config renames `foo` metric to `bar` across all the scraped metrics, while leaving other metric names as is:
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - if: 'foo'
+    replacement: bar
+    target_label: __name__
+```
+
+The following config renames metrics starting from `foo_` to metrics starting from `bar_` across all the scraped metrics. For example, `foo_count` is renamed to `bar_count`:
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - source_labels: [__name__]
+    regex: 'foo_(.*)'
+    replacement: bar_$1
+    target_label: __name__
+```
+
+The following config replaces all the `-` chars in metric names with `_` chars across all the scraped metrics. For example, `foo-bar-baz` is renamed to `foo_bar_baz`:
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - source_labels: [__name__]
+    action: replace_all
+    regex: '-'
+    replacement: '_'
+    target_label: __name__
+```
+
+See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
+
+
+## How to add labels to scraped metrics
+
+The following config sets `foo="bar"` [label](https://docs.victoriametrics.com/keyconcepts/#labels) across all the scraped metrics:
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - target_label: foo
+    replacement: bar
+```
+
+The following config sets `foo="bar"` label only for metrics matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - if: '{job=~"my-app-.*",env!="dev"}'
+    target_label: foo
+    replacement: bar
+```
+
+See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
+
+
+## How to change label values in scraped metrics
+
+The following config adds `foo_` prefix to all the values of `job` label across all the scraped metrics:
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - source_labels: [job]
+    target_label: job
+    replacement: foo_$1
+```
+
+The following config adds `foo_` prefix to `job` label values only for metrics
+matching `{job=~"my-app-.*",env!="dev"}` [series selector](https://docs.victoriametrics.com/keyconcepts/#filtering):
+
+```yaml
+scrape_configs:
+- job_name: test
+  static_configs:
+  - targets: [host123]
+  metric_relabel_configs:
+  - if: '{job=~"my-app-.*",env!="dev"}'
+    source_labels: [job]
+    target_label: job
+    replacement: foo_$1
+```
+
+See also [useful tips for metric relabeling](#useful-tips-for-metric-relabeling).
 
 
 ## How to remove labels from scraped metrics
diff --git a/lib/flagutil/array.go b/lib/flagutil/array.go
index 5ab4524a4..324596aa1 100644
--- a/lib/flagutil/array.go
+++ b/lib/flagutil/array.go
@@ -21,6 +21,7 @@ func NewArrayString(name, description string) *ArrayString {
 func NewArrayDuration(name string, defaultValue time.Duration, description string) *ArrayDuration {
 	description += fmt.Sprintf(" (default %s)", defaultValue)
 	description += "\nSupports `array` of values separated by comma or specified via multiple flags."
+	description += "\nEmpty values are set to default value."
 	a := &ArrayDuration{
 		defaultValue: defaultValue,
 	}
@@ -31,6 +32,7 @@ func NewArrayDuration(name string, defaultValue time.Duration, description strin
 // NewArrayBool returns new ArrayBool with the given name and description.
 func NewArrayBool(name, description string) *ArrayBool {
 	description += "\nSupports `array` of values separated by comma or specified via multiple flags."
+	description += "\nEmpty values are set to false."
 	var a ArrayBool
 	flag.Var(&a, name, description)
 	return &a
@@ -40,6 +42,7 @@ func NewArrayBool(name, description string) *ArrayBool {
 func NewArrayInt(name string, defaultValue int, description string) *ArrayInt {
 	description += fmt.Sprintf(" (default %d)", defaultValue)
 	description += "\nSupports `array` of values separated by comma or specified via multiple flags."
+	description += "\nEmpty values are set to default value."
 	a := &ArrayInt{
 		defaultValue: defaultValue,
 	}
@@ -52,6 +55,7 @@ func NewArrayBytes(name string, defaultValue int64, description string) *ArrayBy
 	description += "\nSupports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB."
 	description += fmt.Sprintf(" (default %d)", defaultValue)
 	description += "\nSupports `array` of values separated by comma or specified via multiple flags."
+	description += "\nEmpty values are set to default value."
 	a := &ArrayBytes{
 		defaultValue: defaultValue,
 	}
@@ -241,6 +245,9 @@ func (a *ArrayBool) String() string {
 func (a *ArrayBool) Set(value string) error {
 	values := parseArrayValues(value)
 	for _, v := range values {
+		if v == "" {
+			v = "false"
+		}
 		b, err := strconv.ParseBool(v)
 		if err != nil {
 			return err
@@ -284,6 +291,9 @@ func (a *ArrayDuration) String() string {
 func (a *ArrayDuration) Set(value string) error {
 	values := parseArrayValues(value)
 	for _, v := range values {
+		if v == "" {
+			v = a.defaultValue.String()
+		}
 		b, err := time.ParseDuration(v)
 		if err != nil {
 			return err
@@ -332,6 +342,9 @@ func (a *ArrayInt) String() string {
 func (a *ArrayInt) Set(value string) error {
 	values := parseArrayValues(value)
 	for _, v := range values {
+		if v == "" {
+			v = fmt.Sprintf("%d", a.defaultValue)
+		}
 		n, err := strconv.Atoi(v)
 		if err != nil {
 			return err
@@ -376,6 +389,9 @@ func (a *ArrayBytes) Set(value string) error {
 	values := parseArrayValues(value)
 	for _, v := range values {
 		var b Bytes
+		if v == "" {
+			v = fmt.Sprintf("%d", a.defaultValue)
+		}
 		if err := b.Set(v); err != nil {
 			return err
 		}
diff --git a/lib/flagutil/bytes.go b/lib/flagutil/bytes.go
index 8b86c9c13..6f60a0aa0 100644
--- a/lib/flagutil/bytes.go
+++ b/lib/flagutil/bytes.go
@@ -47,6 +47,11 @@ func (b *Bytes) String() string {
 
 // Set implements flag.Value interface
 func (b *Bytes) Set(value string) error {
+	if value == "" {
+		b.N = 0
+		b.valueString = ""
+		return nil
+	}
 	value = normalizeBytesString(value)
 	switch {
 	case strings.HasSuffix(value, "KB"):
diff --git a/lib/flagutil/duration.go b/lib/flagutil/duration.go
index 4af032331..8044e2708 100644
--- a/lib/flagutil/duration.go
+++ b/lib/flagutil/duration.go
@@ -49,6 +49,11 @@ func (d *Duration) String() string {
 
 // Set implements flag.Value interface
 func (d *Duration) Set(value string) error {
+	if value == "" {
+		d.msecs = 0
+		d.valueString = ""
+		return nil
+	}
 	// An attempt to parse value in months.
 	months, err := strconv.ParseFloat(value, 64)
 	if err == nil {
diff --git a/lib/protoparser/datadogsketches/parser.go b/lib/protoparser/datadogsketches/parser.go
index 87609d1bb..c9242bc57 100644
--- a/lib/protoparser/datadogsketches/parser.go
+++ b/lib/protoparser/datadogsketches/parser.go
@@ -9,7 +9,7 @@ import (
 )
 
 var (
-	// TODO: @AndrewChubatiuk, please provide a permalink for the original source code where these constants were extracted
+	// https://github.com/DataDog/opentelemetry-mapping-go/blob/48d52eeea60d28da2e14c154a24557c4d290c6e2/pkg/quantile/config.go
 	epsillon   = 1.0 / 128
 	gamma      = 1 + 2*epsillon
 	gammaLn    = math.Log(gamma)
@@ -313,9 +313,6 @@ func (d *Dogsketch) valueForQuantile(q float64) float64 {
 		vLow := f64(ks[i])
 		vHigh := vLow * gamma
 		switch i {
-		// TODO: I'm unsure this code is correct. i cannot equal len(ns) in this loop.
-		// @AndrewChubatiuk, please add a permalink to the original source code, which was used
-		// for writing this code, in the comments to this function.
 		case len(ns):
 			vHigh = d.Max
 		case 0:
@@ -330,8 +327,6 @@ func f64(k int32) float64 {
 	switch {
 	case k < 0:
 		return -f64(-k)
-	// TODO: I'm unsure this logic is correct, since k can be smaller than math.MinInt16 and bigger than math.MaxInt16
-	// @AndrewChubatiuk, please add a permalink to the original source code, which was used for writing this code.
 	case k == math.MaxInt16 || k == math.MinInt16:
 		return math.Inf(int(k))
 	case k == 0:


Test patch:
diff --git a/lib/flagutil/array_test.go b/lib/flagutil/array_test.go
index ec4d87d76..6bb884fa6 100644
--- a/lib/flagutil/array_test.go
+++ b/lib/flagutil/array_test.go
@@ -165,6 +165,7 @@ func TestArrayDuration_Set(t *testing.T) {
 	f := func(s, expectedResult string) {
 		t.Helper()
 		var a ArrayDuration
+		a.defaultValue = 42 * time.Second
 		if err := a.Set(s); err != nil {
 			t.Fatalf("unexpected error: %s", err)
 		}
@@ -176,6 +177,7 @@ func TestArrayDuration_Set(t *testing.T) {
 	f("", "")
 	f(`1m`, `1m0s`)
 	f(`5m,1s,1h`, `5m0s,1s,1h0m0s`)
+	f(`5m,,1h`, `5m0s,42s,1h0m0s`)
 }
 
 func TestArrayDuration_GetOptionalArg(t *testing.T) {
@@ -238,6 +240,7 @@ func TestArrayBool_Set(t *testing.T) {
 	f("", "")
 	f(`true`, `true`)
 	f(`false,True,False`, `false,true,false`)
+	f(`1,,False`, `true,false,false`)
 }
 
 func TestArrayBool_GetOptionalArg(t *testing.T) {
@@ -289,6 +292,7 @@ func TestArrayInt_Set(t *testing.T) {
 	f := func(s, expectedResult string, expectedValues []int) {
 		t.Helper()
 		var a ArrayInt
+		a.defaultValue = 42
 		if err := a.Set(s); err != nil {
 			t.Fatalf("unexpected error: %s", err)
 		}
@@ -304,6 +308,7 @@ func TestArrayInt_Set(t *testing.T) {
 	f("", "", nil)
 	f(`1`, `1`, []int{1})
 	f(`-2,3,-64`, `-2,3,-64`, []int{-2, 3, -64})
+	f(`,,-64,`, `42,42,-64,42`, []int{42, 42, -64, 42})
 }
 
 func TestArrayInt_GetOptionalArg(t *testing.T) {
@@ -357,6 +362,7 @@ func TestArrayBytes_Set(t *testing.T) {
 	f := func(s, expectedResult string) {
 		t.Helper()
 		var a ArrayBytes
+		a.defaultValue = 42
 		if err := a.Set(s); err != nil {
 			t.Fatalf("unexpected error: %s", err)
 		}
@@ -368,6 +374,7 @@ func TestArrayBytes_Set(t *testing.T) {
 	f("", "")
 	f(`1`, `1`)
 	f(`-2,3,10kb`, `-2,3,10KB`)
+	f(`,,10kb`, `42,42,10KB`)
 }
 
 func TestArrayBytes_GetOptionalArg(t *testing.T) {
diff --git a/lib/flagutil/bytes_test.go b/lib/flagutil/bytes_test.go
index 4396b239b..584a9c8f4 100644
--- a/lib/flagutil/bytes_test.go
+++ b/lib/flagutil/bytes_test.go
@@ -12,7 +12,6 @@ func TestBytesSetFailure(t *testing.T) {
 			t.Fatalf("expecting non-nil error in b.Set(%q)", value)
 		}
 	}
-	f("")
 	f("foobar")
 	f("5foobar")
 	f("aKB")
@@ -39,6 +38,7 @@ func TestBytesSetSuccess(t *testing.T) {
 			t.Fatalf("unexpected valueString; got %q; want %q", valueString, valueExpected)
 		}
 	}
+	f("", 0)
 	f("0", 0)
 	f("1", 1)
 	f("-1234", -1234)
diff --git a/lib/flagutil/duration_test.go b/lib/flagutil/duration_test.go
index f9834c679..1e10dcfdc 100644
--- a/lib/flagutil/duration_test.go
+++ b/lib/flagutil/duration_test.go
@@ -14,7 +14,6 @@ func TestDurationSetFailure(t *testing.T) {
 			t.Fatalf("expecting non-nil error in d.Set(%q)", value)
 		}
 	}
-	f("")
 	f("foobar")
 	f("5foobar")
 	f("ah")
@@ -51,6 +50,7 @@ func TestDurationSetSuccess(t *testing.T) {
 			t.Fatalf("unexpected valueString; got %q; want %q", valueString, valueExpected)
 		}
 	}
+	f("", 0)
 	f("0", 0)
 	f("1", msecsPer31Days)
 	f("123.456", 123.456*msecsPer31Days)