Repository: VictoriaMetrics/VictoriaMetrics
Pull request #3163 (2022-09-26 09:57:16)

vmselect/rollup: rm workaround for slow-changing counters
The workaround was introduced to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962. However, it didn't prove itself useful. Instead, it is recommended using `increase_pure` function.

Removing the workaround makes VM to produce accurate results when calculating `delta` or `increase` functions over slow-changing counters with vary intervals between data points.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Base: **57.55**% // Head: **57.47**% // Decreases project coverage by **`-0.07%`** :warning:
> Coverage data is based on head [(`2ad7503`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) compared to base [(`819aa95`)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/819aa95552989c4638761953df06d636bb88d2da?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> Patch has no changes to coverable lines.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #3163      +/-   ##
==========================================
- Coverage   57.55%   57.47%   -0.08%     
==========================================
  Files         282      282              
  Lines       48026    48023       -3     
==========================================
- Hits        27639    27603      -36     
- Misses      18592    18629      +37     
+ Partials     1795     1791       -4     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmselect/promql/rollup.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9yb2xsdXAuZ28=) | `85.64% <ø> (-0.03%)` | :arrow_down: |
| [lib/mergeset/block\_stream\_reader.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L2Jsb2NrX3N0cmVhbV9yZWFkZXIuZ28=) | `54.67% <0.00%> (-11.22%)` | :arrow_down: |
| [lib/bloomfilter/filter.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2Jsb29tZmlsdGVyL2ZpbHRlci5nbw==) | `98.00% <0.00%> (-2.00%)` | :arrow_down: |
| [app/vmselect/promql/aggr\_incremental.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC9hZ2dyX2luY3JlbWVudGFsLmdv) | `85.24% <0.00%> (-1.21%)` | :arrow_down: |
| [lib/mergeset/table.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL21lcmdlc2V0L3RhYmxlLmdv) | `66.15% <0.00%> (-0.88%)` | :arrow_down: |
| [app/vmselect/promql/transform.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21xbC90cmFuc2Zvcm0uZ28=) | `88.07% <0.00%> (+0.16%)` | :arrow_up: |
| [app/vmalert/remotewrite/remotewrite.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvcmVtb3Rld3JpdGUuZ28=) | `66.44% <0.00%> (+1.34%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/3163?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/README.md b/README.md
index cd285f1cd..aa8f1487f 100644
--- a/README.md
+++ b/README.md
@@ -432,6 +432,10 @@ This command should return the following output if everything is OK:
 {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]}
 ```
 
+VictoriaMetrics automatically sanitizes metric names for the data ingested via DataDog protocol
+according to [DataDog metric naming recommendations](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics).
+If you need accepting metric names as is without sanitizing, then pass `-datadog.sanitizeMetricName=false` command-line flag to VictoriaMetrics.
+
 Extra labels may be added to all the written time series by passing `extra_label=name=value` query args.
 For example, `/datadog/api/v1/series?extra_label=foo=bar` would add `{foo="bar"}` label to all the ingested metrics.
 
@@ -465,6 +469,9 @@ VictoriaMetrics performs the following transformations to the ingested InfluxDB
 * Field names are mapped to time series names prefixed with `{measurement}{separator}` value, where `{separator}` equals to `_` by default. It can be changed with `-influxMeasurementFieldSeparator` command-line flag. See also `-influxSkipSingleField` command-line flag. If `{measurement}` is empty or if `-influxSkipMeasurement` command-line flag is set, then time series names correspond to field names.
 * Field values are mapped to time series values.
 * Tags are mapped to Prometheus labels as-is.
+* If `-usePromCompatibleNaming` command-line flag is set, then all the metric names and label names
+  are normalized to [Prometheus-compatible naming](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels) by replacing unsupported chars with `_`.
+  For example, `foo.bar-baz/1` metric name or label name is substituted with `foo_bar_baz_1`.
 
 For example, the following InfluxDB line:
 
@@ -1994,6 +2001,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -dedup.minScrapeInterval duration
      Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling
   -deleteAuthKey string
@@ -2333,6 +2342,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
   -vmalert.proxyURL string
diff --git a/app/vmagent/README.md b/app/vmagent/README.md
index 0db26b0d9..8b136e910 100644
--- a/app/vmagent/README.md
+++ b/app/vmagent/README.md
@@ -915,6 +915,8 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -denyQueryTracing
      Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing
   -dryRun
@@ -1267,6 +1269,8 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
 ```
diff --git a/app/vmagent/remotewrite/relabel.go b/app/vmagent/remotewrite/relabel.go
index 76e140532..abcdbb0de 100644
--- a/app/vmagent/remotewrite/relabel.go
+++ b/app/vmagent/remotewrite/relabel.go
@@ -3,6 +3,7 @@ package remotewrite
 import (
 	"flag"
 	"fmt"
+	"regexp"
 	"strings"
 	"sync"
 
@@ -25,6 +26,10 @@ var (
 	relabelDebug = flagutil.NewArrayBool("remoteWrite.urlRelabelDebug", "Whether to log metrics before and after relabeling with -remoteWrite.urlRelabelConfig. "+
 		"If the -remoteWrite.urlRelabelDebug is enabled, then the metrics aren't sent to the corresponding -remoteWrite.url. "+
 		"This is useful for debugging the relabeling configs")
+
+	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
+		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
+		"See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels")
 )
 
 var labelsGlobal []prompbmarshal.Label
@@ -107,6 +112,18 @@ func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, extraLab
 				labels = append(labels, *extraLabel)
 			}
 		}
+		if *usePromCompatibleNaming {
+			// Replace unsupported Prometheus chars in label names and metric names with underscores.
+			tmpLabels := labels[labelsLen:]
+			for j := range tmpLabels {
+				label := &tmpLabels[j]
+				if label.Name == "__name__" {
+					label.Value = unsupportedPromChars.ReplaceAllString(label.Value, "_")
+				} else {
+					label.Name = unsupportedPromChars.ReplaceAllString(label.Name, "_")
+				}
+			}
+		}
 		labels = pcs.Apply(labels, labelsLen, true)
 		if len(labels) == labelsLen {
 			// Drop the current time series, since relabeling removed all the labels.
@@ -121,6 +138,9 @@ func (rctx *relabelCtx) applyRelabeling(tss []prompbmarshal.TimeSeries, extraLab
 	return tssDst
 }
 
+// See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
+var unsupportedPromChars = regexp.MustCompile(`[^a-zA-Z0-9_:]`)
+
 type relabelCtx struct {
 	// pool for labels, which are used during the relabeling.
 	labels []prompbmarshal.Label
diff --git a/app/vminsert/relabel/relabel.go b/app/vminsert/relabel/relabel.go
index ac17d834a..bb02836c5 100644
--- a/app/vminsert/relabel/relabel.go
+++ b/app/vminsert/relabel/relabel.go
@@ -3,6 +3,7 @@ package relabel
 import (
 	"flag"
 	"fmt"
+	"regexp"
 	"sync/atomic"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
@@ -20,6 +21,10 @@ var (
 		"See https://docs.victoriametrics.com/#relabeling for details. The config is reloaded on SIGHUP signal")
 	relabelDebug = flag.Bool("relabelDebug", false, "Whether to log metrics before and after relabeling with -relabelConfig. If the -relabelDebug is enabled, "+
 		"then the metrics aren't sent to storage. This is useful for debugging the relabeling configs")
+
+	usePromCompatibleNaming = flag.Bool("usePromCompatibleNaming", false, "Whether to replace characters unsupported by Prometheus with underscores "+
+		"in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. "+
+		"See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels")
 )
 
 // Init must be called after flag.Parse and before using the relabel package.
@@ -67,7 +72,7 @@ func loadRelabelConfig() (*promrelabel.ParsedConfigs, error) {
 // HasRelabeling returns true if there is global relabeling.
 func HasRelabeling() bool {
 	pcs := pcsGlobal.Load().(*promrelabel.ParsedConfigs)
-	return pcs.Len() > 0
+	return pcs.Len() > 0 || *usePromCompatibleNaming
 }
 
 // Ctx holds relabeling context.
@@ -87,11 +92,11 @@ func (ctx *Ctx) Reset() {
 // The returned labels are valid until the next call to ApplyRelabeling.
 func (ctx *Ctx) ApplyRelabeling(labels []prompb.Label) []prompb.Label {
 	pcs := pcsGlobal.Load().(*promrelabel.ParsedConfigs)
-	if pcs.Len() == 0 {
+	if pcs.Len() == 0 && !*usePromCompatibleNaming {
 		// There are no relabeling rules.
 		return labels
 	}
-	// Convert src to prompbmarshal.Label format suitable for relabeling.
+	// Convert labels to prompbmarshal.Label format suitable for relabeling.
 	tmpLabels := ctx.tmpLabels[:0]
 	for _, label := range labels {
 		name := bytesutil.ToUnsafeString(label.Name)
@@ -105,13 +110,28 @@ func (ctx *Ctx) ApplyRelabeling(labels []prompb.Label) []prompb.Label {
 		})
 	}
 
-	// Apply relabeling
-	tmpLabels = pcs.Apply(tmpLabels, 0, true)
-	ctx.tmpLabels = tmpLabels
-	if len(tmpLabels) == 0 {
-		metricsDropped.Inc()
+	if *usePromCompatibleNaming {
+		// Replace unsupported Prometheus chars in label names and metric names with underscores.
+		for i := range tmpLabels {
+			label := &tmpLabels[i]
+			if label.Name == "__name__" {
+				label.Value = unsupportedPromChars.ReplaceAllString(label.Value, "_")
+			} else {
+				label.Name = unsupportedPromChars.ReplaceAllString(label.Name, "_")
+			}
+		}
+	}
+
+	if pcs.Len() > 0 {
+		// Apply relabeling
+		tmpLabels = pcs.Apply(tmpLabels, 0, true)
+		if len(tmpLabels) == 0 {
+			metricsDropped.Inc()
+		}
 	}
 
+	ctx.tmpLabels = tmpLabels
+
 	// Return back labels to the desired format.
 	dst := labels[:0]
 	for _, label := range tmpLabels {
@@ -129,3 +149,6 @@ func (ctx *Ctx) ApplyRelabeling(labels []prompb.Label) []prompb.Label {
 }
 
 var metricsDropped = metrics.NewCounter(`vm_relabel_metrics_dropped_total`)
+
+// See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
+var unsupportedPromChars = regexp.MustCompile(`[^a-zA-Z0-9_:]`)
diff --git a/app/vmselect/promql/rollup.go b/app/vmselect/promql/rollup.go
index be6ade111..1c4946e75 100644
--- a/app/vmselect/promql/rollup.go
+++ b/app/vmselect/promql/rollup.go
@@ -1495,11 +1495,8 @@ func rollupDelta(rfa *rollupFuncArg) float64 {
 			// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 			return values[len(values)-1] - rfa.realPrevValue
 		}
-		// Assume that the previous non-existing value was 0 only in the following cases:
-		//
-		// - If the delta with the next value equals to 0.
-		//   This is the case for slow-changing counter - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
-		// - If the first value doesn't exceed too much the delta with the next value.
+		// Assume that the previous non-existing value was 0
+		// only if the first value doesn't exceed too much the delta with the next value.
 		//
 		// This should prevent from improper increase() results for os-level counters
 		// such as cpu time or bytes sent over the network interface.
@@ -1513,9 +1510,6 @@ func rollupDelta(rfa *rollupFuncArg) float64 {
 		} else if !math.IsNaN(rfa.realNextValue) {
 			d = rfa.realNextValue - values[0]
 		}
-		if d == 0 {
-			d = 10
-		}
 		if math.Abs(values[0]) < 10*(math.Abs(d)+1) {
 			prevValue = 0
 		} else {
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 3df6c7f2f..883de8601 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -19,6 +19,9 @@ The following tip changes can be tested by building VictoriaMetrics components f
 
 **Update note 2:** [vmalert](https://docs.victoriametrics.com/vmalert.html) changes default value for command-line flag `-datasource.queryStep` from `0s` to `5m`. The change supposed to improve reliability of the rules evaluation when evaluation interval is lower than scraping interval.
 
+* FEATURE: sanitize metric names for data ingested via [DataDog protocol](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) according to [DataDog metric naming](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics). The behaviour can be disabled by passing `-datadog.sanitizeMetricName=false` command-line flag. Thanks to @PerGon for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3105).
+* FEATURE: add `-usePromCompatibleNaming` command-line flag to [vmagent](https://docs.victoriametrics.com/vmagent.html), to single-node VictoriaMetrics and to `vminsert` component of VictoriaMetrics cluster. This flag can be used for normalizing the ingested metric names and label names to [Prometheus-compatible form](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). If this flag is set, then all the chars unsupported by Prometheus are replaced with `_` chars in metric names and labels of the ingested samples. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113).
+* FEATURE: accept whitespace in metric names and tags ingested via [Graphite plaintext protocol](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd) according to [the specs](https://graphite.readthedocs.io/en/latest/tags.html). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102).
 * FEATURE: check the correctess of raw sample timestamps stored on disk when reading them. This reduces the probability of possible silent corruption of the data stored on disk. This should help [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2998) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3011).
 * FEATURE: atomically delete directories with snapshots, parts and partitions at [storage level](https://docs.victoriametrics.com/#storage). Previously such directories can be left in partially deleted state when the deletion operation was interrupted by unclean shutdown. This may result in `cannot open file ...: no such file or directory` error on the next start. The probability of this error was quite high when NFS or EFS was used as persistent storage for VictoriaMetrics data. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038).
 * FEATURE: set the `start` arg to `end - 5 minutes` if isn't passed explicitly to [/api/v1/labels](https://docs.victoriametrics.com/url-examples.html#apiv1labels) and [/api/v1/label/.../values](https://docs.victoriametrics.com/url-examples.html#apiv1labelvalues). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3052).
diff --git a/docs/Cluster-VictoriaMetrics.md b/docs/Cluster-VictoriaMetrics.md
index 9684386e3..afbbbfc37 100644
--- a/docs/Cluster-VictoriaMetrics.md
+++ b/docs/Cluster-VictoriaMetrics.md
@@ -639,6 +639,8 @@ Below is the output for `/path/to/vminsert -help`:
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -denyQueryTracing
      Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing
   -disableRerouting
@@ -775,6 +777,8 @@ Below is the output for `/path/to/vminsert -help`:
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
   -vmstorageDialTimeout duration
diff --git a/docs/README.md b/docs/README.md
index cd285f1cd..aa8f1487f 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -432,6 +432,10 @@ This command should return the following output if everything is OK:
 {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]}
 ```
 
+VictoriaMetrics automatically sanitizes metric names for the data ingested via DataDog protocol
+according to [DataDog metric naming recommendations](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics).
+If you need accepting metric names as is without sanitizing, then pass `-datadog.sanitizeMetricName=false` command-line flag to VictoriaMetrics.
+
 Extra labels may be added to all the written time series by passing `extra_label=name=value` query args.
 For example, `/datadog/api/v1/series?extra_label=foo=bar` would add `{foo="bar"}` label to all the ingested metrics.
 
@@ -465,6 +469,9 @@ VictoriaMetrics performs the following transformations to the ingested InfluxDB
 * Field names are mapped to time series names prefixed with `{measurement}{separator}` value, where `{separator}` equals to `_` by default. It can be changed with `-influxMeasurementFieldSeparator` command-line flag. See also `-influxSkipSingleField` command-line flag. If `{measurement}` is empty or if `-influxSkipMeasurement` command-line flag is set, then time series names correspond to field names.
 * Field values are mapped to time series values.
 * Tags are mapped to Prometheus labels as-is.
+* If `-usePromCompatibleNaming` command-line flag is set, then all the metric names and label names
+  are normalized to [Prometheus-compatible naming](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels) by replacing unsupported chars with `_`.
+  For example, `foo.bar-baz/1` metric name or label name is substituted with `foo_bar_baz_1`.
 
 For example, the following InfluxDB line:
 
@@ -1994,6 +2001,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -dedup.minScrapeInterval duration
      Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling
   -deleteAuthKey string
@@ -2333,6 +2342,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
   -vmalert.proxyURL string
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 12d65ce82..0445dac26 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -436,6 +436,10 @@ This command should return the following output if everything is OK:
 {"metric":{"__name__":"system.load.1","environment":"test","host":"test.example.com"},"values":[0.5],"timestamps":[1632833641000]}
 ```
 
+VictoriaMetrics automatically sanitizes metric names for the data ingested via DataDog protocol
+according to [DataDog metric naming recommendations](https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics).
+If you need accepting metric names as is without sanitizing, then pass `-datadog.sanitizeMetricName=false` command-line flag to VictoriaMetrics.
+
 Extra labels may be added to all the written time series by passing `extra_label=name=value` query args.
 For example, `/datadog/api/v1/series?extra_label=foo=bar` would add `{foo="bar"}` label to all the ingested metrics.
 
@@ -469,6 +473,9 @@ VictoriaMetrics performs the following transformations to the ingested InfluxDB
 * Field names are mapped to time series names prefixed with `{measurement}{separator}` value, where `{separator}` equals to `_` by default. It can be changed with `-influxMeasurementFieldSeparator` command-line flag. See also `-influxSkipSingleField` command-line flag. If `{measurement}` is empty or if `-influxSkipMeasurement` command-line flag is set, then time series names correspond to field names.
 * Field values are mapped to time series values.
 * Tags are mapped to Prometheus labels as-is.
+* If `-usePromCompatibleNaming` command-line flag is set, then all the metric names and label names
+  are normalized to [Prometheus-compatible naming](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels) by replacing unsupported chars with `_`.
+  For example, `foo.bar-baz/1` metric name or label name is substituted with `foo_bar_baz_1`.
 
 For example, the following InfluxDB line:
 
@@ -1998,6 +2005,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -dedup.minScrapeInterval duration
      Leave only the last sample in every time series per each discrete interval equal to -dedup.minScrapeInterval > 0. See https://docs.victoriametrics.com/#deduplication and https://docs.victoriametrics.com/#downsampling
   -deleteAuthKey string
@@ -2337,6 +2346,8 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
   -vmalert.proxyURL string
diff --git a/docs/vmagent.md b/docs/vmagent.md
index 9a8e39a4d..7747e55f2 100644
--- a/docs/vmagent.md
+++ b/docs/vmagent.md
@@ -919,6 +919,8 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
   -datadog.maxInsertRequestSize size
      The maximum size in bytes of a single DataDog POST request to /api/v1/series
      Supports the following optional suffixes for size values: KB, MB, GB, KiB, MiB, GiB (default 67108864)
+  -datadog.sanitizeMetricName
+     Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)
   -denyQueryTracing
      Whether to disable the ability to trace queries. See https://docs.victoriametrics.com/#query-tracing
   -dryRun
@@ -1271,6 +1273,8 @@ See the docs at https://docs.victoriametrics.com/vmagent.html .
      Supports an array of values separated by comma or specified via multiple flags.
   -tlsKeyFile string
      Path to file with TLS key if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated
+  -usePromCompatibleNaming
+     Whether to replace characters unsupported by Prometheus with underscores in the ingested metric names and label names. For example, foo.bar{a.b='c'} is transformed into foo_bar{a_b='c'} during data ingestion if this flag is set. See https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
   -version
      Show VictoriaMetrics version
 ```
diff --git a/lib/protoparser/datadog/streamparser.go b/lib/protoparser/datadog/streamparser.go
index ace0165b0..24cc557ae 100644
--- a/lib/protoparser/datadog/streamparser.go
+++ b/lib/protoparser/datadog/streamparser.go
@@ -2,8 +2,10 @@ package datadog
 
 import (
 	"bufio"
+	"flag"
 	"fmt"
 	"io"
+	"regexp"
 	"sync"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
@@ -14,8 +16,18 @@ import (
 	"github.com/VictoriaMetrics/metrics"
 )
 
-// The maximum request size is defined at https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
-var maxInsertRequestSize = flagutil.NewBytes("datadog.maxInsertRequestSize", 64*1024*1024, "The maximum size in bytes of a single DataDog POST request to /api/v1/series")
+var (
+	// The maximum request size is defined at https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
+	maxInsertRequestSize = flagutil.NewBytes("datadog.maxInsertRequestSize", 64*1024*1024, "The maximum size in bytes of a single DataDog POST request to /api/v1/series")
+
+	// If all metrics in Datadog have the same naming schema as custom metrics, then the following rules apply:
+	// https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics
+	// But there's some hidden behaviour. In addition to what it states in the docs, the following is also done:
+	// - Consecutive underscores are replaced with just one underscore
+	// - Underscore immediately before or after a dot are removed
+	sanitizeMetricName = flag.Bool("datadog.sanitizeMetricName", true, "Sanitize metric names for the ingested DataDog data to comply with DataDog behaviour described at "+
+		"https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics")
+)
 
 // ParseStream parses DataDog POST request for /api/v1/series from reader and calls callback for the parsed request.
 //
@@ -52,6 +64,9 @@ func ParseStream(r io.Reader, contentEncoding string, callback func(series []Ser
 	series := req.Series
 	for i := range series {
 		rows += len(series[i].Points)
+		if *sanitizeMetricName {
+			series[i].Metric = sanitizeName(series[i].Metric)
+		}
 	}
 	rowsRead.Add(rows)
 
@@ -136,3 +151,19 @@ func putRequest(req *Request) {
 }
 
 var requestPool sync.Pool
+
+// sanitizeName performs DataDog-compatible santizing for metric names
+//
+// See https://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics
+func sanitizeName(s string) string {
+	s = unsupportedDatadogChars.ReplaceAllString(s, "_")
+	s = multiUnderscores.ReplaceAllString(s, "_")
+	s = underscoresWithDots.ReplaceAllString(s, ".")
+	return s
+}
+
+var (
+	unsupportedDatadogChars = regexp.MustCompile(`[^0-9a-zA-Z_\.]+`)
+	multiUnderscores        = regexp.MustCompile(`_+`)
+	underscoresWithDots     = regexp.MustCompile(`_?\._?`)
+)
diff --git a/lib/protoparser/graphite/parser.go b/lib/protoparser/graphite/parser.go
index 5be4cae48..1198da50c 100644
--- a/lib/protoparser/graphite/parser.go
+++ b/lib/protoparser/graphite/parser.go
@@ -61,9 +61,6 @@ func (r *Row) reset() {
 
 // UnmarshalMetricAndTags unmarshals metric and optional tags from s.
 func (r *Row) UnmarshalMetricAndTags(s string, tagsPool []Tag) ([]Tag, error) {
-	if strings.Contains(s, " ") {
-		return tagsPool, fmt.Errorf("unexpected whitespace found in %q", s)
-	}
 	n := strings.IndexByte(s, ';')
 	if n < 0 {
 		// No tags
@@ -72,11 +69,7 @@ func (r *Row) UnmarshalMetricAndTags(s string, tagsPool []Tag) ([]Tag, error) {
 		// Tags found
 		r.Metric = s[:n]
 		tagsStart := len(tagsPool)
-		var err error
-		tagsPool, err = unmarshalTags(tagsPool, s[n+1:])
-		if err != nil {
-			return tagsPool, fmt.Errorf("cannot unmarshal tags: %w", err)
-		}
+		tagsPool = unmarshalTags(tagsPool, s[n+1:])
 		tags := tagsPool[tagsStart:]
 		r.Tags = tags[:len(tags):len(tags)]
 	}
@@ -88,40 +81,43 @@ func (r *Row) UnmarshalMetricAndTags(s string, tagsPool []Tag) ([]Tag, error) {
 
 func (r *Row) unmarshal(s string, tagsPool []Tag) ([]Tag, error) {
 	r.reset()
-	n := strings.IndexAny(s, graphiteSeparators)
+	sOrig := s
+	s = stripTrailingWhitespace(s)
+	n := strings.LastIndexAny(s, graphiteSeparators)
 	if n < 0 {
-		return tagsPool, fmt.Errorf("cannot find separator between metric and value in %q", s)
+		return tagsPool, fmt.Errorf("cannot find separator between value and timestamp in %q", s)
 	}
-	metricAndTags := s[:n]
-	tail := stripLeadingWhitespace(s[n+1:])
-
+	timestampStr := s[n+1:]
+	valueStr := ""
+	metricAndTags := ""
+	s = stripTrailingWhitespace(s[:n])
+	n = strings.LastIndexAny(s, graphiteSeparators)
+	if n < 0 {
+		// Missing timestamp
+		metricAndTags = stripLeadingWhitespace(s)
+		valueStr = timestampStr
+		timestampStr = ""
+	} else {
+		metricAndTags = stripLeadingWhitespace(s[:n])
+		valueStr = s[n+1:]
+	}
+	metricAndTags = stripTrailingWhitespace(metricAndTags)
 	tagsPool, err := r.UnmarshalMetricAndTags(metricAndTags, tagsPool)
 	if err != nil {
-		return tagsPool, err
+		return tagsPool, fmt.Errorf("cannot parse metric and tags from %q: %w; original line: %q", metricAndTags, err, sOrig)
 	}
-
-	n = strings.IndexAny(tail, graphiteSeparators)
-	if n < 0 {
-		// There is no timestamp. Use default timestamp instead.
-		v, err := fastfloat.Parse(tail)
+	if len(timestampStr) > 0 {
+		ts, err := fastfloat.Parse(timestampStr)
 		if err != nil {
-			return tagsPool, fmt.Errorf("cannot unmarshal value from %q: %w", tail, err)
+			return tagsPool, fmt.Errorf("cannot unmarshal timestamp from %q: %w; orignal line: %q", timestampStr, err, sOrig)
 		}
-		r.Value = v
-		return tagsPool, nil
+		r.Timestamp = int64(ts)
 	}
-	v, err := fastfloat.Parse(tail[:n])
+	v, err := fastfloat.Parse(valueStr)
 	if err != nil {
-		return tagsPool, fmt.Errorf("cannot unmarshal value from %q: %w", tail[:n], err)
-	}
-	tail = stripLeadingWhitespace(tail[n+1:])
-	tail = stripTrailingWhitespace(tail)
-	ts, err := fastfloat.Parse(tail)
-	if err != nil {
-		return tagsPool, fmt.Errorf("cannot unmarshal timestamp from %q: %w", tail, err)
+		return tagsPool, fmt.Errorf("cannot unmarshal value from %q: %w; original line: %q", valueStr, err, sOrig)
 	}
 	r.Value = v
-	r.Timestamp = int64(ts)
 	return tagsPool, nil
 }
 
@@ -142,11 +138,11 @@ func unmarshalRow(dst []Row, s string, tagsPool []Tag) ([]Row, []Tag) {
 	if len(s) > 0 && s[len(s)-1] == '\r' {
 		s = s[:len(s)-1]
 	}
+	s = stripLeadingWhitespace(s)
 	if len(s) == 0 {
 		// Skip empty line
 		return dst, tagsPool
 	}
-
 	if cap(dst) > len(dst) {
 		dst = dst[:len(dst)+1]
 	} else {
@@ -165,7 +161,7 @@ func unmarshalRow(dst []Row, s string, tagsPool []Tag) ([]Row, []Tag) {
 
 var invalidLines = metrics.NewCounter(`vm_rows_invalid_total{type="graphite"}`)
 
-func unmarshalTags(dst []Tag, s string) ([]Tag, error) {
+func unmarshalTags(dst []Tag, s string) []Tag {
 	for {
 		if cap(dst) > len(dst) {
 			dst = dst[:len(dst)+1]
@@ -182,7 +178,7 @@ func unmarshalTags(dst []Tag, s string) ([]Tag, error) {
 				// Skip empty tag
 				dst = dst[:len(dst)-1]
 			}
-			return dst, nil
+			return dst
 		}
 		tag.unmarshal(s[:n])
 		s = s[n+1:]


Test patch:
diff --git a/app/vmselect/promql/rollup_test.go b/app/vmselect/promql/rollup_test.go
index ae3362814..806398c1e 100644
--- a/app/vmselect/promql/rollup_test.go
+++ b/app/vmselect/promql/rollup_test.go
@@ -1425,19 +1425,16 @@ func TestRollupDelta(t *testing.T) {
 
 	// Small initial value
 	f(nan, nan, nan, []float64{1}, 1)
-	f(nan, nan, nan, []float64{10}, 10)
-	f(nan, nan, nan, []float64{100}, 100)
+	f(nan, nan, nan, []float64{10}, 0)
+	f(nan, nan, nan, []float64{100}, 0)
 	f(nan, nan, nan, []float64{1, 2, 3}, 3)
 	f(1, nan, nan, []float64{1, 2, 3}, 2)
 	f(nan, nan, nan, []float64{5, 6, 8}, 8)
 	f(2, nan, nan, []float64{5, 6, 8}, 6)
 
-	// Moderate initial value with zero delta after that.
-	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
-	f(nan, nan, nan, []float64{100}, 100)
-	f(nan, nan, nan, []float64{100, 100}, 100)
+	f(nan, nan, nan, []float64{100, 100}, 0)
 
-	// Big initial value with with zero delta after that.
+	// Big initial value with zero delta after that.
 	f(nan, nan, nan, []float64{1000}, 0)
 	f(nan, nan, nan, []float64{1000, 1000}, 0)
 
diff --git a/lib/protoparser/datadog/streamparser_test.go b/lib/protoparser/datadog/streamparser_test.go
new file mode 100644
index 000000000..670be59d4
--- /dev/null
+++ b/lib/protoparser/datadog/streamparser_test.go
@@ -0,0 +1,23 @@
+package datadog
+
+import (
+	"testing"
+)
+
+func TestSanitizeName(t *testing.T) {
+	f := func(s, resultExpected string) {
+		t.Helper()
+		result := sanitizeName(s)
+		if result != resultExpected {
+			t.Fatalf("unexpected result for sanitizeName(%q); got\n%q\nwant\n%q", s, result, resultExpected)
+		}
+	}
+	f("before.dot.metric!.name", "before.dot.metric.name")
+	f("after.dot.metric.!name", "after.dot.metric.name")
+	f("in.the.middle.met!ric.name", "in.the.middle.met_ric.name")
+	f("before.and.after.and.middle.met!ric!.!name", "before.and.after.and.middle.met_ric.name")
+	f("many.consecutive.met!!!!ric!!.!!name", "many.consecutive.met_ric.name")
+	f("many.non.consecutive.m!e!t!r!i!c!.!name", "many.non.consecutive.m_e_t_r_i_c.name")
+	f("how.about.underscores_.!_metric!_!.__!!name", "how.about.underscores.metric.name")
+	f("how.about.underscores.middle.met!_!_ric.name", "how.about.underscores.middle.met_ric.name")
+}
diff --git a/lib/protoparser/graphite/parser_test.go b/lib/protoparser/graphite/parser_test.go
index 3a29be751..467d5d9e4 100644
--- a/lib/protoparser/graphite/parser_test.go
+++ b/lib/protoparser/graphite/parser_test.go
@@ -19,13 +19,6 @@ func TestUnmarshalMetricAndTagsFailure(t *testing.T) {
 	}
 	f("")
 	f(";foo=bar")
-	f(" ")
-	f("foo ;bar=baz")
-	f("f oo;bar=baz")
-	f("foo;bar=baz   ")
-	f("foo;bar= baz")
-	f("foo;bar=b az")
-	f("foo;b ar=baz")
 }
 
 func TestUnmarshalMetricAndTagsSuccess(t *testing.T) {
@@ -40,10 +33,67 @@ func TestUnmarshalMetricAndTagsSuccess(t *testing.T) {
 			t.Fatalf("unexpected row;\ngot\n%+v\nwant\n%+v", &r, rExpected)
 		}
 	}
+	f(" ", &Row{
+		Metric: " ",
+	})
+	f("foo ;bar=baz", &Row{
+		Metric: "foo ",
+		Tags: []Tag{
+			{
+				Key:   "bar",
+				Value: "baz",
+			},
+		},
+	})
+	f("f oo;bar=baz", &Row{
+		Metric: "f oo",
+		Tags: []Tag{
+			{
+				Key:   "bar",
+				Value: "baz",
+			},
+		},
+	})
+	f("foo;bar=baz   ", &Row{
+		Metric: "foo",
+		Tags: []Tag{
+			{
+				Key:   "bar",
+				Value: "baz   ",
+			},
+		},
+	})
+	f("foo;bar= baz", &Row{
+		Metric: "foo",
+		Tags: []Tag{
+			{
+				Key:   "bar",
+				Value: " baz",
+			},
+		},
+	})
+	f("foo;bar=b az", &Row{
+		Metric: "foo",
+		Tags: []Tag{
+			{
+				Key:   "bar",
+				Value: "b az",
+			},
+		},
+	})
+	f("foo;b ar=baz", &Row{
+		Metric: "foo",
+		Tags: []Tag{
+			{
+				Key:   "b ar",
+				Value: "baz",
+			},
+		},
+	})
 	f("foo", &Row{
 		Metric: "foo",
 	})
-	f("foo;bar=123;baz=aabb", &Row{
+	f("foo;bar=123;baz=aa=bb", &Row{
 		Metric: "foo",
 		Tags: []Tag{
 			{
@@ -52,7 +102,7 @@ func TestUnmarshalMetricAndTagsSuccess(t *testing.T) {
 			},
 			{
 				Key:   "baz",
-				Value: "aabb",
+				Value: "aa=bb",
 			},
 		},
 	})
@@ -74,16 +124,9 @@ func TestRowsUnmarshalFailure(t *testing.T) {
 		}
 	}
 
-	// Missing metric
-	f(" 123 455")
-
 	// Missing value
 	f("aaa")
 
-	// unexpected space in tag value
-	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
-	f("s;tag1=aaa1;tag2=bb b2;tag3=ccc3 1")
-
 	// invalid value
 	f("aa bb")
 
@@ -103,7 +146,7 @@ func TestRowsUnmarshalSuccess(t *testing.T) {
 		// Try unmarshaling again
 		rows.Unmarshal(s)
 		if !reflect.DeepEqual(rows.Rows, rowsExpected.Rows) {
-			t.Fatalf("unexpected rows;\ngot\n%+v;\nwant\n%+v", rows.Rows, rowsExpected.Rows)
+			t.Fatalf("unexpected rows on second unmarshal;\ngot\n%+v;\nwant\n%+v", rows.Rows, rowsExpected.Rows)
 		}
 
 		rows.Reset()
@@ -119,6 +162,12 @@ func TestRowsUnmarshalSuccess(t *testing.T) {
 	f("\n\r\n", &Rows{})
 
 	// Single line
+	f(" 123 455", &Rows{
+		Rows: []Row{{
+			Metric: "123",
+			Value:  455,
+		}},
+	})
 	f("foobar -123.456 789", &Rows{
 		Rows: []Row{{
 			Metric:    "foobar",
@@ -134,6 +183,26 @@ func TestRowsUnmarshalSuccess(t *testing.T) {
 		}},
 	})
 
+	// Whitespace in metric name, tag name and tag value
+	// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3102
+	f("s a;ta g1=aaa1;tag2=bb b2;tag3 1 23", &Rows{
+		Rows: []Row{{
+			Metric:    "s a",
+			Value:     1,
+			Timestamp: 23,
+			Tags: []Tag{
+				{
+					Key:   "ta g1",
+					Value: "aaa1",
+				},
+				{
+					Key:   "tag2",
+					Value: "bb b2",
+				},
+			},
+		}},
+	})
+
 	// Missing timestamp
 	f("aaa 1123", &Rows{
 		Rows: []Row{{