Repository: VictoriaMetrics/VictoriaMetrics
Pull request #1280 (2021-05-09 20:07:03)

Add vendor license checker, update codecov action, add dependbot for …
…github actions


codecov[bot]: # [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
> Merging [#1280](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (1381497) into [master](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/377ef9f39938fafa12fb6db3a90a396baf05f3bb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) (377ef9f) will **increase** coverage by `1.49%`.
> The diff coverage is `66.27%`.

> :exclamation: Current head 1381497 differs from pull request most recent head 67a3364. Consider uploading reports for the commit 67a3364 to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/graphs/tree.svg?width=650&height=150&src=pr&token=YND8dYVcXn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)

```diff
@@            Coverage Diff             @@
##           master    #1280      +/-   ##
==========================================
+ Coverage   59.36%   60.85%   +1.49%     
==========================================
  Files         215      215              
  Lines       25570    25660      +90     
==========================================
+ Hits        15179    15615     +436     
+ Misses       8984     8617     -367     
- Partials     1407     1428      +21     
```


| [Impacted Files](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/victoria-metrics/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZpY3RvcmlhLW1ldHJpY3MvbWFpbi5nbw==) | `6.55% <0.00%> (+0.20%)` | :arrow_up: |
| [app/vmalert/datasource/datasource.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS9kYXRhc291cmNlLmdv) | `12.50% <ø> (ø)` | |
| [app/vmalert/datasource/init.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvZGF0YXNvdXJjZS9pbml0Lmdv) | `0.00% <ø> (ø)` | |
| [app/vmalert/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvbWFpbi5nbw==) | `22.01% <0.00%> (ø)` | |
| [app/vmalert/remotewrite/init.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvaW5pdC5nbw==) | `0.00% <ø> (ø)` | |
| [app/vmauth/main.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYXV0aC9tYWluLmdv) | `11.11% <0.00%> (+0.58%)` | :arrow_up: |
| [app/vmctl/vm/vm.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtY3RsL3ZtL3ZtLmdv) | `4.70% <ø> (ø)` | |
| [app/vmselect/prometheus/prometheus.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3Ztc2VsZWN0L3Byb21ldGhldXMvcHJvbWV0aGV1cy5nbw==) | `5.01% <0.00%> (ø)` | |
| [lib/fs/dir\_remover.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2ZzL2Rpcl9yZW1vdmVyLmdv) | `30.50% <0.00%> (ø)` | |
| [lib/fs/fs.go](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2ZzL2ZzLmdv) | `4.54% <0.00%> (-0.32%)` | :arrow_down: |
| ... and [39 more](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Last update [6ff1909...67a3364](https://codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/1280?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).

f41gh7:  Codecov is back!

Patch:
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 000000000..123014908
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "daily"
diff --git a/.github/workflows/check-licenses.yml b/.github/workflows/check-licenses.yml
new file mode 100644
index 000000000..ce272b70d
--- /dev/null
+++ b/.github/workflows/check-licenses.yml
@@ -0,0 +1,26 @@
+name: license-check
+on:
+  push:
+    paths:
+      - 'vendor'
+  pull_request:
+    paths:
+      - 'vendor'
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Go
+        uses: actions/setup-go@main
+        with:
+          go-version: 1.16
+        id: go
+      - name: Dependencies
+        run: |
+          go get -u github.com/frapposelli/wwhrd
+      - name: Code checkout
+        uses: actions/checkout@master
+      - name: Check License
+        run: |
+          wwhrd check
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index f75a7ef1b..ff8bf995a 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -60,7 +60,7 @@ jobs:
           GOOS=darwin go build -mod=vendor ./app/vmctl
           CGO_ENABLED=0 GOOS=windows go build -mod=vendor ./app/vmagent
       - name: Publish coverage
-        uses: codecov/codecov-action@v1.0.6
+        uses: codecov/codecov-action@v1.5.0
         with:
           file: ./coverage.txt
 
diff --git a/.gitignore b/.gitignore
index ba509278a..3e4d0d2a3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,3 +15,4 @@
 /package/temp-rpm-*
 /package/*.deb
 /package/*.rpm
+.DS_store
diff --git a/.wwhrd.yml b/.wwhrd.yml
new file mode 100644
index 000000000..5afba720d
--- /dev/null
+++ b/.wwhrd.yml
@@ -0,0 +1,7 @@
+blacklist:
+  - GPL-2.0
+whitelist:
+  - Apache-2.0
+  - MIT
+  - BSD-3-Clause
+  - BSD-2-Clause
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 0fd6e4dde..e0a60b310 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -212,6 +212,8 @@ The shortlist of configuration flags is the following:
     	Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.queryStep duration
     	queryStep defines how far a value can fallback to when evaluating queries. For example, if datasource.queryStep=15s then param "step" with value "15s" will be added to every query.If queryStep isn't specified, rule's evaluationInterval will be used instead.
+  -datasource.roundDigits int
+    	Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
   -datasource.tlsCAFile string
     	Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
   -datasource.tlsCertFile string
@@ -313,6 +315,8 @@ The shortlist of configuration flags is the following:
     	Optional basic auth password for -remoteRead.url
   -remoteRead.basicAuth.username string
     	Optional basic auth username for -remoteRead.url
+  -remoteRead.ignoreRestoreErrors
+    	Whether to ignore errors from remote storage when restoring alerts state on startup. (default true)
   -remoteRead.lookback duration
     	Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
   -remoteRead.tlsCAFile string
diff --git a/app/vmalert/alerting.go b/app/vmalert/alerting.go
index 82a53348b..75c80f2de 100644
--- a/app/vmalert/alerting.go
+++ b/app/vmalert/alerting.go
@@ -2,7 +2,6 @@ package main
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"hash/fnv"
 	"sort"
@@ -405,9 +404,6 @@ func alertForToTimeSeries(name string, a *notifier.Alert, timestamp time.Time) p
 	return newTimeSeries(float64(a.Start.Unix()), labels, timestamp)
 }
 
-// ErrStateRestore indicates that the vmalert state failed to restore during startup.
-var ErrStateRestore = errors.New("failed to restore the state")
-
 // Restore restores the state of active alerts basing on previously written timeseries.
 // Restore restores only Start field. Field State will be always Pending and supposed
 // to be updated on next Exec, as well as Value field.
@@ -432,7 +428,7 @@ func (ar *AlertingRule) Restore(ctx context.Context, q datasource.Querier, lookb
 		alertForStateMetricName, ar.Name, labelsFilter, int(lookback.Seconds()))
 	qMetrics, err := q.Query(ctx, expr)
 	if err != nil {
-		return fmt.Errorf("%s: %w", err, ErrStateRestore)
+		return err
 	}
 
 	for _, m := range qMetrics {
diff --git a/app/vmalert/datasource/init.go b/app/vmalert/datasource/init.go
index 62603f749..a6f44aa8a 100644
--- a/app/vmalert/datasource/init.go
+++ b/app/vmalert/datasource/init.go
@@ -4,6 +4,7 @@ import (
 	"flag"
 	"fmt"
 	"net/http"
+	"strings"
 
 	"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/utils"
 )
@@ -26,6 +27,8 @@ var (
 		"For example, if datasource.queryStep=15s then param \"step\" with value \"15s\" will be added to every query."+
 		"If queryStep isn't specified, rule's evaluationInterval will be used instead.")
 	maxIdleConnections = flag.Int("datasource.maxIdleConnections", 100, `Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state.`)
+	roundDigits        = flag.Int("datasource.roundDigits", 0, `Adds "round_digits" GET param to datasource requests. `+
+		`In VM "round_digits" limits the number of digits after the decimal point in response values.`)
 )
 
 // Init creates a Querier from provided flag values.
@@ -33,11 +36,27 @@ func Init() (QuerierBuilder, error) {
 	if *addr == "" {
 		return nil, fmt.Errorf("datasource.url is empty")
 	}
+
 	tr, err := utils.Transport(*addr, *tlsCertFile, *tlsKeyFile, *tlsCAFile, *tlsServerName, *tlsInsecureSkipVerify)
 	if err != nil {
 		return nil, fmt.Errorf("failed to create transport: %w", err)
 	}
 	tr.MaxIdleConns = *maxIdleConnections
-	c := &http.Client{Transport: tr}
-	return NewVMStorage(*addr, *basicAuthUsername, *basicAuthPassword, *lookBack, *queryStep, *appendTypePrefix, c), nil
+
+	var rd string
+	if *roundDigits > 0 {
+		rd = fmt.Sprintf("%d", *roundDigits)
+	}
+
+	return &VMStorage{
+		c:                &http.Client{Transport: tr},
+		basicAuthUser:    *basicAuthUsername,
+		basicAuthPass:    *basicAuthPassword,
+		datasourceURL:    strings.TrimSuffix(*addr, "/"),
+		appendTypePrefix: *appendTypePrefix,
+		lookBack:         *lookBack,
+		queryStep:        *queryStep,
+		roundDigits:      rd,
+		dataSourceType:   NewPrometheusType(),
+	}, nil
 }
diff --git a/app/vmalert/datasource/vm.go b/app/vmalert/datasource/vm.go
index 72ab55175..8b59b389c 100644
--- a/app/vmalert/datasource/vm.go
+++ b/app/vmalert/datasource/vm.go
@@ -81,6 +81,7 @@ type VMStorage struct {
 	appendTypePrefix bool
 	lookBack         time.Duration
 	queryStep        time.Duration
+	roundDigits      string
 
 	dataSourceType     Type
 	evaluationInterval time.Duration
@@ -218,6 +219,9 @@ func (s *VMStorage) setPrometheusReqParams(r *http.Request, query string, timest
 		// override step with user-specified value
 		q.Set("step", s.queryStep.String())
 	}
+	if s.roundDigits != "" {
+		q.Set("round_digits", s.roundDigits)
+	}
 	r.URL.RawQuery = q.Encode()
 }
 
diff --git a/app/vmalert/manager.go b/app/vmalert/manager.go
index 22de88ae8..4d9b0595e 100644
--- a/app/vmalert/manager.go
+++ b/app/vmalert/manager.go
@@ -2,7 +2,6 @@ package main
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"strings"
 	"sync"
@@ -52,12 +51,7 @@ func (m *manager) AlertAPI(gID, aID uint64) (*APIAlert, error) {
 }
 
 func (m *manager) start(ctx context.Context, path []string, validateTpl, validateExpr bool) error {
-	err := m.update(ctx, path, validateTpl, validateExpr, true)
-	if *remoteReadIgnoreRestoreErrors && errors.Is(err, ErrStateRestore) {
-		logger.Errorf("%s", err)
-		return nil
-	}
-	return err
+	return m.update(ctx, path, validateTpl, validateExpr, true)
 }
 
 func (m *manager) close() {
@@ -74,7 +68,10 @@ func (m *manager) startGroup(ctx context.Context, group *Group, restore bool) er
 	if restore && m.rr != nil {
 		err := group.Restore(ctx, m.rr, *remoteReadLookBack, m.labels)
 		if err != nil {
-			return fmt.Errorf("error while restoring state for group %q: %w", group.Name, err)
+			if !*remoteReadIgnoreRestoreErrors {
+				return fmt.Errorf("failed to restore state for group %q: %w", group.Name, err)
+			}
+			logger.Errorf("error while restoring state for group %q: %s", group.Name, err)
 		}
 	}
 
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 80a8520f0..806e77dcb 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -5,7 +5,8 @@ sort: 15
 # CHANGELOG
 
 * FEATURE: vminsert: add support for data ingestion via other `vminsert` nodes. This allows building multi-level data ingestion paths in VictoriaMetrics cluster by writing data from one level of `vminsert` nodes to another level of `vminsert` nodes. See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multi-level-cluster-setup) and [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/541#issuecomment-835487858) for details.
-* FEATURE: vmalert: add flag to control behaviour on startup for state restore errors. Alerting rules now can return specific error type ErrStateRestore to indicate whether restore state procedure failed. Such errors were returned and logged before as well. But now user can specify whether to just log these errors (`-remoteRead.ignoreRestoreErrors=true`) or to stop the process (`-remoteRead.ignoreRestoreErrors=false`). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252).
+* FEATURE: vmalert: add flag to control behaviour on startup for state restore errors. Such errors were returned and logged before as well. But now user can specify whether to just log these errors (`-remoteRead.ignoreRestoreErrors=true`) or to stop the process (`-remoteRead.ignoreRestoreErrors=false`). The latter is important when VM isn't ready yet to serve queries from vmalert and it needs to wait. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252).
+* FEATURE: vmalert: add ability to pass `round_digits` query arg to datasource via `-datasource.roundDigits` command-line flag. This can be used for limiting the number of decimal digits after the point in recording rule results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525).
 
 * BUGFIX: vmagent: fix possible race when refreshing `role: endpoints` and `role: endpointslices` scrape targets in `kubernetes_sd_config`. Prevoiusly `pod` objects could be updated after the related `endpoints` object update. This could lead to missing scrape targets. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240).
 * BUGFIX: properly remove stale parts outside the configured retention if `-retentionPeriod` is smaller than one month. Previously stale parts could remain active for up to a month after they go outside the retention.
diff --git a/docs/vmalert.md b/docs/vmalert.md
index 18cc96804..416903055 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -216,6 +216,8 @@ The shortlist of configuration flags is the following:
     	Defines the number of idle (keep-alive connections) to each configured datasource. Consider setting this value equal to the value: groups_total * group.concurrency. Too low a value may result in a high number of sockets in TIME_WAIT state. (default 100)
   -datasource.queryStep duration
     	queryStep defines how far a value can fallback to when evaluating queries. For example, if datasource.queryStep=15s then param "step" with value "15s" will be added to every query.If queryStep isn't specified, rule's evaluationInterval will be used instead.
+  -datasource.roundDigits int
+    	Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
   -datasource.tlsCAFile string
     	Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
   -datasource.tlsCertFile string
@@ -317,6 +319,8 @@ The shortlist of configuration flags is the following:
     	Optional basic auth password for -remoteRead.url
   -remoteRead.basicAuth.username string
     	Optional basic auth username for -remoteRead.url
+  -remoteRead.ignoreRestoreErrors
+    	Whether to ignore errors from remote storage when restoring alerts state on startup. (default true)
   -remoteRead.lookback duration
     	Lookback defines how far to look into past for alerts timeseries. For example, if lookback=1h then range from now() to now()-1h will be scanned. (default 1h0m0s)
   -remoteRead.tlsCAFile string


Test patch:
diff --git a/app/vmalert/datasource/vm_test.go b/app/vmalert/datasource/vm_test.go
index db030cd5a..9a004bcc5 100644
--- a/app/vmalert/datasource/vm_test.go
+++ b/app/vmalert/datasource/vm_test.go
@@ -243,6 +243,16 @@ func TestPrepareReq(t *testing.T) {
 				checkEqualString(t, exp, r.URL.RawQuery)
 			},
 		},
+		{
+			"round digits",
+			&VMStorage{
+				roundDigits: "10",
+			},
+			func(t *testing.T, r *http.Request) {
+				exp := fmt.Sprintf("query=%s&round_digits=10&time=%d", query, timestamp.Unix())
+				checkEqualString(t, exp, r.URL.RawQuery)
+			},
+		},
 	}
 
 	for _, tc := range testCases {