Repository: VictoriaMetrics/VictoriaMetrics
Pull request #6782 (2024-08-08 14:32:51)

docs: vmanomaly - release v1.15.1
### Describe Your Changes

 vmanomaly - release v1.15.1 updates to docs:
- changelog page
- reader page (new arguments docs)
- typos & fixes

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/).



Patch:
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 5e0f3b2c3..5700f2bc2 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -35,6 +35,7 @@ See also [LTS releases](https://docs.victoriametrics.com/lts-releases/).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent/): reduce memory usage when scraping targets with big response body. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6759).
 * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup/), [vmrestore](https://docs.victoriametrics.com/vmrestore/), [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager/): use exponential backoff for retries when uploading or downloading data from S3. This should reduce the number of failed uploads and downloads when S3 is temporarily unavailable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6732).
 
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent/) fix service discovery of Azure Virtual Machines for response contains `nextLink`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6784).
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert): respect HTTP headers defined in [notifier configuration file](https://docs.victoriametrics.com/vmalert/#notifier-configuration-file) for each request to notifiers. Previously, this param was ignored by mistake.
 * BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): correctly apply `-streamAggr.dropInputLabels` when global stream deduplication is enabled without `-streamAggr.config`. Previously, `-remoteWrite.streamAggr.dropInputLabels` was used instead.
 * BUGFIX: [stream aggregation](https://docs.victoriametrics.com/stream-aggregation/): fix command-line flag `-remoteWrite.streamAggr.ignoreFirstIntervals` to accept multiple values and be applied per each corresponding `-remoteWrite.url`. Previously, this flag only could have been used globally for all URLs. 
diff --git a/docs/anomaly-detection/CHANGELOG.md b/docs/anomaly-detection/CHANGELOG.md
index 726c4ff79..1c0b96c17 100644
--- a/docs/anomaly-detection/CHANGELOG.md
+++ b/docs/anomaly-detection/CHANGELOG.md
@@ -13,6 +13,20 @@ Please find the changelog for VictoriaMetrics Anomaly Detection below.
 
 > **Important note: Users are strongly encouraged to upgrade to `vmanomaly` [v1.9.2](https://hub.docker.com/repository/docker/victoriametrics/vmanomaly/tags?page=1&ordering=name) or newer for optimal performance and accuracy. <br><br> This recommendation is crucial for configurations with a low `infer_every` parameter [in your scheduler](./components/scheduler.md#parameters-1), and in scenarios where data exhibits significant high-order seasonality patterns (such as hourly or daily cycles). Previous versions from v1.5.1 to v1.8.0 were identified to contain a critical issue impacting model training, where models were inadvertently trained on limited data subsets, leading to suboptimal fits, affecting the accuracy of anomaly detection. <br><br> Upgrading to v1.9.2 addresses this issue, ensuring proper model training and enhanced reliability. For users utilizing Helm charts, it is recommended to upgrade to version [1.0.0](https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-anomaly/CHANGELOG.md#100) or newer.**
 
+## v1.15.1
+Released: 2024-08-08
+- FEATURE: Introduced backward-compatible `data_range` [query-specific parameter](https://docs.victoriametrics.com/anomaly-detection/components/reader/#per-query-parameters) to the [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader). It enables the definition of **valid** data ranges for input per individual query in `queries`, resulting in:
+  - **High anomaly scores** (>1) when the *data falls outside the expected range*, indicating a data constraint violation.
+  - **Lowest anomaly scores** (=0) when the *model's predictions (`yhat`) fall outside the expected range*, signaling uncertain predictions.
+  - For more details, please refer to the [documentation](https://docs.victoriametrics.com/anomaly-detection/components/reader/?highlight=data_range#per-query-parameters).
+
+- IMPROVEMENT: Added `latency_offset` argument to the [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) to override the default `-search.latencyOffset` [flag of VictoriaMetrics](https://docs.victoriametrics.com/?highlight=search.latencyOffset#list-of-command-line-flags) (30s). The default value is set to 1ms, which should help in cases where `sampling_frequency` is low (10-60s) and `sampling_frequency` equals `infer_every` in the [PeriodicScheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/?highlight=infer_every#periodic-scheduler). This prevents users from receiving `service - WARNING - [Scheduler [scheduler_alias]] No data available for inference.` warnings in logs and allows for consecutive `infer` calls without gaps. To restore the backward compatible behavior, set it equal to your `-search.latencyOffset` value in [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader) config section.
+
+- FIX: Ensure the `use_transform` argument of the [`OnlineQuantileModel`](https://docs.victoriametrics.com/anomaly-detection/components/models/
+#online-seasonal-quantile) functions as intended.
+- FIX: Add a docstring for `query_from_last_seen_timestamp` arg of [VmReader](https://docs.victoriametrics.com/anomaly-detection/components/reader/#vm-reader).
+
+
 ## v1.15.0
 Released: 2024-08-06
 - FEATURE: Introduced models that support [online learning](https://en.wikipedia.org/wiki/Online_machine_learning) for stream-like input. These models significantly reduce the amount of data required for the initial fit stage. For example, they enable reducing `fit_every` from **weeks to hours** and increasing `fit_every` from **hours to weeks** in the [PeriodicScheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/#periodic-scheduler), significantly reducing the **peak amount** of data queried from VictoriaMetrics during `fit` stages. The next models were added:
diff --git a/docs/anomaly-detection/components/models.md b/docs/anomaly-detection/components/models.md
index 1c25872de..9bb53a03e 100644
--- a/docs/anomaly-detection/components/models.md
+++ b/docs/anomaly-detection/components/models.md
@@ -633,7 +633,7 @@ It uses the `quantiles` triplet to calculate `yhat_lower`, `yhat`, and `yhat_upp
 
 *Parameters specific for vmanomaly*:
 
-* `class` (string) - model class name `"model.online.OnlineSeasonalQuantile"` (or `quantile_online` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
+* `class` (string) - model class name `"model.online.OnlineQuantileModel"` (or `quantile_online` starting from [v1.13.0](../CHANGELOG.md#1130) with class alias support)
 * `quantiles` (list[float], optional) - The quantiles to estimate. `yhat_lower`, `yhat`, `yhat_upper` are the quantile order. By default (0.01, 0.5, 0.99).
 * `seasonal_interval` (string, optional) - the interval for the seasonal adjustment. If not set, the model will equal to a simple online quantile model. By default not set.
 * `min_subseason` (str, optional) - the minimum interval to estimate quantiles for. By default not set. Note that the minimum interval should be a multiple of the seasonal interval, i.e. if seasonal_interval='2h', then min_subseason='15m' is valid, but '37m' is not.
@@ -651,7 +651,7 @@ Suppose we have a data with strong intraday (hourly) and intraweek (daily) seaso
 ```yaml
 models:
   your_desired_alias_for_a_model:
-    class: "quantile_online"  # or 'model.online.OnlineSeasonalQuantile'
+    class: "quantile_online"  # or 'model.online.OnlineQuantileModel'
     quantiles: [0.025, 0.5, 0.975]  # lowered to exclude anomalous edges, can be compensated by `scale` param > 1
     seasonal_interval: '7d'  # longest seasonality (week, day) = week, starting from `season_starts_from`
     min_subseason: '1h'  # smallest seasonality (week, day, hour) = hour, will have its own quantile estimates
diff --git a/docs/anomaly-detection/components/reader.md b/docs/anomaly-detection/components/reader.md
index 883d08778..1943d7427 100644
--- a/docs/anomaly-detection/components/reader.md
+++ b/docs/anomaly-detection/components/reader.md
@@ -62,6 +62,11 @@ Starting from [v1.13.0](/anomaly-detection/changelog#v1130) there is change of [
 
     > **Note**: having **different** individual `step` args for queries (i.e. `30s` for `q1` and `2m` for `q2`) is not yet supported for [multivariate model](/anomaly-detection/components/models/index.html#multivariate-models) if you want to run it on several queries simultaneously (i.e. setting [`queries`](/anomaly-detection/components/models/#queries) arg of a model to [`q1`, `q2`]).
 
+- `data_range` (list[float | string]): Introduced in [v1.15.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1151), it allows defining **valid** data ranges for input per individual query in `queries`, resulting in:
+  - **High anomaly scores** (>1) when the *data falls outside the expected range*, indicating a data constraint violation.
+  - **Lowest anomaly scores** (=0) when the *model's predictions (`yhat`) fall outside the expected range*, meaning uncertain predictions.
+
+
 ### Per-query config example
 ```yaml
 reader:
@@ -72,6 +77,7 @@ reader:
     ingestion_rate:
       expr: 'sum(rate(vm_rows_inserted_total[5m])) by (type) > 0'
       step: '2m'  # overrides global `sampling_period` of 1m
+      data_range: [10, 'inf']  # meaning only positive values > 10 are expected, i.e. a value `y` < 10 will trigger anomaly score > 1
 ```
 
 ### Config parameters
@@ -91,182 +97,166 @@ reader:
 `class`
             </td>
             <td>
-
 `reader.vm.VmReader` (or `vm` starting from [v1.13.0](../CHANGELOG.md#v1130))
             </td>
             <td>
-
 Name of the class needed to enable reading from VictoriaMetrics or Prometheus. VmReader is the default option, if not specified.
             </td>
         </tr>
         <tr>
             <td>
-
 `queries`
             </td>
             <td>
-                See [per-query config example](#per-query-config-example) above
+See [per-query config example](#per-query-config-example) above
             </td>
-
-<td>
-
+            <td>
 See [per-query config section](#per-query-parameters) above
             </td>
         </tr>
         <tr>
             <td>
-
 `datasource_url`
             </td>
             <td>
-
 `http://localhost:8481/`
             </td>
             <td>
-
 Datasource URL address
             </td>
         </tr>
         <tr>
             <td>
-
 `tenant_id`
             </td>
             <td>
-
 `0:0`
             </td>
             <td>
-
 For VictoriaMetrics Cluster version only, tenants are identified by accountID or accountID:projectID. See VictoriaMetrics Cluster [multitenancy docs](../../Cluster-VictoriaMetrics.md#multitenancy)
             </td>
         </tr>
         <tr>
             <td>
-
 `sampling_period`
             </td>
             <td>
-
 `1h`
             </td>
             <td>
-
 Frequency of the points returned. Will be converted to `/query_range?step=%s` param (in seconds). **Required** since [v1.9.0](../CHANGELOG.md#v190).
             </td>
         </tr>
         <tr>
             <td>
-
 `query_range_path`
             </td>
             <td>
-
 `/api/v1/query_range`
             </td>
             <td>
-
 Performs PromQL/MetricsQL range query
             </td>
         </tr>
         <tr>
             <td>
-
 `health_path`
             </td>
             <td>
-
 `health`
             </td>
             <td>
-
 Absolute or relative URL address where to check availability of the datasource.
             </td>
         </tr>
         <tr>
             <td>
-
 `user`
             </td>
             <td>
-
 `USERNAME`
             </td>
             <td>
-
 BasicAuth username
             </td>
         </tr>
         <tr>
             <td>
-
 `password`
             </td>
             <td>
-
 `PASSWORD`
             </td>
             <td>
-
 BasicAuth password
             </td>
         </tr>
         <tr>
             <td>
-
 `timeout`
             </td>
             <td>
-
 `30s`
             </td>
             <td>
-
 Timeout for the requests, passed as a string
             </td>
         </tr>
         <tr>
             <td>
-
 `verify_tls`
             </td>
             <td>
-
 `false`
             </td>
             <td>
-
 Allows disabling TLS verification of the remote certificate.
             </td>
         </tr>
         <tr>
             <td>
-
 `bearer_token`
             </td>
             <td>
-
 `token`
             </td>
             <td>
-
 Token is passed in the standard format with header: `Authorization: bearer {token}`
             </td>
         </tr>
         <tr>
             <td>
-
 `extra_filters`
             </td>
             <td>
-
 `[]`
             </td>
             <td>
-
 List of strings with series selector. See: [Prometheus querying API enhancements](../../README.md##prometheus-querying-api-enhancements)
             </td>
         </tr>
+        <tr>
+            <td>
+`query_from_last_seen_timestamp`
+            </td>
+            <td>
+`True`
+            </td>
+            <td>
+If True, then query will be performed from the last seen timestamp for a given series. If False, then query will be performed from the start timestamp, based on a schedule period. Defaults to `True`. (`False` prior to [v1.15.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1151)). Useful for `infer` stages in case there were skipped `infer` calls prior to given.
+            </td>
+        </tr>
+        <tr>
+            <td>
+`latency_offset`
+            </td>
+            <td>
+`1ms`
+            </td>
+            <td>
+Introduced in [v1.15.1](https://docs.victoriametrics.com/anomaly-detection/changelog/#v1151), it allows overriding the default `-search.latencyOffset` [flag of VictoriaMetrics](https://docs.victoriametrics.com/?highlight=search.latencyOffset#list-of-command-line-flags) (30s). The default value is set to 1ms, which should help in cases where `sampling_frequency` is low (10-60s) and `sampling_frequency` equals `infer_every` in the [PeriodicScheduler](https://docs.victoriametrics.com/anomaly-detection/components/scheduler/?highlight=infer_every#periodic-scheduler). This prevents users from receiving `service - WARNING - [Scheduler [scheduler_alias]] No data available for inference.` warnings in logs and allows for consecutive `infer` calls without gaps. To restore the old behavior, set it equal to your `-search.latencyOffset` [flag value]((https://docs.victoriametrics.com/?highlight=search.latencyOffset#list-of-command-line-flags)).
+            </td>
+        </tr>
     </tbody>
 </table>
 
diff --git a/lib/promscrape/discovery/azure/api.go b/lib/promscrape/discovery/azure/api.go
index 76eb2667d..e21257d69 100644
--- a/lib/promscrape/discovery/azure/api.go
+++ b/lib/promscrape/discovery/azure/api.go
@@ -67,6 +67,9 @@ type apiConfig struct {
 	tokenLock           sync.Mutex
 	token               string
 	tokenExpireDeadline time.Time
+
+	// apiServerHost is only used for verifying the `nextLink` in response of the list API.
+	apiServerHost string
 }
 
 type refreshTokenFunc func() (string, time.Duration, error)
@@ -114,8 +117,12 @@ func newAPIConfig(sdc *SDConfig, baseDir string) (*apiConfig, error) {
 	if err != nil {
 		return nil, fmt.Errorf("cannot create client for %q: %w", env.ResourceManagerEndpoint, err)
 	}
+	// It's already verified in discoveryutils.NewClient so no need to check err.
+	u, _ := url.Parse(c.APIServer())
+
 	cfg := &apiConfig{
 		c:              c,
+		apiServerHost:  u.Host,
 		port:           port,
 		resourceGroup:  sdc.ResourceGroup,
 		subscriptionID: sdc.SubscriptionID,
diff --git a/lib/promscrape/discovery/azure/machine.go b/lib/promscrape/discovery/azure/machine.go
index a9b86a8a7..491d80b36 100644
--- a/lib/promscrape/discovery/azure/machine.go
+++ b/lib/promscrape/discovery/azure/machine.go
@@ -96,8 +96,8 @@ func visitAllAPIObjects(ac *apiConfig, apiURL string, cb func(data json.RawMessa
 			return fmt.Errorf("cannot parse nextLink from response %q: %w", lar.NextLink, err)
 		}
 
-		if nextURL.Host != "" && nextURL.Host != ac.c.APIServer() {
-			return fmt.Errorf("unexpected nextLink host %q, expecting %q", nextURL.Host, ac.c.APIServer())
+		if nextURL.Host != "" && nextURL.Host != ac.apiServerHost {
+			return fmt.Errorf("unexpected nextLink host %q, expecting %q", nextURL.Host, ac.apiServerHost)
 		}
 
 		nextLinkURI = nextURL.RequestURI()


Test patch:
diff --git a/lib/promscrape/discovery/azure/machine_test.go b/lib/promscrape/discovery/azure/machine_test.go
index bb13a2f5a..06710c192 100644
--- a/lib/promscrape/discovery/azure/machine_test.go
+++ b/lib/promscrape/discovery/azure/machine_test.go
@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"net/http"
 	"net/http/httptest"
+	"net/url"
 	"reflect"
 	"strings"
 	"testing"
@@ -13,6 +14,11 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape/discoveryutils"
 )
 
+var (
+	testServerURL string
+	listAPICall   int
+)
+
 func TestGetVirtualMachinesSuccess(t *testing.T) {
 	prettifyVMs := func(src []virtualMachine) string {
 		var sb strings.Builder
@@ -41,39 +47,51 @@ func TestGetVirtualMachinesSuccess(t *testing.T) {
 		}
 		return sb.String()
 	}
-	f := func(name string, expectedVMs []virtualMachine, apiResponses [4]string) {
+	f := func(name string, expectedVMs []virtualMachine, apiResponses [5]string) {
 		t.Run(name, func(t *testing.T) {
 			testServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 				switch {
 				// list vms response
 				case strings.Contains(r.URL.Path, "/providers/Microsoft.Compute/virtualMachines"):
 					w.WriteHeader(http.StatusOK)
-					fmt.Fprintf(w, apiResponses[0])
+					if listAPICall == 0 {
+						// with nextLink
+						apiResponse := strings.Replace(apiResponses[0], "{nextLinkPlaceHolder}", testServerURL+"/providers/Microsoft.Compute/virtualMachines", 1)
+						fmt.Fprint(w, apiResponse)
+						listAPICall++
+					} else {
+						// without nextLink
+						fmt.Fprint(w, apiResponses[1])
+					}
 					// list scaleSets response
 				case strings.Contains(r.URL.RequestURI(), "/providers/Microsoft.Compute/virtualMachineScaleSets?api-version=2022-03-01"):
 					w.WriteHeader(http.StatusOK)
-					fmt.Fprintf(w, apiResponses[1])
+					fmt.Fprint(w, apiResponses[2])
 					// list scalesets vms response
 				case strings.Contains(r.URL.Path, "/providers/Microsoft.Compute/virtualMachineScaleSets/{virtualMachineScaleSetName}/virtualMach"):
 					w.WriteHeader(http.StatusOK)
-					fmt.Fprintf(w, apiResponses[2])
+					fmt.Fprint(w, apiResponses[3])
 					// nic response
 				case strings.Contains(r.URL.Path, "/networkInterfaces/"):
 					w.WriteHeader(http.StatusOK)
-					fmt.Fprintf(w, apiResponses[3])
+					fmt.Fprint(w, apiResponses[4])
 				default:
 					w.WriteHeader(http.StatusNotFound)
 					fmt.Fprintf(w, "API path not found: %s", r.URL.Path)
 				}
 			}))
 			defer testServer.Close()
+			testServerURL = testServer.URL
 			c, err := discoveryutils.NewClient(testServer.URL, nil, nil, nil, &promauth.HTTPClientConfig{})
 			if err != nil {
 				t.Fatalf("unexpected error at client create: %s", err)
 			}
+			u, _ := url.Parse(c.APIServer())
+
 			defer c.Stop()
 			ac := &apiConfig{
 				c:              c,
+				apiServerHost:  u.Host,
 				subscriptionID: "some-id",
 				refreshToken: func() (string, time.Duration, error) {
 					return "auth-token", 0, nil
@@ -102,7 +120,7 @@ func TestGetVirtualMachinesSuccess(t *testing.T) {
 			},
 			Tags: map[string]string{},
 		},
-	}, [4]string{
+	}, [5]string{
 		`
 {
   "value": [
@@ -193,6 +211,10 @@ func TestGetVirtualMachinesSuccess(t *testing.T) {
       "name": "{virtualMachineName}"
     }
   ],
+  "nextLink": "{nextLinkPlaceHolder}"
+}`,
+		`{
+  "value": [],
   "nextLink": ""
 }`,
 		`{}`,
@@ -255,7 +277,8 @@ func TestGetVirtualMachinesSuccess(t *testing.T) {
 			},
 			Tags: map[string]string{},
 		},
-	}, [4]string{
+	}, [5]string{
+		`{}`,
 		`{}`,
 		`{
   "value": [