Repository: VictoriaMetrics/VictoriaMetrics
Pull request #5284 (2023-11-03 09:25:06)

app/vmalert: update remote-write process
* automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers.
* increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded.
* increment `vmalert_remotewrite_dropped_rows_total` amd `vmalert_remotewrite_dropped_bytes_total` metrics if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Attention: `7 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`da24c12`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/da24c129dbeface49b2b772ea01f9ebcafcca933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.30% compared to head [(`62dd444`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.33%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #5284      +/-   ##
==========================================
+ Coverage   60.30%   60.33%   +0.02%     
==========================================
  Files         398      398              
  Lines       74701    74712      +11     
==========================================
+ Hits        45051    45078      +27     
+ Misses      27142    27130      -12     
+ Partials     2508     2504       -4     
```


| [Files](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [app/vmalert/rule/group.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcnVsZS9ncm91cC5nbw==) | `67.98% <ø> (-0.13%)` | :arrow_down: |
| [app/vmalert/remotewrite/client.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-YXBwL3ZtYWxlcnQvcmVtb3Rld3JpdGUvY2xpZW50Lmdv) | `64.73% <50.00%> (-1.25%)` | :arrow_down: |

... and [7 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/5284?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).


Patch:
diff --git a/.github/workflows/sync-docs.yml b/.github/workflows/sync-docs.yml
index 1e6ca9d4e..7af350ab9 100644
--- a/.github/workflows/sync-docs.yml
+++ b/.github/workflows/sync-docs.yml
@@ -8,7 +8,7 @@ on:
   workflow_dispatch: {}
 env:
   PAGEFIND_VERSION: "1.0.3"
-  HUGO_VERSION: "0.119.0"
+  HUGO_VERSION: "latest"
 permissions:
   contents: read  # This is required for actions/checkout and to commit back image update
   deployments: write
diff --git a/README.md b/README.md
index cc28ee6b7..175fff542 100644
--- a/README.md
+++ b/README.md
@@ -442,7 +442,7 @@ This information is obtained from the `/api/v1/status/active_queries` HTTP endpo
 [VMUI](#vmui) provides an ability to explore metrics exported by a particular `job` / `instance` in the following way:
 
 1. Open the `vmui` at `http://victoriametrics:8428/vmui/`.
-1. Click the `Explore metrics` tab.
+1. Click the `Explore Prometheus metrics` tab.
 1. Select the `job` you want to explore.
 1. Optionally select the `instance` for the selected job to explore.
 1. Select metrics you want to explore and compare.
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 0fa25d1fc..79cd0e0d4 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -1232,7 +1232,7 @@ The shortlist of configuration flags is the following:
   -remoteWrite.bearerTokenFile string
      Optional path to bearer token file to use for -remoteWrite.url.
   -remoteWrite.concurrency int
-     Defines number of writers for concurrent writing into remote querier (default 1)
+     Defines number of writers for concurrent writing into remote write endpoint (default 1)
   -remoteWrite.disablePathAppend
      Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.
   -remoteWrite.flushInterval duration
diff --git a/app/vmalert/remotewrite/client.go b/app/vmalert/remotewrite/client.go
index 525f6b459..6c012a350 100644
--- a/app/vmalert/remotewrite/client.go
+++ b/app/vmalert/remotewrite/client.go
@@ -3,6 +3,7 @@ package remotewrite
 import (
 	"bytes"
 	"context"
+	"errors"
 	"flag"
 	"fmt"
 	"io"
@@ -117,12 +118,19 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
 // Push adds timeseries into queue for writing into remote storage.
 // Push returns and error if client is stopped or if queue is full.
 func (c *Client) Push(s prompbmarshal.TimeSeries) error {
+	rwTotal.Inc()
 	select {
 	case <-c.doneCh:
+		rwErrors.Inc()
+		droppedRows.Add(len(s.Samples))
+		droppedBytes.Add(s.Size())
 		return fmt.Errorf("client is closed")
 	case c.input <- s:
 		return nil
 	default:
+		rwErrors.Inc()
+		droppedRows.Add(len(s.Samples))
+		droppedBytes.Add(s.Size())
 		return fmt.Errorf("failed to push timeseries - queue is full (%d entries). "+
 			"Queue size is controlled by -remoteWrite.maxQueueSize flag",
 			c.maxQueueSize)
@@ -181,11 +189,14 @@ func (c *Client) run(ctx context.Context) {
 }
 
 var (
+	rwErrors = metrics.NewCounter(`vmalert_remotewrite_errors_total`)
+	rwTotal  = metrics.NewCounter(`vmalert_remotewrite_total`)
+
 	sentRows            = metrics.NewCounter(`vmalert_remotewrite_sent_rows_total`)
 	sentBytes           = metrics.NewCounter(`vmalert_remotewrite_sent_bytes_total`)
-	sendDuration        = metrics.NewFloatCounter(`vmalert_remotewrite_send_duration_seconds_total`)
 	droppedRows         = metrics.NewCounter(`vmalert_remotewrite_dropped_rows_total`)
 	droppedBytes        = metrics.NewCounter(`vmalert_remotewrite_dropped_bytes_total`)
+	sendDuration        = metrics.NewFloatCounter(`vmalert_remotewrite_send_duration_seconds_total`)
 	bufferFlushDuration = metrics.NewHistogram(`vmalert_remotewrite_flush_duration_seconds`)
 
 	_ = metrics.NewGauge(`vmalert_remotewrite_concurrency`, func() float64 {
@@ -222,6 +233,11 @@ func (c *Client) flush(ctx context.Context, wr *prompbmarshal.WriteRequest) {
 L:
 	for attempts := 0; ; attempts++ {
 		err := c.send(ctx, b)
+		if errors.Is(err, io.EOF) {
+			// Something in the middle between client and destination might be closing
+			// the connection. So we do a one more attempt in hope request will succeed.
+			err = c.send(ctx, b)
+		}
 		if err == nil {
 			sentRows.Add(len(wr.Timeseries))
 			sentBytes.Add(len(b))
@@ -259,6 +275,7 @@ L:
 
 	}
 
+	rwErrors.Inc()
 	droppedRows.Add(len(wr.Timeseries))
 	droppedBytes.Add(len(b))
 	logger.Errorf("attempts to send remote-write request failed - dropping %d time series",
@@ -303,7 +320,7 @@ func (c *Client) send(ctx context.Context, data []byte) error {
 	// Prometheus remote Write compatible receivers MUST
 	switch resp.StatusCode / 100 {
 	case 2:
-		// respond with a HTTP 2xx status code when the write is successful.
+		// respond with HTTP 2xx status code when write is successful.
 		return nil
 	case 4:
 		if resp.StatusCode != http.StatusTooManyRequests {
diff --git a/app/vmalert/remotewrite/init.go b/app/vmalert/remotewrite/init.go
index 6285c56ba..7bc1f90d6 100644
--- a/app/vmalert/remotewrite/init.go
+++ b/app/vmalert/remotewrite/init.go
@@ -30,7 +30,7 @@ var (
 
 	maxQueueSize  = flag.Int("remoteWrite.maxQueueSize", 1e5, "Defines the max number of pending datapoints to remote write endpoint")
 	maxBatchSize  = flag.Int("remoteWrite.maxBatchSize", 1e3, "Defines max number of timeseries to be flushed at once")
-	concurrency   = flag.Int("remoteWrite.concurrency", 1, "Defines number of writers for concurrent writing into remote querier")
+	concurrency   = flag.Int("remoteWrite.concurrency", 1, "Defines number of writers for concurrent writing into remote write endpoint")
 	flushInterval = flag.Duration("remoteWrite.flushInterval", 5*time.Second, "Defines interval of flushes to remote write endpoint")
 
 	tlsInsecureSkipVerify = flag.Bool("remoteWrite.tlsInsecureSkipVerify", false, "Whether to skip tls verification when connecting to -remoteWrite.url")
diff --git a/app/vmalert/rule/group.go b/app/vmalert/rule/group.go
index 36fb046a4..b635c1fa3 100644
--- a/app/vmalert/rule/group.go
+++ b/app/vmalert/rule/group.go
@@ -663,9 +663,6 @@ var (
 
 	execTotal  = metrics.NewCounter(`vmalert_execution_total`)
 	execErrors = metrics.NewCounter(`vmalert_execution_errors_total`)
-
-	remoteWriteErrors = metrics.NewCounter(`vmalert_remotewrite_errors_total`)
-	remoteWriteTotal  = metrics.NewCounter(`vmalert_remotewrite_total`)
 )
 
 func (e *executor) exec(ctx context.Context, r Rule, ts time.Time, resolveDuration time.Duration, limit int) error {
@@ -686,9 +683,7 @@ func (e *executor) exec(ctx context.Context, r Rule, ts time.Time, resolveDurati
 		pushToRW := func(tss []prompbmarshal.TimeSeries) error {
 			var lastErr error
 			for _, ts := range tss {
-				remoteWriteTotal.Inc()
 				if err := e.Rw.Push(ts); err != nil {
-					remoteWriteErrors.Inc()
 					lastErr = fmt.Errorf("rule %q: remote write failure: %w", r, err)
 				}
 			}
diff --git a/app/vmauth/README.md b/app/vmauth/README.md
index bc366a797..5d5a1e875 100644
--- a/app/vmauth/README.md
+++ b/app/vmauth/README.md
@@ -181,6 +181,15 @@ users:
   password: "***"
   url_prefix: "http://localhost:8428?extra_label=team=dev"
 
+  # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
+  # are proxied to http://localhost:8428 with extra_label=team=dev query arg.
+  # For example, http://vmauth:8427/api/v1/query is routed to https://localhost/api/v1/query?extra_label=team=dev
+  # TLS verification is skipped for https://localhost.
+- username: "local-single-node-with-tls"
+  password: "***"
+  url_prefix: "https://localhost?extra_label=team=test"
+  tls_insecure_skip_verify: true
+
   # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
   # are load-balanced among http://vmselect1:8481/select/123/prometheus and http://vmselect2:8481/select/123/prometheus
   # For example, http://vmauth:8427/api/v1/query is proxied to the following urls in a round-robin manner:
@@ -257,6 +266,7 @@ unauthorized_user:
   - http://vmselect-az1/?deny_partial_response=1
   - http://vmselect-az2/?deny_partial_response=1
   retry_status_codes: [503, 500]
+  tls_insecure_skip_verify: true
 
 ip_filters:
   allow_list: ["1.2.3.0/24", "127.0.0.1"]
diff --git a/app/vmauth/auth_config.go b/app/vmauth/auth_config.go
index ab67bc498..7c96ea66e 100644
--- a/app/vmauth/auth_config.go
+++ b/app/vmauth/auth_config.go
@@ -5,6 +5,7 @@ import (
 	"encoding/base64"
 	"flag"
 	"fmt"
+	"net/http"
 	"net/url"
 	"os"
 	"regexp"
@@ -14,13 +15,14 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/VictoriaMetrics/metrics"
+	"gopkg.in/yaml.v2"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envtemplate"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
-	"github.com/VictoriaMetrics/metrics"
-	"gopkg.in/yaml.v2"
 )
 
 var (
@@ -48,10 +50,13 @@ type UserInfo struct {
 	MaxConcurrentRequests int         `yaml:"max_concurrent_requests,omitempty"`
 	DefaultURL            *URLPrefix  `yaml:"default_url,omitempty"`
 	RetryStatusCodes      []int       `yaml:"retry_status_codes,omitempty"`
+	TLSInsecureSkipVerify bool        `yaml:"tls_insecure_skip_verify,omitempty"`
 
 	concurrencyLimitCh      chan struct{}
 	concurrencyLimitReached *metrics.Counter
 
+	httpTransport *http.Transport
+
 	requests         *metrics.Counter
 	requestsDuration *metrics.Summary
 }
@@ -442,6 +447,7 @@ func parseAuthConfig(data []byte) (*AuthConfig, error) {
 		_ = metrics.GetOrCreateGauge(`vmauth_unauthorized_user_concurrent_requests_current`, func() float64 {
 			return float64(len(ui.concurrencyLimitCh))
 		})
+		ui.httpTransport = getTransport(ui.TLSInsecureSkipVerify)
 	}
 	return &ac, nil
 }
@@ -512,6 +518,8 @@ func parseAuthConfigUsers(ac *AuthConfig) (map[string]*UserInfo, error) {
 		_ = metrics.GetOrCreateGauge(fmt.Sprintf(`vmauth_user_concurrent_requests_current{username=%q}`, name), func() float64 {
 			return float64(len(ui.concurrencyLimitCh))
 		})
+		ui.httpTransport = getTransport(ui.TLSInsecureSkipVerify)
+
 		byAuthToken[at1] = ui
 		byAuthToken[at2] = ui
 	}
diff --git a/app/vmauth/main.go b/app/vmauth/main.go
index 709f83f02..3d3a94c21 100644
--- a/app/vmauth/main.go
+++ b/app/vmauth/main.go
@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"crypto/tls"
 	"errors"
 	"flag"
 	"fmt"
@@ -15,6 +16,8 @@ import (
 	"sync"
 	"time"
 
+	"github.com/VictoriaMetrics/metrics"
+
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/buildinfo"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/envflag"
@@ -24,7 +27,6 @@ import (
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/netutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil"
 	"github.com/VictoriaMetrics/VictoriaMetrics/lib/pushmetrics"
-	"github.com/VictoriaMetrics/metrics"
 )
 
 var (
@@ -191,7 +193,7 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		} else { // Update path for regular routes.
 			targetURL = mergeURLs(targetURL, u)
 		}
-		ok := tryProcessingRequest(w, r, targetURL, hc, retryStatusCodes)
+		ok := tryProcessingRequest(w, r, targetURL, hc, retryStatusCodes, ui.httpTransport)
 		bu.put()
 		if ok {
 			return
@@ -205,12 +207,11 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 	httpserver.Errorf(w, r, "%s", err)
 }
 
-func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int) bool {
+func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, hc HeadersConf, retryStatusCodes []int, transport *http.Transport) bool {
 	// This code has been copied from net/http/httputil/reverseproxy.go
 	req := sanitizeRequestHeaders(r)
 	req.URL = targetURL
 	updateHeadersByConfig(req.Header, hc.RequestHeaders)
-	transportOnce.Do(transportInit)
 	res, err := transport.RoundTrip(req)
 	rtb, rtbOK := req.Body.(*readTrackingBody)
 	if err != nil {
@@ -353,12 +354,7 @@ var (
 	missingRouteRequests     = metrics.NewCounter(`vmauth_http_request_errors_total{reason="missing_route"}`)
 )
 
-var (
-	transport     *http.Transport
-	transportOnce sync.Once
-)
-
-func transportInit() {
+func getTransport(skipTLSVerification bool) *http.Transport {
 	tr := http.DefaultTransport.(*http.Transport).Clone()
 	tr.ResponseHeaderTimeout = *responseTimeout
 	// Automatic compression must be disabled in order to fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535
@@ -369,7 +365,11 @@ func transportInit() {
 	if tr.MaxIdleConns != 0 && tr.MaxIdleConns < tr.MaxIdleConnsPerHost {
 		tr.MaxIdleConns = tr.MaxIdleConnsPerHost
 	}
-	transport = tr
+	if tr.TLSClientConfig == nil {
+		tr.TLSClientConfig = &tls.Config{}
+	}
+	tr.TLSClientConfig.InsecureSkipVerify = skipTLSVerification
+	return tr
 }
 
 var (
diff --git a/app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricGraph/ExploreMetricItemGraph.tsx b/app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricGraph/ExploreMetricItemGraph.tsx
index 936504372..75aa6d472 100644
--- a/app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricGraph/ExploreMetricItemGraph.tsx
+++ b/app/vmui/packages/vmui/src/components/ExploreMetrics/ExploreMetricGraph/ExploreMetricItemGraph.tsx
@@ -70,7 +70,7 @@ with (q = ${queryBase}) (
 )`;
   }, [name, job, instance, rateEnabled, isBucket]);
 
-  const { isLoading, graphData, error, warning, isHistogram } = useFetchQuery({
+  const { isLoading, graphData, error, queryErrors, warning, isHistogram } = useFetchQuery({
     predefinedQuery: [query],
     visible: true,
     customStep: step,
@@ -98,6 +98,7 @@ with (q = ${queryBase}) (
     >
       {isLoading && <Spinner />}
       {error && <Alert variant="error">{error}</Alert>}
+      {queryErrors[0] && <Alert variant="error">{queryErrors[0]}</Alert>}
       {warning && <Alert variant="warning">
         <div className="vm-explore-metrics-graph__warning">
           <p>{warning}</p>
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 8f7c149c5..bb02b0968 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -56,6 +56,7 @@ The sandbox cluster installation is running under the constant load generated by
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): allow specifying full url in notifier static_configs target address, like `http://alertmanager:9093/test/api/v2/alerts`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5184).
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): reduce the number of queries for restoring alerts state on start-up. The change should speed up the restore process and reduce pressure on `remoteRead.url`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5265).
 * FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add label `file` pointing to the group's filename to metrics `vmalert_recording_.*` and `vmalert_alerts_.*`. The filename should help identifying alerting rules belonging to specific groups with identical names but different filenames. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267).
+* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): automatically retry remote-write requests on closed connections. The change should reduce the amount of logs produced in environments with short-living connections or environments without support of keep-alive on network balancers.  
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): support data ingestion from [NewRelic infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent). See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-newrelic-agent), [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3520) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4712).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-remoteWrite.shardByURL.labels` command-line flag, which can be used for specifying a list of labels for sharding outgoing samples among the configured `-remoteWrite.url` destinations if `-remoteWrite.shardByURL` command-line flag is set. See [these docs](https://docs.victoriametrics.com/vmagent.html#sharding-among-remote-storages) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4942) for details.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not exit on startup when [scrape_configs](https://docs.victoriametrics.com/sd_configs.html#scrape_configs) refer to non-existing or invalid files with auth configs, since these files may appear / updated later. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4959) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153).
@@ -76,11 +77,14 @@ The sandbox cluster installation is running under the constant load generated by
 * FEATURE: [vmalert-tool](https://docs.victoriametrics.com/#vmalert-tool): add `unittest` command to run unittest for alerting and recording rules. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4789) for details.
 * FEATURE: dashboards/vmalert: add new panel `Missed evaluations` for indicating alerting groups that miss their evaluations.
 * FEATURE: all: track requests with wrong auth key and wrong basic auth at `vm_http_request_errors_total` [metric](https://docs.victoriametrics.com/#monitoring) with `reason="wrong_auth_key"` and `reason="wrong_basic_auth"`. See [this issue](https://github.com/victoriaMetrics/victoriaMetrics/issues/4590). Thanks to @venkatbvc for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5166).
+* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add `tls_insecure_skip_verify` parameter which allows to disable TLS verification for backend connection. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5240).
 
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044).
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): fix vmalert web UI when running on 32-bit architectures machine.
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): do not send requests to configured remote systems when `-datasource.*`, `-remoteWrite.*`, `-remoteRead.*` or `-notifier.*` command-line flags refer files with invalid auth configs. Previously such requests were sent without properly set auth headers. Now the requests are sent only after the files are updated with valid auth configs. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5153).
 * BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): properly maintain alerts state in [replay mode](https://docs.victoriametrics.com/vmalert.html#rules-backfilling) if alert's `for` param was bigger than replay request range (usually a couple of hours). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5186) for details.
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): increment `vmalert_remotewrite_errors_total` metric if all retries to send remote-write request failed. Before, this metric was incremented only if remote-write client's buffer is overloaded.
+* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): increment `vmalert_remotewrite_dropped_rows_total` and `vmalert_remotewrite_dropped_bytes_total` metrics if remote-write client's buffer is overloaded. Before, these metrics were incremented only after unsuccessful HTTP calls.
 * BUGFIX: `vmselect`: improve performance and memory usage during query processing on machines with big number of CPU cores. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087).
 * BUGFIX: dashboards: fix vminsert/vmstorage/vmselect metrics filtering when dashboard is used to display data from many sub-clusters with unique job names. Before, only one specific job could have been accounted for component-specific panels, instead of all available jobs for the component.
 * BUGFIX: dashboards/vmalert: apply `desc` sorting in tooltips for vmalert dashboard in order to improve visibility of the outliers on graph.
@@ -99,6 +103,7 @@ The sandbox cluster installation is running under the constant load generated by
 * BUGFIX: [vmstorage](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): log warning about switching to ReadOnly mode only on state change. Before, vmstorage would log this warning every 1s. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5159) for details.
 * BUGFIX: [vmauth](https://docs.victoriametrics.com/vmauth.html): show browser authorization window for unauthorized requests to unsupported paths if the `unauthorized_user` section is specified. This allows properly authorizing the user. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5236) for details.
 * BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the `Disable cache` toggle at `JSON` and `Table` views. Previously response caching was always enabled and couldn't be disabled at these views.
+* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): correctly display query errors on [Explore Prometheus Metrics](https://docs.victoriametrics.com/#metrics-explorer) page. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5202) for details.
 
 ## [v1.94.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.94.0)
 
diff --git a/docs/README.md b/docs/README.md
index 59eb6f018..108e9678c 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -445,7 +445,7 @@ This information is obtained from the `/api/v1/status/active_queries` HTTP endpo
 [VMUI](#vmui) provides an ability to explore metrics exported by a particular `job` / `instance` in the following way:
 
 1. Open the `vmui` at `http://victoriametrics:8428/vmui/`.
-1. Click the `Explore metrics` tab.
+1. Click the `Explore Prometheus metrics` tab.
 1. Select the `job` you want to explore.
 1. Optionally select the `instance` for the selected job to explore.
 1. Select metrics you want to explore and compare.
diff --git a/docs/Single-server-VictoriaMetrics.md b/docs/Single-server-VictoriaMetrics.md
index 97f8bf8b7..6c8b4e6b6 100644
--- a/docs/Single-server-VictoriaMetrics.md
+++ b/docs/Single-server-VictoriaMetrics.md
@@ -453,7 +453,7 @@ This information is obtained from the `/api/v1/status/active_queries` HTTP endpo
 [VMUI](#vmui) provides an ability to explore metrics exported by a particular `job` / `instance` in the following way:
 
 1. Open the `vmui` at `http://victoriametrics:8428/vmui/`.
-1. Click the `Explore metrics` tab.
+1. Click the `Explore Prometheus metrics` tab.
 1. Select the `job` you want to explore.
 1. Optionally select the `instance` for the selected job to explore.
 1. Select metrics you want to explore and compare.
diff --git a/docs/operator/CHANGELOG.md b/docs/operator/CHANGELOG.md
index e577aad55..d097f8412 100644
--- a/docs/operator/CHANGELOG.md
+++ b/docs/operator/CHANGELOG.md
@@ -13,7 +13,11 @@ menu:
 
 ## Next release
 
-- TODO
+### Features
+
+- [vmoperator](./README.md): properly compare difference for `statefulSet` claimTemplate metadata. See [this commit](https://github.com/VictoriaMetrics/operator/commit/49f9c72b504582b06f72eda94055fd964a11d342) for details.
+- [vmoperator](./README.md): sort `statefulSet` pods by id for rolling update order. See [this commit](https://github.com/VictoriaMetrics/operator/commit/e73b03acd073ec3eda34231083a48c6f79a6757b) for details.
+- [vmoperator](./README.md): optimize statefulset update logic, that should reduce some unneeded operations. See [this PR](https://github.com/VictoriaMetrics/operator/pull/801) for details.
 
 <a name="v0.39.1"></a>
 ## [v0.39.1](https://github.com/VictoriaMetrics/operator/releases/tag/v0.39.1) - 1 Nov 2023
diff --git a/docs/operator/vars.md b/docs/operator/vars.md
index d5af2929c..aebf10ae1 100644
--- a/docs/operator/vars.md
+++ b/docs/operator/vars.md
@@ -10,7 +10,7 @@ menu:
 
 <!-- this doc autogenerated - don't edit it manually -->
 # Auto Generated vars for package config 
- updated at Wed Nov  1 15:52:40 UTC 2023 
+ updated at Mon Nov  6 07:44:56 UTC 2023 
 
 
 | varible name | variable default value | variable required | variable description |
diff --git a/docs/stream-aggregation.md b/docs/stream-aggregation.md
index 504a8f5b4..e0297c17a 100644
--- a/docs/stream-aggregation.md
+++ b/docs/stream-aggregation.md
@@ -137,7 +137,7 @@ to one sample per 5 minutes per each input time series (this operation is also k
 
 The aggregated output metrics have the following names according to [output metric naming](#output-metric-names):
 
-```
+```text
 # For input metrics ending with _total
 some_metric_total:5m_total
 
@@ -169,7 +169,7 @@ See [these docs](#aggregating-by-labels) for more details.
 
 The aggregated output metric has the following name according to [output metric naming](#output-metric-names):
 
-```
+```text
 http_requests_total:30s_without_path_user_total
 ```
 
@@ -185,7 +185,7 @@ For example, if an advertising server generates `hits{some="labels"} 1` and `cli
 per each incoming hit and click, then the following [stream aggregation config](#stream-aggregation-config)
 can be used for counting these metrics per every 30 second interval:
 
-```yml
+```yaml
 - match: '{__name__=~"hits|clicks"}'
   interval: 30s
   outputs: [count_samples]
@@ -194,7 +194,7 @@ can be used for counting these metrics per every 30 second interval:
 This config generates the following output metrics for `hits` and `clicks` input metrics
 according to [output metric naming](#output-metric-names):
 
-```
+```text
 hits:30s_count_samples count1
 clicks:30s_count_samples count2
 ```
@@ -221,7 +221,7 @@ can be used for summing these metrics per every minute:
 
 This config generates the following output metrics according to [output metric naming](#output-metric-names):
 
-```
+```text
 hits:1m_sum_samples sum1
 clicks:1m_sum_samples sum2
 ```
@@ -249,7 +249,7 @@ can be used for calculating 50th and 99th percentiles for these metrics every 30
 
 This config generates the following output metrics according to [output metric naming](#output-metric-names):
 
-```
+```text
 request_duration_seconds:30s_quantiles{quantile="0.50"} value1
 request_duration_seconds:30s_quantiles{quantile="0.99"} value2
 
@@ -280,7 +280,7 @@ for these metrics every 60 seconds:
 
 This config generates the following output metrics according to [output metric naming](#output-metric-names).
 
-```
+```text
 request_duration_seconds:60s_histogram_bucket{vmrange="start1...end1"} count1
 request_duration_seconds:60s_histogram_bucket{vmrange="start2...end2"} count2
 ...
@@ -326,6 +326,7 @@ See also [quantiles over input metrics](#quantiles-over-input-metrics) and [aggr
 [Histogram](https://docs.victoriametrics.com/keyConcepts.html#histogram) is a set of [counter](https://docs.victoriametrics.com/keyConcepts.html#counter)
 metrics with different `vmrange` or `le` labels. As they're counters, the applicable aggregation output is 
 [total](https://docs.victoriametrics.com/stream-aggregation.html#total):
+
 ```yaml
 - match: 'http_request_duration_seconds_bucket'
   interval: 1m
@@ -337,7 +338,8 @@ metrics with different `vmrange` or `le` labels. As they're counters, the applic
 ```
 
 This config generates the following output metrics according to [output metric naming](#output-metric-names):
-```
+
+```text
 http_request_duration_seconds_bucket:1m_without_instance_total{le="0.1"} value1
 http_request_duration_seconds_bucket:1m_without_instance_total{le="0.2"} value2
 http_request_duration_seconds_bucket:1m_without_instance_total{le="0.4"} value3
@@ -368,7 +370,7 @@ See also [histograms over input metrics](#histograms-over-input-metrics) and [qu
 
 Output metric names for stream aggregation are constructed according to the following pattern:
 
-```
+```text
 <metric_name>:<interval>[_by_<by_labels>][_without_<without_labels>]_<output>
 ```
 
@@ -391,7 +393,7 @@ during stream aggregation via `input_relabel_configs` and `output_relabel_config
 
 For example, the following config removes the `:1m_sum_samples` suffix added [to the output metric name](#output-metric-names):
 
-```yml
+```yaml
 - interval: 1m
   outputs: [sum_samples]
   output_relabel_configs:
@@ -677,7 +679,7 @@ support the following approaches for hot reloading stream aggregation configs fr
 
 * By sending `SIGHUP` signal to `vmagent` or `victoria-metrics` process:
 
-  ```console
+  ```bash
   kill -SIGHUP `pidof vmagent`
   ```
 
diff --git a/docs/url-examples.md b/docs/url-examples.md
index d20177a08..93e6396f9 100644
--- a/docs/url-examples.md
+++ b/docs/url-examples.md
@@ -10,7 +10,7 @@ menu:
 
 # VictoriaMetrics API examples
 
-## /api/v1/admin/tsdb/delete_series
+### /api/v1/admin/tsdb/delete_series
 
 **Deletes time series from VictoriaMetrics**
 
@@ -81,7 +81,7 @@ Additional information:
 * [How to delete time series](https://docs.victoriametrics.com/#how-to-delete-time-series)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/export
+### /api/v1/export
 
 **Exports raw samples from VictoriaMetrics in JSON line format**
 
@@ -110,7 +110,7 @@ Additional information:
 * [How to export data in JSON line format](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/export/csv
+### /api/v1/export/csv
 
 **Exports raw samples from VictoriaMetrics in CSV format**
 
@@ -138,7 +138,7 @@ Additional information:
 * [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/export/native
+### /api/v1/export/native
 
 **Exports raw samples from VictoriaMetrics in native format**
 
@@ -166,7 +166,7 @@ More information:
 * [How to import time series](https://docs.victoriametrics.com/#how-to-import-time-series-data)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/import
+### /api/v1/import
 
 **Imports data to VictoriaMetrics in JSON line format**
 
@@ -194,7 +194,7 @@ More information:
 * [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/import/csv
+### /api/v1/import/csv
 
 **Imports CSV data to VictoriaMetrics**
 
@@ -222,7 +222,7 @@ Additional information:
 * [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/import/native
+### /api/v1/import/native
 
 **Imports data to VictoriaMetrics in native format**
 
@@ -248,7 +248,7 @@ Additional information:
 * [How to export time series](https://docs.victoriametrics.com/#how-to-export-time-series)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/import/prometheus
+### /api/v1/import/prometheus
 
 **Imports data to VictoriaMetrics in Prometheus text exposition format**
 
@@ -305,7 +305,7 @@ Additional information:
 * [Querying label values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/label/.../values
+### /api/v1/label/.../values
 
 **Get a list of values for a particular label on the given time range**
 
@@ -335,7 +335,7 @@ Additional information:
 * [Getting label names](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/query
+### /api/v1/query
 
 **Performs PromQL/MetricsQL instant query**
 
@@ -363,7 +363,7 @@ Additional information:
 * [Query language](https://docs.victoriametrics.com/keyConcepts.html#metricsql)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/query_range
+### /api/v1/query_range
 
 **Performs PromQL/MetricsQL range query**
 
@@ -391,7 +391,7 @@ Additional information:
 * [Query language](https://docs.victoriametrics.com/keyConcepts.html#metricsql)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /api/v1/series
+### /api/v1/series
 
 **Returns series names with their labels on the given time range**
 
@@ -422,7 +422,7 @@ Additional information:
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 VictoriaMetrics accepts `limit` query arg for `/api/v1/series` handlers for limiting the number of returned entries. For example, the query to `/api/v1/series?limit=5` returns a sample of up to 5 series, while ignoring the rest. If the provided `limit` value exceeds the corresponding `-search.maxSeries` command-line flag values, then limits specified in the command-line flags are used.
 
-## /api/v1/status/tsdb
+### /api/v1/status/tsdb
 
 **Cardinality statistics**
 
@@ -449,7 +449,7 @@ Additional information:
 * [TSDB Stats](https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /datadog
+### /datadog
 
 **DataDog URL for Single-node VictoriaMetrics**
 
@@ -471,7 +471,7 @@ http://vminsert:8480/insert/0/datadog
 
 </div>
 
-## /datadog/api/v1/series
+### /datadog/api/v1/series
 
 **Imports data in DataDog format into VictoriaMetrics**
 
@@ -534,7 +534,7 @@ Additional information:
 * [How to send data from datadog agent](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /federate
+### /federate
 
 **Returns federated metrics**
 
@@ -562,7 +562,7 @@ Additional information:
 * [Prometheus-compatible federation data](https://prometheus.io/docs/prometheus/latest/federation/#configuring-federation)
 * [URL format for VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /graphite/metrics/find
+### /graphite/metrics/find
 
 **Searches Graphite metrics in VictoriaMetrics**
 
@@ -591,7 +591,7 @@ Additional information:
 * [How to send Graphite data to VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-graphite-compatible-agents-such-as-statsd)
 * [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /influx/write
+### /influx/write
 
 **Writes data with InfluxDB line protocol to VictoriaMetrics**
 
@@ -618,7 +618,7 @@ Additional information:
 * [How to send Influx data to VictoriaMetrics](https://docs.victoriametrics.com/#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf)
 * [URL Format](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format)
 
-## /internal/resetRollupResultCache
+### /internal/resetRollupResultCache
 
 **Resets the response cache for previously served queries. It is recommended to invoke after [backfilling](https://docs.victoriametrics.com/#backfilling) procedure.**
 
@@ -641,9 +641,9 @@ curl -Is http://<vmselect>:8481/select/internal/resetRollupResultCache
 
 </div>
 
-## TCP and UDP
+### TCP and UDP
 
-### How to send data from OpenTSDB-compatible agents to VictoriaMetrics
+#### How to send data from OpenTSDB-compatible agents to VictoriaMetrics
 
 Turned off by default. Enable OpenTSDB receiver in VictoriaMetrics by setting `-opentsdbListenAddr` command-line flag.
 *If run from docker, '-opentsdbListenAddr' port should be exposed*
@@ -691,7 +691,7 @@ Additional information:
 * [OpenTSDB http put API](http://opentsdb.net/docs/build/html/api_http/put.html)
 * [How to send data OpenTSDB data to VictoriaMetrics](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-opentsdb-compatible-agents)
 
-### How to send Graphite data to VictoriaMetrics
+#### How to send Graphite data to VictoriaMetrics
 
 Enable Graphite receiver in VictoriaMetrics by setting `-graphiteListenAddr` command-line flag.
 
diff --git a/docs/vmalert.md b/docs/vmalert.md
index 7db44ddba..afe4e9e40 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -1243,7 +1243,7 @@ The shortlist of configuration flags is the following:
   -remoteWrite.bearerTokenFile string
      Optional path to bearer token file to use for -remoteWrite.url.
   -remoteWrite.concurrency int
-     Defines number of writers for concurrent writing into remote querier (default 1)
+     Defines number of writers for concurrent writing into remote write endpoint (default 1)
   -remoteWrite.disablePathAppend
      Whether to disable automatic appending of '/api/v1/write' path to the configured -remoteWrite.url.
   -remoteWrite.flushInterval duration
diff --git a/docs/vmauth.md b/docs/vmauth.md
index 9e68019f8..ade90019c 100644
--- a/docs/vmauth.md
+++ b/docs/vmauth.md
@@ -192,6 +192,15 @@ users:
   password: "***"
   url_prefix: "http://localhost:8428?extra_label=team=dev"
 
+  # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
+  # are proxied to http://localhost:8428 with extra_label=team=dev query arg.
+  # For example, http://vmauth:8427/api/v1/query is routed to https://localhost/api/v1/query?extra_label=team=dev
+  # TLS verification is skipped for https://localhost.
+- username: "local-single-node-with-tls"
+  password: "***"
+  url_prefix: "https://localhost?extra_label=team=test"
+  tls_insecure_skip_verify: true
+
   # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
   # are load-balanced among http://vmselect1:8481/select/123/prometheus and http://vmselect2:8481/select/123/prometheus
   # For example, http://vmauth:8427/api/v1/query is proxied to the following urls in a round-robin manner:
@@ -268,6 +277,7 @@ unauthorized_user:
   - http://vmselect-az1/?deny_partial_response=1
   - http://vmselect-az2/?deny_partial_response=1
   retry_status_codes: [503, 500]
+  tls_insecure_skip_verify: true
 
 ip_filters:
   allow_list: ["1.2.3.0/24", "127.0.0.1"]


Test patch:
diff --git a/app/vmauth/auth_config_test.go b/app/vmauth/auth_config_test.go
index dc465bcfc..bf9902e2b 100644
--- a/app/vmauth/auth_config_test.go
+++ b/app/vmauth/auth_config_test.go
@@ -227,12 +227,14 @@ users:
   password: bar
   url_prefix: http://aaa:343/bbb
   max_concurrent_requests: 5
+  tls_insecure_skip_verify: true
 `, map[string]*UserInfo{
 		getAuthToken("", "foo", "bar"): {
 			Username:              "foo",
 			Password:              "bar",
 			URLPrefix:             mustParseURL("http://aaa:343/bbb"),
 			MaxConcurrentRequests: 5,
+			TLSInsecureSkipVerify: true,
 		},
 	})
 
@@ -448,6 +450,42 @@ users:
 
 }
 
+func TestParseAuthConfigPassesTLSVerificationConfig(t *testing.T) {
+	c := `
+users:
+- username: foo
+  password: bar
+  url_prefix: https://aaa/bbb
+  max_concurrent_requests: 5
+  tls_insecure_skip_verify: true
+
+unauthorized_user:
+  url_prefix: http://aaa:343/bbb
+  max_concurrent_requests: 5
+  tls_insecure_skip_verify: false
+`
+
+	ac, err := parseAuthConfig([]byte(c))
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+	m, err := parseAuthConfigUsers(ac)
+	if err != nil {
+		t.Fatalf("unexpected error: %s", err)
+	}
+
+	ui := m[getAuthToken("", "foo", "bar")]
+	if ui.TLSInsecureSkipVerify != true || ui.httpTransport.TLSClientConfig.InsecureSkipVerify != true {
+		t.Fatalf("unexpected TLSInsecureSkipVerify value for user foo")
+	}
+
+	if ac.UnauthorizedUser.TLSInsecureSkipVerify != false ||
+		ac.UnauthorizedUser.httpTransport.TLSClientConfig.InsecureSkipVerify != false {
+		t.Fatalf("unexpected TLSInsecureSkipVerify value for unauthorized_user")
+	}
+
+}
+
 func getSrcPaths(paths []string) []*SrcPath {
 	var sps []*SrcPath
 	for _, path := range paths {