Repository: VictoriaMetrics/VictoriaMetrics
Pull request #4835 (2023-08-15 03:59:23)

Add flagutil.Duration to avoid conversion bugs
I had a bug incorrectly using `.Msec`, when converting it to time.Duration.
Added flagutil.Duration() which eliminates the possibility of incorrectly using the `.Msec`.


f41gh7:  I think, we have to re-check all timestamps usage at our code base and convert integer timestamps into typed format with time.Time or time.Duration types. Time manipulation must be executed only with time.Add or similar methods.

 It must prevent time conversion bugs. Such as recent storage bug, that cause panic and data loss at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4877

 I suggest to create an issue for code refactoring. 

Following PR fixes only small part of possible problems, it must be updated with necessary changes.
dima-vm: @hagen1778 I didn't post a bug, because resolved it in the same PR. Basically, forgot to divide/multiply `Msec` by `time.Millisecond` as I felt it's already in milliseconds. Basically, bad readability makes is more bug-prone.

Typed Duration would avoid that issue. I'd even go further and make `Msec` private, refactoring all its usages. Totally agree with @f41gh7 basically.
dima-vm: @hagen1778 @f41gh7 PTAL.

There's only one usage of `.Msecs` left, in [vmstorage/main.go#L78](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/a20234d2d056d0467727fc3f501565e2a4f53b7c/app/vmstorage/main.go#L78). But I think it's better resolved in a follow-up PR, by using time.Time, as @f41gh7 noted above.
hagen1778: @dlazerka commits are still unsigned
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) Report
Patch coverage: **`100.00%`** and project coverage change: **`-0.02%`** :warning:
> Comparison is base [(`5d84836`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/commit/5d848363f0110f4fb3cdb188b91a121a445d9cbd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.56% compared to head [(`cfc302c`)](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) 60.55%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #4835      +/-   ##
==========================================
- Coverage   60.56%   60.55%   -0.02%     
==========================================
  Files         384      384              
  Lines       72296    72295       -1     
==========================================
- Hits        43787    43778       -9     
- Misses      26114    26122       +8     
  Partials     2395     2395              
```


| [Files Changed](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics) | Coverage Δ | |
|---|---|---|
| [lib/flagutil/duration.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL2ZsYWd1dGlsL2R1cmF0aW9uLmdv) | `80.00% <100.00%> (+1.21%)` | :arrow_up: |
| [lib/promutils/duration.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3Byb211dGlscy9kdXJhdGlvbi5nbw==) | `67.85% <100.00%> (ø)` | |
| [lib/storage/storage.go](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics#diff-bGliL3N0b3JhZ2Uvc3RvcmFnZS5nbw==) | `70.83% <100.00%> (+0.06%)` | :arrow_up: |

... and [6 files with indirect coverage changes](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/VictoriaMetrics/VictoriaMetrics/pull/4835?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=VictoriaMetrics).

hagen1778: > It must prevent time conversion bugs. Such as recent storage bug, that cause panic and data loss at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4877

I don't think that using time conversion would protect from that. Thorough unittest could.
dima-vm: Adressed all review comments, PTAL

Patch:
diff --git a/app/vlstorage/main.go b/app/vlstorage/main.go
index efd8dfc21..0a6c9b55a 100644
--- a/app/vlstorage/main.go
+++ b/app/vlstorage/main.go
@@ -39,13 +39,13 @@ func Init() {
 		logger.Panicf("BUG: Init() has been already called")
 	}
 
-	if retentionPeriod.Msecs < 24*3600*1000 {
+	if retentionPeriod.Duration() < 24*time.Hour {
 		logger.Fatalf("-retentionPeriod cannot be smaller than a day; got %s", retentionPeriod)
 	}
 	cfg := &logstorage.StorageConfig{
-		Retention:       time.Millisecond * time.Duration(retentionPeriod.Msecs),
+		Retention:       retentionPeriod.Duration(),
 		FlushInterval:   *inmemoryDataFlushInterval,
-		FutureRetention: time.Millisecond * time.Duration(futureRetention.Msecs),
+		FutureRetention: futureRetention.Duration(),
 		LogNewStreams:   *logNewStreams,
 		LogIngestedRows: *logIngestedRows,
 	}
diff --git a/app/vmauth/README.md b/app/vmauth/README.md
index dea54e0e8..ea14aeff6 100644
--- a/app/vmauth/README.md
+++ b/app/vmauth/README.md
@@ -117,11 +117,16 @@ users:
 
   # Requests with the 'Authorization: Bearer YYY' header are proxied to http://localhost:8428 ,
   # The `X-Scope-OrgID: foobar` http header is added to every proxied request.
+  # The `X-Server-Hostname` http header is removed from the proxied response.
   # For example, http://vmauth:8427/api/v1/query is proxied to http://localhost:8428/api/v1/query
 - bearer_token: "YYY"
   url_prefix: "http://localhost:8428"
+  # extra headers to add to the request or remove from the request (if header value is empty)
   headers:
-  - "X-Scope-OrgID: foobar"
+    - "X-Scope-OrgID: foobar"
+  # extra headers to add to the response or remove from the response (if header value is empty)
+  response_headers:
+    - "X-Server-Hostname:" # empty value means the header will be removed from the response
 
   # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
   # are proxied to http://localhost:8428 .
@@ -175,6 +180,7 @@ users:
   #
   # - Requests to http://vmauth:8427/api/v1/write are proxied to http://vminsert:8480/insert/42/prometheus/api/v1/write .
   #   The "X-Scope-OrgID: abc" http header is added to these requests.
+  #   The "X-Server-Hostname" http header is removed from the proxied response.
   #
   # Request which do not match `src_paths` from the `url_map` are proxied to the urls from `default_url`
   # in a round-robin manner. The original request path is passed in `request_path` query arg.
@@ -194,6 +200,8 @@ users:
     url_prefix: "http://vminsert:8480/insert/42/prometheus"
     headers:
     - "X-Scope-OrgID: abc"
+    response_headers:
+    - "X-Server-Hostname:" # empty value means the header will be removed from the response
     ip_filters:
       deny_list: [127.0.0.1]
   default_url:
diff --git a/app/vmauth/auth_config.go b/app/vmauth/auth_config.go
index 6fab91dc2..af1b0573f 100644
--- a/app/vmauth/auth_config.go
+++ b/app/vmauth/auth_config.go
@@ -37,15 +37,15 @@ type AuthConfig struct {
 
 // UserInfo is user information read from authConfigPath
 type UserInfo struct {
-	Name                  string     `yaml:"name,omitempty"`
-	BearerToken           string     `yaml:"bearer_token,omitempty"`
-	Username              string     `yaml:"username,omitempty"`
-	Password              string     `yaml:"password,omitempty"`
-	URLPrefix             *URLPrefix `yaml:"url_prefix,omitempty"`
-	URLMaps               []URLMap   `yaml:"url_map,omitempty"`
-	Headers               []Header   `yaml:"headers,omitempty"`
-	MaxConcurrentRequests int        `yaml:"max_concurrent_requests,omitempty"`
-	DefaultURL            *URLPrefix `yaml:"default_url,omitempty"`
+	Name                  string      `yaml:"name,omitempty"`
+	BearerToken           string      `yaml:"bearer_token,omitempty"`
+	Username              string      `yaml:"username,omitempty"`
+	Password              string      `yaml:"password,omitempty"`
+	URLPrefix             *URLPrefix  `yaml:"url_prefix,omitempty"`
+	URLMaps               []URLMap    `yaml:"url_map,omitempty"`
+	HeadersConf           HeadersConf `yaml:",inline"`
+	MaxConcurrentRequests int         `yaml:"max_concurrent_requests,omitempty"`
+	DefaultURL            *URLPrefix  `yaml:"default_url,omitempty"`
 
 	concurrencyLimitCh      chan struct{}
 	concurrencyLimitReached *metrics.Counter
@@ -54,6 +54,12 @@ type UserInfo struct {
 	requestsDuration *metrics.Summary
 }
 
+// HeadersConf represents config for request and response headers.
+type HeadersConf struct {
+	RequestHeaders  []Header `yaml:"headers,omitempty"`
+	ResponseHeaders []Header `yaml:"response_headers,omitempty"`
+}
+
 func (ui *UserInfo) beginConcurrencyLimit() error {
 	select {
 	case ui.concurrencyLimitCh <- struct{}{}:
@@ -105,9 +111,9 @@ func (h *Header) MarshalYAML() (interface{}, error) {
 
 // URLMap is a mapping from source paths to target urls.
 type URLMap struct {
-	SrcPaths  []*SrcPath `yaml:"src_paths,omitempty"`
-	URLPrefix *URLPrefix `yaml:"url_prefix,omitempty"`
-	Headers   []Header   `yaml:"headers,omitempty"`
+	SrcPaths    []*SrcPath  `yaml:"src_paths,omitempty"`
+	URLPrefix   *URLPrefix  `yaml:"url_prefix,omitempty"`
+	HeadersConf HeadersConf `yaml:",inline"`
 }
 
 // SrcPath represents an src path
diff --git a/app/vmauth/example_config.yml b/app/vmauth/example_config.yml
index e7721be0e..f9e149626 100644
--- a/app/vmauth/example_config.yml
+++ b/app/vmauth/example_config.yml
@@ -12,11 +12,16 @@ users:
 
   # Requests with the 'Authorization: Bearer YYY' header are proxied to http://localhost:8428 ,
   # The `X-Scope-OrgID: foobar` http header is added to every proxied request.
+  # The `X-Server-Hostname:` http header is removed from the proxied response.
   # For example, http://vmauth:8427/api/v1/query is proxied to http://localhost:8428/api/v1/query
 - bearer_token: "YYY"
   url_prefix: "http://localhost:8428"
+  # extra headers to add to the request or remove from the request (if header value is empty)
   headers:
   - "X-Scope-OrgID: foobar"
+  # extra headers to add to the response or remove from the response (if header value is empty)
+  response_headers:
+  - "X-Server-Hostname:" # empty value means the header will be removed from the response
 
   # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
   # are proxied to http://localhost:8428 .
diff --git a/app/vmauth/main.go b/app/vmauth/main.go
index 925bfb7c6..1f7d685a0 100644
--- a/app/vmauth/main.go
+++ b/app/vmauth/main.go
@@ -151,7 +151,7 @@ func processUserRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 
 func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 	u := normalizeURL(r.URL)
-	up, headers := ui.getURLPrefixAndHeaders(u)
+	up, headersConf := ui.getURLPrefixAndHeaders(u)
 	isDefault := false
 	if up == nil {
 		missingRouteRequests.Inc()
@@ -159,7 +159,7 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 			httpserver.Errorf(w, r, "missing route for %q", u.String())
 			return
 		}
-		up, headers = ui.DefaultURL, ui.Headers
+		up, headersConf = ui.DefaultURL, ui.HeadersConf
 		isDefault = true
 	}
 	r.Body = &readTrackingBody{
@@ -178,7 +178,7 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 		} else { // Update path for regular routes.
 			targetURL = mergeURLs(targetURL, u)
 		}
-		ok := tryProcessingRequest(w, r, targetURL, headers)
+		ok := tryProcessingRequest(w, r, targetURL, headersConf)
 		bu.put()
 		if ok {
 			return
@@ -192,13 +192,11 @@ func processRequest(w http.ResponseWriter, r *http.Request, ui *UserInfo) {
 	httpserver.Errorf(w, r, "%s", err)
 }
 
-func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, headers []Header) bool {
+func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url.URL, headersConf HeadersConf) bool {
 	// This code has been copied from net/http/httputil/reverseproxy.go
 	req := sanitizeRequestHeaders(r)
 	req.URL = targetURL
-	for _, h := range headers {
-		req.Header.Set(h.Name, h.Value)
-	}
+	updateHeadersByConfig(req.Header, headersConf.RequestHeaders)
 	transportOnce.Do(transportInit)
 	res, err := transport.RoundTrip(req)
 	if err != nil {
@@ -222,6 +220,7 @@ func tryProcessingRequest(w http.ResponseWriter, r *http.Request, targetURL *url
 	}
 	removeHopHeaders(res.Header)
 	copyHeader(w.Header(), res.Header)
+	updateHeadersByConfig(w.Header(), headersConf.ResponseHeaders)
 	w.WriteHeader(res.StatusCode)
 
 	copyBuf := copyBufPool.Get()
@@ -247,6 +246,16 @@ func copyHeader(dst, src http.Header) {
 	}
 }
 
+func updateHeadersByConfig(headers http.Header, config []Header) {
+	for _, h := range config {
+		if h.Value == "" {
+			headers.Del(h.Name)
+		} else {
+			headers.Set(h.Name, h.Value)
+		}
+	}
+}
+
 func sanitizeRequestHeaders(r *http.Request) *http.Request {
 	// This code has been copied from net/http/httputil/reverseproxy.go
 	req := r.Clone(r.Context())
diff --git a/app/vmauth/target_url.go b/app/vmauth/target_url.go
index 7e4afd24a..365484d95 100644
--- a/app/vmauth/target_url.go
+++ b/app/vmauth/target_url.go
@@ -29,18 +29,18 @@ func mergeURLs(uiURL, requestURI *url.URL) *url.URL {
 	return &targetURL
 }
 
-func (ui *UserInfo) getURLPrefixAndHeaders(u *url.URL) (*URLPrefix, []Header) {
+func (ui *UserInfo) getURLPrefixAndHeaders(u *url.URL) (*URLPrefix, HeadersConf) {
 	for _, e := range ui.URLMaps {
 		for _, sp := range e.SrcPaths {
 			if sp.match(u.Path) {
-				return e.URLPrefix, e.Headers
+				return e.URLPrefix, e.HeadersConf
 			}
 		}
 	}
 	if ui.URLPrefix != nil {
-		return ui.URLPrefix, ui.Headers
+		return ui.URLPrefix, ui.HeadersConf
 	}
-	return nil, nil
+	return nil, HeadersConf{}
 }
 
 func normalizeURL(uOrig *url.URL) *url.URL {
diff --git a/app/vmctl/README.md b/app/vmctl/README.md
index 4cee35e00..beba31a67 100644
--- a/app/vmctl/README.md
+++ b/app/vmctl/README.md
@@ -489,13 +489,14 @@ doesn't support it).
 See `./vmctl remote-read --help` for details and full list of flags.
 
 To start the migration process configure the following flags:
+
 1. `--remote-read-src-addr` - data source address to read from;
-2. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
+1. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
    and for cluster version is equal to `--httpListenAddr` flag of vminsert component (for example `http://<vminsert>:8480/insert/<accountID>/prometheus`);
-3. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
-4. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
-5. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
-6. `--remote-read-use-stream` - defines whether to use `SAMPLES` or `STREAMED_XOR_CHUNKS` mode. By default, is uses `SAMPLES` mode.
+1. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
+1. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
+1. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
+1. `--remote-read-use-stream` - defines whether to use `SAMPLES` or `STREAMED_XOR_CHUNKS` mode. By default, is uses `SAMPLES` mode.
 
 The importing process example for local installation of Prometheus
 and single-node VictoriaMetrics(`http://localhost:8428`):
@@ -562,7 +563,7 @@ and that you have a separate Thanos Store installation.
     - url: http://victoria-metrics:8428/api/v1/write
     ```
 
-2. Make sure VM is running, of course. Now check the logs to make sure that Prometheus is sending and VM is receiving.
+1. Make sure VM is running, of course. Now check the logs to make sure that Prometheus is sending and VM is receiving.
     In Prometheus, make sure there are no errors. On the VM side, you should see messages like this:
 
     ```
@@ -570,7 +571,7 @@ and that you have a separate Thanos Store installation.
     2020-04-27T18:38:46.506Z info VictoriaMetrics/lib/storage/partition.go:222 partition "2020_04" has been created
     ```
 
-3. Now just wait. Within two hours, Prometheus should finish its current data file and hand it off to Thanos Store for long term
+1. Now just wait. Within two hours, Prometheus should finish its current data file and hand it off to Thanos Store for long term
     storage.
 
 ### Historical data
@@ -579,12 +580,13 @@ Let's assume your data is stored on S3 served by minio. You first need to copy t
 then import it into VM using `vmctl` in `prometheus` mode.
 
 1. Copy data from minio.
-    1.1 Run the `minio/mc` Docker container.
-    1.2 `mc config host add minio http://minio:9000 accessKey secretKey`, substituting appropriate values for the last 3 items.
-    1.3 `mc cp -r minio/prometheus thanos-data`
-2. Import using `vmctl`.
-    2.1 Follow the [instructions](#how-to-build) to compile `vmctl` on your machine.
-    2.2 Use [prometheus](#migrating-data-from-prometheus) mode to import data:
+    1. Run the `minio/mc` Docker container.
+    1. `mc config host add minio http://minio:9000 accessKey secretKey`, substituting appropriate values for the last 3 items.
+    1. `mc cp -r minio/prometheus thanos-data`
+
+1. Import using `vmctl`.
+    1. Follow the [instructions](#how-to-build) to compile `vmctl` on your machine.
+    1. Use [prometheus](#migrating-data-from-prometheus) mode to import data:
 
     ```
     vmctl prometheus --prom-snapshot thanos-data --vm-addr http://victoria-metrics:8428
diff --git a/app/vmstorage/main.go b/app/vmstorage/main.go
index d7381a9f9..eef298616 100644
--- a/app/vmstorage/main.go
+++ b/app/vmstorage/main.go
@@ -102,13 +102,13 @@ func Init(resetCacheIfNeeded func(mrs []storage.MetricRow)) {
 	mergeset.SetIndexBlocksCacheSize(cacheSizeIndexDBIndexBlocks.IntN())
 	mergeset.SetDataBlocksCacheSize(cacheSizeIndexDBDataBlocks.IntN())
 
-	if retentionPeriod.Msecs < 24*3600*1000 {
+	if retentionPeriod.Duration() < 24*time.Hour {
 		logger.Fatalf("-retentionPeriod cannot be smaller than a day; got %s", retentionPeriod)
 	}
 	logger.Infof("opening storage at %q with -retentionPeriod=%s", *DataPath, retentionPeriod)
 	startTime := time.Now()
 	WG = syncwg.WaitGroup{}
-	strg := storage.MustOpenStorage(*DataPath, retentionPeriod.Msecs, *maxHourlySeries, *maxDailySeries)
+	strg := storage.MustOpenStorage(*DataPath, retentionPeriod.Duration(), *maxHourlySeries, *maxDailySeries)
 	Storage = strg
 	initStaleSnapshotsRemover(strg)
 
@@ -380,10 +380,10 @@ func RequestHandler(w http.ResponseWriter, r *http.Request) bool {
 
 func initStaleSnapshotsRemover(strg *storage.Storage) {
 	staleSnapshotsRemoverCh = make(chan struct{})
-	if snapshotsMaxAge.Msecs <= 0 {
+	if snapshotsMaxAge.Duration() <= 0 {
 		return
 	}
-	snapshotsMaxAgeDur := time.Duration(snapshotsMaxAge.Msecs) * time.Millisecond
+	snapshotsMaxAgeDur := snapshotsMaxAge.Duration()
 	staleSnapshotsRemoverWG.Add(1)
 	go func() {
 		defer staleSnapshotsRemoverWG.Done()
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 304ed980d..697fd8a20 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -36,6 +36,7 @@ The following `tip` changes can be tested by building VictoriaMetrics components
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve accessibility score to 100 according to [Google's Lighthouse](https://developer.chrome.com/docs/lighthouse/accessibility/) tests.
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): organize `min`, `max`, `median` values on the chart legend and tooltips for better visibility.
 * FEATURE: dashboards: provide copies of Grafana dashboards alternated with VictoriaMetrics datasource at [dashboards/vm](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/dashboards/vm).
+* FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): added ability to set, override and clear request and response headers on a per-user and per-path basis. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4825) and [these docs](https://docs.victoriametrics.com/vmauth.html#auth-config) for details.
 
 * BUGFIX: [build](https://docs.victoriametrics.com/): fix Docker builds for old Docker releases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907).
 * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): consistently set `User-Agent` header to `vm_promscrape` during scraping with enabled or disabled [stream parsing mode](https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4884).
diff --git a/docs/vmauth.md b/docs/vmauth.md
index 4dd06f501..e8f4cb243 100644
--- a/docs/vmauth.md
+++ b/docs/vmauth.md
@@ -128,11 +128,16 @@ users:
 
   # Requests with the 'Authorization: Bearer YYY' header are proxied to http://localhost:8428 ,
   # The `X-Scope-OrgID: foobar` http header is added to every proxied request.
+  # The `X-Server-Hostname` http header is removed from the proxied response.
   # For example, http://vmauth:8427/api/v1/query is proxied to http://localhost:8428/api/v1/query
 - bearer_token: "YYY"
   url_prefix: "http://localhost:8428"
+  # extra headers to add to the request or remove from the request (if header value is empty)
   headers:
-  - "X-Scope-OrgID: foobar"
+    - "X-Scope-OrgID: foobar"
+  # extra headers to add to the response or remove from the response (if header value is empty)
+  response_headers:
+    - "X-Server-Hostname:" # empty value means the header will be removed from the response
 
   # All the requests to http://vmauth:8427 with the given Basic Auth (username:password)
   # are proxied to http://localhost:8428 .
@@ -186,6 +191,7 @@ users:
   #
   # - Requests to http://vmauth:8427/api/v1/write are proxied to http://vminsert:8480/insert/42/prometheus/api/v1/write .
   #   The "X-Scope-OrgID: abc" http header is added to these requests.
+  #   The "X-Server-Hostname" http header is removed from the proxied response.
   #
   # Request which do not match `src_paths` from the `url_map` are proxied to the urls from `default_url`
   # in a round-robin manner. The original request path is passed in `request_path` query arg.
@@ -205,6 +211,8 @@ users:
     url_prefix: "http://vminsert:8480/insert/42/prometheus"
     headers:
     - "X-Scope-OrgID: abc"
+    response_headers:
+    - "X-Server-Hostname:" # empty value means the header will be removed from the response
     ip_filters:
       deny_list: [127.0.0.1]
   default_url:
diff --git a/docs/vmctl.md b/docs/vmctl.md
index 1a3b666d4..9e71165b8 100644
--- a/docs/vmctl.md
+++ b/docs/vmctl.md
@@ -500,13 +500,14 @@ doesn't support it).
 See `./vmctl remote-read --help` for details and full list of flags.
 
 To start the migration process configure the following flags:
+
 1. `--remote-read-src-addr` - data source address to read from;
-2. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
+1. `--vm-addr` - VictoriaMetrics address to write to. For single-node VM is usually equal to `--httpListenAddr`, 
    and for cluster version is equal to `--httpListenAddr` flag of vminsert component (for example `http://<vminsert>:8480/insert/<accountID>/prometheus`);
-3. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
-4. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
-5. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
-6. `--remote-read-use-stream` - defines whether to use `SAMPLES` or `STREAMED_XOR_CHUNKS` mode. By default, is uses `SAMPLES` mode.
+1. `--remote-read-filter-time-start` - the time filter in RFC3339 format to select time series with timestamp equal or higher than provided value. E.g. '2020-01-01T20:07:00Z';
+1. `--remote-read-filter-time-end` - the time filter in RFC3339 format to select time series with timestamp equal or smaller than provided value. E.g. '2020-01-01T20:07:00Z'. Current time is used when omitted.;
+1. `--remote-read-step-interval` - split export data into chunks. Valid values are `month, day, hour, minute`;
+1. `--remote-read-use-stream` - defines whether to use `SAMPLES` or `STREAMED_XOR_CHUNKS` mode. By default, is uses `SAMPLES` mode.
 
 The importing process example for local installation of Prometheus
 and single-node VictoriaMetrics(`http://localhost:8428`):
@@ -573,7 +574,7 @@ and that you have a separate Thanos Store installation.
     - url: http://victoria-metrics:8428/api/v1/write
     ```
 
-2. Make sure VM is running, of course. Now check the logs to make sure that Prometheus is sending and VM is receiving.
+1. Make sure VM is running, of course. Now check the logs to make sure that Prometheus is sending and VM is receiving.
     In Prometheus, make sure there are no errors. On the VM side, you should see messages like this:
 
     ```
@@ -581,7 +582,7 @@ and that you have a separate Thanos Store installation.
     2020-04-27T18:38:46.506Z info VictoriaMetrics/lib/storage/partition.go:222 partition "2020_04" has been created
     ```
 
-3. Now just wait. Within two hours, Prometheus should finish its current data file and hand it off to Thanos Store for long term
+1. Now just wait. Within two hours, Prometheus should finish its current data file and hand it off to Thanos Store for long term
     storage.
 
 ### Historical data
@@ -590,12 +591,13 @@ Let's assume your data is stored on S3 served by minio. You first need to copy t
 then import it into VM using `vmctl` in `prometheus` mode.
 
 1. Copy data from minio.
-    1.1 Run the `minio/mc` Docker container.
-    1.2 `mc config host add minio http://minio:9000 accessKey secretKey`, substituting appropriate values for the last 3 items.
-    1.3 `mc cp -r minio/prometheus thanos-data`
-2. Import using `vmctl`.
-    2.1 Follow the [instructions](#how-to-build) to compile `vmctl` on your machine.
-    2.2 Use [prometheus](#migrating-data-from-prometheus) mode to import data:
+    1. Run the `minio/mc` Docker container.
+    1. `mc config host add minio http://minio:9000 accessKey secretKey`, substituting appropriate values for the last 3 items.
+    1. `mc cp -r minio/prometheus thanos-data`
+
+1. Import using `vmctl`.
+    1. Follow the [instructions](#how-to-build) to compile `vmctl` on your machine.
+    1. Use [prometheus](#migrating-data-from-prometheus) mode to import data:
 
     ```
     vmctl prometheus --prom-snapshot thanos-data --vm-addr http://victoria-metrics:8428
diff --git a/lib/flagutil/duration.go b/lib/flagutil/duration.go
index 87d602c6e..d42e20831 100644
--- a/lib/flagutil/duration.go
+++ b/lib/flagutil/duration.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"strconv"
 	"strings"
+	"time"
 
 	"github.com/VictoriaMetrics/metricsql"
 )
@@ -30,6 +31,11 @@ type Duration struct {
 	valueString string
 }
 
+// Duration convert to time.Duration.
+func (d *Duration) Duration() time.Duration {
+	return time.Millisecond * time.Duration(d.Msecs)
+}
+
 // String implements flag.Value interface
 func (d *Duration) String() string {
 	return d.valueString
diff --git a/lib/storage/storage.go b/lib/storage/storage.go
index e61c0f0c0..9514ea8c9 100644
--- a/lib/storage/storage.go
+++ b/lib/storage/storage.go
@@ -33,8 +33,8 @@ import (
 )
 
 const (
-	msecsPerMonth     = 31 * 24 * 3600 * 1000
-	maxRetentionMsecs = 100 * 12 * msecsPerMonth
+	retentionMonth = 31 * 24 * time.Hour
+	retentionMax   = 100 * 12 * retentionMonth
 )
 
 // Storage represents TSDB storage.
@@ -154,21 +154,18 @@ type Storage struct {
 }
 
 // MustOpenStorage opens storage on the given path with the given retentionMsecs.
-func MustOpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySeries int) *Storage {
+func MustOpenStorage(path string, retention time.Duration, maxHourlySeries, maxDailySeries int) *Storage {
 	path, err := filepath.Abs(path)
 	if err != nil {
 		logger.Panicf("FATAL: cannot determine absolute path for %q: %s", path, err)
 	}
-	if retentionMsecs <= 0 {
-		retentionMsecs = maxRetentionMsecs
-	}
-	if retentionMsecs > maxRetentionMsecs {
-		retentionMsecs = maxRetentionMsecs
+	if retention <= 0 || retention > retentionMax {
+		retention = retentionMax
 	}
 	s := &Storage{
 		path:           path,
 		cachePath:      filepath.Join(path, cacheDirname),
-		retentionMsecs: retentionMsecs,
+		retentionMsecs: retention.Milliseconds(),
 		stop:           make(chan struct{}),
 	}
 	fs.MustMkdirIfNotExist(path)
@@ -246,7 +243,8 @@ func MustOpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDail
 
 	// Initialize nextRotationTimestamp
 	nowSecs := time.Now().UnixNano() / 1e9
-	nextRotationTimestamp := nextRetentionDeadlineSeconds(nowSecs, retentionMsecs/1000, retentionTimezoneOffsetSecs)
+	retentionSecs := retention.Milliseconds() / 1000 // not .Seconds() because unnecessary float64 conversion
+	nextRotationTimestamp := nextRetentionDeadlineSeconds(nowSecs, retentionSecs, retentionTimezoneOffsetSecs)
 	atomic.StoreInt64(&s.nextRotationTimestamp, nextRotationTimestamp)
 
 	// Load nextDayMetricIDs cache


Test patch:
diff --git a/app/vmauth/auth_config_test.go b/app/vmauth/auth_config_test.go
index e794231aa..dc465bcfc 100644
--- a/app/vmauth/auth_config_test.go
+++ b/app/vmauth/auth_config_test.go
@@ -299,14 +299,16 @@ users:
 						"http://vminsert1/insert/0/prometheus",
 						"http://vminsert2/insert/0/prometheus",
 					}),
-					Headers: []Header{
-						{
-							Name:  "foo",
-							Value: "bar",
-						},
-						{
-							Name:  "xxx",
-							Value: "y",
+					HeadersConf: HeadersConf{
+						RequestHeaders: []Header{
+							{
+								Name:  "foo",
+								Value: "bar",
+							},
+							{
+								Name:  "xxx",
+								Value: "y",
+							},
 						},
 					},
 				},
@@ -325,14 +327,16 @@ users:
 						"http://vminsert1/insert/0/prometheus",
 						"http://vminsert2/insert/0/prometheus",
 					}),
-					Headers: []Header{
-						{
-							Name:  "foo",
-							Value: "bar",
-						},
-						{
-							Name:  "xxx",
-							Value: "y",
+					HeadersConf: HeadersConf{
+						RequestHeaders: []Header{
+							{
+								Name:  "foo",
+								Value: "bar",
+							},
+							{
+								Name:  "xxx",
+								Value: "y",
+							},
 						},
 					},
 				},
@@ -389,14 +393,16 @@ users:
 						"http://vminsert1/insert/0/prometheus",
 						"http://vminsert2/insert/0/prometheus",
 					}),
-					Headers: []Header{
-						{
-							Name:  "foo",
-							Value: "bar",
-						},
-						{
-							Name:  "xxx",
-							Value: "y",
+					HeadersConf: HeadersConf{
+						RequestHeaders: []Header{
+							{
+								Name:  "foo",
+								Value: "bar",
+							},
+							{
+								Name:  "xxx",
+								Value: "y",
+							},
 						},
 					},
 				},
@@ -419,14 +425,16 @@ users:
 						"http://vminsert1/insert/0/prometheus",
 						"http://vminsert2/insert/0/prometheus",
 					}),
-					Headers: []Header{
-						{
-							Name:  "foo",
-							Value: "bar",
-						},
-						{
-							Name:  "xxx",
-							Value: "y",
+					HeadersConf: HeadersConf{
+						RequestHeaders: []Header{
+							{
+								Name:  "foo",
+								Value: "bar",
+							},
+							{
+								Name:  "xxx",
+								Value: "y",
+							},
 						},
 					},
 				},
diff --git a/app/vmauth/target_url_test.go b/app/vmauth/target_url_test.go
index 482138e17..9a35d19d3 100644
--- a/app/vmauth/target_url_test.go
+++ b/app/vmauth/target_url_test.go
@@ -7,7 +7,7 @@ import (
 )
 
 func TestCreateTargetURLSuccess(t *testing.T) {
-	f := func(ui *UserInfo, requestURI, expectedTarget, expectedHeaders string) {
+	f := func(ui *UserInfo, requestURI, expectedTarget, expectedRequestHeaders, expectedResponseHeaders string) {
 		t.Helper()
 		u, err := url.Parse(requestURI)
 		if err != nil {
@@ -24,37 +24,39 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 		if target.String() != expectedTarget {
 			t.Fatalf("unexpected target; got %q; want %q", target, expectedTarget)
 		}
-		headersStr := fmt.Sprintf("%q", headers)
-		if headersStr != expectedHeaders {
-			t.Fatalf("unexpected headers; got %s; want %s", headersStr, expectedHeaders)
+		headersStr := fmt.Sprintf("%q", headers.RequestHeaders)
+		if headersStr != expectedRequestHeaders {
+			t.Fatalf("unexpected headers; got %s; want %s", headersStr, expectedRequestHeaders)
 		}
 	}
 	// Simple routing with `url_prefix`
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
-	}, "", "http://foo.bar/.", "[]")
+	}, "", "http://foo.bar/.", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
-		Headers: []Header{{
-			Name:  "bb",
-			Value: "aaa",
-		}},
-	}, "/", "http://foo.bar", `[{"bb" "aaa"}]`)
+		HeadersConf: HeadersConf{
+			RequestHeaders: []Header{{
+				Name:  "bb",
+				Value: "aaa",
+			}},
+		},
+	}, "/", "http://foo.bar", `[{"bb" "aaa"}]`, `[]`)
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar/federate"),
-	}, "/", "http://foo.bar/federate", "[]")
+	}, "/", "http://foo.bar/federate", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar"),
-	}, "a/b?c=d", "http://foo.bar/a/b?c=d", "[]")
+	}, "a/b?c=d", "http://foo.bar/a/b?c=d", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/z", "https://sss:3894/x/y/z", "[]")
+	}, "/z", "https://sss:3894/x/y/z", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/../../aaa", "https://sss:3894/x/y/aaa", "[]")
+	}, "/../../aaa", "https://sss:3894/x/y/aaa", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("https://sss:3894/x/y"),
-	}, "/./asd/../../aaa?a=d&s=s/../d", "https://sss:3894/x/y/aaa?a=d&s=s%2F..%2Fd", "[]")
+	}, "/./asd/../../aaa?a=d&s=s/../d", "https://sss:3894/x/y/aaa?a=d&s=s%2F..%2Fd", "[]", "[]")
 
 	// Complex routing with `url_map`
 	ui := &UserInfo{
@@ -62,14 +64,22 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 			{
 				SrcPaths:  getSrcPaths([]string{"/api/v1/query"}),
 				URLPrefix: mustParseURL("http://vmselect/0/prometheus"),
-				Headers: []Header{
-					{
-						Name:  "xx",
-						Value: "aa",
+				HeadersConf: HeadersConf{
+					RequestHeaders: []Header{
+						{
+							Name:  "xx",
+							Value: "aa",
+						},
+						{
+							Name:  "yy",
+							Value: "asdf",
+						},
 					},
-					{
-						Name:  "yy",
-						Value: "asdf",
+					ResponseHeaders: []Header{
+						{
+							Name:  "qwe",
+							Value: "rty",
+						},
 					},
 				},
 			},
@@ -79,14 +89,20 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 			},
 		},
 		URLPrefix: mustParseURL("http://default-server"),
-		Headers: []Header{{
-			Name:  "bb",
-			Value: "aaa",
-		}},
+		HeadersConf: HeadersConf{
+			RequestHeaders: []Header{{
+				Name:  "bb",
+				Value: "aaa",
+			}},
+			ResponseHeaders: []Header{{
+				Name:  "x",
+				Value: "y",
+			}},
+		},
 	}
-	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", `[{"xx" "aa"} {"yy" "asdf"}]`)
-	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]")
-	f(ui, "/api/v1/query_range", "http://default-server/api/v1/query_range", `[{"bb" "aaa"}]`)
+	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", `[{"xx" "aa"} {"yy" "asdf"}]`, `[{"qwe" "rty"}]`)
+	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]", "[]")
+	f(ui, "/api/v1/query_range", "http://default-server/api/v1/query_range", `[{"bb" "aaa"}]`, `[{"x" "y"}]`)
 
 	// Complex routing regexp paths in `url_map`
 	ui = &UserInfo{
@@ -102,18 +118,17 @@ func TestCreateTargetURLSuccess(t *testing.T) {
 		},
 		URLPrefix: mustParseURL("http://default-server"),
 	}
-	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", "[]")
-	f(ui, "/api/v1/query_range?query=up", "http://vmselect/0/prometheus/api/v1/query_range?query=up", "[]")
-	f(ui, "/api/v1/label/foo/values", "http://vmselect/0/prometheus/api/v1/label/foo/values", "[]")
-	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]")
-	f(ui, "/api/v1/foo/bar", "http://default-server/api/v1/foo/bar", "[]")
+	f(ui, "/api/v1/query?query=up", "http://vmselect/0/prometheus/api/v1/query?query=up", "[]", "[]")
+	f(ui, "/api/v1/query_range?query=up", "http://vmselect/0/prometheus/api/v1/query_range?query=up", "[]", "[]")
+	f(ui, "/api/v1/label/foo/values", "http://vmselect/0/prometheus/api/v1/label/foo/values", "[]", "[]")
+	f(ui, "/api/v1/write", "http://vminsert/0/prometheus/api/v1/write", "[]", "[]")
+	f(ui, "/api/v1/foo/bar", "http://default-server/api/v1/foo/bar", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar?extra_label=team=dev"),
-	}, "/api/v1/query", "http://foo.bar/api/v1/query?extra_label=team=dev", "[]")
+	}, "/api/v1/query", "http://foo.bar/api/v1/query?extra_label=team=dev", "[]", "[]")
 	f(&UserInfo{
 		URLPrefix: mustParseURL("http://foo.bar?extra_label=team=mobile"),
-	}, "/api/v1/query?extra_label=team=dev", "http://foo.bar/api/v1/query?extra_label=team%3Dmobile", "[]")
-
+	}, "/api/v1/query?extra_label=team=dev", "http://foo.bar/api/v1/query?extra_label=team%3Dmobile", "[]", "[]")
 }
 
 func TestCreateTargetURLFailure(t *testing.T) {
@@ -128,7 +143,10 @@ func TestCreateTargetURLFailure(t *testing.T) {
 		if up != nil {
 			t.Fatalf("unexpected non-empty up=%#v", up)
 		}
-		if headers != nil {
+		if headers.RequestHeaders != nil {
+			t.Fatalf("unexpected non-empty headers=%q", headers)
+		}
+		if headers.ResponseHeaders != nil {
 			t.Fatalf("unexpected non-empty headers=%q", headers)
 		}
 	}
diff --git a/lib/flagutil/duration_test.go b/lib/flagutil/duration_test.go
index bd26d47b8..b899c45a8 100644
--- a/lib/flagutil/duration_test.go
+++ b/lib/flagutil/duration_test.go
@@ -3,6 +3,7 @@ package flagutil
 import (
 	"strings"
 	"testing"
+	"time"
 )
 
 func TestDurationSetFailure(t *testing.T) {
@@ -59,3 +60,22 @@ func TestDurationSetSuccess(t *testing.T) {
 	f("1w", 7*24*3600*1000)
 	f("0.25y", 0.25*365*24*3600*1000)
 }
+
+func TestDurationDuration(t *testing.T) {
+	f := func(value string, expected time.Duration) {
+		t.Helper()
+		var d Duration
+		if err := d.Set(value); err != nil {
+			t.Fatalf("unexpected error in d.Set(%q): %s", value, err)
+		}
+		if d.Duration() != expected {
+			t.Fatalf("unexpected result; got %v; want %v", d.Duration().String(), expected.String())
+		}
+	}
+	f("0", 0)
+	f("1", 31*24*time.Hour)
+	f("1h", time.Hour)
+	f("1.5d", 1.5*24*time.Hour)
+	f("1w", 7*24*time.Hour)
+	f("0.25y", 0.25*365*24*time.Hour)
+}
diff --git a/lib/storage/index_db_test.go b/lib/storage/index_db_test.go
index 074613f33..742592b95 100644
--- a/lib/storage/index_db_test.go
+++ b/lib/storage/index_db_test.go
@@ -508,7 +508,7 @@ func TestIndexDB(t *testing.T) {
 
 	t.Run("serial", func(t *testing.T) {
 		const path = "TestIndexDB-serial"
-		s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+		s := MustOpenStorage(path, retentionMax, 0, 0)
 
 		db := s.idb()
 		mns, tsids, err := testIndexDBGetOrCreateTSIDByName(db, metricGroups)
@@ -521,7 +521,7 @@ func TestIndexDB(t *testing.T) {
 
 		// Re-open the storage and verify it works as expected.
 		s.MustClose()
-		s = MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+		s = MustOpenStorage(path, retentionMax, 0, 0)
 
 		db = s.idb()
 		if err := testIndexDBCheckTSIDByName(db, mns, tsids, false); err != nil {
@@ -534,7 +534,7 @@ func TestIndexDB(t *testing.T) {
 
 	t.Run("concurrent", func(t *testing.T) {
 		const path = "TestIndexDB-concurrent"
-		s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+		s := MustOpenStorage(path, retentionMax, 0, 0)
 		db := s.idb()
 
 		ch := make(chan error, 3)
@@ -1429,7 +1429,7 @@ func TestMatchTagFilters(t *testing.T) {
 func TestIndexDBRepopulateAfterRotation(t *testing.T) {
 	r := rand.New(rand.NewSource(1))
 	path := "TestIndexRepopulateAfterRotation"
-	s := MustOpenStorage(path, msecsPerMonth, 1e5, 1e5)
+	s := MustOpenStorage(path, retentionMonth, 1e5, 1e5)
 
 	db := s.idb()
 	if db.generation == 0 {
@@ -1516,7 +1516,7 @@ func TestIndexDBRepopulateAfterRotation(t *testing.T) {
 
 func TestSearchTSIDWithTimeRange(t *testing.T) {
 	const path = "TestSearchTSIDWithTimeRange"
-	s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+	s := MustOpenStorage(path, retentionMax, 0, 0)
 	db := s.idb()
 
 	is := db.getIndexSearch(noDeadline)
@@ -1966,7 +1966,7 @@ func newTestStorage() *Storage {
 		metricNameCache:   workingsetcache.New(1234),
 		tsidCache:         workingsetcache.New(1234),
 		dateMetricIDCache: newDateMetricIDCache(),
-		retentionMsecs:    maxRetentionMsecs,
+		retentionMsecs:    retentionMax.Milliseconds(),
 	}
 	s.setDeletedMetricIDs(&uint64set.Set{})
 	return s
diff --git a/lib/storage/index_db_timing_test.go b/lib/storage/index_db_timing_test.go
index 0a2c48c19..d67bebaa7 100644
--- a/lib/storage/index_db_timing_test.go
+++ b/lib/storage/index_db_timing_test.go
@@ -40,7 +40,7 @@ func BenchmarkRegexpFilterMismatch(b *testing.B) {
 
 func BenchmarkIndexDBAddTSIDs(b *testing.B) {
 	const path = "BenchmarkIndexDBAddTSIDs"
-	s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+	s := MustOpenStorage(path, retentionMax, 0, 0)
 	db := s.idb()
 
 	const recordsPerLoop = 1e3
@@ -94,7 +94,7 @@ func BenchmarkHeadPostingForMatchers(b *testing.B) {
 	// This benchmark is equivalent to https://github.com/prometheus/prometheus/blob/23c0299d85bfeb5d9b59e994861553a25ca578e5/tsdb/head_bench_test.go#L52
 	// See https://www.robustperception.io/evaluating-performance-and-correctness for more details.
 	const path = "BenchmarkHeadPostingForMatchers"
-	s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+	s := MustOpenStorage(path, retentionMax, 0, 0)
 	db := s.idb()
 
 	// Fill the db with data as in https://github.com/prometheus/prometheus/blob/23c0299d85bfeb5d9b59e994861553a25ca578e5/tsdb/head_bench_test.go#L66
@@ -261,7 +261,7 @@ func BenchmarkHeadPostingForMatchers(b *testing.B) {
 
 func BenchmarkIndexDBGetTSIDs(b *testing.B) {
 	const path = "BenchmarkIndexDBGetTSIDs"
-	s := MustOpenStorage(path, maxRetentionMsecs, 0, 0)
+	s := MustOpenStorage(path, retentionMax, 0, 0)
 	db := s.idb()
 
 	const recordsPerLoop = 1000
diff --git a/lib/storage/storage_test.go b/lib/storage/storage_test.go
index a098fc72a..d2bfe3c71 100644
--- a/lib/storage/storage_test.go
+++ b/lib/storage/storage_test.go
@@ -478,15 +478,15 @@ func TestStorageOpenClose(t *testing.T) {
 
 func TestStorageRandTimestamps(t *testing.T) {
 	path := "TestStorageRandTimestamps"
-	retentionMsecs := int64(10 * msecsPerMonth)
-	s := MustOpenStorage(path, retentionMsecs, 0, 0)
+	retention := 10 * retentionMonth
+	s := MustOpenStorage(path, retention, 0, 0)
 	t.Run("serial", func(t *testing.T) {
 		for i := 0; i < 3; i++ {
 			if err := testStorageRandTimestamps(s); err != nil {
 				t.Fatalf("error on iteration %d: %s", i, err)
 			}
 			s.MustClose()
-			s = MustOpenStorage(path, retentionMsecs, 0, 0)
+			s = MustOpenStorage(path, retention, 0, 0)
 		}
 	})
 	t.Run("concurrent", func(t *testing.T) {
@@ -936,8 +936,8 @@ func testStorageRegisterMetricNames(s *Storage) error {
 func TestStorageAddRowsSerial(t *testing.T) {
 	rng := rand.New(rand.NewSource(1))
 	path := "TestStorageAddRowsSerial"
-	retentionMsecs := int64(msecsPerMonth * 10)
-	s := MustOpenStorage(path, retentionMsecs, 1e5, 1e5)
+	retention := 10 * retentionMonth
+	s := MustOpenStorage(path, retention, 1e5, 1e5)
 	if err := testStorageAddRows(rng, s); err != nil {
 		t.Fatalf("unexpected error: %s", err)
 	}
@@ -949,8 +949,8 @@ func TestStorageAddRowsSerial(t *testing.T) {
 
 func TestStorageAddRowsConcurrent(t *testing.T) {
 	path := "TestStorageAddRowsConcurrent"
-	retentionMsecs := int64(msecsPerMonth * 10)
-	s := MustOpenStorage(path, retentionMsecs, 1e5, 1e5)
+	retention := 10 * retentionMonth
+	s := MustOpenStorage(path, retention, 1e5, 1e5)
 	ch := make(chan error, 3)
 	for i := 0; i < cap(ch); i++ {
 		go func(n int) {
@@ -1164,8 +1164,8 @@ func testStorageAddMetrics(s *Storage, workerNum int) error {
 func TestStorageDeleteStaleSnapshots(t *testing.T) {
 	rng := rand.New(rand.NewSource(1))
 	path := "TestStorageDeleteStaleSnapshots"
-	retentionMsecs := int64(msecsPerMonth * 10)
-	s := MustOpenStorage(path, retentionMsecs, 1e5, 1e5)
+	retention := 10 * retentionMonth
+	s := MustOpenStorage(path, retention, 1e5, 1e5)
 	const rowsPerAdd = 1e3
 	const addsCount = 10
 	maxTimestamp := timestampFromTime(time.Now())
diff --git a/lib/storage/table_test.go b/lib/storage/table_test.go
index c664a55fe..49b2881ee 100644
--- a/lib/storage/table_test.go
+++ b/lib/storage/table_test.go
@@ -7,7 +7,7 @@ import (
 
 func TestTableOpenClose(t *testing.T) {
 	const path = "TestTableOpenClose"
-	const retentionMsecs = 123 * msecsPerMonth
+	const retention = 123 * retentionMonth
 
 	if err := os.RemoveAll(path); err != nil {
 		t.Fatalf("cannot remove %q: %s", path, err)
@@ -18,7 +18,7 @@ func TestTableOpenClose(t *testing.T) {
 
 	// Create a new table
 	strg := newTestStorage()
-	strg.retentionMsecs = retentionMsecs
+	strg.retentionMsecs = retention.Milliseconds()
 	tb := mustOpenTable(path, strg)
 
 	// Close it