Project: VictoriaMetrics (Observability and Analysis / Observability)
File: vmauth.md

sort: 5 weight: 5 menu: docs: parent: 'victoriametrics' weight: 5 title: vmauth aliases: /vmauth.html `vmauth` is an HTTP proxy, which can , and requests across components or any other HTTP backends. Just download `vmutils-*` archive from , unpack it and pass the following flag to `vmauth` binary in order to start authorizing and proxying requests: ```sh /path/to/vmauth -auth.config=/path/to/auth/config.yml ``` The `-auth.config` command-line flag must point to valid . See with typical `-auth.config` examples. `vmauth` accepts HTTP requests on port `8427` and proxies them according to the provided . The port can be modified via `-httpListenAddr` command-line flag. See . Docker images for `vmauth` are available . See how `vmauth` used in . Pass `-help` to `vmauth` in order to see all the supported command-line flags with their descriptions. Feel free if you need customized auth proxy for VictoriaMetrics with the support of LDAP, SSO, RBAC, SAML, accounting and rate limiting such as . The following instructs `vmauth` to proxy all the incoming requests to the given backend. For example, requests to `http://vmauth:8427/foo/bar` are proxied to `http://backend/foo/bar`: ```yaml unauthorized_user: url_prefix: "http://backend/" ``` `vmauth` can balance load among multiple backends - see for details. See also and docs. `vmauth` can proxy requests to different backends depending on the requested path, and any HTTP request header. For example, the following instructs `vmauth` to make the following: Requests starting with `/app1/` are proxied to `http://app1-backend/`, while the `/app1/` path prefix is dropped according to . For example, the request to `http://vmauth:8427/app1/foo/bar?baz=qwe` is proxied to `http://app1-backend/foo/bar?baz=qwe`. Requests starting with `/app2/` are proxied to `http://app2-backend/`, while the `/app2/` path prefix is dropped according to . For example, the request to `http://vmauth:8427/app2/index.html` is proxied to `http://app2-backend/index.html`. Other requests are proxied to `http://some-backend/404-page.html`, while the requested path is passed via `request_path` query arg. For example, the request to `http://vmauth:8427/foo/bar?baz=qwe` is proxied to `http://some-backend/404-page.html?request_path=%2Ffoo%2Fbar%3Fbaz%3Dqwe`. ```yaml unauthorized_user: url_map: src_paths: "/app1/.*" dropsrcpathprefixparts: 1 url_prefix: "http://app1-backend/" src_paths: "/app2/.*" dropsrcpathprefixparts: 1 url_prefix: "http://app2-backend/" default_url: http://some-backend/404-page.html ``` See for details. See also and docs. `vmauth` can balance load among multiple HTTP backends in least-loaded round-robin mode. For example, the following instructs `vmauth` to spread load among multiple application instances: ```yaml unauthorized_user: url_prefix: "http://app-instance-1/" "http://app-instance-2/" "http://app-instance-3/" ``` See for more details. See also and docs. If is used for processing , then it is possible to scale the performance of data processing at `vmagent` by spreading load among multiple identically configured `vmagent` instances. This can be done with the following for `vmauth`: ```yaml unauthorized_user: url_map: src_paths: "/prometheus/api/v1/write" "/influx/write" "/api/v1/import" "/api/v1/import/.+" url_prefix: "http://vmagent-1:8429/" "http://vmagent-2:8429/" "http://vmagent-3:8429/" ``` See for more details. See also and docs. accepts incoming data via `vminsert` nodes and processes incoming requests via `vmselect` nodes according to . `vmauth` can be used for balancing both `insert` and `select` requests among `vminsert` and `vmselect` nodes, when the following is used: ```yaml unauthorized_user: url_map: src_paths: "/insert/.+" url_prefix: "http://vminsert-1:8480/" "http://vminsert-2:8480/" "http://vminsert-3:8480/" src_paths: "/select/.+" url_prefix: "http://vmselect-1:8481/" "http://vmselect-2:8481/" ``` See for more details. See also and docs. `vmauth` automatically switches from temporarily unavailable backend to other hot standby backends listed in `url_prefix` if it runs with `-loadBalancingPolicy=firstavailable` command-line flag. The load balancing policy can be overridden at `user` and `urlmap` sections of via `loadbalancingpolicy` option. For example, the following config instructs `vmauth` to proxy requests to `http://victoria-metrics-main:8428/` backend. If this backend becomes unavailable, then `vmauth` starts proxying requests to

If this backend becomes also unavailable, then requests are proxied to the last specified backend - `http://victoria-metrics-standby2:8428/`: ```yaml unauthorized_user: url_prefix: "http://victoria-metrics-main:8428/" "http://victoria-metrics-standby1:8428/" "http://victoria-metrics-standby2:8428/" loadbalancingpolicy: first_available ``` See for more details. See also and docs. `vmauth` can terminate HTTPS requests to backend services when it runs with the following command-line flags: ``` /path/to/vmauth -tls -tlsKeyFile=/path/to/tlskeyfile -tlsCertFile=/path/to/tlscertfile -httpListenAddr=0.0.0.0:443 ``` `-httpListenAddr` sets the address to listen for incoming HTTPS requests `-tls` enables accepting TLS connections at `-httpListenAddr` `-tlsKeyFile` sets the path to TLS certificate key file `-tlsCertFile` sets the path to TLS certificate file See also . See also , and docs. `vmauth` can authorize access to backends depending on the provided request headers. For example, the following proxies requests to if they contain Basic Auth header with the given `username` and `password`: ```yaml users: username: foo password: bar url_prefix: "http://victoria-metrics:8428/" ``` See also , and docs. `vmauth` can authorize access to backends depending on the provided `Bearer Token` request headers. For example, the following proxies requests to if they contain the given `bearer_token`: ```yaml users: bearer_token: ABCDEF url_prefix: "http://victoria-metrics:8428/" ``` See also , and docs. The following instructs proxying `insert` and `select` requests from the user `tenant1` to the `1`, while requests from the user `tenant2` are sent to tenant `2`: ```yaml users: username: tenant1 password: "*" url_map: src_paths: "/api/v1/write" url_prefix: "http://vminsert-backend:8480/insert/1/prometheus/" src_paths: "/api/v1/query" "/api/v1/query_range" "/api/v1/series" "/api/v1/labels" "/api/v1/label/.+/values" url_prefix: "http://vmselect-backend:8481/select/1/prometheus/" username: tenant2 password: "*" url_map: src_paths: "/api/v1/write" url_prefix: "http://vminsert-backend:8480/insert/2/prometheus/" src_paths: "/api/v1/query" "/api/v1/query_range" "/api/v1/series" "/api/v1/labels" "/api/v1/label/.+/values" url_prefix: "http://vmselect-backend:8481/select/2/prometheus/" ``` See also , and docs. can be configured for routing requests to different backends depending on the following in the TLS certificate provided by client: `organizational_unit` aka `OU` `organization` aka `O` `common_name` aka `CN` For example, the following routes requests from clients with `organizational_unit: finance` TLS certificates to `http://victoriametrics-finance:8428` backend, while requests from clients with `organizational_unit: devops` TLS certificates are routed to `http://victoriametrics-devops:8428` backend: ```yaml users: mtls: organizational_unit: finance url_prefix: "http://victoriametrics-finance:8428" mtls: organizational_unit: devops url_prefix: "http://victoriametrics-devops:8428" ``` must be enabled for mTLS-based routing. See also , and docs. `vmauth` can be configured for adding some mandatory query args before proxying requests to backends. For example, the following adds to all the requests, which are proxied to : ```yaml unauthorized_user: urlprefix: "http://victoria-metrics:8428/?extralabel=foo=bar" ``` See also , and docs. By default `vmauth` doesn't drop the path prefix from the original request when proxying the request to the matching backend. Sometimes it is needed to drop path prefix before proxying the request to the backend. This can be done by specifying the number of `/`-delimited prefix parts to drop from the request path via `dropsrcpathprefixparts` option at `url_map` level or at `user` level or . For example, if you need serving requests to at `/vmalert/` path prefix, while serving requests to at `/vmagent/` path prefix, then the following can be used: ```yaml unauthorized_user: url_map: src_paths: "/vmagent/.+" dropsrcpathprefixparts: 1 url_prefix: "http://vmagent-backend:8429/" src_paths: "/vmalert/.+" dropsrcpathprefixparts: 1 url_prefix: "http://vmalert-backend:8880/" ``` `vmauth` supports the following authorization mechanisms: See also , and . `vmauth` can proxy requests to different backends depending on the following parts of HTTP request: See also and . `srcpaths` option can be specified inside `urlmap` in order to route requests by path. The following routes requests to paths starting with `/app1/` to `http://app1-backend`, while requests with paths starting with `/app2` are routed to `http://app2-backend`, and the rest of requests are routed to `http://some-backend/404-page.html`: ```yaml unauthorized_user: url_map: src_paths: "/app1/.*" url_prefix: "http://app1-backend/" src_paths: "/app2/.*" url_prefix: "http://app2-backend/" default_url: http://some-backend/404-page.html ``` `srcpaths` accepts a list of . The incoming request is routed to the given `urlprefix` if the whole requested path matches at least one `src_paths` entry. See also

`srchosts` option can be specified inside `urlmap` in order to route requests by host. The following routes requests to `app1.my-host.com` host to `http://app1-backend`, while routing requests to `app2.my-host.com` host to `http://app2-backend`, and the rest of requests are routed to `http://some-backend/404-page.html`: ```yaml unauthorized_user: url_map: src_hosts: "app1\\.my-host\\.com" url_prefix: "http://app1-backend/" src_hosts: "app2\\.my-host\\.com" url_prefix: "http://app2-backend/" default_url: http://some-backend/404-page.html ``` `srchosts` accepts a list of . The incoming request is routed to the given `urlprefix` if the whole request host matches at least one `src_hosts` entry. `srcqueryargs` option can be specified inside `url_map` in order to route requests by the given . For example, the following routes requests to `http://app1-backend/` if `db=foo` query arg is present in the request, while routing requests with `db` query arg starting with `bar` to `http://app2-backend`, and the rest of requests are routed to `http://some-backend/404-page.html`: ```yaml unauthorized_user: url_map: srcqueryargs: ["db=foo"] url_prefix: "http://app1-backend/" srcqueryargs: ["db=~bar.*"] url_prefix: "http://app2-backend/" default_url: http://some-backend/404-page.html ``` `srcqueryargs` accepts a list of strings in the format `arg=value` or `arg=~regex`. The `arg=value` format means exact matching of the whole `arg` query arg value to the given `value`. The `arg=~regex` format means regex matching of the whole `arg` query arg value to the given `regex`. If at least a single query arg in the request matches at least one `srcqueryargs` entry, then the request is routed to the given `url_prefix`. `srcheaders` option can be specified inside `urlmap` in order to route requests by the given HTTP request header. For example, the following routes requests to `http://app1-backend` if `TenantID` request header equals to `42`, while routing requests to `http://app2-backend` if `TenantID` request header equals to `123:456`, and the rest of requests are routed to `http://some-backend/404-page.html`: ```yaml unauthorized_user: url_map: src_headers: ["TenantID: 42"] url_prefix: "http://app1-backend/" src_headers: ["TenantID: 123:456"] url_prefix: "http://app2-backend/" default_url: http://some-backend/404-page.html ``` If `srcheaders` contains multiple entries, then it is enough to match only a single entry in order to route the request to the given `urlprefix`. Any subset of , , and options can be specified simultaneously in a single `urlmap` entry. In this case the request is routed to the given `urlprefix` if the request matches all the provided configs simultaneously. For example, the following routes requests to `http://app1-backend` if all the conditions mentioned below are simultaneously met: the request path starts with `/app/` the requested hostname ends with `.bar.baz` the request contains `db=abc` query arg the `TenantID` request header equals to `42` ```yaml unauthorized_user: url_map: src_paths: ["/app/.*"] src_hosts: [".+\\.bar\\.baz"] srcqueryargs: ["db=abc"] src_headers: ["TenantID: 42"] url_prefix: "http://app1-backend/" ``` Each `url_prefix` in the can be specified in the following forms: A single url. For example: ```yaml unauthorized_user: url_prefix: 'http://vminsert:8480/insert/0/prometheus/` ``` In this case `vmauth` proxies requests to the specified url. A list of urls. For example: ```yaml unauthorized_user: url_prefix: 'http://vminsert-1:8480/insert/0/prometheus/' 'http://vminsert-2:8480/insert/0/prometheus/' 'http://vminsert-3:8480/insert/0/prometheus/' ``` In this case `vmauth` spreads requests among the specified urls using least-loaded round-robin policy. This guarantees that incoming load is shared uniformly among the specified backends. See also . `vmauth` automatically detects temporarily unavailable backends and spreads incoming queries among the remaining available backends. This allows restarting the backends and peforming mantenance tasks on the backends without the need to remove them from the `url_prefix` list. By default `vmauth` returns backend responses with all the http status codes to the client. It is possible to configure automatic retry of requests at other backends if the backend responds with status code specified in the `-retryStatusCodes` command-line

It is possible to customize the list of http response status codes to retry via `retrystatuscodes` list at `user` and `url_map` level of . For example, the following config re-tries requests on other backends if the current backend returns response with `500` or `502` HTTP status code: ```yaml unauthorized_user: url_prefix: http://vmselect1:8481/ http://vmselect2:8481/ http://vmselect3:8481/ retrystatuscodes: [500, 502] ``` By default `vmauth` uses `least_loaded` policy for spreading incoming requests among available backends. The policy can be changed to `first_available` via `-loadBalancingPolicy` command-line flag. In this case `vmauth` sends all the requests to the first specified backend while it is available. `vmauth` starts sending requests to the next specified backend when the first backend is temporarily unavailable. It is possible to customize the load balancing policy at the `user` and `url_map` level. For example, the following config specifies `first_available` load balancing policy for unauthorized requests: ```yaml unauthorized_user: url_prefix: http://victoria-metrics-main:8428/ http://victoria-metrics-standby:8428/ loadbalancingpolicy: first_available ``` Load balancing feature can be used in the following cases: Balancing the load among multiple `vmselect` and/or `vminsert` nodes in . The following can be used for spreading incoming requests among 3 vmselect nodes and re-trying failed requests or requests with 500 and 502 response status codes: ```yaml unauthorized_user: url_prefix: http://vmselect1:8481/ http://vmselect2:8481/ http://vmselect3:8481/ retrystatuscodes: [500, 502] ``` Sending select queries to the closest availability zone (AZ), while falling back to other AZs with identical data if the closest AZ is unavaialable. For example, the following sends select queries to `https://vmselect-az1/` and uses the `https://vmselect-az2/` as a fallback when `https://vmselect-az1/` is temporarily unavailable or cannot return full responses. See for details about `denypartialresponse` query arg, which is added to requests before they are proxied to backends. ```yaml unauthorized_user: url_prefix: https://vmselect-az1/?denypartialresponse=1 https://vmselect-az2/?denypartialresponse=1 retrystatuscodes: [500, 502, 503] loadbalancingpolicy: first_available ``` Load balancig can be configured independently per each `user` entry and per each `url_map` entry. See for more details. See also , and . By default `vmauth` spreads load among the listed backends at `url_prefix` as described in . Sometimes multiple backend instances can be hidden behind a single hostname. For example, `vmselect-service` hostname may point to a cluster of `vmselect` instances in . So the following config may fail spreading load among available `vmselect` instances, since `vmauth` will send all the requests to the same url, which may end up to a single backend instance: ```yaml unauthorized_user: url_prefix: http://vmselect-service/select/0/prometheus/ ``` There are the following solutions for this issue: To enumerate every `vmselect` hosname or IP in the `url_prefix` list: ```yaml unauthorized_user: url_prefix: http://vmselect-1:8481/select/0/prometheus/ http://vmselect-2:8481/select/0/prometheus/ http://vmselect-3:8481/select/0/prometheus/ ``` This scheme works great, but it needs manual updating of the every time `vmselect` services are restarted, downscaled or upscaled. To set `discoverbackendips: true` option, so `vmauth` automatically discovers IPs behind the given hostname and then spreads load among the discovered IPs: ```yaml unauthorized_user: url_prefix: http://vmselect-service:8481/select/0/prometheus/ discoverbackendips: true ``` If the `url_prefix` contains hostname with `srv+` prefix, then the hostname without `srv+` prefix is automatically resolved via to the list of hostnames with TCP ports, and `vmauth` balances load among the discovered TCP addresses: ```yaml unauthorized_user: url_prefix: "http://srv+vmselect/select/0/prometheus" discoverbackendips: true ``` This functionality is useful for balancing load among backend instances, which run on different TCP ports, since DNS SRV records contain TCP ports. The `discoverbackendips` option can be specified at `user` and `url_map` level in the . It can also be enabled globally via `-discoverBackendIPs` command-line flag. See also . If `url_prefix` contains url with the hostname starting with `srv+` prefix, then `vmauth` uses lookup for the hostname without the `srv+` prefix and selects random TCP address (e.g. hostname plus TCP port) form the resolved

For example, if `some-addr` record contains `some-host:12345` TCP address, then `urlprefix: http://srv+some-addr/some/path` is automatically resolved into `urlprefix: http://some-host:12345/some/path`. The DNS SRV resolution is performed every time new connection to the `url_prefix` backend is established. See also . `vmauth` supports the ability to set and remove HTTP request headers before sending the requests to backends. This is done via `headers` option. For example, the following sets `TenantID: foobar` header to requests proxied to `http://backend:1234/`. It also overrides `X-Forwarded-For` request header with an empty value. This effectively removes the `X-Forwarded-For` header from requests proxied to `http://backend:1234/`: ```yaml unauthorized_user: url_prefix: "http://backend:1234/" headers: "TenantID: foobar" "X-Forwarded-For:" ``` `vmauth` also supports the ability to set and remove HTTP response headers before returning the response from the backend to client. This is done via `response_headers` option. For example, the following sets `Foo: bar` response header and removes `Server` response header before returning the response to client: ```yaml unauthorized_user: url_prefix: "http://backend:1234/" response_headers: "Foo: bar" "Server:" ``` `vmauth` supports dynamic reload of via the following ways: By sending `SIGHUP` signal to `vmauth` process: ``` kill -HUP `pidof vmauth` ``` By querying `/-/reload` endpoint. It is recommended protecting it with `-reloadAuthKey`. See for details. By passing the interval for config check to `-configCheckInterval` command-line flag. `vmauth` may limit the number of concurrent requests according to the following command-line flags: `-maxConcurrentRequests` limits the global number of concurrent requests `vmauth` can serve across all the configured users. `-maxConcurrentPerUserRequests` limits the number of concurrent requests `vmauth` can serve per each configured user. It is also possible to set individual limits on the number of concurrent requests per each user with the `maxconcurrentrequests` option. For example, the following limits the number of concurrent requests from the user `foo` to 10: ```yaml users: username: foo password: bar url_prefix: "http://some-backend/" maxconcurrentrequests: 10 ``` `vmauth` responds with `429 Too Many Requests` HTTP error when the number of concurrent requests exceeds the configured limits. The following related to concurrency limits are exposed by `vmauth`: `vmauthconcurrentrequests_capacity` - the global limit on the number of concurrent requests `vmauth` can serve. It is set via `-maxConcurrentRequests` command-line flag. `vmauthconcurrentrequests_current` - the current number of concurrent requests `vmauth` processes. `vmauthconcurrentrequestslimitreached_total` - the number of requests rejected with `429 Too Many Requests` error because of the global concurrency limit has been reached. `vmauthuserconcurrentrequestscapacity{username="..."}` - the limit on the number of concurrent requests for the given `username`. `vmauthuserconcurrentrequestscurrent{username="..."}` - the current number of concurrent requests for the given `username`. `vmauthuserconcurrentrequestslimitreachedtotal{username="..."}` - the number of requests rejected with `429 Too Many Requests` error because of the concurrency limit has been reached for the given `username`. `vmauthunauthorizeduserconcurrentrequestscapacity` - the limit on the number of concurrent requests for unauthorized users (if `unauthorizeduser` section is used). `vmauthunauthorizeduserconcurrentrequestscurrent` - the current number of concurrent requests for unauthorized users (if `unauthorizeduser` section is used). `vmauthunauthorizeduserconcurrentrequestslimitreached_total` - the number of requests rejected with `429 Too Many Requests` error because of the concurrency limit has been reached for unauthorized users (if `unauthorized_user` section is used). By default `vmauth` uses system settings when performing requests to HTTPS backends specified via `url_prefix` option in the . These settings can be overridden with the following command-line flags: `-backend.tlsInsecureSkipVerify` allows skipping TLS verification when connecting to HTTPS backends. This global setting can be overridden at per-user level inside via `tlsinsecureskip_verify` option. For example: ```yaml username: "foo" url_prefix: "https://localhost" tlsinsecureskip_verify: true ``` `-backend.tlsCAFile` allows specifying the path to TLS Root CA for verifying backend TLS

This global setting can be overridden at per-user level inside via `tlscafile` option. For example: ```yaml username: "foo" url_prefix: "https://localhost" tlscafile: "/path/to/tls/root/ca" ``` `-backend.tlsCertFile` and `-backend.tlsKeyFile` allows specifying client TLS certificate for passing in requests to HTTPS backends, so these ceritifcate could be verified at the backend side (aka ). This global setting can be overridden at per-user level inside via `tlscertfile` and `tlskeyfile` options. For example: ```yaml username: "foo" url_prefix: "https://localhost" tlscertfile: "/path/to/tls/cert" tlskeyfile: "/path/to/tls/key" ``` `-backend.tlsServerName` allows specifying optional for passing in requests to HTTPS backends. This global setting can be overridden at per-user level inside via `tlsservername` option. For example: ```yaml username: "foo" url_prefix: "https://localhost" tlsservername: "foo.bar.com" ``` The `-backend.tlsCAFile`, `-backend.tlsCertFile`, `-backend.tlsKeyFile`, `tlscafile`, `tlscertfile` and `tlskeyfile` may point either to local file or to `http` / `https` url. The file is checked for modifications every second and is automatically re-read when it is updated. of `vmauth` can be configured to allow / deny incoming requests via global and per-user IP filters. For example, the following config allows requests to `vmauth` from `10.0.0.0/24` network and from `1.2.3.4` IP address, while denying requests from `10.0.0.42` IP address: ```yaml users: ip_filters: allow_list: 10.0.0.0/24 1.2.3.4 deny_list: [10.0.0.42] ``` The following config allows requests for the user `foobar` only from the IP `127.0.0.1`: ```yaml users: username: "foobar" password: "*" url_prefix: "http://localhost:8428" ip_filters: allow_list: [127.0.0.1] ``` See config example of using IP filters . `vmauth` reads `username`, `password` and `bearer_token` from `Authorization` request header. It is possible to read these auth tokens from any other request header by specifying it via `-httpAuthHeader` command-line flag. For example, the following command instructs `vmauth` to read auth token from `X-Amz-Firehose-Access-Key` header: ``` ./vmauth -httpAuthHeader='X-Amz-Firehose-Access-Key' ``` It is possible to read auth tokens from multiple headers. For example, the following command instructs `vmauth` to read auth token from both `Authorization` and `X-Amz-Firehose-Access-Key` headers: ``` ./vmauth -httpAuthHeader='Authorization' -httpAuthHeader='X-Amz-Firehose-Access-Key' ``` See also and . `-auth.config` is represented in the following `yml` format: ```yaml users: bearer_token: "XXXX" url_prefix: "http://localhost:8428" auth_token: "Foo XXXX" url_prefix: "http://localhost:8428" bearer_token: "YYY" url_prefix: "http://localhost:8428" headers: "X-Scope-OrgID: foobar" response_headers: "X-Server-Hostname:" # empty value means the header will be removed from the response username: "local-single-node" password: "*" url_prefix: "http://localhost:8428" maxconcurrentrequests: 10 username: "local-single-node2" password: "*" urlprefix: "http://localhost:8428?extralabel=team=dev" username: "local-single-node-with-tls" password: "*" url_prefix: "https://localhost" tlsinsecureskip_verify: true username: "cluster-select-account-123" password: "*" url_prefix: "http://vmselect1:8481/select/123/prometheus" "http://vmselect2:8481/select/123/prometheus" username: "cluster-insert-account-42" password: "*" url_prefix: "http://vminsert1:8480/insert/42/prometheus" "http://vminsert2:8480/insert/42/prometheus" username: "foobar" url_map: src_paths: "/api/v1/query" "/api/v1/query_range" "/api/v1/label/[^/]+/values" url_prefix: "http://vmselect1:8481/select/42/prometheus" "http://vmselect2:8481/select/42/prometheus" retrystatuscodes: [500, 502] src_paths: ["/api/v1/write"] url_prefix: "http://vminsert:8480/insert/42/prometheus" headers: "X-Scope-OrgID: abc" response_headers: "X-Server-Hostname:" # empty value means the header will be removed from the response ip_filters: deny_list: [127.0.0.1] default_url: "http://default1:8888/unsupportedurlhandler" "http://default2:8888/unsupportedurlhandler" unauthorized_user: url_prefix: http://vmselect-az1/?denypartialresponse=1 http://vmselect-az2/?denypartialresponse=1 retrystatuscodes: [503, 500] ip_filters: allow_list: ["1.2.3.0/24", "127.0.0.1"] deny_list: 10.1.0.1 ``` The config may contain `%{ENVVAR}` placeholders, which are substituted by the corresponding `ENVVAR` environment variable values. This may be useful for passing secrets to the config. By default `vmauth` accepts http requests at `8427` port (this port can be changed via `-httpListenAddr` command-line flags). supports the ability to accept requests at this port, by specifying `-tls` and `-mtls` command-line flags. For example, the following command runs `vmauth`, which accepts only mTLS requests at port `8427`: ``` ./vmauth -tls -mtls -auth.config=... ``` By default system-wide is used for verifying client certificates if `-mtls` command-line flag is specified. It is possible to specify custom TLS Root CA via `-mtlsCAFile` command-line flag. See also and

It is expected that all the backend services protected by `vmauth` are located in an isolated private network, so they can be accessed by external users only via `vmauth`. Do not transfer auth headers in plaintext over untrusted networks. Enable https at `-httpListenAddr`. This can be done by passing the following `-tls*` command-line flags to `vmauth`: ```sh -tls Whether to enable TLS for incoming HTTP requests at -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set -tlsCertFile string Path to file with TLS certificate. Used only if -tls is set. Prefer ECDSA certs instead of RSA certs, since RSA certs are slow -tlsKeyFile string Path to file with TLS key. Used only if -tls is set ``` See also . See on how to enable protection at `vmauth`. Alternatively, may be put in front of `vmauth`. It is recommended protecting the following endpoints with authKeys: `/-/reload` with `-reloadAuthKey` command-line flag, so external users couldn't trigger config reload. `/flags` with `-flagsAuthKey` command-line flag, so unauthorized users couldn't get command-line flag values. `/metrics` with `-metricsAuthKey` command-line flag, so unauthorized users couldn't access . `/debug/pprof` with `-pprofAuthKey` command-line flag, so unauthorized users couldn't access . `vmauth` also supports the ability to restrict access by IP - see . See also . `vmauth` supports automatic issuing of TLS certificates via . The following command-line flags must be set in order to enable automatic issuing of TLS certificates: `-httpListenAddr` must be set for listening TCP port `443`. For example, `-httpListenAddr=:443`. This port must be accessible by the . `-tls` must be set in order to accept HTTPS requests at `-httpListenAddr`. Note that `-tlcCertFile` and `-tlsKeyFile` aren't needed when automatic TLS certificate issuing is enabled. `-tlsAutocertHosts` must be set to comma-separated list of hosts, which can be reached via `-httpListenAddr`. TLS certificates are automatically issued for these hosts. `-tlsAutocertEmail` must be set to contact email for the issued TLS certificates. `-tlsAutocertCacheDir` may be set to the directory path for persisting the issued TLS certificates between `vmauth` restarts. If this flag isn't set, then TLS certificates are re-issued on every restart. This functionality can be evaluated for free according to . See also . `vmauth` exports various metrics in Prometheus exposition format at `http://vmauth-host:8427/metrics` page. It is recommended setting up regular scraping of this page either via or via Prometheus-compatible scraper, so the exported metrics could be analyzed later. If you use Google Cloud Managed Prometheus for scraping metrics from VictoriaMetrics components, then pass `-metrics.exposeMetadata` command-line to them, so they add `TYPE` and `HELP` comments per each exposed metric at `/metrics` page. See for details. `vmauth` exports the following metrics per each defined user in : `vmauthuserrequests_total` - the number of requests served for the given `username` `vmauthuserrequestbackenderrors_total` - the number of request errors for the given `username` `vmauthuserrequestdurationseconds` - the duration of requests for the given `username` `vmauthuserconcurrentrequestslimitreachedtotal` - the number of failed requests for the given `username` because of exceeded `vmauthuserconcurrentrequestscapacity` - the maximum number of for the given `username` `vmauthuserconcurrentrequestscurrent` - the current number of for the given `username` By default, per-user metrics contain only `username` label. This label is set to `username` field value at the corresponding user section in the file. It is possible to override the `username` label value by specifying `name` field additionally to `username` field. For example, the following config will result in `vmauthuserrequeststotal{username="foobar"}` instead of `vmauthuserrequeststotal{username="secret_user"}`: ```yaml users: username: "secret_user" name: "foobar" ``` Additional labels for per-user metrics can be specified via `metric_labels`

For example, the following config defines `{dc="eu",team="dev"}` labels additionally to `username="foobar"` label: ```yaml users: username: "foobar" metric_labels: dc: eu team: dev ``` `vmauth` exports the following metrics if `unauthorized_user` section is defined in : `vmauthunauthorizeduserrequeststotal` - the number of unauthorized requests served `vmauthunauthorizeduserrequestbackenderrorstotal` - the number of unauthorized request errors `vmauthunauthorizeduserrequestduration_seconds` - the duration of unauthorized requests `vmauthunauthorizeduserconcurrentrequestslimitreached_total` - the number of failed unauthorized requests because of exceeded `vmauthunauthorizeduserconcurrentrequests_capacity` - the maximum number of `vmauthunauthorizeduserconcurrentrequests_current` - the current number of It is recommended using - `vmauth` is located in `vmutils-*` archives there. . The minimum supported version is Go 1.22. Run `make vmauth` from the root folder of . It builds `vmauth` binary and puts it into the `bin` folder. . Run `make vmauth-prod` from the root folder of . It builds `vmauth-prod` binary and puts it into the `bin` folder. Run `make package-vmauth`. It builds `victoriametrics/vmauth:<PKG_TAG>` docker image locally. `<PKG_TAG>` is auto-generated image tag, which depends on source code in the repository. The `<PKGTAG>` may be manually set via `PKGTAG=foobar make package-vmauth`. The base docker image is but it is possible to use any other base image by setting it via `<ROOT_IMAGE>` environment variable. For example, the following command builds the image on top of image: ```sh ROOT_IMAGE=scratch make package-vmauth ``` `vmauth` provides handlers for collecting the following : Memory profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed): ```sh curl http://0.0.0.0:8427/debug/pprof/heap > mem.pprof ``` CPU profile. It can be collected with the following command (replace `0.0.0.0` with hostname if needed): ```sh curl http://0.0.0.0:8427/debug/pprof/profile > cpu.pprof ``` The command for collecting CPU profile waits for 30 seconds before returning. The collected profiles may be analyzed with . It is safe sharing the collected profiles from security point of view, since they do not contain sensitive information. Pass `-help` command-line arg to `vmauth` in order to see all the configuration options: ```sh ./vmauth -help vmauth authenticates and authorizes incoming requests and proxies them to VictoriaMetrics. See the docs at https://docs.victoriametrics.com/vmauth/ . -auth.config string Path to auth config. It can point either to local file or to http url. See https://docs.victoriametrics.com/vmauth/ for details on the format of this auth config -backend.TLSCAFile string Optional path to TLS root CA file, which is used for TLS verification when connecting to backends over HTTPS. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup -backend.TLSCertFile string Optional path to TLS client certificate file, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup -backend.TLSKeyFile string Optional path to TLS client key file, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup -backend.TLSServerName string Optional TLS ServerName, which must be sent to HTTPS backend. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup -backend.tlsInsecureSkipVerify Whether to skip TLS verification when connecting to backends over HTTPS. See https://docs.victoriametrics.com/vmauth/#backend-tls-setup -configCheckInterval duration interval for config file re-read. Zero value disables config re-reading. By default, refreshing is disabled, send SIGHUP for config refresh. -discoverBackendIPs Whether to discover backend IPs via periodic DNS queries to hostnames specified in urlprefix. This may be useful when urlprefix points to a hostname with dynamically scaled instances behind it. See https://docs.victoriametrics.com/vmauth/#discovering-backend-ips -discoverBackendIPsInterval duration The interval for re-discovering backend IPs if -discoverBackendIPs command-line flag is set. Too low value may lead to DNS errors (default 10s) -enableTCP6 Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used -envflag.enable Whether to enable reading flags from environment variables in addition to the command line. Command line flag values have priority over values from environment vars. Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details

string Prefix for environment variables if -envflag.enable is set -eula Deprecated, please use -license or -licenseFile flags instead. By specifying this flag, you confirm that you have an enterprise license and accept the ESA https://victoriametrics.com/legal/esa/ . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -failTimeout duration Sets a delay period for load balancing to skip a malfunctioning backend (default 3s) -filestream.disableFadvise Whether to disable fadvise() syscall when reading large data files. The fadvise() syscall prevents from eviction of recently accessed data from OS page cache during background merges and backups. In some rare cases it is better to disable the syscall if it uses too much CPU -flagsAuthKey value Auth key for /flags endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings Flag value can be read from the given file when using -flagsAuthKey=file:///abs/path/to/file or -flagsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -flagsAuthKey=http://host/path or -flagsAuthKey=https://host/path -fs.disableMmap Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread() -http.connTimeout duration Incoming connections to -httpListenAddr are closed after the configured timeout. This may help evenly spreading load among a cluster of services behind TCP-level load balancer. Zero value disables closing of incoming connections (default 2m0s) -http.disableResponseCompression Disable compression of HTTP responses to save CPU resources. By default, compression is enabled to save network bandwidth -http.header.csp string Value for 'Content-Security-Policy' header, recommended: "default-src 'self'" -http.header.frameOptions string Value for 'X-Frame-Options' header -http.header.hsts string Value for 'Strict-Transport-Security' header, recommended: 'max-age=31536000; includeSubDomains' -http.idleConnTimeout duration Timeout for incoming idle http connections (default 1m0s) -http.maxGracefulShutdownDuration duration The maximum duration for a graceful shutdown of the HTTP server. A highly loaded server may require increased value for a graceful shutdown (default 7s) -http.pathPrefix string An optional prefix to add to all the paths handled by http server. For example, if '-http.pathPrefix=/foo/bar' is set, then all the http requests will be handled on '/foo/bar/*' paths. This may be useful for proxied requests. See https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus -http.shutdownDelay duration Optional delay before http server shutdown. During this delay, the server returns non-OK responses from /health page, so load balancers can route new requests to other servers -httpAuth.password value Password for HTTP server's Basic Auth. The authentication is disabled if -httpAuth.username is empty Flag value can be read from the given file when using -httpAuth.password=file:///abs/path/to/file or -httpAuth.password=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -httpAuth.password=http://host/path or -httpAuth.password=https://host/path -httpAuth.username string Username for HTTP server's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password -httpAuthHeader array HTTP request header to use for obtaining authorization tokens. By default auth tokens are read from Authorization request header Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -httpListenAddr array TCP address to listen for incoming http requests. See also -tls and -httpListenAddr.useProxyProtocol Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -httpListenAddr.useProxyProtocol array Whether to use proxy protocol for connections accepted at the corresponding -httpListenAddr . See https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt . With enabled proxy protocol http server cannot serve regular /metrics

Use -pushmetrics.url for metrics pushing Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -internStringCacheExpireDuration duration The expiry duration for caches for interned strings. See https://en.wikipedia.org/wiki/String_interning . See also -internStringMaxLen and -internStringDisableCache (default 6m0s) -internStringDisableCache Whether to disable caches for interned strings. This may reduce memory usage at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringCacheExpireDuration and -internStringMaxLen -internStringMaxLen int The maximum length for strings to intern. A lower limit may save memory at the cost of higher CPU usage. See https://en.wikipedia.org/wiki/String_interning . See also -internStringDisableCache and -internStringCacheExpireDuration (default 500) -license string Lisense key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/ . Trial Enterprise license can be obtained from https://victoriametrics.com/products/enterprise/trial/ . This flag is available only in Enterprise binaries. The license key can be also passed via file specified by -licenseFile command-line flag -license.forceOffline Whether to enable offline verification for VictoriaMetrics Enterprise license key, which has been passed either via -license or via -licenseFile command-line flag. The issued license key must support offline verification feature. Contact info@victoriametrics.com if you need offline license verification. This flag is avilable only in Enterprise binaries -licenseFile string Path to file with license key for VictoriaMetrics Enterprise. See https://victoriametrics.com/products/enterprise/ . Trial Enterprise license can be obtained from https://victoriametrics.com/products/enterprise/trial/ . This flag is available only in Enterprise binaries. The license key can be also passed inline via -license command-line flag -loadBalancingPolicy string The default load balancing policy to use for backend urls specified inside urlprefix section. Supported policies: leastloaded, firstavailable. See https://docs.victoriametrics.com/vmauth/#load-balancing (default "leastloaded") -logInvalidAuthTokens Whether to log requests with invalid auth tokens. Such requests are always counted at vmauthhttprequesterrorstotal{reason="invalidauthtoken"} metric, which is exposed at /metrics page -loggerDisableTimestamps Whether to disable writing timestamps in logs -loggerErrorsPerSecondLimit int Per-second limit on the number of ERROR messages. If more than the given number of errors are emitted per second, the remaining errors are suppressed. Zero values disable the rate limit -loggerFormat string Format for logs. Possible values: default, json (default "default") -loggerJSONFields string Allows renaming fields in JSON formatted logs. Example: "ts:timestamp,msg:message" renames "ts" to "timestamp" and "msg" to "message". Supported fields: ts, level, caller, msg -loggerLevel string Minimum level of errors to log. Possible values: INFO, WARN, ERROR, FATAL, PANIC (default "INFO") -loggerMaxArgLen int The maximum length of a single logged argument. Longer arguments are replaced with 'argstart..argend', where 'argstart' and 'argend' is prefix and suffix of the arg with the length not exceeding -loggerMaxArgLen / 2 (default 1000) -loggerOutput string Output for the logs. Supported values: stderr, stdout (default "stderr") -loggerTimezone string Timezone to use for timestamps in logs. Timezone must be a valid IANA Time Zone. For example: America/New_York, Europe/Berlin, Etc/GMT+3 or Local (default "UTC") -loggerWarnsPerSecondLimit int Per-second limit on the number of WARN messages. If more than the given number of warns are emitted per second, then the remaining warns are suppressed. Zero values disable the rate limit -maxConcurrentPerUserRequests int The maximum number of concurrent requests vmauth can process per each configured user. Other requests are rejected with '429 Too Many Requests' http status code. See also -maxConcurrentRequests command-line option and maxconcurrentrequests option in per-user config (default 300) -maxConcurrentRequests int The maximum number of concurrent requests vmauth can process. Other requests are rejected with '429 Too Many Requests' http status code. See also -maxConcurrentPerUserRequests and -maxIdleConnsPerBackend command-line options (default 1000) -maxIdleConnsPerBackend int The maximum number of idle connections vmauth can open per each backend

See also -maxConcurrentRequests (default 100) -maxRequestBodySizeToRetry size The maximum request body size, which can be cached and re-tried at other backends. Bigger values may require more memory Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 16384) -memory.allowedBytes size Allowed size of system memory VictoriaMetrics caches may occupy. This option overrides -memory.allowedPercent if set to a non-zero value. Too low a value may increase the cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache resulting in higher disk IO usage Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 0) -memory.allowedPercent float Allowed percent of system memory VictoriaMetrics caches may occupy. See also -memory.allowedBytes. Too low a value may increase cache miss rate usually resulting in higher CPU and disk IO usage. Too high a value may evict too much data from the OS page cache which will result in higher disk IO usage (default 60) -metrics.exposeMetadata Whether to expose TYPE and HELP metadata at the /metrics page, which is exposed at -httpListenAddr . The metadata may be needed when the /metrics page is consumed by systems, which require this information. For example, Managed Prometheus in Google Cloud - https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type -metricsAuthKey value Auth key for /metrics endpoint. It must be passed via authKey query arg. It overrides httpAuth.* settings Flag value can be read from the given file when using -metricsAuthKey=file:///abs/path/to/file or -metricsAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -metricsAuthKey=http://host/path or -metricsAuthKey=https://host/path -mtls array Whether to require valid client certificate for https requests to the corresponding -httpListenAddr . This flag works only if -tls flag is set. See also -mtlsCAFile . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -mtlsCAFile array Optional path to TLS Root CA for verifying client certificates at the corresponding -httpListenAddr when -mtls is enabled. By default the host system TLS Root CA is used for client certificate verification. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -pprofAuthKey value Auth key for /debug/pprof/ endpoints. It must be passed via authKey query arg. It overrides httpAuth. settings Flag value can be read from the given file when using -pprofAuthKey=file:///abs/path/to/file or -pprofAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -pprofAuthKey=http://host/path or -pprofAuthKey=https://host/path -pushmetrics.disableCompression Whether to disable request body compression when pushing metrics to every -pushmetrics.url -pushmetrics.extraLabel array Optional labels to add to metrics pushed to every -pushmetrics.url . For example, -pushmetrics.extraLabel='instance="foo"' adds instance="foo" label to all the metrics pushed to every -pushmetrics.url Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -pushmetrics.header array Optional HTTP request header to send to every -pushmetrics.url . For example, -pushmetrics.header='Authorization: Basic foobar' adds 'Authorization: Basic foobar' header to every request to every -pushmetrics.url Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces.

duration Interval for pushing metrics to every -pushmetrics.url (default 10s) -pushmetrics.url array Optional URL to push metrics exposed at /metrics page. See https://docs.victoriametrics.com/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storage Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -reloadAuthKey value Auth key for /-/reload http endpoint. It must be passed as authKey=... Flag value can be read from the given file when using -reloadAuthKey=file:///abs/path/to/file or -reloadAuthKey=file://./relative/path/to/file . Flag value can be read from the given http/https url when using -reloadAuthKey=http://host/path or -reloadAuthKey=https://host/path -responseTimeout duration The timeout for receiving a response from backend (default 5m0s) -retryStatusCodes array Comma-separated list of default HTTP response status codes when vmauth re-tries the request on other backends. See https://docs.victoriametrics.com/vmauth/#load-balancing for details (default 0) Supports array of values separated by comma or specified via multiple flags. Empty values are set to default value. -tls array Whether to enable TLS for incoming HTTP requests at the given -httpListenAddr (aka https). -tlsCertFile and -tlsKeyFile must be set if -tls is set. See also -mtls Supports array of values separated by comma or specified via multiple flags. Empty values are set to false. -tlsAutocertCacheDir string Directory to store TLS certificates issued via Let's Encrypt. Certificates are lost on restarts if this flag isn't set. This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertEmail string Contact email for the issued Let's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir .This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ -tlsAutocertHosts array Optional hostnames for automatic issuing of Let's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/enterprise/ Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCertFile array Path to file with TLS certificate for the corresponding -httpListenAddr if -tls is set. Prefer ECDSA certs instead of RSA certs as RSA certs are slower. The provided certificate file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsCipherSuites array Optional list of TLS cipher suites for incoming requests over HTTPS if -tls is set. See the list of supported cipher suites at https://pkg.go.dev/crypto/tls#pkg-constants Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsKeyFile array Path to file with TLS key for the corresponding -httpListenAddr if -tls is set. The provided key file is automatically re-read every second, so it can be dynamically updated. See also -tlsAutocertHosts Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -tlsMinVersion array Optional minimum TLS version to use for the corresponding -httpListenAddr if -tls is set. Supported values: TLS10, TLS11, TLS12, TLS13 Supports an array of values separated by comma or specified via multiple flags. Value can contain comma inside single-quoted or double-quoted string, {}, [] and () braces. -version Show VictoriaMetrics version ```