Repository: VictoriaMetrics/VictoriaMetrics (Go)

Issue:
Wrong vmagent query with dockerswarm_sd_configs filters
**Describe the bug**
I use docker swarm and try to migrate to VM from Prometheus (as full replacement) with **dockerswarm_sd_configs** [filters](https://docs.victoriametrics.com/sd_configs.html#dockerswarm_sd_configs), but vmagent stop working if i specify any filters. Without filters vmagent is working, but prometheus working with filters too.

**To Reproduce**
My docker-compose.yml file
```
version: '3.8'
  vmagent:
    image: victoriametrics/vmagent:v1.85.3
    expose:
      - 8429
    volumes:
      - monitoring_vmagent_data:/vmagentdata
      - /var/run/docker.sock:/var/run/docker.sock:ro
    configs:
      - source: monitoring_prometheus_config
        target: /etc/prometheus/prometheus.yml
    command:
      - "--promscrape.config=/etc/prometheus/prometheus.yml"
      - "--promscrape.config.strictParse=false"
      - "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
    networks:
      - monitoring_network
    restart: always
```


My prometheus.yml file
```
  - job_name: 'swarm_task_exporter'
    scrape_interval: 15s
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        filters:
          - name: "desired-state"
            values: ["running"]
```

**Expected behavior**
If i connect to docker container and using curl create request, everything is ok
`curl --silent -XGET --unix-socket /run/docker.sock http://localhost/services?filters%5Bdesired-state%5D%5B0%5D=running`
But vmagent generate wrong request if i specify any filters

**Logs**
`error	VictoriaMetrics/lib/promscrape/config.go:1007	skipping dockerswarm_sd_config targets for job_name "swarm_task_exporter" because of error: cannot query dockerswarm api for services: unexpected status code returned from "http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D": 400; expecting 200; response body: "{\"message\":\"Invalid filter 'desired-state'\"}\n`"

The logs show that there is a request for 

> http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D

If you decode the parameters, you get 'filters' like this: `{"desired-state":{"running":true}}`
The correct parameters should be 'filters': `{"desired-state": "running"}`

**Version**
Swarm 20.10.22
vmagent v1.85.3


Discussion:
The bug shuld be fixed in the commit 185cdcd81331dda45c3db13611976051a534f9a3 . This commit will be included in the next release.

Re-opening the issue until the bugfix is included in the next release.
`vmagent` should properly encode `filters` arg before passing to to Docker Swarm starting from [v1.86.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Closing the issue as fixed.
This issue has been fixed too in the [v1.79.7 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.7).
@valyala @zekker6  Only partially fixed :)

Generation of query parameters now works correctly, but the request goes to the **wrong url**
In yml file i specify role: **tasks** 
```
  - job_name: 'swarm_task_exporter'
    scrape_interval: 15s
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        filters:
          - name: "desired-state"
            values: ["running"]
```

so request should go to

>  unix/**tasks**?filters=%7B%22desired-state%22%3A%5B%22running%22%5D%7D

But now vmagent create request to 

> unix/**services**?filters=%7B%22desired-state%22%3A%5B%22running%22%5D%7D

 and return error because **services** don't really have such a filter, while **tasks** [do](https://docs.docker.com/engine/api/v1.41/#tag/Task/operation/TaskList)

>error	VictoriaMetrics/lib/promscrape/config.go:1039	skipping dockerswarm_sd_config targets for job_name "swarm_task_exporter" because of error: cannot query dockerswarm api for services: unexpected status code returned from "http://unix/services?filters=%7B%22desired-state%22%3A%5B%22running%22%5D%7D": 400; expecting 200; response body: "{\"message\":\"Invalid filter 'desired-state'\"}\n"

**Version:**
vmagent v1.86.1


Hi @fannyfan414, thank you for the report.

This issue was related to the label discovery for the tasks and have been fixed at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3632

In the meantime, you can build vmagent from sources by using instruction [here]( https://github.com/VictoriaMetrics/VictoriaMetrics#how-to-build-from-sources)
Or use pre-built docker image: `victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190`
The bugfix has been merged in the commit 8537533beb92a72c110b3c13e84fba2b25580637 . This commit will be included in the next release.
Hi @zekker6 , thanks for fixing, in `version victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190` query is correct. But I found another bug or discrepancy with prometheus on the same topic.

I have 2 nodes(servers): redis + redis_exporter on each node. And third node for monitoring.

**Redis exporter** connected to 2 dockerswarm networks - **monitoring_network** and **redis_network**.
**Vmagent** connected to 1 network - **monitoring_network**

My docker-compose.yml
```
version: '3.8'
networks:
  redis_network:
    external:
      name: redis_network
  monitoring_network:
    external:
      name: monitoring_network

services:
  redis_exporter:
    image: oliver006/redis_exporter:v1.45.0
    expose:
      - 9121
#   Hostname important for prometheus metrics
    hostname: "{{.Service.Name}}.{{.Task.Slot}}"
    restart: unless-stopped
    networks:
      - redis_network
      - monitoring_network
    deploy:
      mode: replicated
      replicas: 2
        max_replicas_per_node: 1
        constraints:
          - "node.labels.node_group==redis"
      labels:
        - metrics_port=9121
        - metrics_path=/metrics
        
  vmagent:
    image: victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190
    expose:
      - 8429
    volumes:
      - monitoring_vmagent_data:/vmagentdata
      - /var/run/docker.sock:/var/run/docker.sock:ro
    configs:
      - source: monitoring_prometheus_config
        target: /etc/prometheus/prometheus.yml
    command:
      - "--promscrape.config=/etc/prometheus/prometheus.yml"
      - "--promscrape.config.strictParse=false"
      - "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
    networks:
      - monitoring_network
    restart: always
    deploy:
      mode: replicated
      replicas: 1
      placement:
        max_replicas_per_node: 1
        constraints:
          - "node.labels.node_group==monitoring"
```

Since the redis_exporter container is connected to **2 networks** (monitoring_network and redis_network), it has **two different** IP addresses for each network.

Docker gives Prometheus and vmagent through dockerwarm_sd_configs **all IPs from all networks**, but they can only access containers that are on the **monitoring_network**.

Therefore, I ask prometheus and vmagent to access metrics **not by IP**, but by **hostname** = docker service name + replica index number. It can be seen here **hostname** = {{.Service.Name}}.{{.Task.Slot}} and add labels to **metrics port and path**
```
services:
  redis_exporter:
    image: oliver006/redis_exporter:v1.45.0
    expose:
      - 9121
# Specify hostname
    hostname: "{{.Service.Name}}.{{.Task.Slot}}"
    deploy:
      ....
# Add labels for scrape config
      labels:
        - metrics_port=9121
        - metrics_path=/metrics
```

Therefore I use **relabel_configs** to change **\_\_address\_\_**
```
  - job_name: 'swarm_task_exporter'
    scrape_interval: 15s
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        filters:
          - name: "desired-state"
            values: ["running"]
          - name: "service"
            values: [
              "redis_redis_exporter"
            ]
    relabel_configs:
      # Remove port from "__address__"
      - action: replace
        target_label: __address__
        source_labels:
          - __address__
        regex: ([^:]+)(?::\d+)?
        replacement: $1
      # Generate address from swarm container as "service_name.task_slot"
      - action: replace
        target_label: swarm_container_address
        source_labels:
          - __meta_dockerswarm_service_name
          - __meta_dockerswarm_task_slot
        separator: .
      # Remove task_slot=0 for global replicates containers
      - action: replace
        target_label: swarm_container_address
        source_labels:
          - swarm_container_address
          - __address__
        regex: (.*)\.[0];(.*)
        replacement: $2
      # Replace "__address__" as "swarm_container_address"
      - action: replace
        target_label: __address__
        source_labels:
          - swarm_container_address
          - __meta_dockerswarm_service_label_metrics_port
        separator: ":"
      # Replace metrics_path as label "metrics_path"
      - action: replace
        target_label: __metrics_path__
        source_labels:
          - __meta_dockerswarm_service_label_metrics_path
        regex: (.+)
```

If I open **prometheus UI** service-discovery I see **2 active targets** with  **\_\_address\_\_**="redis_exporter.2:9121" and  **\_\_address\_\_**="redis_redis_exporter.1:9121"

If open **vmagent UI** service-discovery, i see swarm_task_exporter (2/6 active) - 2 up and 4 dropped and **\_\_address\_\_** is **not changed** after relabelling: 
• active **\_\_address\_\_**="10.0.9.229:80" and **\_\_address\_\_**="10.0.19.32:80"
• dropped **\_\_address\_\_**="10.0.19.29:80", **\_\_address\_\_**="10.0.19.32:80", **\_\_address\_\_**="10.0.9.229:80", **\_\_address\_\_**="10.0.9.230:80"

And every scrape_interval i got error in vmagent log because ****\_\_address\_\_** is not relabeled**:
> error	VictoriaMetrics/lib/promscrape/scraper.go:364	skipping duplicate scrape target with identical labels; endpoint=http://redis_exporter.1:9121/metrics, labels={instance="my",job="swarm_task_exporter",swarm_container_address="redis_exporter.1"}; make sure service discovery and relabeling is set up properly; see also https://docs.victoriametrics.com/vmagent.html#troubleshooting; original labels for target1: {__address__="10.0.9.230:80",__meta_dockerswarm_network_id="ule7l71hf291dtzqc7e0a4jfa",__meta_dockerswarm_network_ingress="false",__meta_dockerswarm_network_internal="false",__meta_dockerswarm_network_name="monitoring_network",__meta_dockerswarm_network_scope="swarm",__meta_dockerswarm_node_availability="active",__meta_dockerswarm_node_engine_version="20.10.22",__meta_dockerswarm_node_manager_leader="false",__meta_dockerswarm_node_manager_reachability="reachable",__meta_dockerswarm_node_platform_architecture="x86_64",__meta_dockerswarm_node_platform_os="linux",__meta_dockerswarm_node_role="manager",__meta_dockerswarm_node_status="ready",__meta_dockerswarm_service_id="xxah5tpto4kjbfour35c8piif",__meta_dockerswarm_service_label_com_docker_stack_image="oliver006/redis_exporter:v1.45.0",__meta_dockerswarm_service_label_metrics_path="/metrics",__meta_dockerswarm_service_label_metrics_port="9121",__meta_dockerswarm_service_mode="replicated",__meta_dockerswarm_service_name="redis_exporter",__meta_dockerswarm_service_task_container_hostname="{{.Service.Name}}.{{.Task.Slot}}",__meta_dockerswarm_service_task_container_image="oliver006/redis_exporter:v1.45.0@sha256:b3e847f29ae66fefcaf991e2ccee04b03e57314aab14abadfc87b0eed4453e65",__meta_dockerswarm_service_updating_status="completed",__meta_dockerswarm_task_container_id="a102c38f383d2331344ba9e1254174b75bc764a4c48f81d960b4d5d41aa7b502",__meta_dockerswarm_task_desired_state="running",__meta_dockerswarm_task_id="wf6d35lcmyr30bx9zxdbx4ppj",__meta_dockerswarm_task_slot="1",__meta_dockerswarm_task_state="running",__metrics_path__="/metrics",__scheme__="http",__scrape_interval__="15s",__scrape_timeout__="10s",job="swarm_task_exporter"}; original labels for target2: {__address__="10.0.19.32:80",__meta_dockerswarm_network_id="s8w9wn2j6t8qhfyvl5v8b1ls2",__meta_dockerswarm_network_ingress="false",__meta_dockerswarm_network_internal="false",__meta_dockerswarm_network_name="redis_network",__meta_dockerswarm_network_scope="swarm",__meta_dockerswarm_node_availability="active",__meta_dockerswarm_node_engine_version="20.10.22",__meta_dockerswarm_node_manager_leader="false",__meta_dockerswarm_node_manager_reachability="reachable",__meta_dockerswarm_node_platform_architecture="x86_64",__meta_dockerswarm_node_platform_os="linux",__meta_dockerswarm_node_role="manager",__meta_dockerswarm_node_status="ready",__meta_dockerswarm_service_id="xxah5tpto4kjbfour35c8piif",__meta_dockerswarm_service_label_com_docker_stack_image="oliver006/redis_exporter:v1.45.0",__meta_dockerswarm_service_label_metrics_path="/metrics",__meta_dockerswarm_service_label_metrics_port="9121",__meta_dockerswarm_service_mode="replicated",__meta_dockerswarm_service_name="redis_exporter",__meta_dockerswarm_service_task_container_hostname="{{.Service.Name}}.{{.Task.Slot}}",__meta_dockerswarm_service_task_container_image="oliver006/redis_exporter:v1.45.0@sha256:b3e847f29ae66fefcaf991e2ccee04b03e57314aab14abadfc87b0eed4453e65",__meta_dockerswarm_service_updating_status="completed",__meta_dockerswarm_task_container_id="a102c38f383d2331344ba9e1254174b75bc764a4c48f81d960b4d5d41aa7b502",__meta_dockerswarm_task_desired_state="running",__meta_dockerswarm_task_id="wf6d35lcmyr30bx9zxdbx4ppj",__meta_dockerswarm_task_slot="1",__meta_dockerswarm_task_state="running",__metrics_path__="/metrics",__scheme__="http",__scrape_interval__="15s",__scrape_timeout__="10s",job="swarm_task_exporter"}


@fannyfan414 Could you check open vmagent UI by using `http://vmagent:8429/targets` and use `debug target relabeling` button next to the targets where relabeling rules were not applied?
This will display step-by-step evaluation of relabeling config which might help you find an issue.
@zekker6 

All steps work out correctly, but from somewhere the last step appeared, which, as far as I understand, breaks everything:

`remove labels with __ prefix `



@fannyfan414 Could you check if setting address to `instance` label instead of `__address__` helps?
@zekker6 `instance` is wrong choose, because multiple different containers (different scrape targets) can be on same instance

In previous example I also relabel **\_\_metrics_path\_\_** and last auto-append step delete it

```
  - action: replace
    target_label: __metrics_path__
    source_labels:
      - __meta_dockerswarm_service_label_metrics_path
    regex: (.+)
```
Maybe this auto-added step that removes all __ labels can be made optional via config?

Or it might be easier to remove everything started with __ except these **\_\_address\_\_** and **\_\_metrics_path\_\_** labels.
Or a little more difficult, keep track of the __ lables that have changed and don't delete them.
I've just tested your config with my docker-swarm installation and version from master, it does work for me just as expected:
Source labels:
```
{__address__="10.0.0.5:30000", __meta_dockerswarm_network_id="lmg1s8k7nv2ktuo59qbhxlubc", __meta_dockerswarm_network_ingress="true", __meta_dockerswarm_network_internal="false", __meta_dockerswarm_network_name="ingress", __meta_dockerswarm_network_scope="swarm", __meta_dockerswarm_node_address="10.12.2.15", __meta_dockerswarm_node_availability="active", __meta_dockerswarm_node_engine_version="20.10.22", __meta_dockerswarm_node_hostname="test-vmagent2", __meta_dockerswarm_node_id="r66n93h0n4s8kq47k9l2act4a", __meta_dockerswarm_node_manager_address="", __meta_dockerswarm_node_manager_leader="false", __meta_dockerswarm_node_manager_reachability="", __meta_dockerswarm_node_platform_architecture="x86_64", __meta_dockerswarm_node_platform_os="linux", __meta_dockerswarm_node_role="worker", __meta_dockerswarm_node_status="ready", __meta_dockerswarm_service_endpoint_port_name="", __meta_dockerswarm_service_endpoint_port_publish_mode="ingress", __meta_dockerswarm_service_id="3ggs9pjefwaeg31vzb5ztsxfn", __meta_dockerswarm_service_mode="replicated", __meta_dockerswarm_service_name="my_web", __meta_dockerswarm_service_task_container_hostname="", __meta_dockerswarm_service_task_container_image="nginx:latest@sha256:b8f2383a95879e1ae064940d9a200f67a6c79e710ed82ac42263397367e7cc4e", __meta_dockerswarm_service_updating_status="completed", __meta_dockerswarm_task_container_id="7df7e6cf7fd4f02a78fa0a355717b08ef9b9cff154cf000e960e8d93c486868d", __meta_dockerswarm_task_desired_state="running", __meta_dockerswarm_task_id="4sjrzt8gzdd32xvp85qq4az5k", __meta_dockerswarm_task_port_publish_mode="ingress", __meta_dockerswarm_task_slot="1", __meta_dockerswarm_task_state="running", __metrics_path__="/metrics", __scheme__="http", __scrape_interval__="1m0s", __scrape_timeout__="10s", job="dockerswarm"}
```

Resulted labels:
```
{instance="my_web.1:", job="dockerswarm", swarm_container_address="my_web.1"}
```

Labels starting with `__` are dropped after discovery as expected (same as Prometheus does). You can read [this](https://www.robustperception.io/life-of-a-label/) article which has a great explanation of labels lifecycle.

Could you share your resulting labels set after relabeling? Would be great to see relabel debug flow as well, to find a step where it does something wrong.




@zekker6  Yes, you are right, I apologize for misleading, I just thought that the error was somehow related to the fact that it gives out in the vmagent logs. Prometheus also removes labels started with __

But why i got error every scrape_interval?
Prometheus works in parallel with the same config and does not give any errors.

`error VictoriaMetrics/lib/promscrape/scraper.go:364 skipping duplicate scrape target with identical labels; endpoint=http://redis_exporter.1:9121/metrics, labels={instance="my",job="swarm_task_exporter",swarm_container_address="redis_exporter.1"}; make sure service discovery and relabeling is set up properly; see also https://docs.victoriametrics.com/vmagent.html#troubleshooting; original labels for target1: {address="10.0.9.230:80",__meta_dockerswarm_network_id="ule7l71hf291dtzqc7e0a4jfa",__meta_dockerswarm_network_ingress="false",__meta_dockerswarm_network_internal="false",__meta_dockerswarm_network_name="monitoring_network",__meta_dockerswarm_network_scope="swarm",__meta_dockerswarm_node_availability="active",__meta_dockerswarm_node_engine_version="20.10.22",__meta_dockerswarm_node_manager_leader="false",__meta_dockerswarm_node_manager_reachability="reachable",__meta_dockerswarm_node_platform_architecture="x86_64",__meta_dockerswarm_node_platform_os="linux",__meta_dockerswarm_node_role="manager",__meta_dockerswarm_node_status="ready",__meta_dockerswarm_service_id="xxah5tpto4kjbfour35c8piif",__meta_dockerswarm_service_label_com_docker_stack_image="oliver006/redis_exporter:v1.45.0",__meta_dockerswarm_service_label_metrics_path="/metrics",__meta_dockerswarm_service_label_metrics_port="9121",__meta_dockerswarm_service_mode="replicated",__meta_dockerswarm_service_name="redis_exporter",__meta_dockerswarm_service_task_container_hostname="{{.Service.Name}}.{{.Task.Slot}}",__meta_dockerswarm_service_task_container_image="oliver006/redis_exporter:v1.45.0@sha256:b3e847f29ae66fefcaf991e2ccee04b03e57314aab14abadfc87b0eed4453e65",__meta_dockerswarm_service_updating_status="completed",__meta_dockerswarm_task_container_id="a102c38f383d2331344ba9e1254174b75bc764a4c48f81d960b4d5d41aa7b502",__meta_dockerswarm_task_desired_state="running",__meta_dockerswarm_task_id="wf6d35lcmyr30bx9zxdbx4ppj",__meta_dockerswarm_task_slot="1",__meta_dockerswarm_task_state="running",metrics_path="/metrics",scheme="http",scrape_interval="15s",scrape_timeout="10s",job="swarm_task_exporter"}; original labels for target2: {address="10.0.19.32:80",__meta_dockerswarm_network_id="s8w9wn2j6t8qhfyvl5v8b1ls2",__meta_dockerswarm_network_ingress="false",__meta_dockerswarm_network_internal="false",__meta_dockerswarm_network_name="redis_network",__meta_dockerswarm_network_scope="swarm",__meta_dockerswarm_node_availability="active",__meta_dockerswarm_node_engine_version="20.10.22",__meta_dockerswarm_node_manager_leader="false",__meta_dockerswarm_node_manager_reachability="reachable",__meta_dockerswarm_node_platform_architecture="x86_64",__meta_dockerswarm_node_platform_os="linux",__meta_dockerswarm_node_role="manager",__meta_dockerswarm_node_status="ready",__meta_dockerswarm_service_id="xxah5tpto4kjbfour35c8piif",__meta_dockerswarm_service_label_com_docker_stack_image="oliver006/redis_exporter:v1.45.0",__meta_dockerswarm_service_label_metrics_path="/metrics",__meta_dockerswarm_service_label_metrics_port="9121",__meta_dockerswarm_service_mode="replicated",__meta_dockerswarm_service_name="redis_exporter",__meta_dockerswarm_service_task_container_hostname="{{.Service.Name}}.{{.Task.Slot}}",__meta_dockerswarm_service_task_container_image="oliver006/redis_exporter:v1.45.0@sha256:b3e847f29ae66fefcaf991e2ccee04b03e57314aab14abadfc87b0eed4453e65",__meta_dockerswarm_service_updating_status="completed",__meta_dockerswarm_task_container_id="a102c38f383d2331344ba9e1254174b75bc764a4c48f81d960b4d5d41aa7b502",__meta_dockerswarm_task_desired_state="running",__meta_dockerswarm_task_id="wf6d35lcmyr30bx9zxdbx4ppj",__meta_dockerswarm_task_slot="1",__meta_dockerswarm_task_state="running",metrics_path="/metrics",scheme="http",scrape_interval="15s",scrape_timeout="10s",job="swarm_task_exporter"} `


@fannyfan414 This container was added into discovery with 2 different networks. Since relabeling will produce two identical sets of labels vmagent will write this warning every time.
I can suggest you to keep one of targets since you only need to scrape targets from monitoring network:
```
- action: keep
  if: "{__meta_dockerswarm_network_name='monitoring_network'}"
```
If i open `http://vmagent:8429/service-discovery` i saw:
• active` {__address__="10.0.9.230:80",  __meta_dockerswarm_network_name="monitoring_network", ...}` and `{__address__="10.0.9.229:80",  __meta_dockerswarm_network_name="monitoring_network", ...}` - my really 2 active container in **monitoring_network** 
• dropped `{__address__="10.0.19.29:80", __meta_dockerswarm_network_name="redis_network", ...}` and` {__address__="10.0.19.32:80", __meta_dockerswarm_network_name="redis_network", ...}` - container in **redis_network**, to which vmagent does not have access.

If I understand correctly, since they are marked as **dropped** , then vmagent **should not** query them.

But in the long log that I threw off, I see that there are two IPs: `10.0.9.230:80 `- active and `10.0.19.32:80` - dropped

> @fannyfan414Этот контейнер был добавлен в обнаружение с двумя разными сетями. Поскольку перемаркировка создаст два идентичных набора меток, vmagent будет каждый раз выводить это предупреждение. Я могу предложить вам сохранить одну из целей, так как вам нужно только очищать цели из сети мониторинга:
> 
> ```
> - action: keep
>   if: "{__meta_dockerswarm_network_name='monitoring_network'}"
> ```

Thanks a lot, it helped!
@zekker6 one more question, is it ok that victoriametrics consume more ram than prometheus? (same yaml file with dockerswarm_sd_configs)

![image](https://user-images.githubusercontent.com/32100589/212347552-327ee414-6c2f-4d63-b1d5-094b625b3ee0.png)


VM victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190:
```
command:
  - "--storageDataPath=/storage"
  - "--retentionPeriod=14d"
  - "--graphiteListenAddr=:2003"
  - "--opentsdbListenAddr=:4242"
  - "--httpListenAddr=:8428"
  - "--influxListenAddr=:8089"
```

vmagent:
```
command:
  - "--promscrape.config=/etc/prometheus/prometheus.yml"
  - "--promscrape.config.strictParse=false"
  - "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
```

Prometheus v2.41.0:
```
command:
  - '--config.file=/etc/prometheus/prometheus.yml'
  - '--storage.tsdb.path=/prometheus'
  - '--storage.tsdb.retention.time=15d'
  - '--storage.tsdb.retention.size=4GB'
  - '--web.console.libraries=/etc/prometheus/console_libraries'
  - '--web.console.templates=/etc/prometheus/consoles'
  - '--web.enable-lifecycle'
  - '--web.enable-admin-api'
  - '--log.level=debug'
```

@fannyfan414 If both VM and Prometheus have similar workloads (ingest and query rates) that is not really a common case. Usually VM uses less resources than Prometheus when having similar load.
Could you check if your workload is similar between VM and Prometheus?
If it is similar, you could share [profiling](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#profiling) stats so that we could check if there is a way we can optimize resources usage at VM side.
@zekker6 ready (.pprof not supported by github, upload as zip)
[vm-profiling.zip](https://github.com/VictoriaMetrics/VictoriaMetrics/files/10412673/vm-profiling.zip)

Ingestion rate VM = 1.7K:
![image](https://user-images.githubusercontent.com/32100589/212354446-ed028ba5-71f0-4816-a104-ef2afe7798c6.png)

Prometheus = 1.7K:
![image](https://user-images.githubusercontent.com/32100589/212354611-dcb3ac18-b173-4b06-a338-ef73b13a7033.png)


@fannyfan414 Thanks for sharing profiling stats, as far as I can see memory usage is at expected level. 

Did you consider using discovery and scraping at vmsingle side?
vmsingle and vmagent share this functionality in-common, so when you're using both it will have larger memory footprint as you run 2 components.
In order to make this comparison kind of "fair" you could either run vmsingle with built-in scraping or to use vmagent remote-write for both vmsingle and prometheus.


@zekker6 I didn’t quite understand you, what vmsingle is and why I need it)
To replace prometheus I need to use victoriametrics + vmagent right?

Victoriametrics - stores metrics, vmagent - picks up and puts metrics in victoriametrics 
@fannyfan414 vmsingle is another name for single binary distribution of VictoriaMetric (according to configuration shared above you are using this one).


> Victoriametrics - stores metrics, vmagent - picks up and puts metrics in victoriametrics

This is one of options which we suggest by default as it has gives some better features than using single VictoriaMetrics. One of important features is that vmagent can buffer data in case VictoriaMetrics is not available due to some reason.

In case you need to minimize memory footprint you can configure VictoriaMetrics to scrape metrics just the same as vmagent.
@zekker6 Understood thanks. According to the metrics, for some magic, I recently added another target to yml config, rebooted prometheus and victoria - and it began to consume less memory than prometheus :)
![image](https://user-images.githubusercontent.com/32100589/212644065-7682e694-366f-440d-86a9-78ba9264b6bf.png)

@fannyfan414 That might be related to the caches being reset after reboot. 
By default, VictoriaMetrics uses part of available memory for caching, so after time queries can populate cache to the previous level of RAM usage.
You can [tune caches](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#cache-tuning) configuration, but usually this costs query execution speed since more data will be fetched from disk and use IO instead of RAM.


The original issue related to incorrect `filters` query arg passed to inpcompatible objects should be fixed in [v1.86.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases). Closing this issue as fixed then.

@fannyfan414 , could you open a new issue regarding memory/cpu usage in order to don't mix multiple issues into a single one? Just put a link to this issue in the newly created issue, so readers could get additional details.
FYI, this issue has been also fixed in [v1.79.8 LTS release](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.8).



Patch:
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 2e0bbb241dce5..844814b1357bb 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -29,6 +29,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
 * BUGFIX: properly parse floating-point numbers without integer or fractional parts such as `.123` and `20.` during [data import](https://docs.victoriametrics.com/#how-to-import-time-series-data). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544).
 * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly parse durations with uppercase suffixes such as `10S`, `5MS`, `1W`, etc. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589).
 * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): fix a panic during target discovery when `vmagent` runs with `-promscrape.dropOriginalLabels` command-line flag. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580). The bug has been introduced in [v1.85.0](https://docs.victoriametrics.com/CHANGELOG.html#v1850).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): [dockerswarm_sd_configs](https://docs.victoriametrics.com/sd_configs.html#dockerswarm_sd_configs): properly encode `filters` field. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3579)
 
 
 ## [v1.85.3](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.3)
diff --git a/lib/promscrape/discovery/dockerswarm/api.go b/lib/promscrape/discovery/dockerswarm/api.go
index d0bb79cef07c4..35d2ab97421ca 100644
--- a/lib/promscrape/discovery/dockerswarm/api.go
+++ b/lib/promscrape/discovery/dockerswarm/api.go
@@ -63,20 +63,16 @@ func (cfg *apiConfig) getAPIResponse(path string) ([]byte, error) {
 	return cfg.client.GetAPIResponse(path)
 }
 
+// Encodes filters as `map[string][]string` and then marshals it to JSON.
+// Reference: https://docs.docker.com/engine/api/v1.41/#tag/Task
 func getFiltersQueryArg(filters []Filter) string {
 	if len(filters) == 0 {
 		return ""
 	}
-	m := make(map[string]map[string]bool)
+
+	m := make(map[string][]string)
 	for _, f := range filters {
-		x := m[f.Name]
-		if x == nil {
-			x = make(map[string]bool)
-			m[f.Name] = x
-		}
-		for _, value := range f.Values {
-			x[value] = true
-		}
+		m[f.Name] = f.Values
 	}
 	buf, err := json.Marshal(m)
 	if err != nil {


Test patch:
diff --git a/lib/promscrape/discovery/dockerswarm/api_test.go b/lib/promscrape/discovery/dockerswarm/api_test.go
index fc8888ee59621..114ec43a46f47 100644
--- a/lib/promscrape/discovery/dockerswarm/api_test.go
+++ b/lib/promscrape/discovery/dockerswarm/api_test.go
@@ -22,5 +22,11 @@ func TestGetFiltersQueryArg(t *testing.T) {
 			Name:   "xxx",
 			Values: []string{"aa"},
 		},
-	}, "%7B%22name%22%3A%7B%22bar%22%3Atrue%2C%22foo%22%3Atrue%7D%2C%22xxx%22%3A%7B%22aa%22%3Atrue%7D%7D")
+	}, "%7B%22name%22%3A%5B%22foo%22%2C%22bar%22%5D%2C%22xxx%22%3A%5B%22aa%22%5D%7D")
+	f([]Filter{
+		{
+			Name:   "desired-state",
+			Values: []string{"running", "shutdown"},
+		},
+	}, "%7B%22desired-state%22%3A%5B%22running%22%2C%22shutdown%22%5D%7D")
 }