opened
vissible
username_0
**Version**
VictoriaMetrics (VM): 1.33.0 (DockerHub: victoriametrics/victoria-metrics:v1.33.0)
Prometheus: v2.15.2 (DockerHub: prom/prometheus:v2.15.2)

**Problem**
The container_cpu_usage_seconds_total is a counter, the value should be always increased.
I can get correct value from prometheus.
```
container_cpu_usage_seconds_total{cpu="total",job="kubernetes-cadvisor",kubernetes_io_hostname="k8s-node01.teambition.corp",usage="monitoring2"}   117783.208311443
```
![image](https://user-images.githubusercontent.com/459022/74024975-e4982e00-49dd-11ea-81df-502352494e05.png)
But it's wrong with VM.
```
$curl "127.0.0.1:8428/api/v1/query?query=container_cpu_usage_seconds_total%7Bimage%3D~%22%22%2Csystemd_service_name%3D%22%22%2Cnamespace%3D%22%22%2Ckubernetes_io_hostname%3D~%22%5Ek8s-node01%5C%5C.teambition%5C%5C.corp%24%22%7D"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"container_cpu_usage_seconds_total","cpu":"total","job":"kubernetes-cadvisor","kubernetes_io_hostname":"k8s-node01.teambition.corp","prom_name":"prometheus-core2","usage":"monitoring2"},"value":[1581071237,"134743.34274985"]}]}}

$curl "127.0.0.1:8428/api/v1/query?query=container_cpu_usage_seconds_total%7Bimage%3D~%22%22%2Csystemd_service_name%3D%22%22%2Cnamespace%3D%22%22%2Ckubernetes_io_hostname%3D~%22%5Ek8s-node01%5C%5C.teambition%5C%5C.corp%24%22%7D"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"container_cpu_usage_seconds_total","cpu":"total","job":"kubernetes-cadvisor","kubernetes_io_hostname":"k8s-node01.teambition.corp","prom_name":"prometheus-core2","usage":"monitoring2"},"value":[1581071239,"857.386208124"]}]}}

$curl "127.0.0.1:8428/api/v1/query?query=container_cpu_usage_seconds_total%7Bimage%3D~%22%22%2Csystemd_service_name%3D%22%22%2Cnamespace%3D%22%22%2Ckubernetes_io_hostname%3D~%22%5Ek8s-node01%5C%5C.teambition%5C%5C.corp%24%22%7D"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"container_cpu_usage_seconds_total","cpu":"total","job":"kubernetes-cadvisor","kubernetes_io_hostname":"k8s-node01.teambition.corp","prom_name":"prometheus-core2","usage":"monitoring2"},"value":[1581071355,"4605.01774016"]}]}}

$curl "127.0.0.1:8428/api/v1/query?query=container_cpu_usage_seconds_total%7Bimage%3D~%22%22%2Csystemd_service_name%3D%22%22%2Cnamespace%3D%22%22%2Ckubernetes_io_hostname%3D~%22%5Ek8s-node01%5C%5C.teambition%5C%5C.corp%24%22%7D"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"container_cpu_usage_seconds_total","cpu":"total","job":"kubernetes-cadvisor","kubernetes_io_hostname":"k8s-node01.teambition.corp","prom_name":"prometheus-core2","usage":"monitoring2"},"value":[1581071414,"14380.84401501"]}]}}
```
![image](https://user-images.githubusercontent.com/459022/74024939-d1855e00-49dd-11ea-9f2e-3e26c53f7cc2.png)
container_cpu_usage_seconds_total return strange value
issue
created
valyala
username_1
@username_0 , thanks for the bug report!

Unfortunately I cannot reproduce it. Could you answer the following questions in order to better understand the issue?

* How many Prometheus instances write data to VictoriaMetrics?
* Could you provide [remote_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) section from Prometheus configs?
* Could you provide the command line used for starting VictoriaMetrics? (remove sensitive information with passwords before posting it)
* Could you provide screenshots from the official Grafana dashboard for VictoriaMetrics? See more information about this [here](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#monitoring).
* Could you check for error logs for both Prometheus and VictoriaMetrics?
comment
created
vissible
username_0
* two prometheus instances, but each one has different jobs.
  container_cpu_usage_seconds_total comes from kubernetes-cadvisor.

```
      - job_name: 'kubernetes-nodes'
      - job_name: 'kubernetes-service-endpoints'
      - job_name: etcd-servers
      - job_name: 'prometheus'
```
```
      - job_name: 'kubernetes-apiservers'
      - job_name: 'kubernetes-cadvisor'
      - job_name: 'kubernetes-http-services'
      - job_name: 'kubernetes-ingresses'
      - job_name: 'kubernetes-pods'
      - job_name: 'prometheus'
```
*  the remote_wtrite config

```
    remote_write:
      - url: "http://x.x.x.x:8428/api/v1/write"
        queue_config:
          max_samples_per_send: 10000
```
* VictoriaMetrics commond line args

```
version: '3.5'
services:
  victoriametrics:
    container_name: victoriametrics
    image: victoriametrics/victoria-metrics:v1.33.0
    ports:
      - 8428:8428
    volumes:
    - /data/db/victoriametrics:/storage
    command:
      - '--storageDataPath=/storage'
      - '--retentionPeriod=1'
      - '--httpListenAddr=:8428'
    restart: always
```
* VictoriaMetrics  screenshots 
![image](https://user-images.githubusercontent.com/459022/74206963-822e7e80-4cb8-11ea-89b0-844dfe4e5772.png)
![image](https://user-images.githubusercontent.com/459022/74206983-9d00f300-4cb8-11ea-8273-e55eaec5b844.png)
![image](https://user-images.githubusercontent.com/459022/74207000-abe7a580-4cb8-11ea-889a-22c87f7288ff.png)
* no error logs found for both Prometheus and VictoriaMetrics
comment
created
valyala
username_1
The provided configs and screenshots from the official dashboard for VictoriaMetrics look OK. Could you answer the following additional questions?

* Does the issue affect only `container_cpu_usage_seconds_total` metric or other metrics too?
* Does the issue affect only metrics obtained from `kubernetes-cadvisor` job or from other jobs too?
* Does the issue affect only metrics obtained from the second Prometheus which scrapes `kubernetes-cadvisor` job or it affects metrics from the first Prometheus too?
* Do you use [relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) in Prometheus configs?
* Do you set `honor_labels` in any [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) inside Prometheus configs?
comment
created
vissible
username_0
* only second Prometheus instance, only `kubernetes-cadvisor` job and only `container_cpu_usage_seconds_total` metric was affected.
* rabel_configs

```
 relabel_configs:
 - action: labelmap
   regex: __meta_kubernetes_node_label_(.+)
 - target_label: __address__
   replacement: kubernetes.default.svc:443
 - source_labels: [__meta_kubernetes_node_name]
   regex: (.+)
   target_label: __metrics_path__
   replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
```

* no honor_labels was set
comment
created
valyala
username_1
So, only a single metric is affected. This looks really weird...

Could you export raw data for this metric on a short period of time where it returns strange values? Raw data can be exported via [/api/v1/export](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/README.md#how-to-export-time-series).

Sorry for late reply - I'm busy with `vmagent` tool at the moment - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/260#issuecomment-565014679 .
comment
created
vissible
username_0
```
curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__="container_cpu_usage_seconds_total",image="",systemd_service_name="",namespace="",kubernetes_io_hostname=~"k8s-node01.teambition.corp"}' -d 'start=1582255040&end=1582256840'  > data.jsonl.gz
```
[data.jsonl.gz](https://github.com/VictoriaMetrics/VictoriaMetrics/files/4234192/data.jsonl.gz)
```
curl -H 'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d 'match[]={__name__="container_cpu_usage_seconds_total",kubernetes_io_hostname=~"k8s-node01.teambition.corp"}' -d 'start=1582255040&end=1582256840'  > data2.jsonl.gz
```
[data2.jsonl.gz](https://github.com/VictoriaMetrics/VictoriaMetrics/files/4234196/data2.jsonl.gz)
comment
created
valyala
username_1
The `data.jsonl.gz` contains unexpected 24 datapoints with different values for each timestamp. For instance, the following values are for the timestamp `1582256830054` (unix timestamp in milliseconds):

```
3072.08584029
96.763481128
5308.36252567
0.00187057
0.008396971
0.165825978
16184.86242209
60380.980416
615023.369593
975.600095051
618099.947855
60380.66141
17134.01953843
102899.31647448
32556.05006456
0.005473415
6050.13130531
31844.76831767
0.004733176
2797.04320866
0.010330897
6050.13336446
864455.844879
12770.80208589
```

While the `data2.jsonl.gz` contains a single value for each timestamp in each time series.

It looks like 24 different time series with the same set of labels - `container_cpu_usage_seconds_total{job="kubernetes-cadvisor",cpu="total",kubernetes_io_hostname="k8s-node01.teambition.corp",prom_name="prometheus-core2",usage="monitoring2"}` - are written into VictoriaMetrics. The most common cause for this is improper metrics' relabeling via `labelmap`, `labeldrop` or `labelkeep` actions, which removes too many labels from the scraped metrics.

Let's look at the `relabel_config` provided above:

```
relabel_configs:
 - action: labelmap
   regex: __meta_kubernetes_node_label_(.+)
 - target_label: __address__
   replacement: kubernetes.default.svc:443
 - source_labels: [__meta_kubernetes_node_name]
   regex: (.+)
   target_label: __metrics_path__
   replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
```

The following two rules can result in multiple metrics with identical set of labels as seen in the `data.jsonl.gz`:

The `labelmap` rule can overwrite existing labels from the scraped metrics with the value matched by `(.+)`:
```
 - action: labelmap
   regex: __meta_kubernetes_node_label_(.+)
```

The `replace` rule substitutes different scrape target addresses (aka `instance` label) with a single constant - `kuebernetes.defualt.svc.:443`. This may result in metrics with identical set of labels for scrape targets belonging to a single `job`:
```
 - target_label: __address__
   replacement: kubernetes.default.svc:443
```

Try adding the following rule before the previous rule in order to preserve the original `instance` label:
```
- source_labels: [__address__]
  target_label: instance
```
comment
created
valyala
username_1
@username_0 , could you try substituting Promtheus with [vmagent](https://victoriametrics.github.io/vmagent.html) and verifying whether the issue persists there? It is likely vmagent would emit `duplicate scrape target` errors. The solution for this error is described in [troubleshooting docs for vmagent](https://victoriametrics.github.io/vmagent.html#troubleshooting).
comment
created
Gagi2k
username_2
I just stumbled over this one as well, but i'm seeing the same results directly using the prometheus Web UI. I don't think VictoriaMetrics is doing anything wrong here, i guess it's cadvisor which reports wrong values.
comment