Repository: cortexlabs/cortex (language: go)

Issue:
Fix istio request rate discrepancy
#### Description

The request rate per second as shown on an API's dashboard in Grafana doesn't reflect the reality. A couple of observations is showing that the `istio_requests_total` metric doesn’t make any sense.

1. It looks like the final value baseline (without factoring in the ever increasing value) is 60 times bigger than the real value.
1. The reported requests/s keeps increasing over time despite the cluster only taking in the same steady-state requests/s.
1. When the requests stop flowing in, the graphs does not go to zero. It just stagnates at that high point.
1. When the ingress gateway scales down, and then back up, the metrics from different ingress pods get mixed up together, thus making it look like there’s a single metric, when instead there are n ingresses. This creates mayhem in what the dashboard reports.

#### Reproducibility

Run a hello world realtime API with some replicas (i.e. 5) on a cortex cluster and hit it with the following load test:

```bash
hey -n 100000000000 -c 20 -m POST -q 1000 $ENDPOINT/hello-world
```

Let it run for a couple of minutes and then cancel the load test. Compare the number of requests as shown in Grafana with those reported by `hey`. There will be a big discrepancy.

#### How to "fix" this

Editing the `istio-stats` `PodMonitor` in the `prometheus` namespace and removing the `metricRelabelings` section will get it to work well. Obviously, that section it still very much needed, so it needs a fix.


Discussion / hints:

Fix patch:
diff --git a/manager/manifests/prometheus-monitoring.yaml b/manager/manifests/prometheus-monitoring.yaml
index 53f096836a..86ef65f15a 100644
--- a/manager/manifests/prometheus-monitoring.yaml
+++ b/manager/manifests/prometheus-monitoring.yaml
@@ -138,7 +138,7 @@ spec:
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
-  name: istio-stats
+  name: istio-ingress-stats
   namespace: prometheus
   labels:
     monitoring.cortex.dev: "istio"
@@ -150,7 +150,7 @@ spec:
       - { key: release, operator: In, values: [ "istio" ]}
   namespaceSelector:
     any: true
-  jobLabel: envoy-stats
+  jobLabel: istio-ingress-stats
   podMetricsEndpoints:
     - path: /stats/prometheus
       interval: 15s
@@ -173,6 +173,11 @@ spec:
         - sourceLabels: [ __meta_kubernetes_pod_name ]
           action: replace
           targetLabel: pod_name
+        - sourceLabels: [ __name__, __meta_kubernetes_pod_label_istio, __meta_kubernetes_pod_name ]
+          action: replace
+          regex: (istio_requests_total)?;(ingressgateway-apis)?;(.+)
+          replacement: $3
+          targetLabel: istioingress_podname
       metricRelabelings:
         - action: keep
           sourceLabels: [__name__]
@@ -183,7 +188,7 @@ spec:
             request_duration_milliseconds_count\
             )"
         - action: labelkeep
-          regex: (__name__|destination_service|response_code|le)
+          regex: (__name__|destination_service|response_code|le|istioingress_podname)
 
 ---
 


Test patch: