Repository: ChainSafe/lodestar
Pull request #7836: chore: update dashboards for gossip_block_elapsed_time_till_received labels
Author: login: KatyaRyazantseva
Created: 2025-05-15T14:29:02Z | Merged: 2025-05-21T09:58:33Z

**Motivation**

In the validator monitor metrics refactoring done in #7722, labels were added to the `gossip_block_elapsed_time_till_received` metric. Several dashboards need to be updated to reflect these changes.

**Description**

The following dashboards, which include the `gossip_block_elapsed_time_till_received` metric, have been updated to use the new labels:
- Lodestar - block processor
- Lodestar - debug gossipsub (gossip label filter)
- Lodestar - networking

Closes #issue_number

**Steps to test or reproduce**

Verify the labels on the following graphs:
- Lodestar - block processor: Avg block recv delay
- Lodestar - debug gossipsub: Gossip Block Received Delay
- Lodestar - networking: Gossip Block process time, Gossip Block Received Delay, Received Time


Diff:
diff --git a/dashboards/lodestar_block_processor.json b/dashboards/lodestar_block_processor.json
index 84fe28531bd8..58cfed7581df 100644
--- a/dashboards/lodestar_block_processor.json
+++ b/dashboards/lodestar_block_processor.json
@@ -4838,7 +4838,7 @@
           "calcs": [],
           "displayMode": "list",
           "placement": "bottom",
-          "showLegend": false
+          "showLegend": true
         },
         "tooltip": {
           "mode": "single",
@@ -4856,7 +4856,8 @@
           "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval]) / rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "format": "heatmap",
           "instant": false,
-          "legendFormat": "time",
+          "interval": "",
+          "legendFormat": "{{source}}",
           "range": true,
           "refId": "A"
         }
diff --git a/dashboards/lodestar_debug_gossipsub.json b/dashboards/lodestar_debug_gossipsub.json
index 26cd9b9b791a..414bf351650c 100644
--- a/dashboards/lodestar_debug_gossipsub.json
+++ b/dashboards/lodestar_debug_gossipsub.json
@@ -90,6 +90,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -218,10 +219,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.1.1",
+      "pluginVersion": "10.4.1",
       "targets": [
         {
           "datasource": {
@@ -287,10 +290,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "value"
+        "textMode": "value",
+        "wideLayout": true
       },
-      "pluginVersion": "10.1.1",
+      "pluginVersion": "10.4.1",
       "targets": [
         {
           "datasource": {
@@ -339,10 +344,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "name"
+        "textMode": "name",
+        "wideLayout": true
       },
-      "pluginVersion": "10.1.1",
+      "pluginVersion": "10.4.1",
       "targets": [
         {
           "datasource": {
@@ -394,10 +401,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "name"
+        "textMode": "name",
+        "wideLayout": true
       },
-      "pluginVersion": "10.1.1",
+      "pluginVersion": "10.4.1",
       "targets": [
         {
           "datasource": {
@@ -413,7 +422,6 @@
         }
       ],
       "title": "Lodestar version",
-      "transformations": [],
       "type": "stat"
     },
     {
@@ -449,10 +457,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "name"
+        "textMode": "name",
+        "wideLayout": true
       },
-      "pluginVersion": "10.1.1",
+      "pluginVersion": "10.4.1",
       "targets": [
         {
           "datasource": {
@@ -480,6 +490,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -692,6 +703,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -776,6 +788,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -948,6 +961,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1029,6 +1043,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1137,6 +1152,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1220,6 +1236,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1304,6 +1321,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1365,10 +1383,12 @@
             "type": "prometheus",
             "uid": "${DS_PROMETHEUS}"
           },
+          "editorMode": "code",
           "exemplar": false,
-          "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval])\n/\nrate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
+          "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum{source=\"gossip\"}[$rate_interval])\n/\nrate(lodestar_gossip_block_elapsed_time_till_received_count{source=\"gossip\"}[$rate_interval])",
           "interval": "",
           "legendFormat": "till received",
+          "range": true,
           "refId": "A"
         },
         {
@@ -1398,6 +1418,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1508,6 +1529,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -1588,6 +1610,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -8764,8 +8787,7 @@
     }
   ],
   "refresh": "10s",
-  "schemaVersion": 38,
-  "style": "dark",
+  "schemaVersion": 39,
   "tags": [
     "lodestar",
     "debug"
diff --git a/dashboards/lodestar_networking.json b/dashboards/lodestar_networking.json
index 4a96b9f5c220..291888304547 100644
--- a/dashboards/lodestar_networking.json
+++ b/dashboards/lodestar_networking.json
@@ -3327,7 +3327,7 @@
           "exemplar": false,
           "expr": "(\n  rate(lodestar_gossip_block_elapsed_time_till_processed_sum[$rate_interval])\n  -\n  rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval])\n)\n/\nrate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "interval": "",
-          "legendFormat": "processed minus received",
+          "legendFormat": "processed minus received {{source}}",
           "range": true,
           "refId": "A"
         },
@@ -3421,10 +3421,12 @@
             "type": "prometheus",
             "uid": "${DS_PROMETHEUS}"
           },
+          "editorMode": "code",
           "exemplar": false,
           "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval])\n/\nrate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "interval": "",
-          "legendFormat": "till received",
+          "legendFormat": "till received {{source}}",
+          "range": true,
           "refId": "A"
         },
         {
@@ -5370,8 +5372,8 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"0.5\"}[$rate_interval]))\n/\nsum(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
-          "legendFormat": "<=0.5s",
+          "expr": "sum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"0.5\"}[$rate_interval]))\n/\nsum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
+          "legendFormat": "<=0.5s {{source}}",
           "range": true,
           "refId": "A"
         },
@@ -5381,9 +5383,9 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"1\"}[$rate_interval]))\n/\nsum(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
+          "expr": "sum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"1\"}[$rate_interval]))\n/\nsum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
           "hide": false,
-          "legendFormat": "<=1s",
+          "legendFormat": "<=1s {{source}}",
           "range": true,
           "refId": "B"
         },
@@ -5393,9 +5395,9 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"2\"}[$rate_interval]))\n/\nsum(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
+          "expr": "sum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"2\"}[$rate_interval]))\n/\nsum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
           "hide": false,
-          "legendFormat": "<=2s",
+          "legendFormat": "<=2s {{source}}",
           "range": true,
           "refId": "C"
         },
@@ -5405,9 +5407,10 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"4\"}[$rate_interval]))\n/\nsum(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
+          "expr": "sum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"4\"}[$rate_interval]))\n/\nsum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
           "hide": false,
-          "legendFormat": "<=4s",
+          "interval": "",
+          "legendFormat": "<=4s {{source}}",
           "range": true,
           "refId": "D"
         },
@@ -5417,9 +5420,10 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"6\"}[$rate_interval]))\n/\nsum(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
+          "expr": "sum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_bucket{le=\"6\"}[$rate_interval]))\n/\nsum by (source)(rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval]))",
           "hide": false,
-          "legendFormat": "<=6s",
+          "interval": "",
+          "legendFormat": "<=6s {{source}}",
           "range": true,
           "refId": "E"
         }


Reviews:
user: login: nflaig
body: the diff in this PR looks really wrong to me
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T09:01:09Z
user: login: KatyaRyazantseva
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T10:50:47Z
user: login: nflaig
body: Diff looks good to me, but should add screenshots of modified panels to this PR. Also consider renaming PR title to reflect changes you did.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T10:52:40Z
user: login: KatyaRyazantseva
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T11:00:03Z
user: login: nflaig
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T11:02:45Z
user: login: nflaig
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T11:50:23Z

Review comments:
diff_hunk: @@ -7,13 +7,6 @@
       "pluginId": "prometheus",
       "pluginName": "Prometheus",
       "type": "datasource"
-    },
-    {
-      "description": "",
path: dashboards/lodestar_block_processor.json
user: login: nflaig
body: why is this removed?
created_at: 2025-05-16T09:00:00Z
author_association: MEMBER
diff_hunk: @@ -2187,10 +2194,6 @@
     },
     {
       "collapsed": false,
-      "datasource": {
path: dashboards/lodestar_block_processor.json
user: login: nflaig
body: why are these removed? seems like something went wrong, how did you export the dashboard?
created_at: 2025-05-16T09:00:27Z
author_association: MEMBER
diff_hunk: @@ -2232,19 +2226,7 @@
         "content": "Verifies signature sets in a thread pool of workers. Must ensure that signatures are verified fast and efficiently.",
         "mode": "markdown"
       },
-      "pluginVersion": "10.4.1",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(lodestar_bls_thread_pool_time_seconds_sum[$rate_interval])",
path: dashboards/lodestar_block_processor.json
user: login: nflaig
body: why is this removed?
created_at: 2025-05-16T09:00:41Z
author_association: MEMBER
diff_hunk: @@ -2232,19 +2226,7 @@
         "content": "Verifies signature sets in a thread pool of workers. Must ensure that signatures are verified fast and efficiently.",
         "mode": "markdown"
       },
-      "pluginVersion": "10.4.1",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(lodestar_bls_thread_pool_time_seconds_sum[$rate_interval])",
path: dashboards/lodestar_block_processor.json
user: login: KatyaRyazantseva
body: This part was automatically removed by the script, likely as part of a Grafana 11.6.0 fix. This is the `About` section, which is of type `text`. The query doesn't appear to be necessary for this section, and visually, the dashboard remains unchanged.
Previous version:
```
      "id": 154,
      "options": {
        "code": {
          "language": "plaintext",
          "showLineNumbers": false,
          "showMiniMap": false
        },
        "content": "Verifies signature sets in a thread pool of workers. Must ensure that signatures are verified fast and efficiently.",
        "mode": "markdown"
      },
      "pluginVersion": "10.4.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "expr": "rate(lodestar_bls_thread_pool_time_seconds_sum[$rate_interval])",
          "interval": "",
          "legendFormat": "{{workerId}}",
          "refId": "A"
        }
      ],
      "title": "About",
      "type": "text"
```
  
Current version:
```
      "id": 154,
      "options": {
        "code": {
          "language": "plaintext",
          "showLineNumbers": false,
          "showMiniMap": false
        },
        "content": "Verifies signature sets in a thread pool of workers. Must ensure that signatures are verified fast and efficiently.",
        "mode": "markdown"
      },
      "pluginVersion": "11.6.0",
      "title": "About",
      "type": "text"
```   
The section is still there:
<img width="1125" alt="Screenshot 2025-05-16 at 12 22 24" src="https://github.com/user-attachments/assets/1a652fd4-728b-4c7e-ba19-b5860c2008b7" /> 
created_at: 2025-05-16T10:50:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4856,7 +4856,8 @@
           "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval]) / rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "format": "heatmap",
           "instant": false,
-          "legendFormat": "time",
+          "interval": "",
+          "legendFormat": "{{source}}",
path: dashboards/lodestar_block_processor.json
user: login: nflaig
body: should this be?

```suggestion
          "legendFormat": "{{source}} time",
```
created_at: 2025-05-20T10:47:17Z
author_association: MEMBER
diff_hunk: @@ -4856,7 +4856,8 @@
           "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval]) / rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "format": "heatmap",
           "instant": false,
-          "legendFormat": "time",
+          "interval": "",
+          "legendFormat": "{{source}}",
path: dashboards/lodestar_block_processor.json
user: login: KatyaRyazantseva
body: I deleted it because the legend toggle was not enabled, so, it wasn't displayed. But I can bring it back.
created_at: 2025-05-20T11:00:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4856,7 +4856,8 @@
           "expr": "rate(lodestar_gossip_block_elapsed_time_till_received_sum[$rate_interval]) / rate(lodestar_gossip_block_elapsed_time_till_received_count[$rate_interval])",
           "format": "heatmap",
           "instant": false,
-          "legendFormat": "time",
+          "interval": "",
+          "legendFormat": "{{source}}",
path: dashboards/lodestar_block_processor.json
user: login: nflaig
body: Not sure if it's better, but that's why it important to share screenshots on PRs that modify panels
created_at: 2025-05-20T11:02:45Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-05-16T13:13:43Z
body: ## [Codecov](https://app.codecov.io/gh/ChainSafe/lodestar/pull/7836?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ChainSafe) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 55.74%. Comparing base [(`d1a37bc`)](https://app.codecov.io/gh/ChainSafe/lodestar/commit/d1a37bc1f6b1ba29aa94f1a396ce29aec1dbac16?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ChainSafe) to head [(`9d2cb0a`)](https://app.codecov.io/gh/ChainSafe/lodestar/commit/9d2cb0a8bf8263afc407ebf61d98895859062b62?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ChainSafe).
> Report is 8 commits behind head on unstable.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##           unstable    #7836   +/-   ##
=========================================
  Coverage     55.74%   55.74%           
=========================================
  Files           823      823           
  Lines         58084    58084           
  Branches       4464     4464           
=========================================
  Hits          32377    32377           
  Misses        25640    25640           
  Partials         67       67           
```

</details>
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: KatyaRyazantseva
created_at: 2025-05-20T11:20:10Z
body: Lodestar - block processor: Avg block recv delay
<img width="725" alt="Screenshot 2025-05-20 at 13 07 14" src="https://github.com/user-attachments/assets/562de60b-e48f-4d7e-b9ec-911b537d193b" />

Lodestar - debug gossipsub: Gossip Block Received Delay, filtered by source="gossip"
<img width="719" alt="Screenshot 2025-05-20 at 13 08 03" src="https://github.com/user-attachments/assets/fa95fb54-eb0a-4e68-85b4-67b80d1cfca0" />

Lodestar - networking: Gossip Block Received Delay, Gossip Block process time
<img width="1440" alt="Screenshot 2025-05-20 at 13 09 39" src="https://github.com/user-attachments/assets/8b018d6b-95fe-4009-a325-9d2e644fa28a" />

Lodestar - networking: Received Time
<img width="1440" alt="Screenshot 2025-05-20 at 13 11 06" src="https://github.com/user-attachments/assets/8066f6ca-ef8c-4889-83f2-a6cc0463eafd" />

author_association: CONTRIBUTOR
user: login: KatyaRyazantseva
created_at: 2025-05-20T11:49:47Z
body: @nflaig considering the screenshots, please, let me know if I should bring back `{{source}} time` in Avg block recv delay.
author_association: CONTRIBUTOR
user: login: nflaig
created_at: 2025-05-20T11:51:06Z
body: > @nflaig considering the screenshots, please, let me know if I should bring back `{{source}} time` in Avg block recv delay.

No, I think it's fine as is
author_association: MEMBER
user: login: wemeetagain
created_at: 2025-06-03T20:14:33Z
body: :tada: This PR is included in [v1.31.0](https://github.com/ChainSafe/lodestar/releases/tag/v1.31.0) :tada:
author_association: MEMBER