Repository: NetApp/harvest
Pull request #1679 (2023-02-01 10:04:30)

fix: backward compatibility for qtree template
In 22.05,

qtree_labels:
<img width="1768" alt="image" src="https://user-images.githubusercontent.com/83282894/215990035-6b45e5ec-b91c-4fd6-a7fb-10e5b15e62e7.png">

qtree_disk_llimit:
<img width="1776" alt="image" src="https://user-images.githubusercontent.com/83282894/215990125-7abf76cd-46a1-4a6b-b459-0e15b22f4c8b.png">

No quota_xxx metrics exist.


In 23.02, without any changes,
qtree_labels:
<img width="1773" alt="image" src="https://user-images.githubusercontent.com/83282894/215991159-ddd7fd31-0995-4ab5-8fa7-9d2c598662d0.png">

qtree_disk_limit:
<img width="1775" alt="image" src="https://user-images.githubusercontent.com/83282894/215991254-315aa865-6736-4666-b71c-d58d228e8ab1.png">

quota_disk_limit:
<img width="1779" alt="image" src="https://user-images.githubusercontent.com/83282894/215991339-a8422033-0f87-429f-b531-baf2a6122c6e.png">


In 23.02, with changes with qtree.yaml,
qtree_labels:
<img width="1765" alt="image" src="https://user-images.githubusercontent.com/83282894/215995965-1494f5b0-624b-4346-a4ba-2cfbe2e4b88f.png">

qtree_disk_limit:
<img width="1777" alt="image" src="https://user-images.githubusercontent.com/83282894/215996023-e9237a9b-90bc-4634-b491-0fca4311be09.png">

quota_disk_limit:
<img width="1768" alt="image" src="https://user-images.githubusercontent.com/83282894/215996087-3f4a3f36-ca78-43ec-88ec-6aed55ab956b.png">


In 23.02, when changed from qtree.yaml to qtree.22.05.yaml in default.yaml
qtree_labels:
<img width="1778" alt="image" src="https://user-images.githubusercontent.com/83282894/216007667-5cdc523f-b3f7-412e-a1fa-98362a51e82b.png">

qtree_disk_limit:
<img width="1775" alt="image" src="https://user-images.githubusercontent.com/83282894/216007729-cdaaf3e2-6320-4347-abf3-a39b9f49ec40.png">

quota_disk_limit:
<img width="1783" alt="image" src="https://user-images.githubusercontent.com/83282894/216007830-20b15413-cd2f-4fdb-863b-23cbe7452f6b.png">


Points:
- labels of `qtree_labels` would be same in both the template.
- no of records of `qtree_labels` would be differ in both template --> reason: latest template excludes empty qtree names.
- no of records of quota metrics would be differ in both template --> reason: latest template excludes empty qtree names.

Current Issue:
- latest template would expect all type of quota when no quota-type filter applied in template, whereas old template would expect only tree quotas.   

Hardikl: Added minor change in older template to fetch only `tree` quotas as Harvest 22.05 only gives tree quotas.

**Reason**: We have only one `Qtree` plugin which also contains all the latest changes. It would be easier to get same metric behaviour from the template change than plugin changes. 

Metrics when changed to qtree.22.05.yaml in default.yaml.
qtree_labels:
<img width="1771" alt="image" src="https://user-images.githubusercontent.com/83282894/216057341-cba6f139-10ec-4e9f-a8b1-eac6cc520a61.png">

qtree_disk_limit:
<img width="1780" alt="image" src="https://user-images.githubusercontent.com/83282894/216058092-8e685e31-8322-434d-9f6b-abe37ef23af2.png">

quota_disk_limit:
<img width="1770" alt="image" src="https://user-images.githubusercontent.com/83282894/216058207-3e92db31-2335-4ccd-b80b-38250e678654.png">


Points:
- labels of qtree_labels would be same in both the template.
- 3 extra labels[index, type, unit] would be present in qtree_xxx metric in 22.05 template 
- no of records of qtree_labels, qtree_xxx would be same in both the template.
- qtree.22.05.yaml gives same behaviour as Harevst 22.05
- qtree.yaml gives current behaviour 
Hardikl: Case 1: **Same behaviour as Harvest 22.05**
with hidden key-val pair
qtree_labels:
<img width="1777" alt="image" src="https://user-images.githubusercontent.com/83282894/216583073-a15065c4-eb71-4abd-8eaf-8370b2fdf601.png">

qtree_disk_limit:
<img width="1776" alt="image" src="https://user-images.githubusercontent.com/83282894/216588425-ac61b0d2-f342-494c-ac8d-0586691bce9c.png">

quota_xxxx metrics not exist.



Case 2: **Current Harvest behaviour** 
without hidden key-val pair:
qtree_labels:
<img width="1767" alt="image" src="https://user-images.githubusercontent.com/83282894/216590678-10515827-79c9-4310-907a-1f26c61559be.png">

qtree_disk_limit:
<img width="1773" alt="image" src="https://user-images.githubusercontent.com/83282894/216590737-d9b10e9c-378c-4f00-b1bc-6f4cff44778e.png">

quota_disk_limit:
<img width="1771" alt="image" src="https://user-images.githubusercontent.com/83282894/216590858-a3c590e1-dd80-4225-a50d-77d0d276a05f.png">



Hardikl: Change update in PR:
with hidden flag --> both qtree and quota metrics will be visible with new labels as well as older labels.
without hidden flag --> both qtree and quota metrics will be visible with new labels only.

Patch:
diff --git a/cmd/collectors/zapi/plugins/qtree/qtree.go b/cmd/collectors/zapi/plugins/qtree/qtree.go
index d7315e08..9d3214d1 100644
--- a/cmd/collectors/zapi/plugins/qtree/qtree.go
+++ b/cmd/collectors/zapi/plugins/qtree/qtree.go
@@ -21,13 +21,14 @@ const BatchSize = "500"
 // Qtree plugin is needed to match qtrees with quotas.
 type Qtree struct {
 	*plugin.AbstractPlugin
-	data           *matrix.Matrix
-	instanceKeys   map[string]string
-	instanceLabels map[string]*dict.Dict
-	batchSize      string
-	client         *zapi.Client
-	query          string
-	quotaType      []string
+	data             *matrix.Matrix
+	instanceKeys     map[string]string
+	instanceLabels   map[string]*dict.Dict
+	batchSize        string
+	client           *zapi.Client
+	query            string
+	quotaType        []string
+	historicalLabels bool // supports labels, metrics for 22.05
 }
 
 func New(p *plugin.AbstractPlugin) plugin.Plugin {
@@ -61,10 +62,34 @@ func (my *Qtree) Init() error {
 	my.data = matrix.New(my.Parent+".Qtree", "quota", "quota")
 	my.instanceKeys = make(map[string]string)
 	my.instanceLabels = make(map[string]*dict.Dict)
+	my.historicalLabels = false
+
+	if my.Params.HasChildS("historicalLabels") {
+		exportOptions := node.NewS("export_options")
+		instanceKeys := exportOptions.NewChildS("instance_keys", "")
+
+		// apply all instance keys, instance labels from parent (qtree.yaml) to all quota metrics
+		if exportOption := my.ParentParams.GetChildS("export_options"); exportOption != nil {
+			//parent instancekeys would be added in plugin metrics
+			if parentKeys := exportOption.GetChildS("instance_keys"); parentKeys != nil {
+				for _, parentKey := range parentKeys.GetAllChildContentS() {
+					instanceKeys.NewChildS("", parentKey)
+				}
+			}
+			// parent instacelabels would be added in plugin metrics
+			if parentLabels := exportOption.GetChildS("instance_labels"); parentLabels != nil {
+				for _, parentLabel := range parentLabels.GetAllChildContentS() {
+					instanceKeys.NewChildS("", parentLabel)
+				}
+			}
+		}
 
-	objects := my.Params.GetChildS("objects")
-	if objects == nil {
-		return errs.New(errs.ErrMissingParam, "objects")
+		instanceKeys.NewChildS("", "type")
+		instanceKeys.NewChildS("", "index")
+		instanceKeys.NewChildS("", "unit")
+
+		my.data.SetExportOptions(exportOptions)
+		my.historicalLabels = true
 	}
 
 	quotaType := my.Params.GetChildS("quotaType")
@@ -75,6 +100,11 @@ func (my *Qtree) Init() error {
 		my.quotaType = []string{"user", "group", "tree"}
 	}
 
+	objects := my.Params.GetChildS("objects")
+	if objects == nil {
+		return errs.New(errs.ErrMissingParam, "objects")
+	}
+
 	for _, obj := range objects.GetAllChildContentS() {
 
 		metricName, display, _, _ := util.ParseMetric(obj)
@@ -135,6 +165,13 @@ func (my *Qtree) Run(data *matrix.Matrix) ([]*matrix.Matrix, error) {
 	tag := "initial"
 	quotaIndex := 0
 
+	// In 22.05, all qtrees were exported
+	if my.historicalLabels {
+		for _, qtreeInstance := range data.GetInstances() {
+			qtreeInstance.SetExportable(true)
+		}
+	}
+
 	cluster, _ := data.GetGlobalLabels().GetHas("cluster")
 
 	for {
@@ -166,120 +203,231 @@ func (my *Qtree) Run(data *matrix.Matrix) ([]*matrix.Matrix, error) {
 
 		my.Logger.Debug().Int("quotas", len(quotas)).Msg("fetching quotas")
 
-		for _, quota := range quotas {
-			var vserver, quotaInstanceKey, uid, uName string
-			quotaType := quota.GetChildContentS("quota-type")
-			tree := quota.GetChildContentS("tree")
-			volume := quota.GetChildContentS("volume")
-			if my.client.IsClustered() {
-				vserver = quota.GetChildContentS("vserver")
-			}
-			users := quota.GetChildS("quota-users")
-			if users != nil {
-				quotaUser := users.GetChildS("quota-user")
-				if quotaUser != nil {
-					uid = quotaUser.GetChildContentS("quota-user-id")
-					uName = quotaUser.GetChildContentS("quota-user-name")
-				}
+		if my.historicalLabels {
+			// In 22.05, populate metrics with qtree prefix and old labels
+			err = my.handlingHistoricalMetrics(quotas, data, cluster, &quotaIndex, &numMetrics)
+		} else {
+			// Populate metrics with quota prefix and current labels
+			err = my.handlingQuotaMetrics(quotas, cluster, &quotaIndex, &numMetrics)
+		}
+
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	my.Logger.Info().
+		Int("numQuotas", quotaIndex).
+		Int("metrics", numMetrics).
+		Str("apiD", apiT.Round(time.Millisecond).String()).
+		Str("parseD", parseT.Round(time.Millisecond).String()).
+		Str("batchSize", my.batchSize).
+		Msg("Collected")
+
+	// metrics with qtree prefix and quota prefix are available to support backward compatibility
+	qtreePluginData := my.data.Clone(true, true, true)
+	qtreePluginData.UUID = my.Parent + ".Qtree"
+	qtreePluginData.Object = "qtree"
+	qtreePluginData.Identifier = "qtree"
+	return []*matrix.Matrix{qtreePluginData, my.data}, nil
+}
+
+func (my Qtree) handlingHistoricalMetrics(quotas []*node.Node, data *matrix.Matrix, cluster string, quotaIndex *int, numMetrics *int) error {
+	for qIndex, quota := range quotas {
+		var vserver, quotaInstanceKey string
+		var qtreeInstance *matrix.Instance
+
+		quotaType := quota.GetChildContentS("quota-type")
+		tree := quota.GetChildContentS("tree")
+		volume := quota.GetChildContentS("volume")
+		if my.client.IsClustered() {
+			vserver = quota.GetChildContentS("vserver")
+		}
+
+		*quotaIndex++
+		for attribute, m := range my.data.GetMetrics() {
+
+			objectElem := quota.GetChildS(attribute)
+			if objectElem == nil {
+				continue
 			}
 
-			// ignore default quotas and set user/group
-			// Rest uses service side filtering to remove default records
-			if quotaType == "user" {
-				if (uName == "*" && uid == "*") || (uName == "" && uid == "") {
-					continue
+			if attrValue := quota.GetChildContentS(attribute); attrValue != "" {
+				if my.client.IsClustered() {
+					qtreeInstance = data.GetInstance(tree + "." + volume + "." + vserver)
+				} else {
+					qtreeInstance = data.GetInstance(volume + "." + tree)
 				}
-			} else if quotaType == "group" {
-				if uName == "*" || uName == "" {
+				if qtreeInstance == nil {
+					my.Logger.Warn().
+						Str("tree", tree).
+						Str("volume", volume).
+						Str("vserver", vserver).
+						Msg("No instance matching tree.volume.vserver")
 					continue
 				}
-			} else if quotaType == "tree" {
-				if tree == "" {
+				if !qtreeInstance.IsExportable() {
 					continue
 				}
-			}
+				// Ex. InstanceKey: SVMA.vol1Abc.qtree1.5.disk-limit
+				if my.client.IsClustered() {
+					quotaInstanceKey = vserver + "." + volume + "." + tree + "." + strconv.Itoa(qIndex) + "." + attribute
+				} else {
+					quotaInstanceKey = volume + "." + tree + "." + strconv.Itoa(qIndex) + "." + attribute
+				}
+				quotaInstance, err := my.data.NewInstance(quotaInstanceKey)
 
-			quotaIndex++
+				if err != nil {
+					my.Logger.Debug().Msgf("add (%s) instance: %v", attribute, err)
+					return err
+				}
 
-			for attribute, m := range my.data.GetMetrics() {
+				my.Logger.Debug().Msgf("add (%s) instance: %s.%s.%s", attribute, vserver, volume, tree)
 
-				objectElem := quota.GetChildS(attribute)
-				if objectElem == nil {
-					continue
+				for _, label := range my.data.GetExportOptions().GetChildS("instance_keys").GetAllChildContentS() {
+					if value := qtreeInstance.GetLabel(label); value != "" {
+						quotaInstance.SetLabel(label, value)
+					}
 				}
 
-				if attrValue := quota.GetChildContentS(attribute); attrValue != "" {
+				//set labels
+				quotaInstance.SetLabel("type", quotaType)
+				quotaInstance.SetLabel("qtree", tree)
+				quotaInstance.SetLabel("volume", volume)
+				quotaInstance.SetLabel("svm", vserver)
+				quotaInstance.SetLabel("index", cluster+"_"+strconv.Itoa(*quotaIndex))
+
+				// If the Qtree is the volume itself, than qtree label is empty, so copy the volume name to qtree.
+				if tree == "" {
+					quotaInstance.SetLabel("qtree", volume)
+				}
 
-					// Ex. InstanceKey: SVMA.vol1Abc.qtree1.5.disk-limit
-					if my.client.IsClustered() {
-						quotaInstanceKey = vserver + "." + volume + "." + tree + "." + strconv.Itoa(quotaIndex) + "." + attribute
+				// populate numeric data
+				if value := strings.Split(attrValue, " ")[0]; value != "" {
+					// Few quota metrics would have value '-' which means unlimited (ex: disk-limit)
+					if value == "-" {
+						value = "0"
+					}
+					if attribute == "soft-disk-limit" || attribute == "disk-limit" || attribute == "disk-used" {
+						quotaInstance.SetLabel("unit", "Kbyte")
+					}
+					if err := m.SetValueString(quotaInstance, value); err != nil {
+						my.Logger.Debug().Msgf("(%s) failed to parse value (%s): %v", attribute, value, err)
 					} else {
-						quotaInstanceKey = volume + "." + tree + "." + strconv.Itoa(quotaIndex) + "." + attribute
+						*numMetrics++
+						my.Logger.Debug().Msgf("(%s) added value (%s)", attribute, value)
 					}
-					quotaInstance, err := my.data.NewInstance(quotaInstanceKey)
-					if err != nil {
-						my.Logger.Debug().Msgf("add (%s) instance: %v", attribute, err)
-						return nil, err
+				}
+
+			} else {
+				my.Logger.Debug().Msgf("instance without [%s], skipping", attribute)
+			}
+		}
+	}
+	return nil
+}
+
+func (my Qtree) handlingQuotaMetrics(quotas []*node.Node, cluster string, quotaIndex *int, numMetrics *int) error {
+	for _, quota := range quotas {
+		var vserver, quotaInstanceKey, uid, uName string
+
+		quotaType := quota.GetChildContentS("quota-type")
+		tree := quota.GetChildContentS("tree")
+		volume := quota.GetChildContentS("volume")
+		if my.client.IsClustered() {
+			vserver = quota.GetChildContentS("vserver")
+		}
+
+		users := quota.GetChildS("quota-users")
+		if users != nil {
+			quotaUser := users.GetChildS("quota-user")
+			if quotaUser != nil {
+				uid = quotaUser.GetChildContentS("quota-user-id")
+				uName = quotaUser.GetChildContentS("quota-user-name")
+			}
+		}
+
+		// ignore default quotas and set user/group
+		// Rest uses service side filtering to remove default records
+		if quotaType == "user" {
+			if (uName == "*" && uid == "*") || (uName == "" && uid == "") {
+				continue
+			}
+		} else if quotaType == "group" {
+			if uName == "*" || uName == "" {
+				continue
+			}
+		} else if quotaType == "tree" {
+			if tree == "" {
+				continue
+			}
+		}
+		*quotaIndex++
+
+		for attribute, m := range my.data.GetMetrics() {
+
+			objectElem := quota.GetChildS(attribute)
+			if objectElem == nil {
+				continue
+			}
+
+			if attrValue := quota.GetChildContentS(attribute); attrValue != "" {
+
+				// Ex. InstanceKey: SVMA.vol1Abc.qtree1.5.disk-limit
+				if my.client.IsClustered() {
+					quotaInstanceKey = vserver + "." + volume + "." + tree + "." + strconv.Itoa(*quotaIndex) + "." + attribute
+				} else {
+					quotaInstanceKey = volume + "." + tree + "." + strconv.Itoa(*quotaIndex) + "." + attribute
+				}
+				quotaInstance, err := my.data.NewInstance(quotaInstanceKey)
+				if err != nil {
+					my.Logger.Debug().Msgf("add (%s) instance: %v", attribute, err)
+					return err
+				}
+				//set labels
+				quotaInstance.SetLabel("type", quotaType)
+				quotaInstance.SetLabel("qtree", tree)
+				quotaInstance.SetLabel("volume", volume)
+				quotaInstance.SetLabel("svm", vserver)
+				quotaInstance.SetLabel("index", cluster+"_"+strconv.Itoa(*quotaIndex))
+
+				if quotaType == "user" {
+					if uName != "" {
+						quotaInstance.SetLabel("user", uName)
+					} else if uid != "" {
+						quotaInstance.SetLabel("user", uid)
+					}
+				} else if quotaType == "group" {
+					if uName != "" {
+						quotaInstance.SetLabel("group", uName)
+					} else if uid != "" {
+						quotaInstance.SetLabel("group", uid)
 					}
-					//set labels
-					quotaInstance.SetLabel("type", quotaType)
-					quotaInstance.SetLabel("qtree", tree)
-					quotaInstance.SetLabel("volume", volume)
-					quotaInstance.SetLabel("svm", vserver)
-					quotaInstance.SetLabel("index", cluster+"_"+strconv.Itoa(quotaIndex))
-
-					if quotaType == "user" {
-						if uName != "" {
-							quotaInstance.SetLabel("user", uName)
-						} else if uid != "" {
-							quotaInstance.SetLabel("user", uid)
-						}
-					} else if quotaType == "group" {
-						if uName != "" {
-							quotaInstance.SetLabel("group", uName)
-						} else if uid != "" {
-							quotaInstance.SetLabel("group", uid)
-						}
+				}
+
+				my.Logger.Debug().Msgf("add (%s) instance: %s.%s.%s", attribute, vserver, volume, tree)
+
+				// populate numeric data
+				if value := strings.Split(attrValue, " ")[0]; value != "" {
+					// Few quota metrics would have value '-' which means unlimited (ex: disk-limit)
+					if value == "-" {
+						value = "-1"
 					}
 
-					my.Logger.Debug().Msgf("add (%s) instance: %s.%s.%s", attribute, vserver, volume, tree)
-
-					// populate numeric data
-					if value := strings.Split(attrValue, " ")[0]; value != "" {
-						// Few quota metrics would have value '-' which means unlimited (ex: disk-limit)
-						if value == "-" {
-							value = "-1"
-						}
-
-						if attribute == "soft-disk-limit" || attribute == "disk-limit" || attribute == "disk-used" {
-							quotaInstance.SetLabel("unit", "Kbyte")
-						}
-						if err := m.SetValueString(quotaInstance, value); err != nil {
-							my.Logger.Debug().Msgf("(%s) failed to parse value (%s): %v", attribute, value, err)
-						} else {
-							numMetrics++
-							my.Logger.Debug().Msgf("(%s) added value (%s)", attribute, value)
-						}
+					if attribute == "soft-disk-limit" || attribute == "disk-limit" || attribute == "disk-used" {
+						quotaInstance.SetLabel("unit", "Kbyte")
+					}
+					if err := m.SetValueString(quotaInstance, value); err != nil {
+						my.Logger.Debug().Msgf("(%s) failed to parse value (%s): %v", attribute, value, err)
+					} else {
+						*numMetrics++
+						my.Logger.Debug().Msgf("(%s) added value (%s)", attribute, value)
 					}
-				} else {
-					my.Logger.Debug().Msgf("instance without [%s], skipping", attribute)
 				}
+			} else {
+				my.Logger.Debug().Msgf("instance without [%s], skipping", attribute)
 			}
 		}
 	}
-
-	my.Logger.Info().
-		Int("numQuotas", quotaIndex).
-		Int("metrics", numMetrics).
-		Str("apiD", apiT.Round(time.Millisecond).String()).
-		Str("parseD", parseT.Round(time.Millisecond).String()).
-		Str("batchSize", my.batchSize).
-		Msg("Collected")
-
-	// This would generate quota metrics prefix with `qtree_`. These are deprecated now and will be removed later.
-	qtreePluginData := my.data.Clone(true, true, true)
-	qtreePluginData.UUID = my.Parent + ".Qtree"
-	qtreePluginData.Object = "qtree"
-	qtreePluginData.Identifier = "qtree"
-	return []*matrix.Matrix{qtreePluginData, my.data}, nil
+	return nil
 }
diff --git a/grafana/dashboards/storagegrid/overview.json b/grafana/dashboards/storagegrid/overview.json
index 4268f7e0..cc0bdb0e 100644
--- a/grafana/dashboards/storagegrid/overview.json
+++ b/grafana/dashboards/storagegrid/overview.json
@@ -1600,6 +1600,26 @@
   ],
   "templating": {
     "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "description": null,
+        "error": null,
+        "hide": 2,
+        "includeAll": false,
+        "label": "Data Source",
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
       {
         "allValue": null,
         "current": {},
diff --git a/grafana/dashboards/storagegrid/tenant.json b/grafana/dashboards/storagegrid/tenant.json
index 5b2ee261..f48b48cc 100644
--- a/grafana/dashboards/storagegrid/tenant.json
+++ b/grafana/dashboards/storagegrid/tenant.json
@@ -454,6 +454,26 @@
   ],
   "templating": {
     "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "description": null,
+        "error": null,
+        "hide": 2,
+        "includeAll": false,
+        "label": "Data Source",
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
       {
         "allValue": null,
         "current": {},
diff --git a/harvest.yml b/harvest.yml
index 0c451bc1..bfc22e06 100644
--- a/harvest.yml
+++ b/harvest.yml
@@ -61,7 +61,7 @@ Pollers:
   # no need to add collectors, since they're already defined in Defaults
 
   # example with certificate authentication
-  # See https://github.com/NetApp/harvest/blob/main/docs/AuthAndPermissions.md#using-certificate-authentication
+  # See https://netapp.github.io/harvest/latest/prepare-cdot-clusters/#using-certificate-authentication
   # cluster-01:
   #   datacenter: DC-01
   #   addr: 10.0.0.2
@@ -70,4 +70,15 @@ Pollers:
   #   ssl_key: /path/to/key.key
   #   ca_cert: /path/to/server.crt
   #   exporters:
-  #     - prometheus1
\ No newline at end of file
+  #     - prometheus1
+
+  # StorageGRID example
+  #  grid-01:
+  #    datacenter: DC-01
+  #    addr: 10.0.0.3
+  #    username: myuser
+  #    password: mypass
+  #    exporters:
+  #      - prometheus1
+  #    collectors:
+  #      - StorageGrid


Test patch:
diff --git a/cmd/tools/grafana/dashboard_test.go b/cmd/tools/grafana/dashboard_test.go
index 31427079..a0d56ecf 100644
--- a/cmd/tools/grafana/dashboard_test.go
+++ b/cmd/tools/grafana/dashboard_test.go
@@ -28,6 +28,26 @@ func checkDashboardForDatasource(t *testing.T, path string, data []byte) {
 		}
 		return true
 	})
+
+	// Check that the variable DS_PROMETHEUS exist
+	doesDsPromExist := false
+	gjson.GetBytes(data, "templating.list").ForEach(func(key, value gjson.Result) bool {
+		if value.Get("name").String() == "DS_PROMETHEUS" {
+			doesDsPromExist = true
+			query := value.Get("query").String()
+			if query != "prometheus" {
+				t.Errorf("dashboard=%s var=DS_PROMETHEUS query want=prometheus got=%s", path, query)
+			}
+			theType := value.Get("type").String()
+			if theType != "datasource" {
+				t.Errorf("dashboard=%s var=DS_PROMETHEUS type want=datasource got=%s", path, theType)
+			}
+		}
+		return true
+	})
+	if !doesDsPromExist {
+		t.Errorf("dashboard=%s should define variable has DS_PROMETHEUS", path)
+	}
 }
 
 func TestUnitsAndExprMatch(t *testing.T) {