Repository: cloudnative-pg/cloudnative-pg

Issue:
[Feature]: decouple metrics scraping from metrics recomputation
### Is there an existing issue already for this feature request/idea?

- [X] I have searched for an existing issue, and could not find anything. I believe this is a new feature request to be evaluated.

### What problem is this feature going to solve? Why should it be added?

Several metrics, including user defined metrics, are run each time the prometheus metrics endpoint 9187, is scraped. Depending on the configuration of Prometheus, and even perhaps just having some other process doing a GET on `:9187/metrics`, those SQL queries might run often.
If a user included some long running queries for monitoring, this could be a CPU hog and even pose a threat to the normal functioning of their cluster.

### Describe the solution you'd like

Generally, to respond to a GET on the metrics port, we should simply return the current values of the various prometheus counters, histograms etc.
A few select metrics might warrant a refresh on each scrape.

The recomputation of metrics should happen independently of scrapes.
The refresh interval should be configurable, and if not set default to something reasonable, e.g. 30 seconds, which seems the current default scraping interval for the Prometheus operator

``` go
type MonitoringConfiguration struct {
	<... snipped ...>
	RefreshInterval *metav1.Duration `json:"refreshInterval,omitempty"`
```

For the purpose of decoupling scraping from updating of metrics, it is sufficient to have a goroutine rerunning the metrics queries and creating Prometheus metrics, with the above refresh interval.

A future PR/Issue could tackle having query-dependent refresh intervals, so that very expensive queries could be monitored, while keeping a normal frequency with the smaller queries.

### Describe alternatives you've considered

The prometheus HTTP library has a parameter `MaxRequestsInFlight` which, if set will respond with an HTTP 503 if exceeded.
That could be a very low effort low resolution safety mechanism.

### Additional context

_No response_

### Backport?

Yes

### Are you willing to actively contribute to this feature?

Yes

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct



Discussion:
@dosubot: Hi, @jsilvela. I'm [Dosu](https://dosu.dev), and I'm helping the cloudnative-pg team manage their backlog. I'm marking this issue as stale.

**Issue Summary:**
- You proposed a feature to separate metrics scraping from recomputation to prevent CPU overload.
- The solution involves returning existing metric values during scrapes and recomputing metrics independently.
- This would require implementing goroutines and adding a `RefreshInterval` field to the `UserQuery` type.
- There has been no activity or comments on this issue since it was opened.

**Next Steps:**
- Please let me know if this issue is still relevant to the latest version of the cloudnative-pg repository by commenting here.
- If there is no further activity, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!@jsilvela: This is still relevant, please unmark@jsilvela: Dosu, please unmark.
This issue is still relevant to the latest version of the cloudnative-pg repository.@github-actions: This issue is stale because it has been open for 60 days with no activity.@jsilvela: No, not stale. Working on it.@gbartolini: Conservatively moving this to 1.28 as it might be impacted by #8033 @jsilvela: Pinging here as a reminder, this doesn't have any reviews@jsilvela: @mnencia any chance for a review? It is, kind of, a security weakness...@github-actions: This issue is stale because it has been open for 60 days with no activity.

Solution patch:
diff --git a/.wordlist-en-custom.txt b/.wordlist-en-custom.txt
index 07d3689a..c0d13600 100644
--- a/.wordlist-en-custom.txt
+++ b/.wordlist-en-custom.txt
@@ -1043,6 +1043,7 @@ memstats
 metav
 metric's
 metricNameKS
+metricsQueriesTTL
 microservice
 microservices
 microsoft
diff --git a/api/v1/cluster_funcs.go b/api/v1/cluster_funcs.go
index cd1570de..d1c0963e 100644
--- a/api/v1/cluster_funcs.go
+++ b/api/v1/cluster_funcs.go
@@ -1184,6 +1184,19 @@ func (cluster *Cluster) IsMetricsTLSEnabled() bool {
 	return false
 }
 
+// GetMetricsQueriesTTL returns the Time To Live of the metrics computed from
+// queries. Once exceeded, a scrape of the metric will trigger a rerun of the queries.
+// Default value is 30 seconds
+func (cluster *Cluster) GetMetricsQueriesTTL() metav1.Duration {
+	if cluster.Spec.Monitoring != nil && cluster.Spec.Monitoring.MetricsQueriesTTL != nil {
+		return *cluster.Spec.Monitoring.MetricsQueriesTTL
+	}
+
+	return metav1.Duration{
+		Duration: 30 * time.Second,
+	}
+}
+
 // GetEnableSuperuserAccess returns if the superuser access is enabled or not
 func (cluster *Cluster) GetEnableSuperuserAccess() bool {
 	if cluster.Spec.EnableSuperuserAccess != nil {
diff --git a/api/v1/cluster_types.go b/api/v1/cluster_types.go
index 558f7c25..d0bfa24c 100644
--- a/api/v1/cluster_types.go
+++ b/api/v1/cluster_types.go
@@ -2222,6 +2222,14 @@ type MonitoringConfiguration struct {
 	// you need this functionality, you can create a PodMonitor manually.
 	// +optional
 	PodMonitorRelabelConfigs []monitoringv1.RelabelConfig `json:"podMonitorRelabelings,omitempty"`
+
+	// The interval during which metrics computed from queries are considered current.
+	// Once it is exceeded, a new scrape will trigger a rerun
+	// of the queries.
+	// If not set, defaults to 30 seconds, in line with Prometheus scraping defaults.
+	// Setting this to zero disables the caching mechanism and can cause heavy load on the PostgreSQL server.
+	// +optional
+	MetricsQueriesTTL *metav1.Duration `json:"metricsQueriesTTL,omitempty"`
 }
 
 // ClusterMonitoringTLSConfiguration is the type containing the TLS configuration
diff --git a/api/v1/zz_generated.deepcopy.go b/api/v1/zz_generated.deepcopy.go
index 1239af87..2c1bf602 100644
--- a/api/v1/zz_generated.deepcopy.go
+++ b/api/v1/zz_generated.deepcopy.go
@@ -1990,6 +1990,11 @@ func (in *MonitoringConfiguration) DeepCopyInto(out *MonitoringConfiguration) {
 			(*in)[i].DeepCopyInto(&(*out)[i])
 		}
 	}
+	if in.MetricsQueriesTTL != nil {
+		in, out := &in.MetricsQueriesTTL, &out.MetricsQueriesTTL
+		*out = new(metav1.Duration)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonitoringConfiguration.
diff --git a/config/crd/bases/postgresql.cnpg.io_clusters.yaml b/config/crd/bases/postgresql.cnpg.io_clusters.yaml
index b1273531..0e82c377 100644
--- a/config/crd/bases/postgresql.cnpg.io_clusters.yaml
+++ b/config/crd/bases/postgresql.cnpg.io_clusters.yaml
@@ -3840,6 +3840,14 @@ spec:
                       Deprecated: This feature will be removed in an upcoming release. If
                       you need this functionality, you can create a PodMonitor manually.
                     type: boolean
+                  metricsQueriesTTL:
+                    description: |-
+                      The interval during which metrics computed from queries are considered current.
+                      Once it is exceeded, a new scrape will trigger a rerun
+                      of the queries.
+                      If not set, defaults to 30 seconds, in line with Prometheus scraping defaults.
+                      Setting this to zero disables the caching mechanism and can cause heavy load on the PostgreSQL server.
+                    type: string
                   podMonitorMetricRelabelings:
                     description: |-
                       The list of metric relabelings for the `PodMonitor`. Applied to samples before ingestion.
diff --git a/docs/src/cloudnative-pg.v1.md b/docs/src/cloudnative-pg.v1.md
index fe06cf03..1e3e98f0 100644
--- a/docs/src/cloudnative-pg.v1.md
+++ b/docs/src/cloudnative-pg.v1.md
@@ -4000,6 +4000,17 @@ you need this functionality, you can create a PodMonitor manually.</p>
 you need this functionality, you can create a PodMonitor manually.</p>
 </td>
 </tr>
+<tr><td><code>metricsQueriesTTL</code><br/>
+<a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Duration"><i>meta/v1.Duration</i></a>
+</td>
+<td>
+   <p>The interval during which metrics computed from queries are considered current.
+Once it is exceeded, a new scrape will trigger a rerun
+of the queries.
+If not set, defaults to 30 seconds, in line with Prometheus scraping defaults.
+Setting this to zero disables the caching mechanism and can cause heavy load on the PostgreSQL server.</p>
+</td>
+</tr>
 </tbody>
 </table>
 
diff --git a/docs/src/monitoring.md b/docs/src/monitoring.md
index aa43e1da..68e54647 100644
--- a/docs/src/monitoring.md
+++ b/docs/src/monitoring.md
@@ -55,6 +55,20 @@ by specifying a list of one or more databases in the `target_databases` option.
     with Prometheus and Grafana, you can find a quick setup guide
     in [Part 4 of the quickstart](quickstart.md#part-4-monitor-clusters-with-prometheus-and-grafana)
 
+### Output caching
+
+By default, the outputs of monitoring queries are cached for thirty
+seconds. This is done to enhance resource efficiency and to avoid
+PostgreSQL to run monitoring queries every time the prometheus
+endpoint is scraped.
+
+The cache itself can be observed by the `cache_hits`, `cache_misses`
+and `last_update_timestamp` metrics.
+
+Setting the `cluster.spec.monitoring.metricsQueriesTTL` to zero will
+disable the cache, and in that case the metrics will be run on every
+metrics endpoint scrape.
+
 ### Monitoring with the Prometheus operator
 
 You can monitor a specific PostgreSQL cluster using the
diff --git a/go.mod b/go.mod
index 45cbd766..95210062 100644
--- a/go.mod
+++ b/go.mod
@@ -30,6 +30,7 @@ require (
 	github.com/onsi/gomega v1.38.2
 	github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.86.1
 	github.com/prometheus/client_golang v1.23.2
+	github.com/prometheus/client_model v0.6.2
 	github.com/robfig/cron v1.2.0
 	github.com/sethvargo/go-password v0.3.1
 	github.com/spf13/cobra v1.10.1
@@ -104,7 +105,6 @@ require (
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.66.1 // indirect
 	github.com/prometheus/procfs v0.16.1 // indirect
 	github.com/spf13/pflag v1.0.10 // indirect
diff --git a/internal/management/controller/instance_controller.go b/internal/management/controller/instance_controller.go
index 9673514e..12483e24 100644
--- a/internal/management/controller/instance_controller.go
+++ b/internal/management/controller/instance_controller.go
@@ -865,7 +865,8 @@ func (r *InstanceReconciler) reconcileClusterRoleWithoutDB(
 	return false, nil
 }
 
-// reconcileMetrics updates any required metrics
+// reconcileMetrics updates the prometheus metrics that deal with instance
+// manager or cluster data: manual_switchover_required, sync_replicas, replica_mode
 func (r *InstanceReconciler) reconcileMetrics(
 	ctx context.Context,
 	cluster *apiv1.Cluster,
diff --git a/internal/webhook/v1/cluster_webhook.go b/internal/webhook/v1/cluster_webhook.go
index 920fd390..a59033bd 100644
--- a/internal/webhook/v1/cluster_webhook.go
+++ b/internal/webhook/v1/cluster_webhook.go
@@ -2519,9 +2519,22 @@ func (v *ClusterCustomValidator) getAdmissionWarnings(r *apiv1.Cluster) admissio
 	list = append(list, getRetentionPolicyWarnings(r)...)
 	list = append(list, getStorageWarnings(r)...)
 	list = append(list, getSharedBuffersWarnings(r)...)
+	list = append(list, getMonitoringFieldsWarnings(r)...)
 	return append(list, getDeprecatedMonitoringFieldsWarnings(r)...)
 }
 
+func getMonitoringFieldsWarnings(r *apiv1.Cluster) admission.Warnings {
+	var result admission.Warnings
+
+	if r.GetMetricsQueriesTTL().Duration == 0 {
+		result = append(result,
+			"spec.monitoring.metricsQueriesTTL is explicitly set to 0; this disables automatic TTL behavior "+
+				"and can cause heavy load on the PostgreSQL server.")
+	}
+
+	return result
+}
+
 func getStorageWarnings(r *apiv1.Cluster) admission.Warnings {
 	generateWarningsFunc := func(path field.Path, configuration *apiv1.StorageConfiguration) admission.Warnings {
 		if configuration == nil {
diff --git a/pkg/management/postgres/metrics/collector.go b/pkg/management/postgres/metrics/collector.go
index 6dc12534..1b962d6a 100644
--- a/pkg/management/postgres/metrics/collector.go
+++ b/pkg/management/postgres/metrics/collector.go
@@ -28,6 +28,7 @@ import (
 	"path"
 	"regexp"
 	"slices"
+	"sync"
 	"time"
 
 	"github.com/blang/semver"
@@ -56,45 +57,95 @@ type QueriesCollector struct {
 
 	errorUserQueries      *prometheus.CounterVec
 	errorUserQueriesGauge prometheus.Gauge
+	lastUpdateTimestamp   prometheus.Gauge
+	cacheHits             prometheus.Gauge
+	cacheMiss             prometheus.Gauge
+
+	computedMetrics []prometheus.Metric
+	timeLastUpdated time.Time
+
+	// metricsMutex guards access to computedMetrics, error metrics, and timestamp
+	// of last update,
+	// to avoid races between Update (writer) and Collect (reader)
+	metricsMutex sync.RWMutex
 }
 
 // Name returns the name of this collector, as supplied by the user in the configMap
-func (q QueriesCollector) Name() string {
+func (q *QueriesCollector) Name() string {
 	return q.collectorName
 }
 
 var isPathPattern = regexp.MustCompile(`[][*?]`)
 
-// Collect loads data from the actual PostgreSQL instance
-func (q QueriesCollector) Collect(ch chan<- prometheus.Metric) error {
-	// Reset before collecting
+// Update recomputes the metrics from the user queries
+func (q *QueriesCollector) Update() error {
+	q.metricsMutex.Lock()
+	defer q.metricsMutex.Unlock()
+
+	// Start a fresh error state for this cycle
+	q.errorUserQueriesGauge.Set(0)
 	q.errorUserQueries.Reset()
 
-	err := q.collectUserQueries(ch)
+	// Reset cache hit/miss counters when we update (cache miss)
+	q.cacheHits.Set(0)
+	q.cacheMiss.Set(1)
+
+	isPrimary, err := q.instance.IsPrimary()
 	if err != nil {
-		return err
+		q.errorUserQueriesGauge.Set(1)
+		q.errorUserQueries.WithLabelValues("isPrimary: " + err.Error()).Inc()
+		return fmt.Errorf("while updating query metrics, could not check for primary: %w", err)
 	}
 
-	// Add errors into errorUserQueriesVec and errorUserQueriesGauge metrics
-	q.errorUserQueriesGauge.Collect(ch)
-	q.errorUserQueries.Collect(ch)
-
+	q.createMetricsFromUserQueries(isPrimary)
+	q.timeLastUpdated = time.Now()
+	// Update the timestamp metric to reflect when metrics were last computed
+	q.lastUpdateTimestamp.Set(float64(q.timeLastUpdated.Unix()))
 	return nil
 }
 
-func (q *QueriesCollector) collectUserQueries(ch chan<- prometheus.Metric) error {
-	isPrimary, err := q.instance.IsPrimary()
-	if err != nil {
-		return err
+// ShouldUpdate finds if the metrics from queries need to be rerun,
+// or the cached values can be used
+func (q *QueriesCollector) ShouldUpdate(ttl time.Duration) bool {
+	q.metricsMutex.RLock()
+	defer q.metricsMutex.RUnlock()
+	return q.timeLastUpdated.IsZero() || time.Since(q.timeLastUpdated) > ttl
+}
+
+// RecordCacheHit increments the cache hit counter for the current cache period
+func (q *QueriesCollector) RecordCacheHit() {
+	q.metricsMutex.Lock()
+	defer q.metricsMutex.Unlock()
+	q.cacheHits.Add(1)
+}
+
+// Collect sends the pre-computed metrics to the output channel.
+// These metrics were cached during the last Update() call and are not
+// fetched from the database during collection.
+func (q *QueriesCollector) Collect(ch chan<- prometheus.Metric) {
+	// Guard the snapshot read and error metrics collection
+	q.metricsMutex.RLock()
+	defer q.metricsMutex.RUnlock()
+	for _, m := range q.computedMetrics {
+		ch <- m
 	}
+	// Add errors into errorUserQueriesVec and errorUserQueriesGauge metrics
+	q.errorUserQueriesGauge.Collect(ch)
+	q.errorUserQueries.Collect(ch)
+	q.lastUpdateTimestamp.Collect(ch)
+	q.cacheHits.Collect(ch)
+	q.cacheMiss.Collect(ch)
+}
 
+func (q *QueriesCollector) createMetricsFromUserQueries(isPrimary bool) {
 	// In case more than one user query specify a pattern in target_databases,
 	// we need to get them just once
 	var allAccessibleDatabasesCache []string
 
+	var generatedMetrics []prometheus.Metric
 	for name, userQuery := range q.userQueries {
 		queryLogger := log.WithValues("query", name)
-		collector := QueryCollector{
+		queryRunner := QueryRunner{
 			namespace:      name,
 			userQuery:      userQuery,
 			columnMapping:  q.mappings[name],
@@ -129,25 +180,26 @@ func (q *QueriesCollector) collectUserQueries(ch chan<- prometheus.Metric) error
 
 		allTargetDatabases := q.expandTargetDatabases(targetDatabases, allAccessibleDatabasesCache)
 		for targetDatabase := range allTargetDatabases {
-			conn, err := q.instance.ConnectionPool().Connection(targetDatabase)
+			db, err := q.instance.ConnectionPool().Connection(targetDatabase)
 			if err != nil {
 				q.reportUserQueryErrorMetric(name + ": " + err.Error())
 				continue
 			}
 
-			err = collector.collect(conn, ch)
+			computedMetrics, err := queryRunner.computeMetrics(db)
 			if err != nil {
 				queryLogger.Error(err, "Error collecting user query",
 					"targetDatabase", targetDatabase)
 				// Increment metrics counters.
 				q.reportUserQueryErrorMetric(name + " on db " + targetDatabase + ": " + err.Error())
 			}
+			generatedMetrics = append(generatedMetrics, computedMetrics...)
 		}
 	}
-	return nil
+	q.computedMetrics = generatedMetrics
 }
 
-func (q QueriesCollector) toBeChecked(name string, userQuery UserQuery, isPrimary bool, queryLogger log.Logger) bool {
+func (q *QueriesCollector) toBeChecked(name string, userQuery UserQuery, isPrimary bool, queryLogger log.Logger) bool {
 	if (userQuery.Primary || userQuery.Master) && !isPrimary { // wokeignore:rule=master
 		queryLogger.Debug("Skipping because runs only on primary")
 		return false
@@ -171,12 +223,12 @@ func (q QueriesCollector) toBeChecked(name string, userQuery UserQuery, isPrimar
 	return true
 }
 
-func (q QueriesCollector) reportUserQueryErrorMetric(label string) {
+func (q *QueriesCollector) reportUserQueryErrorMetric(label string) {
 	q.errorUserQueries.WithLabelValues(label).Inc()
 	q.errorUserQueriesGauge.Set(1)
 }
 
-func (q QueriesCollector) checkRunOnServerMatches(runOnServer string, name string) (bool, error) {
+func (q *QueriesCollector) checkRunOnServerMatches(runOnServer string, name string) (bool, error) {
 	// The instance will only get the PostgreSQL version one time
 	// and then cache the result, so this is not a real query
 	pgVersion, err := q.instance.GetPgVersion()
@@ -195,7 +247,7 @@ func (q QueriesCollector) checkRunOnServerMatches(runOnServer string, name strin
 	return isVersionInRange(pgVersion), nil
 }
 
-func (q QueriesCollector) expandTargetDatabases(
+func (q *QueriesCollector) expandTargetDatabases(
 	targetDatabases []string,
 	allAccessibleDatabasesCache []string,
 ) (allTargetDatabases map[string]bool) {
@@ -215,7 +267,7 @@ func (q QueriesCollector) expandTargetDatabases(
 	return allTargetDatabases
 }
 
-func (q QueriesCollector) getAllAccessibleDatabases() ([]string, error) {
+func (q *QueriesCollector) getAllAccessibleDatabases() ([]string, error) {
 	conn, err := q.instance.ConnectionPool().Connection(q.defaultDBName)
 	if err != nil {
 		return nil, fmt.Errorf("while connecting to expand target_database *: %w", err)
@@ -237,9 +289,9 @@ func (q QueriesCollector) getAllAccessibleDatabases() ([]string, error) {
 }
 
 // Describe implements the prometheus.Collector and defines the metrics with return
-func (q QueriesCollector) Describe(ch chan<- *prometheus.Desc) {
+func (q *QueriesCollector) Describe(ch chan<- *prometheus.Desc) {
 	for name, userQuery := range q.userQueries {
-		collector := QueryCollector{
+		collector := QueryRunner{
 			namespace:     name,
 			userQuery:     userQuery,
 			columnMapping: q.mappings[name],
@@ -251,6 +303,9 @@ func (q QueriesCollector) Describe(ch chan<- *prometheus.Desc) {
 	// add error user queries description
 	q.errorUserQueries.Describe(ch)
 	q.errorUserQueriesGauge.Describe(ch)
+	q.lastUpdateTimestamp.Describe(ch)
+	q.cacheHits.Describe(ch)
+	q.cacheMiss.Describe(ch)
 }
 
 // NewQueriesCollector creates a new PgCollector working over a set of custom queries
@@ -277,6 +332,21 @@ func NewQueriesCollector(
 			Name:      "last_error",
 			Help:      "1 if the last collection ended with error, 0 otherwise.",
 		}),
+		lastUpdateTimestamp: prometheus.NewGauge(prometheus.GaugeOpts{
+			Namespace: name,
+			Name:      "last_update_timestamp",
+			Help:      "Timestamp of the last metrics update.",
+		}),
+		cacheHits: prometheus.NewGauge(prometheus.GaugeOpts{
+			Namespace: name,
+			Name:      "cache_hits",
+			Help:      "Total number of hits for the current cache.",
+		}),
+		cacheMiss: prometheus.NewGauge(prometheus.GaugeOpts{
+			Namespace: name,
+			Name:      "cache_miss",
+			Help:      "Indicator: 1 if metrics were recomputed on last update (cache miss), 0 if cache used.",
+		}),
 	}
 }
 
@@ -322,21 +392,22 @@ func (q *QueriesCollector) InjectUserQueries(defaultQueries UserQueries) {
 	}
 }
 
-// QueryCollector is the implementation of PgCollector for a certain
-// custom query supplied by the user
-type QueryCollector struct {
+// QueryRunner computes a custom user query and generates Prometheus metrics
+// from the results
+type QueryRunner struct {
 	namespace      string
 	userQuery      UserQuery
 	columnMapping  MetricMapSet
 	variableLabels VariableSet
 }
 
-// collect retrieves metrics from query and exposes them to prometheus
-func (c QueryCollector) collect(conn *sql.DB, ch chan<- prometheus.Metric) error {
+// computeMetrics runs the queries and generates prometheus metrics from them
+func (c QueryRunner) computeMetrics(conn *sql.DB) ([]prometheus.Metric, error) {
 	tx, err := createMonitoringTx(conn)
 	if err != nil {
-		return err
+		return nil, err
 	}
+	var computedMetrics []prometheus.Metric
 
 	defer func() {
 		if err := tx.Commit(); err != nil {
@@ -346,16 +417,16 @@ func (c QueryCollector) collect(conn *sql.DB, ch chan<- prometheus.Metric) error
 
 	shouldBeCollected, err := c.userQuery.isCollectable(tx)
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	if !shouldBeCollected {
-		return nil
+		return nil, nil
 	}
 
 	rows, err := tx.Query(c.userQuery.Query)
 	if err != nil {
-		return err
+		return nil, err
 	}
 	defer func() {
 		if err := rows.Close(); err != nil {
@@ -366,7 +437,7 @@ func (c QueryCollector) collect(conn *sql.DB, ch chan<- prometheus.Metric) error
 
 	columns, err := rows.Columns()
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	columnData := make([]interface{}, len(columns))
@@ -380,30 +451,31 @@ func (c QueryCollector) collect(conn *sql.DB, ch chan<- prometheus.Metric) error
 			"name", c.namespace,
 			"columnNumberFromDB", len(columns),
 			"columnNumberFromConfiguration", len(c.columnMapping))
-		return nil
+		return nil, nil
 	}
 
 	for rows.Next() {
 		if err = rows.Scan(scanArgs...); err != nil {
-			return err
+			return nil, err
 		}
 
-		labels, done := c.collectLabels(columns, columnData)
+		labels, done := c.listLabels(columns, columnData)
 		if done {
-			c.collectColumns(columns, columnData, labels, ch)
+			computedMetrics = append(computedMetrics, c.createMetricsFromColumns(columns, columnData, labels)...)
 		}
 	}
 	if err := rows.Err(); err != nil {
 		log.Warning("Error while loading metrics",
 			"err", err.Error())
-		return err
+		return nil, err
 	}
-	return nil
+
+	return computedMetrics, nil
 }
 
 // Collect the list of labels from the database, and returns true if the
 // label extraction succeeded, false otherwise
-func (c QueryCollector) collectLabels(columns []string, columnData []interface{}) ([]string, bool) {
+func (c QueryRunner) listLabels(columns []string, columnData []interface{}) ([]string, bool) {
 	var labels []string
 	for idx, columnName := range columns {
 		if mapping, ok := c.columnMapping[columnName]; ok && mapping.Label {
@@ -421,10 +493,13 @@ func (c QueryCollector) collectLabels(columns []string, columnData []interface{}
 	return labels, true
 }
 
-// Collect the metrics from the database columns
-func (c QueryCollector) collectColumns(columns []string, columnData []interface{},
-	labels []string, ch chan<- prometheus.Metric,
-) {
+// createMetricsFromColumns generates Prometheus metrics from the result columns
+func (c QueryRunner) createMetricsFromColumns(
+	columns []string,
+	columnData []interface{},
+	labels []string,
+) []prometheus.Metric {
+	var computedMetrics []prometheus.Metric
 	for idx, columnName := range columns {
 		mapping, ok := c.columnMapping[columnName]
 		if !ok {
@@ -454,13 +529,20 @@ func (c QueryCollector) collectColumns(columns []string, columnData []interface{
 					"columnData", columnData,
 					"labels", labels)
 			} else {
-				c.collectHistogramMetric(mapping, histogramData, labels, ch)
+				m := c.createHistogramMetric(mapping, histogramData, labels)
+				if m != nil {
+					computedMetrics = append(computedMetrics, m)
+				}
 			}
 
 		default:
-			c.collectConstMetric(mapping, columnData[idx], labels, ch)
+			m := c.createConstMetric(mapping, columnData[idx], labels)
+			if m != nil {
+				computedMetrics = append(computedMetrics, m)
+			}
 		}
 	}
+	return computedMetrics
 }
 
 // createMonitoringTx create a monitoring transaction with read-only access
@@ -498,22 +580,22 @@ func createMonitoringTx(conn *sql.DB) (*sql.Tx, error) {
 }
 
 // describe puts in the channel the metadata we have for the queries we collect
-func (c QueryCollector) describe(ch chan<- *prometheus.Desc) {
+func (c QueryRunner) describe(ch chan<- *prometheus.Desc) {
 	for _, mapSet := range c.columnMapping {
 		ch <- mapSet.Desc
 	}
 }
 
-// collectConstMetric reports to the prometheus library a constant metric
-func (c QueryCollector) collectConstMetric(
-	mapping MetricMap, value interface{}, variableLabels []string, ch chan<- prometheus.Metric,
-) {
+// createConstMetric reports to the prometheus library a constant metric
+func (c QueryRunner) createConstMetric(
+	mapping MetricMap, value interface{}, variableLabels []string,
+) prometheus.Metric {
 	if mapping.Conversion == nil {
 		log.Warning("Missing conversion while parsing value",
 			"namespace", c.namespace,
 			"value", value,
 			"mapping", mapping)
-		return
+		return nil
 	}
 
 	floatData, ok := mapping.Conversion(value)
@@ -522,25 +604,24 @@ func (c QueryCollector) collectConstMetric(
 			"namespace", c.namespace,
 			"value", value,
 			"mapping", mapping)
-		return
+		return nil
 	}
 
 	// Generate the metric
 	metric, err := prometheus.NewConstMetric(mapping.Desc, mapping.Vtype, floatData, variableLabels...)
 	if err != nil {
 		log.Error(err, "while collecting constant metric", "metric", mapping.Name)
-		return
+		return nil
 	}
-	ch <- metric
+	return metric
 }
 
-// collectHistogramMetric reports to the prometheus library an histogram-based metric
-func (c QueryCollector) collectHistogramMetric(
+// createHistogramMetric reports to the prometheus library an histogram-based metric
+func (c QueryRunner) createHistogramMetric(
 	mapping MetricMap,
 	columnData *histogram.Value,
 	variableLabels []string,
-	ch chan<- prometheus.Metric,
-) {
+) prometheus.Metric {
 	metric, err := prometheus.NewConstHistogram(
 		mapping.Desc,
 		columnData.Count, columnData.Sum, columnData.Buckets,
@@ -548,9 +629,9 @@ func (c QueryCollector) collectHistogramMetric(
 	)
 	if err != nil {
 		log.Error(err, "while collecting histogram metric", "metric", mapping.Name)
-		return
+		return nil
 	}
-	ch <- metric
+	return metric
 }
 
 // PluginCollector is the interface for collecting metrics from plugins
diff --git a/pkg/management/postgres/webserver/metricserver/pg_collector.go b/pkg/management/postgres/webserver/metricserver/pg_collector.go
index 876b26bd..8ae483a0 100644
--- a/pkg/management/postgres/webserver/metricserver/pg_collector.go
+++ b/pkg/management/postgres/webserver/metricserver/pg_collector.go
@@ -318,8 +318,47 @@ func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {
 // Collect implements prometheus.Collector, collecting the Metrics values to
 // export.
 func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
-	e.collectPgMetrics(ch)
+	log.Debug("collecting Postgres instance metrics")
+	e.updateInstanceMetrics()
+	e.collectInstanceMetrics(ch)
 
+	if e.queries != nil {
+		var defaultClusterForTTL apiv1.Cluster
+		ttl := defaultClusterForTTL.GetMetricsQueriesTTL().Duration
+		if cluster, _ := e.getCluster(); cluster != nil {
+			ttl = cluster.GetMetricsQueriesTTL().Duration
+		}
+
+		if e.queries.ShouldUpdate(ttl) {
+			log.Trace("no cache, updating metrics from queries", "ttlSeconds", ttl.Seconds())
+			e.updateMetricsFromQueries()
+		} else {
+			log.Debug("using cached metrics from queries, as TTL is not exceeded",
+				"ttlSeconds", ttl.Seconds())
+			e.queries.RecordCacheHit()
+		}
+		log.Debug("collecting metrics from queries")
+		e.queries.Collect(ch)
+	}
+}
+
+func (e *Exporter) updateMetricsFromQueries() {
+	// Work on predefined metrics and custom queries
+	label := "Collect." + e.queries.Name()
+	collectionStart := time.Now()
+	log.Debug("updating metrics from queries")
+	if err := e.queries.Update(); err != nil {
+		log.Error(err, "Error during query update", "collector", e.queries.Name())
+		e.Metrics.PgCollectionErrors.WithLabelValues(label).Inc()
+		e.Metrics.Error.Set(1)
+	}
+	log.Debug("created metrics from user queries",
+		"duration", time.Since(collectionStart).Seconds())
+	e.Metrics.CollectionDuration.WithLabelValues(label).Set(time.Since(collectionStart).Seconds())
+}
+
+// collectInstanceMetrics sends the computed instance metrics to the channel
+func (e *Exporter) collectInstanceMetrics(ch chan<- prometheus.Metric) {
 	ch <- e.Metrics.CollectionsTotal
 	ch <- e.Metrics.Error
 	e.Metrics.PgCollectionErrors.Collect(ch)
@@ -359,7 +398,8 @@ func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
 	}
 }
 
-func (e *Exporter) collectPgMetrics(ch chan<- prometheus.Metric) {
+// updateInstanceMetrics updates metrics from the instance manager
+func (e *Exporter) updateInstanceMetrics() {
 	e.Metrics.CollectionsTotal.Inc()
 	collectionStart := time.Now()
 	if e.instance.IsFenced() {
@@ -395,18 +435,6 @@ func (e *Exporter) collectPgMetrics(ch chan<- prometheus.Metric) {
 	e.Metrics.Error.Set(0)
 	e.Metrics.CollectionDuration.WithLabelValues("Collect.up").Set(time.Since(collectionStart).Seconds())
 
-	// Work on predefined metrics and custom queries
-	if e.queries != nil {
-		label := "Collect." + e.queries.Name()
-		collectionStart := time.Now()
-		if err := e.queries.Collect(ch); err != nil {
-			log.Error(err, "Error during collection", "collector", e.queries.Name())
-			e.Metrics.PgCollectionErrors.WithLabelValues(label).Inc()
-			e.Metrics.Error.Set(1)
-		}
-		e.Metrics.CollectionDuration.WithLabelValues(label).Set(time.Since(collectionStart).Seconds())
-	}
-
 	isPrimary, err := e.instance.IsPrimary()
 	if err != nil {
 		log.Error(err, "unable to get if primary")



Test patch:
diff --git a/pkg/management/postgres/metrics/collector_test.go b/pkg/management/postgres/metrics/collector_test.go
index acc2a71a..2f3283e8 100644
--- a/pkg/management/postgres/metrics/collector_test.go
+++ b/pkg/management/postgres/metrics/collector_test.go
@@ -20,10 +20,15 @@ SPDX-License-Identifier: Apache-2.0
 package metrics
 
 import (
+	"database/sql"
+
+	"github.com/DATA-DOG/go-sqlmock"
 	"github.com/cloudnative-pg/cnpg-i/pkg/metrics"
 	"github.com/prometheus/client_golang/prometheus"
+	io_prometheus_client "github.com/prometheus/client_model/go"
 
 	pluginClient "github.com/cloudnative-pg/cloudnative-pg/internal/cnpi/plugin/client"
+	postgresutils "github.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres/utils"
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
@@ -65,45 +70,43 @@ var _ = Describe("Set default queries", Ordered, func() {
 	})
 })
 
-var _ = Describe("QueryCollector tests", func() {
+var _ = Describe("QueryRunner tests", func() {
 	Context("collect metric tests", func() {
 		It("should ensure that a metric without conversion is discarded", func() {
-			qc := QueryCollector{}
-			ch := make(chan prometheus.Metric, 10)
+			qc := QueryRunner{}
 			metricMap := MetricMap{
 				Name:       "MALFORMED_COUNTER",
 				Discard:    true,
 				Conversion: nil,
 				Label:      false,
 			}
-			qc.collectConstMetric(
+			m := qc.createConstMetric(
 				metricMap,
 				"COUNTER_TEST",
 				[]string{"COUNTER_TEST"},
-				ch,
 			)
-			Expect(ch).To(BeEmpty())
+			Expect(m).To(BeNil())
 		})
 
 		It("should ensure that a metric with a bad conversion value is discarded", func() {
-			qc := QueryCollector{}
-			ch := make(chan prometheus.Metric, 10)
+			qc := QueryRunner{}
 			cm := ColumnMapping{Usage: COUNTER}
 			cs := cm.ToMetricMap("TEST_COLUMN", "TEST_NAMESPACE", []string{"TEST_VARIABLE"})
 			for _, metricMap := range cs {
 				// int are not converted
-				qc.collectConstMetric(metricMap, 12, []string{"TEST"}, ch)
+				m := qc.createConstMetric(metricMap, 12, []string{"TEST"})
+				Expect(m).To(BeNil())
 			}
-			Expect(ch).To(BeEmpty())
 		})
 
 		It("should ensure that a correctly formed counter is sent", func() {
-			qc := QueryCollector{}
+			qc := QueryRunner{}
 			ch := make(chan prometheus.Metric, 10)
 			cm := ColumnMapping{Usage: COUNTER}
 			cs := cm.ToMetricMap("TEST_COLUMN", "TEST_NAMESPACE", []string{"TEST_VARIABLE"})
 			for _, metricMap := range cs {
-				qc.collectConstMetric(metricMap, int64(12), []string{"TEST"}, ch)
+				m := qc.createConstMetric(metricMap, int64(12), []string{"TEST"})
+				ch <- m
 			}
 			Expect(ch).To(HaveLen(1))
 			res := <-ch
@@ -111,45 +114,127 @@ var _ = Describe("QueryCollector tests", func() {
 			Expect(desc).To(ContainSubstring("TEST_NAMESPACE_TEST_COLUMN"))
 			Expect(desc).To(ContainSubstring("TEST_VARIABLE"))
 		})
+	})
 
-		Context("fetch label testing", func() {
-			It("should correctly fetch the mapped labels", func() {
-				qc := QueryCollector{
-					columnMapping: map[string]MetricMap{
-						"LABEL_ENABLED": {
-							Label: true,
-						},
-						"LABEL_NOT_ENABLED": {
-							Label: false,
+	Context("fetch label testing", func() {
+		It("should correctly fetch the mapped labels", func() {
+			qc := QueryRunner{
+				columnMapping: map[string]MetricMap{
+					"LABEL_ENABLED": {
+						Label: true,
+					},
+					"LABEL_NOT_ENABLED": {
+						Label: false,
+					},
+				},
+			}
+			labels, success := qc.listLabels(
+				[]string{"LABEL_ENABLED", "LABEL_NOT_ENABLED"},
+				[]interface{}{"SHOULD_FETCH", "SHOULD_NOT_FETCH"},
+			)
+			Expect(success).To(BeTrue())
+			Expect(labels).To(HaveLen(1))
+			Expect(labels).To(ContainElements("SHOULD_FETCH"))
+		})
+
+		It("should report success false when the fetched data conversion is not supported", func() {
+			qc := QueryRunner{
+				columnMapping: map[string]MetricMap{
+					"LABEL_ENABLED": {
+						Label: true,
+					},
+				},
+			}
+			labels, success := qc.listLabels(
+				[]string{"LABEL_ENABLED"},
+				// int is not supported
+				[]interface{}{234},
+			)
+
+			Expect(success).To(BeFalse())
+			Expect(labels).To(BeZero())
+		})
+	})
+
+	Context("receiving query data from the database", func() {
+		var (
+			dbMock sqlmock.Sqlmock
+			db     *sql.DB
+			err    error
+		)
+
+		loPagesQuery := `SELECT pg_catalog.current_database() as datname, relpages as lo_pages
+			FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON (n.oid = c.relnamespace)
+			WHERE n.nspname = 'pg_catalog' AND c.relname = 'pg_largeobject';`
+
+		defaultQueries := UserQueries{
+			"collector": UserQuery{
+				Query:           loPagesQuery,
+				TargetDatabases: []string{"*"},
+				Metrics: []Mapping{
+					{
+						"datname": ColumnMapping{
+							Usage:       LABEL,
+							Description: "Name of the database",
 						},
 					},
-				}
-				labels, success := qc.collectLabels(
-					[]string{"LABEL_ENABLED", "LABEL_NOT_ENABLED"},
-					[]interface{}{"SHOULD_FETCH", "SHOULD_NOT_FETCH"},
-				)
-				Expect(success).To(BeTrue())
-				Expect(labels).To(HaveLen(1))
-				Expect(labels).To(ContainElements("SHOULD_FETCH"))
-			})
-
-			It("should report success false when the fetched data conversion is not supported", func() {
-				qc := QueryCollector{
-					columnMapping: map[string]MetricMap{
-						"LABEL_ENABLED": {
-							Label: true,
+					{
+						"lo_pages": ColumnMapping{
+							Usage:       GAUGE,
+							Description: "Estimated number of pages in the pg_largeobject table",
 						},
 					},
-				}
-				labels, success := qc.collectLabels(
-					[]string{"LABEL_ENABLED"},
-					// int is not supported
-					[]interface{}{234},
-				)
-
-				Expect(success).To(BeFalse())
-				Expect(labels).To(BeZero())
-			})
+				},
+			},
+		}
+
+		BeforeEach(func() {
+			db, dbMock, err = sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
+			Expect(err).ToNot(HaveOccurred())
+		})
+
+		It("should compute the default metrics successfully", func() {
+			metricMap := MetricMapSet{
+				"datname": MetricMap{
+					Name:       "datname",
+					Discard:    true,
+					Conversion: nil,
+					Label:      true,
+				},
+				"lo_pages": MetricMap{
+					Name:  "lo_pages",
+					Vtype: prometheus.GaugeValue,
+					Desc: prometheus.NewDesc(
+						"collector_lo_pages",
+						defaultQueries["collector"].Metrics[1]["lo_pages"].Description, []string{"lo_pages"}, nil),
+					Conversion: postgresutils.DBToFloat64,
+					Label:      false,
+				},
+			}
+
+			qc := QueryRunner{
+				namespace:      "foo",
+				userQuery:      defaultQueries["collector"],
+				columnMapping:  metricMap,
+				variableLabels: []string{"foo"},
+			}
+			_ = metricMap
+			dbMock.ExpectBegin()
+			dbMock.ExpectExec("SET application_name TO cnpg_metrics_exporter").WillReturnResult(sqlmock.NewResult(0, 1))
+			dbMock.ExpectExec("SET standard_conforming_strings TO on").WillReturnResult(sqlmock.NewResult(0, 1))
+			dbMock.ExpectExec("SET ROLE TO pg_monitor").WillReturnResult(sqlmock.NewResult(0, 1))
+			dbMock.ExpectQuery(loPagesQuery).WillReturnRows(sqlmock.NewRows(
+				[]string{"datname", "lo_pages"}).
+				AddRow(`app`, 3))
+			m, err := qc.computeMetrics(db)
+			Expect(err).NotTo(HaveOccurred())
+			Expect(m).To(HaveLen(1))
+			Expect(m[0].Desc().String()).To(ContainSubstring(
+				defaultQueries["collector"].Metrics[1]["lo_pages"].Description))
+			var gauge io_prometheus_client.Metric
+			Expect(m[0].Write(&gauge)).To(Succeed())
+			Expect(gauge.GetGauge().GetValue()).To(BeEquivalentTo(3))
+			Expect(dbMock.ExpectationsWereMet()).To(Succeed())
 		})
 	})
 })
@@ -276,3 +361,231 @@ var _ = Describe("sendPluginMetrics tests", func() {
 		Expect(ch).To(BeEmpty())
 	})
 })
+
+var _ = Describe("QueriesCollector timestamp metric tests", func() {
+	var collector *QueriesCollector
+
+	BeforeEach(func() {
+		collector = NewQueriesCollector("test_collector", nil, "postgres")
+	})
+
+	It("should initialize with zero timestamp", func() {
+		Expect(collector.timeLastUpdated.IsZero()).To(BeTrue())
+
+		// Collect the timestamp metric
+		ch := make(chan prometheus.Metric, 10)
+		collector.lastUpdateTimestamp.Collect(ch)
+
+		Expect(ch).To(HaveLen(1))
+		metric := <-ch
+
+		var m io_prometheus_client.Metric
+		Expect(metric.Write(&m)).To(Succeed())
+		// Should be 0 when not yet updated
+		Expect(m.GetGauge().GetValue()).To(BeEquivalentTo(0))
+	})
+
+	It("should expose timestamp metric in Describe", func() {
+		ch := make(chan *prometheus.Desc, 10)
+		collector.Describe(ch)
+
+		// The descriptor should be registered
+		Expect(collector.lastUpdateTimestamp).NotTo(BeNil())
+
+		// Should have collected some descriptors
+		Expect(ch).ToNot(BeEmpty())
+	})
+
+	It("should include timestamp metric in Collect output", func() {
+		ch := make(chan prometheus.Metric, 10)
+		collector.Collect(ch)
+
+		// Should include at least the timestamp and error gauge metrics
+		Expect(len(ch)).To(BeNumerically(">=", 2))
+
+		// Find the timestamp metric by checking gauge values
+		var foundTimestamp bool
+		for len(ch) > 0 {
+			metric := <-ch
+			var m io_prometheus_client.Metric
+			if metric.Write(&m) == nil && m.Gauge != nil {
+				foundTimestamp = true
+				break
+			}
+		}
+		Expect(foundTimestamp).To(BeTrue())
+	})
+})
+
+var _ = Describe("QueriesCollector cache hit/miss metrics tests", func() {
+	var collector *QueriesCollector
+
+	BeforeEach(func() {
+		collector = NewQueriesCollector("test_collector", nil, "postgres")
+	})
+
+	It("should initialize cache metrics with zero values", func() {
+		ch := make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+		collector.cacheMiss.Collect(ch)
+
+		Expect(ch).To(HaveLen(2))
+
+		// Check cache hits
+		metric := <-ch
+		var m1 io_prometheus_client.Metric
+		Expect(metric.Write(&m1)).To(Succeed())
+		Expect(m1.GetGauge().GetValue()).To(BeEquivalentTo(0))
+
+		// Check cache misses
+		metric = <-ch
+		var m2 io_prometheus_client.Metric
+		Expect(metric.Write(&m2)).To(Succeed())
+		Expect(m2.GetGauge().GetValue()).To(BeEquivalentTo(0))
+	})
+
+	It("should include cache metrics in Describe", func() {
+		ch := make(chan *prometheus.Desc, 20)
+		collector.Describe(ch)
+
+		Expect(collector.cacheHits).NotTo(BeNil())
+		Expect(collector.cacheMiss).NotTo(BeNil())
+		Expect(ch).ToNot(BeEmpty())
+	})
+
+	It("should include cache metrics in Collect output", func() {
+		ch := make(chan prometheus.Metric, 20)
+		collector.Collect(ch)
+
+		// Should include cache hits and cache misses
+		Expect(len(ch)).To(BeNumerically(">=", 4))
+
+		metricNames := make(map[string]float64)
+		for len(ch) > 0 {
+			metric := <-ch
+			var m io_prometheus_client.Metric
+			if metric.Write(&m) == nil && m.Gauge != nil {
+				desc := metric.Desc().String()
+				metricNames[desc] = m.GetGauge().GetValue()
+			}
+		}
+
+		// Check that we collected both metrics
+		Expect(len(metricNames)).To(BeNumerically(">=", 2))
+	})
+
+	It("should increment cache hits when RecordCacheHit is called", func() {
+		// Record multiple cache hits
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+
+		ch := make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+
+		Expect(ch).To(HaveLen(1))
+		metric := <-ch
+		var m io_prometheus_client.Metric
+		Expect(metric.Write(&m)).To(Succeed())
+		Expect(m.GetGauge().GetValue()).To(BeEquivalentTo(3))
+	})
+
+	It("should reset cache metrics when Update is called (cache miss)", func() {
+		// Simulate some cache hits first
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+
+		// Verify cache hits were recorded
+		ch := make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+		metric := <-ch
+		var m1 io_prometheus_client.Metric
+		Expect(metric.Write(&m1)).To(Succeed())
+		Expect(m1.GetGauge().GetValue()).To(BeEquivalentTo(2))
+
+		// Now call Update (which represents a cache miss)
+		// Note: This will fail without a real instance, but we can test the reset logic
+		collector.metricsMutex.Lock()
+		collector.cacheHits.Set(0)
+		collector.cacheMiss.Set(1)
+		collector.metricsMutex.Unlock()
+
+		// Verify cache hits were reset to 0 and cache misses set to 1
+		ch = make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+		collector.cacheMiss.Collect(ch)
+
+		Expect(ch).To(HaveLen(2))
+
+		// Check cache hits (should be reset to 0)
+		metric = <-ch
+		var m2 io_prometheus_client.Metric
+		Expect(metric.Write(&m2)).To(Succeed())
+		Expect(m2.GetGauge().GetValue()).To(BeEquivalentTo(0))
+
+		// Check cache misses (should be 1)
+		metric = <-ch
+		var m3 io_prometheus_client.Metric
+		Expect(metric.Write(&m3)).To(Succeed())
+		Expect(m3.GetGauge().GetValue()).To(BeEquivalentTo(1))
+	})
+
+	It("should track cache hits for current cache period only", func() {
+		// Simulate a cache period:
+		// 1. Cache miss (update)
+		collector.metricsMutex.Lock()
+		collector.cacheHits.Set(0)
+		collector.cacheMiss.Set(1)
+		collector.metricsMutex.Unlock()
+
+		// 2. Multiple cache hits during this period
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+
+		// 3. Verify we have 4 hits and 1 miss
+		ch := make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+		collector.cacheMiss.Collect(ch)
+
+		Expect(ch).To(HaveLen(2))
+
+		hitMetric := <-ch
+		var mHits io_prometheus_client.Metric
+		Expect(hitMetric.Write(&mHits)).To(Succeed())
+		Expect(mHits.GetGauge().GetValue()).To(BeEquivalentTo(4))
+
+		missMetric := <-ch
+		var mMisses io_prometheus_client.Metric
+		Expect(missMetric.Write(&mMisses)).To(Succeed())
+		Expect(mMisses.GetGauge().GetValue()).To(BeEquivalentTo(1))
+
+		// 4. New cache period starts (cache miss/update)
+		collector.metricsMutex.Lock()
+		collector.cacheHits.Set(0)
+		collector.cacheMiss.Set(1)
+		collector.metricsMutex.Unlock()
+
+		// 5. New hits in the new period
+		collector.RecordCacheHit()
+		collector.RecordCacheHit()
+
+		// 6. Verify counters were reset and now show the new period
+		ch = make(chan prometheus.Metric, 10)
+		collector.cacheHits.Collect(ch)
+		collector.cacheMiss.Collect(ch)
+
+		Expect(ch).To(HaveLen(2))
+
+		hitMetric = <-ch
+		var mNewHits io_prometheus_client.Metric
+		Expect(hitMetric.Write(&mNewHits)).To(Succeed())
+		Expect(mNewHits.GetGauge().GetValue()).To(BeEquivalentTo(2))
+
+		missMetric = <-ch
+		var mNewMisses io_prometheus_client.Metric
+		Expect(missMetric.Write(&mNewMisses)).To(Succeed())
+		Expect(mNewMisses.GetGauge().GetValue()).To(BeEquivalentTo(1))
+	})
+})
diff --git a/tests/e2e/asserts_test.go b/tests/e2e/asserts_test.go
index ad4813db..426fd339 100644
--- a/tests/e2e/asserts_test.go
+++ b/tests/e2e/asserts_test.go
@@ -1448,22 +1448,25 @@ func AssertMetricsData(namespace, targetOne, targetTwo, targetSecret string, clu
 		Expect(err).ToNot(HaveOccurred())
 		for _, pod := range podList.Items {
 			podName := pod.GetName()
-			out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, cluster.IsMetricsTLSEnabled())
-			Expect(err).ToNot(HaveOccurred())
-			Expect(strings.Contains(out,
-				fmt.Sprintf(`cnpg_some_query_rows{datname="%v"} 0`, targetOne))).Should(BeTrue(),
-				"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
-			Expect(strings.Contains(out,
-				fmt.Sprintf(`cnpg_some_query_rows{datname="%v"} 0`, targetTwo))).Should(BeTrue(),
-				"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
-			Expect(strings.Contains(out, fmt.Sprintf(`cnpg_some_query_test_rows{datname="%v"} 1`,
-				targetSecret))).Should(BeTrue(),
-				"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+			var out string
+			var err error
+			Eventually(func(g Gomega) {
+				out, err = proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, cluster.IsMetricsTLSEnabled())
+				g.Expect(err).ToNot(HaveOccurred())
+				g.Expect(strings.Contains(out,
+					fmt.Sprintf(`cnpg_some_query_rows{datname="%v"} 0`, targetOne))).Should(BeTrue(),
+					"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+				g.Expect(strings.Contains(out,
+					fmt.Sprintf(`cnpg_some_query_rows{datname="%v"} 0`, targetTwo))).Should(BeTrue(),
+					"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+				g.Expect(strings.Contains(out, fmt.Sprintf(`cnpg_some_query_test_rows{datname="%v"} 1`,
+					targetSecret))).Should(BeTrue(),
+					"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+			}, testTimeouts[timeouts.Short]).To(Succeed())
 
 			if pod.Name != cluster.Status.CurrentPrimary {
 				continue
 			}
-
 			Expect(out).Should(ContainSubstring("last_available_backup_timestamp"))
 			Expect(out).Should(ContainSubstring("last_failed_backup_timestamp"))
 		}
@@ -2486,24 +2489,26 @@ func collectAndAssertDefaultMetricsPresentOnEachPod(
 		Expect(err).ToNot(HaveOccurred())
 		for _, pod := range podList.Items {
 			podName := pod.GetName()
-			out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, tlsEnabled)
-			Expect(err).ToNot(HaveOccurred())
+			Eventually(func(g Gomega) {
+				out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, tlsEnabled)
+				g.Expect(err).ToNot(HaveOccurred())
 
-			// error should be zero on each pod metrics
-			Expect(strings.Contains(out, "cnpg_collector_last_collection_error 0")).Should(BeTrue(),
-				"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
-			// verify that, default set of monitoring queries should not be existed on each pod
-			for _, data := range defaultMetrics {
-				if expectPresent {
-					Expect(strings.Contains(out, data)).Should(BeTrue(),
-						"Metric collection issues on pod %v."+
-							"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
-				} else {
-					Expect(strings.Contains(out, data)).Should(BeFalse(),
-						"Metric collection issues on pod %v."+
-							"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
+				// error should be zero on each pod metrics
+				g.Expect(strings.Contains(out, "cnpg_collector_last_collection_error 0")).Should(BeTrue(),
+					"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+				// verify that, default set of monitoring queries should not be existed on each pod
+				for _, data := range defaultMetrics {
+					if expectPresent {
+						g.Expect(strings.Contains(out, data)).Should(BeTrue(),
+							"Metric collection issues on pod %v."+
+								"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
+					} else {
+						g.Expect(strings.Contains(out, data)).Should(BeFalse(),
+							"Metric collection issues on pod %v."+
+								"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
+					}
 				}
-			}
+			}, testTimeouts[timeouts.Short]).Should(Succeed())
 		}
 	})
 }
@@ -2546,18 +2551,20 @@ func collectAndAssertCollectorMetricsPresentOnEachPod(cluster *apiv1.Cluster) {
 		Expect(err).ToNot(HaveOccurred())
 		for _, pod := range podList.Items {
 			podName := pod.GetName()
-			out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, cluster.IsMetricsTLSEnabled())
-			Expect(err).ToNot(HaveOccurred())
+			Eventually(func(g Gomega) {
+				out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod, cluster.IsMetricsTLSEnabled())
+				g.Expect(err).ToNot(HaveOccurred())
 
-			// error should be zero on each pod metrics
-			Expect(strings.Contains(out, "cnpg_collector_last_collection_error 0")).Should(BeTrue(),
-				"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
-			// verify that, default set of monitoring queries should not be existed on each pod
-			for _, data := range cnpgCollectorMetrics {
-				Expect(strings.Contains(out, data)).Should(BeTrue(),
-					"Metric collection issues on pod %v."+
-						"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
-			}
+				// error should be zero on each pod metrics
+				g.Expect(strings.Contains(out, "cnpg_collector_last_collection_error 0")).Should(BeTrue(),
+					"Metric collection issues on %v.\nCollected metrics:\n%v", podName, out)
+				// verify that, default set of monitoring queries should not be existed on each pod
+				for _, data := range cnpgCollectorMetrics {
+					g.Expect(strings.Contains(out, data)).Should(BeTrue(),
+						"Metric collection issues on pod %v."+
+							"\nFor expected keyword '%v'.\nCollected metrics:\n%v", podName, data, out)
+				}
+			}, testTimeouts[timeouts.Short]).To(Succeed())
 		}
 	})
 }
@@ -2939,7 +2946,7 @@ func assertPredicateClusterHasPhase(namespace, clusterName string, phase []strin
 // If any assertion fails, it will print an error message with details about the failed metric collection.
 //
 // Note: This function is typically used in testing scenarios to validate metric collection behavior.
-func assertIncludesMetrics(rawMetricsOutput string, expectedMetrics map[string]*regexp.Regexp) {
+func assertIncludesMetrics(g Gomega, rawMetricsOutput string, expectedMetrics map[string]*regexp.Regexp) {
 	debugDetails := fmt.Sprintf("Priting rawMetricsOutput:\n%s", rawMetricsOutput)
 	withDebugDetails := func(baseErrMessage string) string {
 		return fmt.Sprintf("%s\n%s\n", baseErrMessage, debugDetails)
@@ -2950,22 +2957,22 @@ func assertIncludesMetrics(rawMetricsOutput string, expectedMetrics map[string]*
 
 		// match a metric with the value of expectedMetrics key
 		match := re.FindString(rawMetricsOutput)
-		Expect(match).NotTo(BeEmpty(), withDebugDetails(fmt.Sprintf("Found no match for metric %s", key)))
+		g.Expect(match).NotTo(BeEmpty(), withDebugDetails(fmt.Sprintf("Found no match for metric %s", key)))
 
 		// extract the value from the metric previously matched
 		value := strings.Fields(match)[1]
-		Expect(strings.Fields(match)[1]).NotTo(BeEmpty(),
+		g.Expect(strings.Fields(match)[1]).NotTo(BeEmpty(),
 			withDebugDetails(fmt.Sprintf("Found no result for metric %s.Metric line: %s", key, match)))
 
 		// expect the expectedMetrics regexp to match the value of the metric
-		Expect(valueRe.MatchString(value)).To(BeTrue(),
+		g.Expect(valueRe.MatchString(value)).To(BeTrue(),
 			withDebugDetails(fmt.Sprintf("Expected %s to have value %v but got %s", key, valueRe, value)))
 	}
 }
 
-func assertExcludesMetrics(rawMetricsOutput string, nonCollected []string) {
+func assertExcludesMetrics(g Gomega, rawMetricsOutput string, nonCollected []string) {
 	for _, nonCollectable := range nonCollected {
 		// match a metric with the value of expectedMetrics key
-		Expect(rawMetricsOutput).NotTo(ContainSubstring(nonCollectable))
+		g.Expect(rawMetricsOutput).NotTo(ContainSubstring(nonCollectable))
 	}
 }
diff --git a/tests/e2e/metrics_test.go b/tests/e2e/metrics_test.go
index d44dc12a..90d3e33d 100644
--- a/tests/e2e/metrics_test.go
+++ b/tests/e2e/metrics_test.go
@@ -123,11 +123,14 @@ var _ = Describe("Metrics", Label(tests.LabelObservability), func() {
 			// Gather metrics in each pod
 			for _, pod := range podList.Items {
 				By(fmt.Sprintf("checking metrics for pod: %s", pod.Name), func() {
-					out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
-						cluster.IsMetricsTLSEnabled())
-					Expect(err).ToNot(HaveOccurred(), "while getting pod metrics")
-					expectedMetrics := buildExpectedMetrics(cluster, !specs.IsPodPrimary(pod))
-					assertIncludesMetrics(out, expectedMetrics)
+					Eventually(func(g Gomega) error {
+						out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
+							cluster.IsMetricsTLSEnabled())
+						g.Expect(err).ToNot(HaveOccurred(), "while getting pod metrics")
+						expectedMetrics := buildExpectedMetrics(cluster, !specs.IsPodPrimary(pod))
+						assertIncludesMetrics(g, out, expectedMetrics)
+						return nil
+					}, testTimeouts[timeouts.Short]).Should(Succeed())
 				})
 			}
 		})
@@ -235,11 +238,14 @@ var _ = Describe("Metrics", Label(tests.LabelObservability), func() {
 			// Gather metrics in each pod
 			for _, pod := range podList.Items {
 				By(fmt.Sprintf("checking metrics for pod: %s", pod.Name), func() {
-					out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
-						cluster.IsMetricsTLSEnabled())
-					Expect(err).ToNot(HaveOccurred(), "while getting pod metrics")
-					assertIncludesMetrics(out, expectedMetrics)
-					assertExcludesMetrics(out, nonCollectableMetrics)
+					Eventually(func(g Gomega) error {
+						out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
+							cluster.IsMetricsTLSEnabled())
+						g.Expect(err).ToNot(HaveOccurred(), "while getting pod metrics")
+						assertIncludesMetrics(g, out, expectedMetrics)
+						assertExcludesMetrics(g, out, nonCollectableMetrics)
+						return nil
+					}, testTimeouts[timeouts.Short]).Should(Succeed())
 				})
 			}
 		})
@@ -354,12 +360,14 @@ var _ = Describe("Metrics", Label(tests.LabelObservability), func() {
 					time.Duration(timeouts.DefaultTestTimeouts[timeouts.Short])*time.Second,
 				).Should(Succeed(), fmt.Sprintf("on pod %v", pod.Name))
 
-				out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
-					replicaCluster.IsMetricsTLSEnabled())
-				Expect(err).ShouldNot(HaveOccurred(),
-					fmt.Sprintf("while getting pod metrics for pod: %v", pod.Name))
-				Expect(strings.Split(out, "\n")).Should(ContainElement(expectedMetric),
-					fmt.Sprintf("expected metric %v not found in pod %v", expectedMetric, pod.Name))
+				Eventually(func(g Gomega) {
+					out, err := proxy.RetrieveMetricsFromInstance(env.Ctx, env.Interface, pod,
+						replicaCluster.IsMetricsTLSEnabled())
+					g.Expect(err).ShouldNot(HaveOccurred(),
+						fmt.Sprintf("while getting pod metrics for pod: %v", pod.Name))
+					g.Expect(strings.Split(out, "\n")).Should(ContainElement(expectedMetric),
+						fmt.Sprintf("expected metric %v not found in pod %v", expectedMetric, pod.Name))
+				}, testTimeouts[timeouts.Short]).Should(Succeed())
 			}
 		})