Repository: ansible/awx

Issue:
[Metrics&Observability] Surface DB connections in /api/v2/metrics
<!-- Issues are for **concrete, actionable bugs and feature requests** only - if you're just asking for debugging help or technical support, please use:

- http://web.libera.chat/?channels=#ansible-awx
- https://groups.google.com/forum/#!forum/awx-project

We have to limit this because of limited volunteer time to respond to issues! -->

##### ISSUE TYPE
 - Feature Idea

##### SUMMARY
<!-- Briefly describe the problem or desired enhancement. -->
Surface the number of database connections awx is making to roll more of the data administrators need to track into /api/v2/metrics 

Users can track this now by creating some kind of exporter that directly queries PG, but this is in general hacky and not portable. We know if we exceed the max connections that things start to go bad, so it is a best practice to track this number.

Maybe we can follow pattern of https://github.com/ansible/awx/blob/4dc956c76f31accc3f57599d5e6df1735ffa2c16/awx/main/analytics/collectors.py#L255-L260

And create a `database_stats` function that returns a dictionary with a key for current connections by awx user. I say a dictionary so we can later expand that if desired, like adding number of idle vs active connections, or the max connections. These could then get surfaced like we do for guages like ORG_COUNT. https://github.com/ansible/awx/blob/4dc956c76f31accc3f57599d5e6df1735ffa2c16/awx/main/analytics/metrics.py#L18


Discussion:
@jainnikhil30 expressed interest in doing this

Resolving patch:
diff --git a/awx/main/analytics/collectors.py b/awx/main/analytics/collectors.py
--- a/awx/main/analytics/collectors.py
+++ b/awx/main/analytics/collectors.py
@@ -129,7 +129,7 @@ def config(since, **kwargs):
     }
 
 
-@register('counts', '1.1', description=_('Counts of objects such as organizations, inventories, and projects'))
+@register('counts', '1.2', description=_('Counts of objects such as organizations, inventories, and projects'))
 def counts(since, **kwargs):
     counts = {}
     for cls in (
@@ -172,6 +172,13 @@ def counts(since, **kwargs):
         .count()
     )
     counts['pending_jobs'] = models.UnifiedJob.objects.exclude(launch_type='sync').filter(status__in=('pending',)).count()
+    if connection.vendor == 'postgresql':
+        with connection.cursor() as cursor:
+            cursor.execute(f"select count(*) from pg_stat_activity where datname=\'{connection.settings_dict['NAME']}\'")
+            counts['database_connections'] = cursor.fetchone()[0]
+    else:
+        # We should be using postgresql, but if we do that change that ever we should change the below value
+        counts['database_connections'] = 1
     return counts
 
 
diff --git a/awx/main/analytics/metrics.py b/awx/main/analytics/metrics.py
--- a/awx/main/analytics/metrics.py
+++ b/awx/main/analytics/metrics.py
@@ -126,6 +126,8 @@ def metrics():
     LICENSE_INSTANCE_TOTAL = Gauge('awx_license_instance_total', 'Total number of managed hosts provided by your license', registry=REGISTRY)
     LICENSE_INSTANCE_FREE = Gauge('awx_license_instance_free', 'Number of remaining managed hosts provided by your license', registry=REGISTRY)
 
+    DATABASE_CONNECTIONS = Gauge('awx_database_connections_total', 'Number of connections to database', registry=REGISTRY)
+
     license_info = get_license()
     SYSTEM_INFO.info(
         {
@@ -163,6 +165,8 @@ def metrics():
     USER_SESSIONS.labels(type='user').set(current_counts['active_user_sessions'])
     USER_SESSIONS.labels(type='anonymous').set(current_counts['active_anonymous_sessions'])
 
+    DATABASE_CONNECTIONS.set(current_counts['database_connections'])
+
     all_job_data = job_counts(None)
     statuses = all_job_data.get('status', {})
     for status, value in statuses.items():


Test patch:
diff --git a/awx/main/tests/functional/analytics/test_counts.py b/awx/main/tests/functional/analytics/test_counts.py
--- a/awx/main/tests/functional/analytics/test_counts.py
+++ b/awx/main/tests/functional/analytics/test_counts.py
@@ -26,6 +26,7 @@ def test_empty():
         "workflow_job_template": 0,
         "unified_job": 0,
         "pending_jobs": 0,
+        "database_connections": 1,
     }
 
 
diff --git a/awx/main/tests/functional/analytics/test_metrics.py b/awx/main/tests/functional/analytics/test_metrics.py
--- a/awx/main/tests/functional/analytics/test_metrics.py
+++ b/awx/main/tests/functional/analytics/test_metrics.py
@@ -31,6 +31,7 @@
     'awx_license_instance_total': 0,
     'awx_license_instance_free': 0,
     'awx_pending_jobs_total': 0,
+    'awx_database_connections_total': 1,
 }