Repository: stackrox/stackrox
Pull request #14590: ROX-27124: Expose Policy as Code Metrics
Author: login: ajheflin
Labels: name: area/central
name: area/postgres
Created: 2025-03-12T17:10:07Z | Merged: 2025-03-27T14:20:03Z

### Description

<!--
A detailed explanation of the changes in your PR. Feel free to remove this
section if the title of your PR is sufficiently descriptive. To learn more
about contributing to this project, check "*.md" files under:
    https://github.com/stackrox/stackrox/tree/master/.github
-->

This PR adds metrics for how many Policy as Code CRs have been posted to the API. One thing to consider may be adding a decrement functionality on the delete CR endpoint in the event that a user is testing the functionality of this endpoint and posting a bunch of CRs, deleting them, and then re-posting the same CRs, but I'll leave that decision to somebody more knowledgeable than I.

### User-facing documentation

- [ ] CHANGELOG is updated **OR** update is not needed
- [ ] [documentation PR](https://spaces.redhat.com/display/StackRox/Submitting+a+User+Documentation+Pull+Request) is created and is linked above **OR** is not needed

### Testing and quality

<!--
General Availability requirements: https://github.com/stackrox/stackrox/blob/master/PR_GA.md
Feature Flags usage: https://github.com/stackrox/stackrox/blob/master/pkg/features/README.md
-->

- [ ] the change is production ready: the change is GA or otherwise the functionality is gated by a feature flag
- [ ] CI results are inspected

#### Automated testing

<!--
If no tests have been contributed, please explain why unless it's obvious,
e.g., the PR is a one-line comment change.
-->

- [ ] added unit tests
- [ ] added e2e tests
- [ ] added regression tests
- [ ] added compatibility tests
- [ ] modified existing tests

#### How I validated my change

<!--
Use this space to explain **how you validated** that **your change functions
exactly how you expect it**. Feel free to attach JSON snippets, curl commands,
screenshots, etc. Apply a simple benchmark: would the information you provided
convince any reviewer or any external reader that you did enough to validate
your change.

It is acceptable to assume trust and keep this section light, e.g. as a
bullet-point list.

It is acceptable to skip testing in cases when CI is sufficient, or it's a
markdown or code comment change only. It is also acceptable to skip testing for
changes that are too taxing to test before merging. In such case you are
responsible for the change after it gets merged which includes reverting,
fixing, etc. Make sure you validate the change ASAP after it gets merged or
explain in PR when the validation will be performed. Explain here why you
skipped testing in case you did so.

Have you created automated tests for your change? Explain here which validation
activities you did manually and why so.
-->


Diff:
diff --git a/central/metrics/central.go b/central/metrics/central.go
index 64e8491a27227..9a2aa589c9469 100644
--- a/central/metrics/central.go
+++ b/central/metrics/central.go
@@ -112,6 +112,13 @@ var (
 		Help:      "A counter of the total number of network endpoints received by Central from Sensor",
 	}, []string{"ClusterID"})
 
+	totalExternalPoliciesGauge = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: metrics.PrometheusNamespace,
+		Subsystem: metrics.CentralSubsystem.String(),
+		Name:      "total_external_policies_count",
+		Help:      "A gauge of the total number of policy as code CRs that have been accepted by Central from Config Controller",
+	})
+
 	riskProcessingHistogramVec = prometheus.NewHistogramVec(prometheus.HistogramOpts{
 		Namespace: metrics.PrometheusNamespace,
 		Subsystem: metrics.CentralSubsystem.String(),
@@ -392,6 +399,14 @@ func IncrementTotalNetworkEndpointsReceivedCounter(clusterID string, numberOfEnd
 	totalNetworkEndpointsReceivedCounter.With(prometheus.Labels{"ClusterID": clusterID}).Add(float64(numberOfEndpoints))
 }
 
+func IncrementTotalExternalPoliciesGauge() {
+	totalExternalPoliciesGauge.Inc()
+}
+
+func DecrementTotalExternalPoliciesGauge() {
+	totalExternalPoliciesGauge.Dec()
+}
+
 // ObserveRiskProcessingDuration adds an observation for risk processing duration.
 func ObserveRiskProcessingDuration(startTime time.Time, riskObjectType string) {
 	riskProcessingHistogramVec.With(prometheus.Labels{"Risk_Reprocessor": riskObjectType}).
diff --git a/central/metrics/init.go b/central/metrics/init.go
index 8aa0a1ed330c3..7f24ddb634b7e 100644
--- a/central/metrics/init.go
+++ b/central/metrics/init.go
@@ -16,6 +16,7 @@ func init() {
 		resourceProcessedCounterVec,
 		totalNetworkFlowsReceivedCounter,
 		totalNetworkEndpointsReceivedCounter,
+		totalExternalPoliciesGauge,
 		sensorEventDurationHistogramVec,
 		riskProcessingHistogramVec,
 		datastoreFunctionDurationHistogramVec,
diff --git a/central/policy/datastore/datastore.go b/central/policy/datastore/datastore.go
index e9fe3dc5bcff2..9aaccf66d9df5 100644
--- a/central/policy/datastore/datastore.go
+++ b/central/policy/datastore/datastore.go
@@ -29,7 +29,7 @@ type DataStore interface {
 
 	AddPolicy(context.Context, *storage.Policy) (string, error)
 	UpdatePolicy(context.Context, *storage.Policy) error
-	RemovePolicy(ctx context.Context, id string) error
+	RemovePolicy(ctx context.Context, policy *storage.Policy) error
 	// This method is allowed to return a v1 proto because it is in the allowed list in
 	// "tools/storedprotos/storeinterface/storeinterface.go".
 	ImportPolicies(ctx context.Context, policies []*storage.Policy, overwrite bool) (responses []*v1.ImportPolicyResponse, allSucceeded bool, err error)
diff --git a/central/policy/datastore/datastore_impl.go b/central/policy/datastore/datastore_impl.go
index b1d763888737b..ab9e9829ee323 100644
--- a/central/policy/datastore/datastore_impl.go
+++ b/central/policy/datastore/datastore_impl.go
@@ -7,6 +7,7 @@ import (
 
 	errorsPkg "github.com/pkg/errors"
 	clusterDS "github.com/stackrox/rox/central/cluster/datastore"
+	"github.com/stackrox/rox/central/metrics"
 	notifierDS "github.com/stackrox/rox/central/notifier/datastore"
 	"github.com/stackrox/rox/central/policy/search"
 	"github.com/stackrox/rox/central/policy/store"
@@ -258,6 +259,11 @@ func (ds *datastoreImpl) AddPolicy(ctx context.Context, policy *storage.Policy)
 	if err := tx.Commit(ctx); err != nil {
 		return "", ds.wrapWithRollback(ctx, tx, err)
 	}
+
+	if clonedPolicy.Source == storage.PolicySource_DECLARATIVE {
+		metrics.IncrementTotalExternalPoliciesGauge()
+	}
+
 	return clonedPolicy.Id, nil
 }
 
@@ -300,7 +306,7 @@ func (ds *datastoreImpl) UpdatePolicy(ctx context.Context, policy *storage.Polic
 }
 
 // RemovePolicy removes a policy from the storage.
-func (ds *datastoreImpl) RemovePolicy(ctx context.Context, id string) error {
+func (ds *datastoreImpl) RemovePolicy(ctx context.Context, policy *storage.Policy) error {
 	if ok, err := workflowAdministrationSAC.WriteAllowed(ctx); err != nil {
 		return err
 	} else if !ok {
@@ -310,7 +316,13 @@ func (ds *datastoreImpl) RemovePolicy(ctx context.Context, id string) error {
 	ds.policyMutex.Lock()
 	defer ds.policyMutex.Unlock()
 
-	return ds.removePolicyNoLock(ctx, id)
+	err := ds.removePolicyNoLock(ctx, policy.GetId())
+
+	if err != nil && policy.Source == storage.PolicySource_DECLARATIVE {
+		metrics.DecrementTotalExternalPoliciesGauge()
+	}
+
+	return err
 }
 
 func (ds *datastoreImpl) removePolicyNoLock(ctx context.Context, id string) error {
diff --git a/central/policy/datastore/datastore_impl_postgres_test.go b/central/policy/datastore/datastore_impl_postgres_test.go
index e5236aa2ca0ca..3790c487a0600 100644
--- a/central/policy/datastore/datastore_impl_postgres_test.go
+++ b/central/policy/datastore/datastore_impl_postgres_test.go
@@ -139,7 +139,7 @@ func (s *PolicyPostgresDataStoreTestSuite) TestInsertUpdatePolicy() {
 	s.Equal(policy.GetId(), results[0].ID)
 
 	// Delete policy
-	s.NoError(s.datastore.RemovePolicy(ctx, policy.GetId()))
+	s.NoError(s.datastore.RemovePolicy(ctx, policy))
 	q = pkgSearch.NewQueryBuilder().AddExactMatches(pkgSearch.Category, "Boo Category 1").ProtoQuery()
 	results, err = s.datastore.Search(ctx, q)
 	s.NoError(err)
@@ -174,7 +174,7 @@ func (s *PolicyPostgresDataStoreTestSuite) TestImportPolicy() {
 	s.Equal(policy.GetId(), results[0].ID)
 
 	// Delete policy
-	s.NoError(s.datastore.RemovePolicy(ctx, policy.GetId()))
+	s.NoError(s.datastore.RemovePolicy(ctx, policy))
 	q = pkgSearch.NewQueryBuilder().AddExactMatches(pkgSearch.Category, "Image Assurance").ProtoQuery()
 	results, err = s.datastore.Search(ctx, q)
 	s.NoError(err)
@@ -287,7 +287,7 @@ func (s *PolicyPostgresDataStoreTestSuite) TestImportOverwriteDefaultPolicy() {
 				s.Equal(c.existingPolicy.GetName(), result.GetName())
 
 				// Delete the policy
-				s.NoError(s.datastore.RemovePolicy(ctx, c.existingPolicy.GetId()))
+				s.NoError(s.datastore.RemovePolicy(ctx, c.existingPolicy))
 			} else {
 				s.NoError(err) // It's not an error just a failure?
 				s.True(allSucceeded)
@@ -300,7 +300,7 @@ func (s *PolicyPostgresDataStoreTestSuite) TestImportOverwriteDefaultPolicy() {
 				s.Equal(c.newPolicy.GetName(), result.GetName())
 
 				// Delete the policy
-				s.NoError(s.datastore.RemovePolicy(ctx, c.newPolicy.GetId()))
+				s.NoError(s.datastore.RemovePolicy(ctx, c.newPolicy))
 			}
 		})
 	}
@@ -376,5 +376,5 @@ func (s *PolicyPostgresDataStoreTestSuite) TestTransactionRollbacks() {
 	s.Equal(1, count)
 
 	// Clean up policy
-	_ = s.datastoreWithMockCategoryDS.RemovePolicy(ctx, policy.GetId())
+	_ = s.datastoreWithMockCategoryDS.RemovePolicy(ctx, policy)
 }
diff --git a/central/policy/datastore/datastore_impl_test.go b/central/policy/datastore/datastore_impl_test.go
index 15e721f5ccc24..2b2061df8f283 100644
--- a/central/policy/datastore/datastore_impl_test.go
+++ b/central/policy/datastore/datastore_impl_test.go
@@ -98,7 +98,7 @@ func (s *PolicyDatastoreTestSuite) TestReplacingResourceAccess() {
 	_, err = s.datastore.AddPolicy(s.hasReadWorkflowAdministrationAccess, policy)
 	s.Error(err)
 	s.ErrorIs(err, sac.ErrResourceAccessDenied)
-	err = s.datastore.RemovePolicy(s.hasReadWorkflowAdministrationAccess, policy.GetId())
+	err = s.datastore.RemovePolicy(s.hasReadWorkflowAdministrationAccess, policy)
 	s.Error(err)
 	s.ErrorIs(err, sac.ErrResourceAccessDenied)
 
@@ -109,7 +109,7 @@ func (s *PolicyDatastoreTestSuite) TestReplacingResourceAccess() {
 
 	_, err = s.datastore.AddPolicy(s.hasReadWriteWorkflowAdministrationAccess, policy)
 	s.NoError(err)
-	err = s.datastore.RemovePolicy(s.hasReadWriteWorkflowAdministrationAccess, policy.GetId())
+	err = s.datastore.RemovePolicy(s.hasReadWriteWorkflowAdministrationAccess, policy)
 	s.NoError(err)
 }
 
diff --git a/central/policy/datastore/mocks/datastore.go b/central/policy/datastore/mocks/datastore.go
index f3894de324cc2..ab7ca90e97dd5 100644
--- a/central/policy/datastore/mocks/datastore.go
+++ b/central/policy/datastore/mocks/datastore.go
@@ -153,17 +153,17 @@ func (mr *MockDataStoreMockRecorder) ImportPolicies(ctx, policies, overwrite any
 }
 
 // RemovePolicy mocks base method.
-func (m *MockDataStore) RemovePolicy(ctx context.Context, id string) error {
+func (m *MockDataStore) RemovePolicy(ctx context.Context, policy *storage.Policy) error {
 	m.ctrl.T.Helper()
-	ret := m.ctrl.Call(m, "RemovePolicy", ctx, id)
+	ret := m.ctrl.Call(m, "RemovePolicy", ctx, policy)
 	ret0, _ := ret[0].(error)
 	return ret0
 }
 
 // RemovePolicy indicates an expected call of RemovePolicy.
-func (mr *MockDataStoreMockRecorder) RemovePolicy(ctx, id any) *gomock.Call {
+func (mr *MockDataStoreMockRecorder) RemovePolicy(ctx, policy any) *gomock.Call {
 	mr.mock.ctrl.T.Helper()
-	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RemovePolicy", reflect.TypeOf((*MockDataStore)(nil).RemovePolicy), ctx, id)
+	return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RemovePolicy", reflect.TypeOf((*MockDataStore)(nil).RemovePolicy), ctx, policy)
 }
 
 // Search mocks base method.
diff --git a/central/policy/datastore/singleton.go b/central/policy/datastore/singleton.go
index c9153f2e34e1c..b31f2f750f390 100644
--- a/central/policy/datastore/singleton.go
+++ b/central/policy/datastore/singleton.go
@@ -5,10 +5,12 @@ import (
 
 	clusterDS "github.com/stackrox/rox/central/cluster/datastore"
 	"github.com/stackrox/rox/central/globaldb"
+	"github.com/stackrox/rox/central/metrics"
 	notifierDS "github.com/stackrox/rox/central/notifier/datastore"
 	"github.com/stackrox/rox/central/policy/search"
 	policyStore "github.com/stackrox/rox/central/policy/store"
 	categoriesDS "github.com/stackrox/rox/central/policycategory/datastore"
+	"github.com/stackrox/rox/generated/storage"
 	"github.com/stackrox/rox/pkg/defaults/policies"
 	"github.com/stackrox/rox/pkg/policyutils"
 	"github.com/stackrox/rox/pkg/sac"
@@ -53,6 +55,10 @@ func addDefaults(s policyStore.Store, categoriesDS categoriesDS.DataStore) {
 
 	for _, p := range storedPolicies {
 		policyIDSet.Add(p.GetId())
+		// Unrelated to adding/checking default policies, this was put here to prevent looping through all policies a second time
+		if p.Source == storage.PolicySource_DECLARATIVE {
+			metrics.IncrementTotalExternalPoliciesGauge()
+		}
 	}
 
 	// Preload the default policies.
diff --git a/central/policy/service/service_impl.go b/central/policy/service/service_impl.go
index 8419b370da59a..2c6f31c87085d 100644
--- a/central/policy/service/service_impl.go
+++ b/central/policy/service/service_impl.go
@@ -336,7 +336,7 @@ func (s *serviceImpl) DeletePolicy(ctx context.Context, request *v1.ResourceByID
 		return nil, errors.Wrap(errox.InvalidArgs, "A default policy cannot be deleted. (You can disable a default policy, but not delete it.)")
 	}
 
-	if err := s.policies.RemovePolicy(ctx, request.GetId()); err != nil {
+	if err := s.policies.RemovePolicy(ctx, policy); err != nil {
 		return nil, err
 	}
 
@@ -347,7 +347,6 @@ func (s *serviceImpl) DeletePolicy(ctx context.Context, request *v1.ResourceByID
 	if err := s.syncPoliciesWithSensors(); err != nil {
 		return nil, err
 	}
-
 	return &v1.Empty{}, nil
 }
 


Reviews:
user: login: clickboo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T21:06:47Z
user: login: clickboo
body: Check comment about policy source field in policy.proto.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T21:07:24Z
user: login: clickboo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T16:26:02Z
user: login: clickboo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T16:26:53Z
user: login: clickboo
body: Once you determine the count init in the singleton is ok, good to go.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T20:02:52Z
user: login: dashrews78
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T16:38:20Z
user: login: dashrews78
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T16:40:36Z
user: login: clickboo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T19:11:25Z
user: login: clickboo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T19:19:30Z
user: login: ajheflin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T13:54:34Z
user: login: ajheflin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T14:03:51Z
user: login: clickboo
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T21:57:15Z
user: login: dashrews78
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T09:36:26Z
user: login: dashrews78
body: left a nit in the service.


state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T09:39:57Z
user: login: kylape
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T13:59:18Z
user: login: kylape
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T19:32:01Z
user: login: kylape
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T20:25:14Z

Review comments:
diff_hunk: @@ -167,6 +167,7 @@ func (gc *grpcClient) PostPolicy(ctx context.Context, policy *storage.Policy) (*
 	req := &v1.PostPolicyRequest{
 		Policy:                 policy,
 		EnableStrictValidation: true,
+		PolicyAsCode:           true,
path: config-controller/pkg/client/client.go
user: login: clickboo
body: This is not needed.  The policy source fields in the policy proto will give you the information if this policy is maintained as code or not.
created_at: 2025-03-12T21:06:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,13 @@ var (
 		Help:      "A counter of the total number of network endpoints received by Central from Sensor",
 	}, []string{"ClusterID"})
 
+	totalPolicyAsCodeCRsReceivedGauge = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: metrics.PrometheusNamespace,
+		Subsystem: metrics.CentralSubsystem.String(),
+		Name:      "total_policy_as_code_crs_received_counter",
path: central/metrics/central.go
user: login: kylape
body: Probably should update the name after the switch to a gauge.
created_at: 2025-03-13T15:24:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,13 @@ var (
 		Help:      "A counter of the total number of network endpoints received by Central from Sensor",
 	}, []string{"ClusterID"})
 
+	totalPolicyAsCodeCRsReceivedGauge = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: metrics.PrometheusNamespace,
+		Subsystem: metrics.CentralSubsystem.String(),
+		Name:      "total_policy_as_code_crs_received_counter",
+		Help:      "A counter of the total number of policy as code CRs that have been accepted by Central from Config Controller",
path: central/metrics/central.go
user: login: kylape
body: Also the help text :)
created_at: 2025-03-13T15:24:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,9 +352,16 @@ func (s *serviceImpl) DeletePolicy(ctx context.Context, request *v1.ResourceByID
 		return nil, err
 	}
 
+	if policy.Source == storage.PolicySource_DECLARATIVE {
path: central/policy/service/service_impl.go
user: login: clickboo
body: ??
created_at: 2025-03-13T16:26:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,7 +76,7 @@ message Policy {
   // Read-only field. Indicates the policy is a default policy if true and a custom policy if false.
   bool is_default = 26; // @gotags: crYaml:"isDefault"
 
-  PolicySource source = 27; // @gotags: crYaml:"-"
+  PolicySource source = 27; // @gotags: search:"Policy Source,store" crYaml:"-"
path: proto/storage/policy.proto
user: login: clickboo
body: Why do we need a search tag?
created_at: 2025-03-13T16:26:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,13 @@ var (
 		Help:      "A counter of the total number of network endpoints received by Central from Sensor",
 	}, []string{"ClusterID"})
 
+	totalPolicyAsCodeCRsReceivedGauge = prometheus.NewGauge(prometheus.GaugeOpts{
path: central/metrics/central.go
user: login: clickboo
body: Suggestion for readability:
totalExternalPoliciesGauge

And the other renaming that follows.
created_at: 2025-03-17T17:30:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +38,9 @@ func initialize() {
 		notifierProcessor.Singleton(),
 		cache.ImageMetadataCacheSingleton(),
 		connection.ManagerSingleton())
+
+	count, _ := datastore.Singleton().Count(context.Background(), search.NewQueryBuilder().AddExactMatches(search.PolicySource, storage.PolicySource_DECLARATIVE.String()).ProtoQuery())
path: central/policy/service/singleton.go
user: login: clickboo
body: This worries me of any race conditions there might be in this initialization. Unfortunately, I can't say for sure. I think this is the first time we are querying the DB in the singleton....
created_at: 2025-03-17T17:36:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,7 +76,7 @@ message Policy {
   // Read-only field. Indicates the policy is a default policy if true and a custom policy if false.
   bool is_default = 26; // @gotags: crYaml:"isDefault"
 
-  PolicySource source = 27; // @gotags: crYaml:"-"
+  PolicySource source = 27; // @gotags: search:"Policy Source,store" crYaml:"-"
path: proto/storage/policy.proto
user: login: clickboo
body: I see now why we do for the count init in the singleton. 
created_at: 2025-03-17T17:37:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +40,15 @@ func initialize() {
 		notifierProcessor.Singleton(),
 		cache.ImageMetadataCacheSingleton(),
 		connection.ManagerSingleton())
+
+	count, _ := datastore.Singleton().Count(
path: central/policy/service/singleton.go
user: login: dashrews78
body: datastore init may be a better place for this.
created_at: 2025-03-18T16:38:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,7 +76,7 @@ message Policy {
   // Read-only field. Indicates the policy is a default policy if true and a custom policy if false.
   bool is_default = 26; // @gotags: crYaml:"isDefault"
 
-  PolicySource source = 27; // @gotags: crYaml:"-"
+  PolicySource source = 27; // @gotags: search:"Policy Source,store" crYaml:"-"
path: proto/storage/policy.proto
user: login: dashrews78
body: Now time for some bad news.  The column will not be populated.  To populate it you will need to write a migration.  The field in the serialized object will be populated, but when you promote a field to a column the value does not get populated by default.  So if you want to capture the count of records already managed by CR you would need to ensure the column is properly populated.
created_at: 2025-03-18T16:40:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,7 +76,7 @@ message Policy {
   // Read-only field. Indicates the policy is a default policy if true and a custom policy if false.
   bool is_default = 26; // @gotags: crYaml:"isDefault"
 
-  PolicySource source = 27; // @gotags: crYaml:"-"
+  PolicySource source = 27; // @gotags: search:"Policy Source,store" crYaml:"-"
path: proto/storage/policy.proto
user: login: clickboo
body: @ajheflin ^
created_at: 2025-03-19T19:11:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,7 +76,7 @@ message Policy {
   // Read-only field. Indicates the policy is a default policy if true and a custom policy if false.
   bool is_default = 26; // @gotags: crYaml:"isDefault"
 
-  PolicySource source = 27; // @gotags: crYaml:"-"
+  PolicySource source = 27; // @gotags: search:"Policy Source,store" crYaml:"-"
path: proto/storage/policy.proto
user: login: clickboo
body: David, thanks for bringing this up. Good catch! @ajheflin Instead of a migration we already possibly loop through policies somewhere in the datastore on start, and if we don't we could do that instead of a migration. If we go the migration route, it should be a simple one IMO, and we would have similar examples in our existing migrations.
created_at: 2025-03-19T19:19:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +40,15 @@ func initialize() {
 		notifierProcessor.Singleton(),
 		cache.ImageMetadataCacheSingleton(),
 		connection.ManagerSingleton())
+
+	count, _ := datastore.Singleton().Count(
path: central/policy/service/singleton.go
user: login: ajheflin
body: My thought on this was that we increment/decrement the metrics in the service, but I suppose we could also move that to the datastore as well
created_at: 2025-03-20T13:54:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +40,15 @@ func initialize() {
 		notifierProcessor.Singleton(),
 		cache.ImageMetadataCacheSingleton(),
 		connection.ManagerSingleton())
+
+	count, _ := datastore.Singleton().Count(
path: central/policy/service/singleton.go
user: login: ajheflin
body: That ended up being fairly invasive to do (and there very well may have been some calls to `RemovePolicy` that I missed), so it may be a bit cleaner to move back to the service, see ed5d150
created_at: 2025-03-20T14:03:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -288,7 +288,8 @@ func (s *serviceImpl) PostPolicy(ctx context.Context, request *v1.PostPolicyRequ
 	if request.GetEnableStrictValidation() {
 		options = append(options, booleanpolicy.ValidateEnvVarSourceRestrictions())
 	}
-	return s.addOrUpdatePolicy(ctx, request.GetPolicy(), ensureIDEmpty, s.addPolicyToStoreAndSetID, options...)
+	policy, err := s.addOrUpdatePolicy(ctx, request.GetPolicy(), ensureIDEmpty, s.addPolicyToStoreAndSetID, options...)
+	return policy, err
path: central/policy/service/service_impl.go
user: login: dashrews78
body: nit:
```suggestion
	return s.addOrUpdatePolicy(ctx, request.GetPolicy(), ensureIDEmpty, s.addPolicyToStoreAndSetID, options...)
```
created_at: 2025-03-21T09:36:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -288,7 +288,8 @@ func (s *serviceImpl) PostPolicy(ctx context.Context, request *v1.PostPolicyRequ
 	if request.GetEnableStrictValidation() {
 		options = append(options, booleanpolicy.ValidateEnvVarSourceRestrictions())
 	}
-	return s.addOrUpdatePolicy(ctx, request.GetPolicy(), ensureIDEmpty, s.addPolicyToStoreAndSetID, options...)
+	policy, err := s.addOrUpdatePolicy(ctx, request.GetPolicy(), ensureIDEmpty, s.addPolicyToStoreAndSetID, options...)
+	return policy, err
path: central/policy/service/service_impl.go
user: login: kylape
body: I don't see the suggested change applied. Was this something you decided not to do?
created_at: 2025-03-24T13:51:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,6 +112,13 @@ var (
 		Help:      "A counter of the total number of network endpoints received by Central from Sensor",
 	}, []string{"ClusterID"})
 
+	totalExternalPoliciesGauge = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: metrics.PrometheusNamespace,
+		Subsystem: metrics.CentralSubsystem.String(),
+		Name:      "total_external_policies_gauge",
path: central/metrics/central.go
user: login: kylape
body: sorry, I should have looked at this closer before my first comment, but it looks like the other gauge names use the unit as their last word, e.g. "count", "length", "capacity", "bytes".  Given that pattern, perhaps "count" makes the most sense here?
created_at: 2025-03-24T19:32:01Z
author_association: CONTRIBUTOR

PR comments:
user: login: openshift-ci[bot]
created_at: 2025-03-12T17:10:11Z
body: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
author_association: NONE
user: login: rhacs-bot
created_at: 2025-03-12T17:36:57Z
body: [//]: # (meta:type=build)

Images are ready for the commit at 1082f83.

To use with deploy scripts, first `export MAIN_IMAGE_TAG=4.8.x-348-g1082f83f1e`.
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-12T18:10:05Z
body: ## [Codecov](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) Report
Attention: Patch coverage is `26.66667%` with `11 lines` in your changes missing coverage. Please review.
> Project coverage is 48.88%. Comparing base [(`e23b725`)](https://app.codecov.io/gh/stackrox/stackrox/commit/e23b72559039b8f0518812b11bea3346ee784fcd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) to head [(`1082f83`)](https://app.codecov.io/gh/stackrox/stackrox/commit/1082f83f1ecc1071a615a009816ca793a253a461?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox).
> Report is 11 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) | Patch % | Lines |
|---|---|---|
| [central/policy/datastore/datastore\_impl.go](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&filepath=central%2Fpolicy%2Fdatastore%2Fdatastore_impl.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox#diff-Y2VudHJhbC9wb2xpY3kvZGF0YXN0b3JlL2RhdGFzdG9yZV9pbXBsLmdv) | 40.00% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) |
| [central/policy/datastore/singleton.go](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&filepath=central%2Fpolicy%2Fdatastore%2Fsingleton.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox#diff-Y2VudHJhbC9wb2xpY3kvZGF0YXN0b3JlL3NpbmdsZXRvbi5nbw==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) |
| [central/policy/service/service\_impl.go](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&filepath=central%2Fpolicy%2Fservice%2Fservice_impl.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox#diff-Y2VudHJhbC9wb2xpY3kvc2VydmljZS9zZXJ2aWNlX2ltcGwuZ28=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master   #14590      +/-   ##
==========================================
- Coverage   48.89%   48.88%   -0.01%     
==========================================
  Files        2547     2547              
  Lines      186964   186975      +11     
==========================================
- Hits        91414    91407       -7     
- Misses      88312    88325      +13     
- Partials     7238     7243       +5     
```

| [Flag](https://app.codecov.io/gh/stackrox/stackrox/pull/14590/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) | Coverage Δ | |
|---|---|---|
| [go-unit-tests](https://app.codecov.io/gh/stackrox/stackrox/pull/14590/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox) | `48.88% <26.66%> (-0.01%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/stackrox/stackrox/pull/14590?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=stackrox).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: ajheflin
created_at: 2025-03-13T13:55:15Z
body: /retest
author_association: CONTRIBUTOR
user: login: kylape
created_at: 2025-03-13T14:08:55Z
body: Yeah I do think a gauge would be more useful. It would involve both decrementing the gauge on deletes as well as initializing the gauge during bootstrap by fetching the number of declarative policies currently in the DB.
author_association: CONTRIBUTOR
user: login: kylape
created_at: 2025-03-13T14:11:41Z
body: Just to add a bit of reasoning behind my statement:  A counter would help measure activity, while a gauge helps measure the current state.  I'm personally more interested in how many policies are managed via config-controller rather than how often they're updated/created/deleted.
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-18T13:25:00Z
body: /retest
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-19T13:29:45Z
body: /retest
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-20T16:56:21Z
body: Assuming the tests pass, this should be good to go in response to everyone's comments, now that it initializes the gauge in the datastore init phase while it's looping through the existing policies to see if they're default policies
author_association: CONTRIBUTOR
user: login: clickboo
created_at: 2025-03-20T21:57:00Z
body: Appreciate your patience, and persistence on this @ajheflin, thank you!
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-24T17:03:01Z
body: /retest
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-25T14:34:15Z
body: /retest
author_association: CONTRIBUTOR
user: login: ajheflin
created_at: 2025-03-26T15:35:08Z
body: /retest
author_association: CONTRIBUTOR