Repository: ratify-project/ratify
Pull request #1710 (2024-08-14 08:29:24)

feat: save reconcile error for KMP/CertStore
# Description

## What this PR does / why we need it:
This PR adds support to save errors happened during KMP/CertStore reconciliation which could be checked by verifiers during artifact validation. Instead of reporting no certs or keys fetched, we could explicitly return the exact error during reconciliation.
1. Refactor KMP and CertificateStore to manage errors from reconciliation.
2. In notation trustStore implementation, we add error check and return it to verifier instead of just logging it.
3. Updated unit tests.

Example:
We provide an invalid certificate to inline KMP.
The previous verifier report didn't clarify that the root cause is from KMP cert.
![image](https://github.com/user-attachments/assets/33e3b34c-2ab5-47b9-8066-8351954718cc)

With this change, Ratify will display the correct root cause:
![image](https://github.com/user-attachments/assets/2b35bffb-6eb9-43b9-b530-cabffc415637)

If I provide an invalid key to inline KMP.
The errorReson before the change:
```
"errorReason": "CONFIG_INVALID: trust policy [default] failed to access key management provider ratify-cosign-inline-key-0, err: failed to access non-existent key management provider: ratify-cosign-inline-key-0",
```

The errorReason after the change:
``` 
"errorReason": "CONFIG_INVALID: trust policy [default] failed to access key management provider ratify-cosign-inline-key-0, err: failed to create key management provider provider: KEY_INVALID: error parsing public key: PEM decoding failed",
```

## Which issue(s) this PR fixes *(optional, using `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when the PR gets merged)*:
Fixes https://github.com/ratify-project/ratify/issues/1653

## Type of change

Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Helm Chart Change (any edit/addition/update that is necessary for changes merged to the `main` branch)
- [ ] This change requires a documentation update

# How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

- [ ] Test A
- [ ] Test B

# Checklist:

- [ ] Does the affected code have corresponding tests?
- [ ] Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
- [ ] Does this introduce breaking changes that would require an announcement or bumping the major version?
- [ ] Do all new files have appropriate license header?

# Post Merge Requirements
- [ ] MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates `Helm Chart Change`


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) Report
Attention: Patch coverage is `87.93103%` with `7 lines` in your changes missing coverage. Please review.

| [Files](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) | Patch % | Lines |
|---|---|---|
| [errors/types.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=errors%2Ftypes.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-ZXJyb3JzL3R5cGVzLmdv) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) |
| [...lusterresource/keymanagementprovider\_controller.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fcontrollers%2Fclusterresource%2Fkeymanagementprovider_controller.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2NvbnRyb2xsZXJzL2NsdXN0ZXJyZXNvdXJjZS9rZXltYW5hZ2VtZW50cHJvdmlkZXJfY29udHJvbGxlci5nbw==) | 60.00% | [2 Missing :warning: ](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) |
| [...espaceresource/keymanagementprovider\_controller.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fcontrollers%2Fnamespaceresource%2Fkeymanagementprovider_controller.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2NvbnRyb2xsZXJzL25hbWVzcGFjZXJlc291cmNlL2tleW1hbmFnZW1lbnRwcm92aWRlcl9jb250cm9sbGVyLmdv) | 60.00% | [2 Missing :warning: ](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) |
| [pkg/keymanagementprovider/keymanagementprovider.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fkeymanagementprovider%2Fkeymanagementprovider.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2tleW1hbmFnZW1lbnRwcm92aWRlci9rZXltYW5hZ2VtZW50cHJvdmlkZXIuZ28=) | 94.44% | [1 Missing :warning: ](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) |

| [Files](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project) | Coverage Δ | |
|---|---|---|
| [...omresources/certificatestores/certificatestores.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fcustomresources%2Fcertificatestores%2Fcertificatestores.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2N1c3RvbXJlc291cmNlcy9jZXJ0aWZpY2F0ZXN0b3Jlcy9jZXJ0aWZpY2F0ZXN0b3Jlcy5nbw==) | `90.62% <100.00%> (+2.16%)` | :arrow_up: |
| [pkg/verifier/notation/truststore.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fverifier%2Fnotation%2Ftruststore.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL3ZlcmlmaWVyL25vdGF0aW9uL3RydXN0c3RvcmUuZ28=) | `85.93% <100.00%> (+1.93%)` | :arrow_up: |
| [pkg/keymanagementprovider/keymanagementprovider.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fkeymanagementprovider%2Fkeymanagementprovider.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2tleW1hbmFnZW1lbnRwcm92aWRlci9rZXltYW5hZ2VtZW50cHJvdmlkZXIuZ28=) | `93.93% <94.44%> (+1.21%)` | :arrow_up: |
| [errors/types.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=errors%2Ftypes.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-ZXJyb3JzL3R5cGVzLmdv) | `89.06% <0.00%> (-1.42%)` | :arrow_down: |
| [...lusterresource/keymanagementprovider\_controller.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fcontrollers%2Fclusterresource%2Fkeymanagementprovider_controller.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2NvbnRyb2xsZXJzL2NsdXN0ZXJyZXNvdXJjZS9rZXltYW5hZ2VtZW50cHJvdmlkZXJfY29udHJvbGxlci5nbw==) | `72.22% <60.00%> (-2.43%)` | :arrow_down: |
| [...espaceresource/keymanagementprovider\_controller.go](https://app.codecov.io/gh/ratify-project/ratify/pull/1710?src=pr&el=tree&filepath=pkg%2Fcontrollers%2Fnamespaceresource%2Fkeymanagementprovider_controller.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project#diff-cGtnL2NvbnRyb2xsZXJzL25hbWVzcGFjZXJlc291cmNlL2tleW1hbmFnZW1lbnRwcm92aWRlcl9jb250cm9sbGVyLmdv) | `72.22% <60.00%> (-2.43%)` | :arrow_down: |

... and [26 files with indirect coverage changes](https://app.codecov.io/gh/ratify-project/ratify/pull/1710/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ratify-project)


Patch:
diff --git a/errors/errors.go b/errors/errors.go
index 8f98e6d..9305f79 100644
--- a/errors/errors.go
+++ b/errors/errors.go
@@ -118,4 +118,18 @@ var (
 		Message:     "data encoding failure",
 		Description: "Failed to encode data. Please verify the encoding data.",
 	})
+
+	// ErrorCodeNotFound is returned when the requested resource is not found.
+	ErrorCodeNotFound = Register("errcode", ErrorDescriptor{
+		Value:       "RESOURCE_NOT_FOUND",
+		Message:     "resource not found",
+		Description: "The requested resource is not found. Please verify the resource exists.",
+	})
+
+	// ErrorCodeForbidden is returned when the requested operation is forbidden.
+	ErrorCodeForbidden = Register("errcode", ErrorDescriptor{
+		Value:       "OPERATION_FORBIDDEN",
+		Message:     "operation forbidden",
+		Description: "The requested operation is forbidden. Please verify the permission to the requested resource.",
+	})
 )
diff --git a/pkg/keymanagementprovider/keymanagementprovider.go b/pkg/keymanagementprovider/keymanagementprovider.go
index 3c9c695..bf77c2c 100644
--- a/pkg/keymanagementprovider/keymanagementprovider.go
+++ b/pkg/keymanagementprovider/keymanagementprovider.go
@@ -74,6 +74,18 @@ var certificatesMap sync.Map
 //	 where PublicKey is a struct containing the public key and the provider type
 var keyMap sync.Map
 
+// static concurrency-safe map to store errors while fetching certificates from key management provider.
+// layout:
+//
+//	map["<namespace>/<name>"] = error
+var certificateErrMap sync.Map
+
+// static concurrency-safe map to store errors while fetching keys from key management provider.
+// layout:
+//
+//	map["<namespace>/<name>"] = error
+var keyErrMap sync.Map
+
 // DecodeCertificates decodes PEM-encoded bytes into an x509.Certificate chain.
 func DecodeCertificates(value []byte) ([]*x509.Certificate, error) {
 	var certs []*x509.Certificate
@@ -114,27 +126,34 @@ func DecodeKey(value []byte) (crypto.PublicKey, error) {
 	return pk, nil
 }
 
-// SetCertificatesInMap sets the certificates in the map
+// setCertificatesInMap sets the certificates in the map
 // it is concurrency-safe
-func SetCertificatesInMap(resource string, certs map[KMPMapKey][]*x509.Certificate) {
+func setCertificatesInMap(resource string, certs map[KMPMapKey][]*x509.Certificate) {
 	certificatesMap.Store(resource, certs)
+	certificateErrMap.Delete(resource)
 }
 
 // GetCertificatesFromMap gets the certificates from the map and returns an empty map of certificate arrays if not found or an error happened.
 func GetCertificatesFromMap(ctx context.Context, resource string) (map[KMPMapKey][]*x509.Certificate, error) {
 	if !hasAccessToProvider(ctx, resource) {
-		return map[KMPMapKey][]*x509.Certificate{}, fmt.Errorf("namespace: [%s] does not have access to key management provider: %s", ctxUtils.GetNamespace(ctx), resource)
+		return map[KMPMapKey][]*x509.Certificate{}, errors.ErrorCodeForbidden.WithDetail(fmt.Sprintf("namespace: [%s] does not have access to key management provider: %s", ctxUtils.GetNamespace(ctx), resource)).WithRemediation(fmt.Sprintf("Ensure the key management provider: %s is created under namespace: [%s] or as a cluster-wide resource.", resource, ctxUtils.GetNamespace(ctx)))
+	}
+	if err, ok := certificateErrMap.Load(resource); ok && err != nil {
+		return map[KMPMapKey][]*x509.Certificate{}, err.(error)
 	}
 	if certs, ok := certificatesMap.Load(resource); ok {
 		return certs.(map[KMPMapKey][]*x509.Certificate), nil
 	}
-	return map[KMPMapKey][]*x509.Certificate{}, fmt.Errorf("failed to access non-existent key management provider: %s", resource)
+	return map[KMPMapKey][]*x509.Certificate{}, errors.ErrorCodeNotFound.WithDetail(fmt.Sprintf("failed to access non-existent key management provider: %s", resource))
 }
 
-// DeleteCertificatesFromMap deletes the certificates from the map
+// DeleteResourceFromMap deletes the certificates, keys and errors from the map
 // it is concurrency-safe
-func DeleteCertificatesFromMap(resource string) {
+func DeleteResourceFromMap(resource string) {
 	certificatesMap.Delete(resource)
+	keyMap.Delete(resource)
+	certificateErrMap.Delete(resource)
+	keyErrMap.Delete(resource)
 }
 
 // FlattenKMPMap flattens the map of certificates fetched for a single key management provider resource and returns a single array
@@ -146,31 +165,46 @@ func FlattenKMPMap(certMap map[KMPMapKey][]*x509.Certificate) []*x509.Certificat
 	return items
 }
 
-// SetKeysInMap sets the keys in the map
-func SetKeysInMap(resource string, providerType string, keys map[KMPMapKey]crypto.PublicKey) {
+// setKeysInMap sets the keys in the map
+func setKeysInMap(resource string, providerType string, keys map[KMPMapKey]crypto.PublicKey) {
 	typedMap := make(map[KMPMapKey]PublicKey)
 	for key, value := range keys {
 		typedMap[key] = PublicKey{Key: value, ProviderType: providerType}
 	}
 	keyMap.Store(resource, typedMap)
+	keyErrMap.Delete(resource)
+}
+
+// SaveSecrets saves the keys and certificates in the map.
+func SaveSecrets(resource, providerType string, keys map[KMPMapKey]crypto.PublicKey, certs map[KMPMapKey][]*x509.Certificate) {
+	setKeysInMap(resource, providerType, keys)
+	setCertificatesInMap(resource, certs)
 }
 
 // GetKeysFromMap gets the keys from the map and returns an empty map if not found or an error happened.
 func GetKeysFromMap(ctx context.Context, resource string) (map[KMPMapKey]PublicKey, error) {
-	// A cluster-wide operation can cluster-wide provider
+	// A cluster-wide operation can access cluster-wide provider
 	// A namespaced operation can only fetch the provider in the same namespace or cluster-wide provider.
 	if !hasAccessToProvider(ctx, resource) {
 		return map[KMPMapKey]PublicKey{}, fmt.Errorf("namespace: [%s] does not have access to key management provider: %s", ctxUtils.GetNamespace(ctx), resource)
 	}
+	if err, ok := keyErrMap.Load(resource); ok && err != nil {
+		return map[KMPMapKey]PublicKey{}, err.(error)
+	}
 	if keys, ok := keyMap.Load(resource); ok {
 		return keys.(map[KMPMapKey]PublicKey), nil
 	}
 	return map[KMPMapKey]PublicKey{}, fmt.Errorf("failed to access non-existent key management provider: %s", resource)
 }
 
-// DeleteKeysFromMap deletes the keys from the map
-func DeleteKeysFromMap(resource string) {
-	keyMap.Delete(resource)
+// SetCertificateError sets the error while fetching certificates from key management provider.
+func SetCertificateError(resource string, err error) {
+	certificateErrMap.Store(resource, err)
+}
+
+// SetKeyError sets the error while fetching keys from key management provider.
+func SetKeyError(resource string, err error) {
+	keyErrMap.Store(resource, err)
 }
 
 // A namespaced verification request could access KMP in the same namespace or cluster-wide KMP.
diff --git a/pkg/keymanagementprovider/refresh/kubeRefresh.go b/pkg/keymanagementprovider/refresh/kubeRefresh.go
index 21098d4..aa06118 100644
--- a/pkg/keymanagementprovider/refresh/kubeRefresh.go
+++ b/pkg/keymanagementprovider/refresh/kubeRefresh.go
@@ -58,8 +58,7 @@ func (kr *KubeRefresher) Refresh(ctx context.Context) error {
 	if err := kr.Get(ctx, kr.Request.NamespacedName, &keyManagementProvider); err != nil {
 		if apierrors.IsNotFound(err) {
 			logger.Infof("deletion detected, removing key management provider %v", resource)
-			kmp.DeleteCertificatesFromMap(resource)
-			kmp.DeleteKeysFromMap(resource)
+			kmp.DeleteResourceFromMap(resource)
 		} else {
 			logger.Error(err, "unable to fetch key management provider")
 		}
@@ -88,6 +87,8 @@ func (kr *KubeRefresher) Refresh(ctx context.Context) error {
 
 	provider, err := cutils.SpecToKeyManagementProvider(keyManagementProvider.Spec.Parameters.Raw, keyManagementProvider.Spec.Type)
 	if err != nil {
+		kmp.SetCertificateError(resource, err)
+		kmp.SetKeyError(resource, err)
 		writeKMProviderStatus(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Request = ctrl.Request{}
 		return err
@@ -96,6 +97,7 @@ func (kr *KubeRefresher) Refresh(ctx context.Context) error {
 	// fetch certificates and store in map
 	certificates, certAttributes, err := provider.GetCertificates(ctx)
 	if err != nil {
+		kmp.SetCertificateError(resource, err)
 		writeKMProviderStatus(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Request = ctrl.Request{}
 		return fmt.Errorf("error fetching certificates in KMProvider %v with %v provider, error: %w", resource, keyManagementProvider.Spec.Type, err)
@@ -104,12 +106,12 @@ func (kr *KubeRefresher) Refresh(ctx context.Context) error {
 	// fetch keys and store in map
 	keys, keyAttributes, err := provider.GetKeys(ctx)
 	if err != nil {
+		kmp.SetKeyError(resource, err)
 		writeKMProviderStatus(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Request = ctrl.Request{}
 		return fmt.Errorf("error fetching keys in KMProvider %v with %v provider, error: %w", resource, keyManagementProvider.Spec.Type, err)
 	}
-	kmp.SetCertificatesInMap(resource, certificates)
-	kmp.SetKeysInMap(resource, keyManagementProvider.Spec.Type, keys)
+	kmp.SaveSecrets(resource, keyManagementProvider.Spec.Type, keys, certificates)
 	// merge certificates and keys status into one
 	maps.Copy(keyAttributes, certAttributes)
 	isFetchSuccessful = true
diff --git a/pkg/keymanagementprovider/refresh/kubeRefreshNamespaced.go b/pkg/keymanagementprovider/refresh/kubeRefreshNamespaced.go
index ab72e10..9343250 100644
--- a/pkg/keymanagementprovider/refresh/kubeRefreshNamespaced.go
+++ b/pkg/keymanagementprovider/refresh/kubeRefreshNamespaced.go
@@ -57,8 +57,7 @@ func (kr *KubeRefresherNamespaced) Refresh(ctx context.Context) error {
 	if err := kr.Get(ctx, kr.Request.NamespacedName, &keyManagementProvider); err != nil {
 		if apierrors.IsNotFound(err) {
 			logger.Infof("deletion detected, removing key management provider %v", resource)
-			kmp.DeleteCertificatesFromMap(resource)
-			kmp.DeleteKeysFromMap(resource)
+			kmp.DeleteResourceFromMap(resource)
 		} else {
 			logger.Error(err, "unable to fetch key management provider")
 		}
@@ -87,6 +86,8 @@ func (kr *KubeRefresherNamespaced) Refresh(ctx context.Context) error {
 
 	provider, err := cutils.SpecToKeyManagementProvider(keyManagementProvider.Spec.Parameters.Raw, keyManagementProvider.Spec.Type)
 	if err != nil {
+		kmp.SetCertificateError(resource, err)
+		kmp.SetKeyError(resource, err)
 		writeKMProviderStatusNamespaced(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Result = ctrl.Result{}
 		return err
@@ -95,6 +96,7 @@ func (kr *KubeRefresherNamespaced) Refresh(ctx context.Context) error {
 	// fetch certificates and store in map
 	certificates, certAttributes, err := provider.GetCertificates(ctx)
 	if err != nil {
+		kmp.SetCertificateError(resource, err)
 		writeKMProviderStatusNamespaced(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Result = ctrl.Result{}
 		return fmt.Errorf("error fetching certificates in KMProvider %v with %v provider, error: %w", resource, keyManagementProvider.Spec.Type, err)
@@ -103,12 +105,12 @@ func (kr *KubeRefresherNamespaced) Refresh(ctx context.Context) error {
 	// fetch keys and store in map
 	keys, keyAttributes, err := provider.GetKeys(ctx)
 	if err != nil {
+		kmp.SetKeyError(resource, err)
 		writeKMProviderStatusNamespaced(ctx, kr, &keyManagementProvider, logger, isFetchSuccessful, err.Error(), lastFetchedTime, nil)
 		kr.Result = ctrl.Result{}
 		return fmt.Errorf("error fetching keys in KMProvider %v with %v provider, error: %w", resource, keyManagementProvider.Spec.Type, err)
 	}
-	kmp.SetCertificatesInMap(resource, certificates)
-	kmp.SetKeysInMap(resource, keyManagementProvider.Spec.Type, keys)
+	kmp.SaveSecrets(resource, keyManagementProvider.Spec.Type, keys, certificates)
 	// merge certificates and keys status into one
 	maps.Copy(keyAttributes, certAttributes)
 	isFetchSuccessful = true
diff --git a/pkg/verifier/notation/truststore.go b/pkg/verifier/notation/truststore.go
index cb71ac6..c7bceb4 100644
--- a/pkg/verifier/notation/truststore.go
+++ b/pkg/verifier/notation/truststore.go
@@ -22,6 +22,7 @@ import (
 	"fmt"
 
 	"github.com/notaryproject/notation-go/verifier/truststore"
+	re "github.com/ratify-project/ratify/errors"
 	"github.com/ratify-project/ratify/internal/logger"
 	"github.com/ratify-project/ratify/pkg/controllers"
 	"github.com/ratify-project/ratify/pkg/keymanagementprovider"
@@ -69,28 +70,32 @@ func (s *trustStore) getCertificatesInternal(ctx context.Context, storeType trus
 	if len(certGroup) > 0 {
 		for _, certStore := range certGroup {
 			logger.GetLogger(ctx, logOpt).Debugf("truststore getting certStore %v", certStore)
-			certMap, err := keymanagementprovider.GetCertificatesFromMap(ctx, certStore)
-			if err != nil {
-				logger.GetLogger(ctx, logOpt).Warnf("unable to fetch certificates for Key Management Provider %+v: %v", certStore, err)
+			certMap, kmpErr := keymanagementprovider.GetCertificatesFromMap(ctx, certStore)
+			if kmpErr != nil {
+				logger.GetLogger(ctx, logOpt).Warnf("unable to fetch certificates for Key Management Provider %+v: %v", certStore, kmpErr)
 			}
 			result := keymanagementprovider.FlattenKMPMap(certMap)
+			var certStoreErr error
 			// notation verifier does not consider specific named/versioned certificates within a key management provider resource
 			if len(result) == 0 {
 				logger.GetLogger(ctx, logOpt).Warnf("no certificate fetched for Key Management Provider %+v", certStore)
 				// check certificate store if key management provider does not have certificates.
 				// NOTE: certificate store and key management provider should not be configured together.
 				// User will be warned by the controller/CLI
-				if result, err = controllers.NamespacedCertStores.GetCertsFromStore(ctx, certStore); err != nil {
-					logger.GetLogger(ctx, logOpt).Warnf("unable to fetch certificates for Certificate Store %+v: %v", certStore, err)
+				if result, certStoreErr = controllers.NamespacedCertStores.GetCertsFromStore(ctx, certStore); certStoreErr != nil {
+					logger.GetLogger(ctx, logOpt).Warnf("unable to fetch certificates for Certificate Store %+v: %v", certStore, certStoreErr)
 				}
 				if len(result) == 0 {
 					logger.GetLogger(ctx, logOpt).Warnf("no certificate fetched for Certificate Store %+v", certStore)
 				}
 			}
+			if err := parseErrFromKmpAndCertStore(kmpErr, certStoreErr); err != nil {
+				return []*x509.Certificate{}, re.ErrorCodeCertInvalid.WithError(err).WithDetail(fmt.Sprintf("unable to fetch certificates from Key Management Provider and Certificate Store: %s", certStore))
+			}
 			certs = append(certs, result...)
 		}
 		if len(certs) == 0 {
-			return certs, fmt.Errorf("unable to fetch certificates for namedStore: %+v", namedStore)
+			return certs, fmt.Errorf("no certificates fetched in namedStore: %+v", namedStore)
 		}
 	} else {
 		for _, path := range s.certPaths {
@@ -122,3 +127,33 @@ func (s *trustStore) filterValidCerts(certs []*x509.Certificate) ([]*x509.Certif
 	}
 	return filteredCerts, nil
 }
+
+// parseErrFromKmpAndCertStore prioritizes and returns the appropriate error from either KMP or CertStore.
+// If the certStoreErr is a reconcile error while kmpErr is not, it returns certStoreErr,
+// otherwise it returns kmpErr.
+// A reconcile error occurs during CR reconciliation, indicating that the resource
+// was applied by users.
+// Since key management provider and certificate store are mutually exclusive,
+// a reconcile error will only originate from one of them.
+// Consequently, the reconcile error from one resource takes precedence over
+// errors from the other.
+// Note: this method should be deleted once certificate store is completely removed.
+func parseErrFromKmpAndCertStore(kmpErr, certStoreErr error) error {
+	if kmpErr == nil || certStoreErr == nil {
+		return nil
+	}
+	if isReconcileError(certStoreErr) && !isReconcileError(kmpErr) {
+		return certStoreErr
+	}
+	return kmpErr
+}
+
+// isReconcileError checks if the error is a reconcile error.
+// If the error is not NotFound or Forbidden, it is considered as an error from KMP/CertStore reconciliation.
+func isReconcileError(err error) bool {
+	ratifyErr := &re.Error{}
+	if errors.As(err, ratifyErr) {
+		return ratifyErr.ErrorCode() != re.ErrorCodeNotFound && ratifyErr.ErrorCode() != re.ErrorCodeForbidden
+	}
+	return true
+}


Test patch:
diff --git a/pkg/customresources/certificatestores/api.go b/pkg/customresources/certificatestores/api.go
index 9cc4709..95d7201 100644
--- a/pkg/customresources/certificatestores/api.go
+++ b/pkg/customresources/certificatestores/api.go
@@ -28,4 +28,7 @@ type CertStoreManager interface {
 
 	// DeleteStore deletes the certificate from the given scope.
 	DeleteStore(storeName string)
+
+	// AddStoreError adds an error to the given certificate store.
+	AddStoreError(storeName string, err error)
 }
diff --git a/pkg/customresources/certificatestores/certificatestores.go b/pkg/customresources/certificatestores/certificatestores.go
index 1bea741..96d33d3 100644
--- a/pkg/customresources/certificatestores/certificatestores.go
+++ b/pkg/customresources/certificatestores/certificatestores.go
@@ -21,6 +21,7 @@ import (
 	"strings"
 	"sync"
 
+	"github.com/ratify-project/ratify/errors"
 	"github.com/ratify-project/ratify/internal/constants"
 	ctxUtils "github.com/ratify-project/ratify/internal/context"
 	"github.com/ratify-project/ratify/pkg/utils"
@@ -37,6 +38,15 @@ type ActiveCertStores struct {
 	//   "namespace2/store2": []*x509.Certificate
 	// }
 	scopedCertStores sync.Map
+
+	// scopedStoreErrors is mapping from cert store name to error generated while reconciling the store.
+	// The certificate store name is prefixed with the namespace.
+	// Example:
+	// {
+	//   "namespace1/store1": error,
+	//   "namespace2/store2": error
+	// }
+	scopedStoreErrors sync.Map
 }
 
 func NewActiveCertStores() CertStoreManager {
@@ -52,24 +62,34 @@ func (c *ActiveCertStores) GetCertsFromStore(ctx context.Context, storeName stri
 	}
 
 	if !hasAccessToStore(ctx, storeName) {
-		return []*x509.Certificate{}, fmt.Errorf("namespace: [%s] does not have access to certificate store: %s", ctxUtils.GetNamespace(ctx), storeName)
+		return []*x509.Certificate{}, errors.ErrorCodeForbidden.WithDetail(fmt.Sprintf("namespace: [%s] does not have access to certificate store: %s", ctxUtils.GetNamespace(ctx), storeName)).WithRemediation(fmt.Sprintf("Ensure the certificate store: %s is created under namespace: [%s]", storeName, ctxUtils.GetNamespace(ctx)))
 	}
+	if err, ok := c.scopedStoreErrors.Load(prependedName); ok && err != nil {
+		return []*x509.Certificate{}, err.(error)
+	}
+
 	if certs, ok := c.scopedCertStores.Load(prependedName); ok {
 		return certs.([]*x509.Certificate), nil
 	}
-	return []*x509.Certificate{}, fmt.Errorf("failed to access non-existent certificate store: %s", storeName)
+	return []*x509.Certificate{}, errors.ErrorCodeNotFound.WithDetail(fmt.Sprintf("failed to access non-existent certificate store: %s", storeName))
 }
 
 // AddStore fulfills the CertStoreManager interface.
 // It adds the given certificate under cert store.
 func (c *ActiveCertStores) AddStore(storeName string, cert []*x509.Certificate) {
 	c.scopedCertStores.Store(storeName, cert)
+	c.scopedStoreErrors.Delete(storeName)
 }
 
 // DeleteStore fulfills the CertStoreManager interface.
 // It deletes the given cert store.
 func (c *ActiveCertStores) DeleteStore(storeName string) {
 	c.scopedCertStores.Delete(storeName)
+	c.scopedStoreErrors.Delete(storeName)
+}
+
+func (c *ActiveCertStores) AddStoreError(storeName string, err error) {
+	c.scopedStoreErrors.Store(storeName, err)
 }
 
 // A namespaced verification request could access certStores in the same namespace.
diff --git a/pkg/customresources/certificatestores/certificatestores_test.go b/pkg/customresources/certificatestores/certificatestores_test.go
index 4e45e9f..e6065e4 100644
--- a/pkg/customresources/certificatestores/certificatestores_test.go
+++ b/pkg/customresources/certificatestores/certificatestores_test.go
@@ -19,6 +19,7 @@ import (
 	"crypto/rsa"
 	"crypto/x509"
 	"crypto/x509/pkix"
+	"errors"
 	"math/big"
 	"os"
 	"testing"
@@ -52,6 +53,11 @@ func TestCertStoresOperations(t *testing.T) {
 	ctx := context.Background()
 	certStore1 := []*x509.Certificate{cert1}
 
+	activeCertStores.AddStoreError(store1, errors.New("error1"))
+	if _, err := activeCertStores.GetCertsFromStore(ctx, store1); err == nil {
+		t.Fatalf("expect to get error, but got nil")
+	}
+
 	activeCertStores.AddStore(store1, certStore1)
 	certs, _ := activeCertStores.GetCertsFromStore(ctx, store1)
 	if len(certs) != 1 {
diff --git a/pkg/keymanagementprovider/keymanagementprovider_test.go b/pkg/keymanagementprovider/keymanagementprovider_test.go
index 6e6a55a..0441562 100644
--- a/pkg/keymanagementprovider/keymanagementprovider_test.go
+++ b/pkg/keymanagementprovider/keymanagementprovider_test.go
@@ -139,7 +139,7 @@ func TestDecodeCertificates_FailedX509ParseError(t *testing.T) {
 // TestSetCertificatesInMap checks if certificates are set in the map
 func TestSetCertificatesInMap(t *testing.T) {
 	certificatesMap.Delete("test")
-	SetCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
+	setCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
 	if _, ok := certificatesMap.Load("test"); !ok {
 		t.Fatalf("certificatesMap should have been set for key")
 	}
@@ -147,8 +147,8 @@ func TestSetCertificatesInMap(t *testing.T) {
 
 // TestGetCertificatesFromMap checks if certificates are fetched from the map
 func TestGetCertificatesFromMap(t *testing.T) {
-	certificatesMap.Delete("test")
-	SetCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
+	DeleteResourceFromMap("test")
+	setCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
 	certs, _ := GetCertificatesFromMap(context.Background(), "test")
 	if len(certs) != 1 {
 		t.Fatalf("certificates should have been fetched from the map")
@@ -157,18 +157,28 @@ func TestGetCertificatesFromMap(t *testing.T) {
 
 // TestGetCertificatesFromMap_FailedToFetch checks if certificates are fetched from the map
 func TestGetCertificatesFromMap_FailedToFetch(t *testing.T) {
-	certificatesMap.Delete("test")
+	DeleteResourceFromMap("test")
 	certs, _ := GetCertificatesFromMap(context.Background(), "test")
 	if len(certs) != 0 {
 		t.Fatalf("certificates should not have been fetched from the map")
 	}
 }
 
+// TestGetCertificatesFromMap_ErrorFromReconcile checks if error is returned from reconcile
+func TestGetCertificatesFromMap_ErrorFromReconcile(t *testing.T) {
+	DeleteResourceFromMap("test")
+	SetCertificateError("test", errors.New("test error"))
+	if _, err := GetCertificatesFromMap(context.Background(), "test"); err == nil {
+		t.Fatalf("expected error, but got nil")
+	}
+	DeleteResourceFromMap("test")
+}
+
 // TestDeleteCertificatesFromMap checks if certificates are deleted from the map
 func TestDeleteCertificatesFromMap(t *testing.T) {
 	certificatesMap.Delete("test")
-	SetCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
-	DeleteCertificatesFromMap("test")
+	setCertificatesInMap("test", map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
+	DeleteResourceFromMap("test")
 	if _, ok := certificatesMap.Load("test"); ok {
 		t.Fatalf("certificatesMap should have been deleted for key")
 	}
@@ -185,20 +195,41 @@ func TestFlattenKMPMap(t *testing.T) {
 // TestSetKeysInMap checks if keys are set in the map
 func TestSetKeysInMap(t *testing.T) {
 	keyMap.Delete("test")
-	SetKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
+	setKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
 	if _, ok := keyMap.Load("test"); !ok {
 		t.Fatalf("keysMap should have been set for key")
 	}
 }
 
+// TestSaveSecrets checks if secrets are saved in the map.
+func TestSaveSecrets(t *testing.T) {
+	DeleteResourceFromMap("test")
+
+	SaveSecrets("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}}, map[KMPMapKey][]*x509.Certificate{{}: {{Raw: []byte("testcert")}}})
+	if _, ok := certificatesMap.Load("test"); !ok {
+		t.Fatalf("certificatesMap should have been set for key")
+	}
+	if _, ok := certificatesMap.Load("test"); !ok {
+		t.Fatalf("certificatesMap should have been set for key")
+	}
+
+	DeleteResourceFromMap("test")
+}
+
 // TestGetKeysFromMap checks if keys are fetched from the map
 func TestGetKeysFromMap(t *testing.T) {
-	keyMap.Delete("test")
-	SetKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
+	DeleteResourceFromMap("test")
+	setKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
 	keys, _ := GetKeysFromMap(context.Background(), "test")
 	if len(keys) != 1 {
 		t.Fatalf("keys should have been fetched from the map")
 	}
+
+	SetKeyError("test", errors.New("test error"))
+	if _, err := GetKeysFromMap(context.Background(), "test"); err == nil {
+		t.Fatalf("expected error, but got nil")
+	}
+	DeleteResourceFromMap("test")
 }
 
 // TestGetKeysFromMap_FailedToFetch checks if keys fail to fetch from map
@@ -222,8 +253,8 @@ func TestGetKeysFromMap_AccessDifferentNamespace_ReturnsFalse(t *testing.T) {
 // TestDeleteKeysFromMap checks if key map entry is deleted from the map
 func TestDeleteKeysFromMap(t *testing.T) {
 	keyMap.Delete("test")
-	SetKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
-	DeleteKeysFromMap("test")
+	setKeysInMap("test", "", map[KMPMapKey]crypto.PublicKey{{}: &rsa.PublicKey{}})
+	DeleteResourceFromMap("test")
 	if _, ok := keyMap.Load("test"); ok {
 		t.Fatalf("keysMap should have been deleted for key")
 	}
diff --git a/pkg/verifier/cosign/trustpolicy_test.go b/pkg/verifier/cosign/trustpolicy_test.go
index 5cb10d2..ea720c5 100644
--- a/pkg/verifier/cosign/trustpolicy_test.go
+++ b/pkg/verifier/cosign/trustpolicy_test.go
@@ -19,6 +19,7 @@ import (
 	"context"
 	"crypto"
 	"crypto/ecdsa"
+	"crypto/x509"
 	"fmt"
 	"testing"
 
@@ -171,7 +172,7 @@ func TestGetKeys(t *testing.T) {
 	inputMap := map[keymanagementprovider.KMPMapKey]crypto.PublicKey{
 		{Name: "key1"}: &ecdsa.PublicKey{},
 	}
-	keymanagementprovider.SetKeysInMap("ns/kmp", "", inputMap)
+	keymanagementprovider.SaveSecrets("ns/kmp", "", inputMap, map[keymanagementprovider.KMPMapKey][]*x509.Certificate{})
 	tc := []struct {
 		name    string
 		cfg     TrustPolicyConfig
diff --git a/pkg/verifier/notation/truststore_test.go b/pkg/verifier/notation/truststore_test.go
index 15bbea4..393103a 100644
--- a/pkg/verifier/notation/truststore_test.go
+++ b/pkg/verifier/notation/truststore_test.go
@@ -17,11 +17,13 @@ import (
 	"context"
 	"crypto/x509"
 	"encoding/pem"
+	"errors"
 	"os"
 	"reflect"
 	"testing"
 
 	"github.com/notaryproject/notation-go/verifier/truststore"
+	re "github.com/ratify-project/ratify/errors"
 	"github.com/ratify-project/ratify/pkg/controllers"
 )
 
@@ -33,10 +35,14 @@ const (
 )
 
 type mockCertStores struct {
-	certMap map[string][]*x509.Certificate
+	certMap      map[string][]*x509.Certificate
+	reconcileErr error
 }
 
 func (m *mockCertStores) GetCertsFromStore(_ context.Context, storeName string) ([]*x509.Certificate, error) {
+	if m.reconcileErr != nil {
+		return nil, m.reconcileErr
+	}
 	if m.certMap == nil {
 		return nil, nil
 	}
@@ -47,6 +53,59 @@ func (m *mockCertStores) AddStore(_ string, _ []*x509.Certificate) {}
 
 func (m *mockCertStores) DeleteStore(_ string) {}
 
+func (m *mockCertStores) AddStoreError(_ string, _ error) {}
+
+func TestIsReconcileError(t *testing.T) {
+	err := errors.New("reconcile error")
+	if !isReconcileError(err) {
+		t.Fatalf("expected to be a reconcile error")
+	}
+
+	err = re.ErrorCodeNotFound.WithDetail("not found")
+	if isReconcileError(err) {
+		t.Fatalf("expected not to be a reconcile error")
+	}
+}
+
+func TestParseErrFromKmpAndCertStore(t *testing.T) {
+	reconcileErr := errors.New("reconcile error")
+	notFoundErr := re.ErrorCodeNotFound.WithDetail("not found")
+	tests := []struct {
+		name         string
+		kmpErr       error
+		certStoreErr error
+		expectedErr  error
+	}{
+		{
+			name:         "nil errors",
+			kmpErr:       nil,
+			certStoreErr: nil,
+			expectedErr:  nil,
+		},
+		{
+			name:         "kmpErr should be returned",
+			kmpErr:       reconcileErr,
+			certStoreErr: notFoundErr,
+			expectedErr:  reconcileErr,
+		},
+		{
+			name:         "certStoreErr should be returned",
+			kmpErr:       notFoundErr,
+			certStoreErr: reconcileErr,
+			expectedErr:  reconcileErr,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := parseErrFromKmpAndCertStore(tt.kmpErr, tt.certStoreErr)
+			if !errors.Is(result, tt.expectedErr) {
+				t.Fatalf("expected %+v, got %+v", tt.expectedErr, result)
+			}
+		})
+	}
+}
+
 func TestGetCertificates_EmptyCertMap(t *testing.T) {
 	resetCertStore()
 	certStore := verificationCertStores{
@@ -64,6 +123,29 @@ func TestGetCertificates_EmptyCertMap(t *testing.T) {
 	}
 }
 
+func TestGetCertificates_ErrorFromKMPReconcile(t *testing.T) {
+	resetCertStore()
+	certStore := verificationCertStores{
+		trustStoreTypeCA: verificationCertStores{
+			"certstore1": []interface{}{"default/kv1"},
+			"certstore2": []interface{}{"projecta/kv2"},
+		},
+	}
+	store, err := newTrustStore(nil, certStore)
+	if err != nil {
+		t.Fatalf("failed to parse verificationCertStores: " + err.Error())
+	}
+
+	controllers.NamespacedCertStores = &mockCertStores{
+		reconcileErr: errors.New("reconcile error"),
+	}
+
+	// only the certificate in the specified namedStore should be returned
+	if _, err := store.getCertificatesInternal(context.Background(), truststore.TypeCA, "certstore1"); err == nil {
+		t.Fatalf("error expected if error from KMP reconcile")
+	}
+}
+
 func TestGetCertificates_NamedStore(t *testing.T) {
 	resetCertStore()
 	certStore := verificationCertStores{