Repository: deislabs/ratify (go)

Issue:
Ratify should retrieve the entire cert chain from AKV
### What happened in your environment?

Currently Ratify makes use of [getCertificate](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/keyvault/v7.0/keyvault#BaseClient.GetCertificate) API provided by AKV sdk. https://github.com/deislabs/ratify/blob/main/pkg/certificateprovider/azurekeyvault/provider.go#L185


However, this api returns the leaf cert only, which will fail the notation verification once we addressed this issue: https://github.com/deislabs/ratify/issues/694

### What did you expect to happen?

Therefore, we should use the [GetSecret](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/services/keyvault/v7.0/keyvault#BaseClient.GetCertificate) to fetch the certificate chain from AKV instead.

At the same time, we need to ensure that users configure the private key non-exportable to avoid Ratify fetch both the private keys and certs. And once we fetch the cert chain, it's required to remove the leaf cert from the trust store as #694 mentioned.

### What version of Kubernetes are you running?

_No response_

### What version of Ratify are you running?

_No response_

### Anything else you would like to add?

_No response_

### Are you willing to submit PRs to contribute to this bug fix?

- [ ] Yes, I am willing to implement it.

Pull request:
fix: Updating akv cert provider to use getSecret
# Description
- This PR includes:
 1. Updating akv cert provider to use getSecret to fetch certificates for the entire chain
 2. adding clean up step for dangling aks resources
 3. Updated documentation to configure certificate policy on cert import
 4. Uploaded existing test cert chain in e2e integration, inline root cert is no longer needed.  
 
## What this PR does / why we need it:
Fixes #695 

## Which issue(s) this PR fixes *(optional, using `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when the PR gets merged)*:
Fixes #695

## Type of change

Please delete options that are not relevant.

- [ X] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Helm Chart Change (any edit/addition/update that is necessary for changes merged to the `main` branch)
- [X ] This change requires a documentation update

# How Has This Been Tested?

-Uploaded existing test cert chain in e2e integration, inline root cert is no longer needed
Manual validation includes:
- Secret with non content type
- Secret disabled
- Secret that is not a a cert
- Not enough permissions for identity
- PEM and PCKS12 formats

# Checklist:

- [ ] Does the affected code have corresponding tests?
- [ ] Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
- [ ] Does this introduce breaking changes that would require an announcement or bumping the major version?
- [ ] Do all new files have appropriate license header?

# Post Merge Requirements
- [ ] MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates `Helm Chart Change`


Patch:
diff --git a/.github/workflows/build-pr.yml b/.github/workflows/build-pr.yml
index 9cf7d52..61f1d62 100644
--- a/.github/workflows/build-pr.yml
+++ b/.github/workflows/build-pr.yml
@@ -163,3 +163,30 @@ jobs:
           name: e2e-logs
           path: |
             logs-*.json
+
+  test-cleanup:
+    env:
+      AZURE_SUBSCRIPTION_ID: daae1e1a-63dc-454f-825d-b39289070f79
+      AZURE_CLIENT_ID: 814e6e97-120c-4534-b8a9-f1645bc99500
+      AZURE_TENANT_ID: 72f988bf-86f1-41af-91ab-2d7cd011db47
+    needs: ['build_test_aks_e2e']
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
+      - name: Set up Go 1.20
+        uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1
+        with:
+          go-version: '1.20'
+
+      - name: Az CLI login
+        uses: azure/login@92a5484dfaf04ca78a94597f4f19fea633851fa2 # v1.4.7
+        with:
+          creds: '{"clientId":"${{ env.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ env.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ env.AZURE_TENANT_ID }}"}'
+
+      - name: clean up
+        run: |
+          make e2e-cleanup AZURE_SUBSCRIPTION_ID=${{ env.AZURE_SUBSCRIPTION_ID }}
\ No newline at end of file
diff --git a/Makefile b/Makefile
index 45fe221..f7501e7 100644
--- a/Makefile
+++ b/Makefile
@@ -548,6 +548,8 @@ e2e-helm-deploy-ratify-replica: e2e-helm-deploy-redis e2e-notation-setup e2e-bui
 e2e-aks:
 	./scripts/azure-ci-test.sh ${KUBERNETES_VERSION} ${GATEKEEPER_VERSION} ${TENANT_ID} ${GATEKEEPER_NAMESPACE} ${CERT_DIR}
 
+e2e-cleanup:
+	./scripts/azure-ci-test-cleanup.sh ${AZURE_SUBSCRIPTION_ID}
 ##@ Development
 
 .PHONY: manifests
diff --git a/docs/quickstarts/ratify-on-azure.md b/docs/quickstarts/ratify-on-azure.md
index d48cbda..1739406 100644
--- a/docs/quickstarts/ratify-on-azure.md
+++ b/docs/quickstarts/ratify-on-azure.md
@@ -108,28 +108,22 @@ Configure user-assigned managed identity and enable `AcrPull` role to the worklo
 
 ## Configure access policy for AKV 
 
-> Note: Azure Key Vault integration currently only support self signed certificate, if you are working with a certificate chain, please specify the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider).
-
 1. Set the environmental variable for Azure Key Vault URI.
 
     ```bash
     export VAULT_URI=$(az keyvault show --name ${AKV_NAME} --resource-group ${GROUP_NAME} --query "properties.vaultUri" -otsv)
     ```
 
-2. Import your own private key and certificates. You can import it on the portal as well.
+2. Ratify requires secret permissions to retrieve the public certificates for the entire certificate chain,
+ please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys [here](https://learn.microsoft.com/en-us/azure/key-vault/certificates/how-to-export-certificate?tabs=azure-cli#exportable-and-non-exportable-keys)
+ 
+> Note: If you were unable to configure certificate policy, please consider specifying the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider) to reduce risk of exposing private key.
 
-    ```bash
-    az keyvault certificate import \
-    --vault-name ${AKV_NAME} \
-    -n ${KEY_NAME} \
-    -f ${CERT_PATH}
-    ```
-   
-3. Configure policy for user-assigned managed identity:
+   Configure policy for user-assigned managed identity:
     
     ```bash
     az keyvault set-policy --name ${AKV_NAME} \
-    --certificate-permissions get \
+    --secret-permissions get \
     --object-id ${IDENTITY_OBJECT_ID}
     ```
 
diff --git a/docs/reference/crds/certificate-stores.md b/docs/reference/crds/certificate-stores.md
index a2e6d1f..8ea8a6a 100644
--- a/docs/reference/crds/certificate-stores.md
+++ b/docs/reference/crds/certificate-stores.md
@@ -19,7 +19,7 @@ status: # supported in version >= config.ratify.deislabs.io/v1beta1
 
 # Certificate Store Provider
 ## AzureKeyVault Certificate Provider
-See notation integration example [here](../../developer/verifier.md#section-6-built-in-verifiers)
+See notation integration example [here](../../reference/verifier.md#section-6-built-in-verifiers)
 ```yml
 apiVersion: config.ratify.deislabs.io/v1beta1
 kind: CertificateStore
@@ -57,10 +57,14 @@ status:
 Use command `kubectl get certificatestores.config.ratify.deislabs.io` to see a overview of `certificatestores` status.
 Use command `kubectl get certificatestores.config.ratify.deislabs.io/certstore-akv` to see full details on each certificate.
 ### Limitation
-Azure Key Vault integration currently only works for self signed certificate, we are following up on Azure Key Vault specific limitations so we could support certificate chains in the future, please use [issue 695](https://github.com/deislabs/ratify/issues/695) for tracking. If you are working with a certificate chain, please specify the public root certificate value inline using the [inline certificate provider](certificate-stores.md#inline-certificate-provider). 
+Azure keyvault Certificates are built on top of keys and secrets. When a certificate is created, an addressable key and secret are also created with the same name. Ratify requires secret permissions to retrieve the public certificates for the entire certificate chain, please set private keys to Non-exportable at certificate creation time to avoid security risk. Learn more about non-exportable keys [here](https://learn.microsoft.com/en-us/azure/key-vault/certificates/how-to-export-certificate?tabs=azure-cli#exportable-and-non-exportable-keys)
 
 Please also ensure the certificate is in PEM format, PKCS12 format with nonexportable private keys can not be parsed due to limitation of Golang certificate library.
 
+Akv set up guide in ratify-on-azure [quick start](https://github.com/deislabs/ratify/blob/main/docs/quickstarts/ratify-on-azure.md#configure-access-policy-for-akv).
+
+> Note: If you were unable to configure certificate policy, please consider specifying the public root certificate value inline using the [inline certificate provider](../reference/crds/certificate-stores.md#inline-certificate-provider) to reduce risk of exposing private key.
+
 ## Inline Certificate Provider
 ```
 apiVersion: config.ratify.deislabs.io/v1beta1
diff --git a/docs/reference/oras-auth-provider.md b/docs/reference/oras-auth-provider.md
index 24d9e68..4d3091f 100644
--- a/docs/reference/oras-auth-provider.md
+++ b/docs/reference/oras-auth-provider.md
@@ -81,149 +81,11 @@ Both the above modes uses a k8s secret of type ```dockerconfigjson``` that is de
 
 ### 2. Azure Workload Identity
 
-Ratify pulls artifacts from a private Azure Container Registry using Workload Federated Identity in an Azure Kubernetes Service cluster. For an overview on how workload identity operates in Azure, refer to the [documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation). You can use workload identity federation to configure an Azure AD app registration or user-assgined managed identity. The following workflow will include steps of both.
-
-#### User steps to set up Workload Identity with AKS, ACR and AKV:
+Ratify pulls artifacts from a private Azure Container Registry using Workload Federated Identity in an Azure Kubernetes Service cluster. For an overview on how workload identity operates in Azure, refer to the [documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation). You can use workload identity federation to configure an Azure AD app registration or user-assgined managed identity. 
 
+Please refer to [quick start](https://github.com/deislabs/ratify/blob/main/docs/quickstarts/ratify-on-azure.md#configure-workload-identity-for-acr) to configure workload identity for ACR.
 The official steps for setting up Workload Identity on AKS can be found [here](https://azure.github.io/azure-workload-identity/docs/quick-start.html).  
 
-1. Configure environment variables.
-```shell
-export IDENTITY_NAME=<Identity Name>
-export GROUP_NAME=<Azure Resource Group Name>
-export SUBSCRIPTION_ID=<Azure Subscription ID>
-export TENANT_ID=<Azure Tenant ID>
-export ACR_NAME=<Azure Container Registry Name>
-export AKS_NAME=<Azure Kubernetes Service Name>
-export KEYVAULT_NAME=<Azure Key Vault Name>
-export RATIFY_NAMESPACE=<Namespace where Ratify deployed, defaults to "gatekeeper-system">
-export NOTATION_PEM_NAME=<Name of the certificate file uploaded to Azure Key Vault>
-```
-
-2. Create a Workload Federated Identity.
-
-Create ACR AAD application:
-```shell
-az ad sp create-for-rbac --name ${IDENTITY_NAME}
-
-export IDENTITY_CLIENT_ID="$(az ad sp list --display-name "${IDENTITY_NAME}" --query '[0].appId' -otsv)"
-export IDENTITY_OBJECT_ID="$(az ad app show --id ${IDENTITY_CLIENT_ID} --query id -otsv)"
-```
-
-Or create user-assigned managed identity:
-```shell
-az identity create --name "${IDENTITY_NAME}" --resource-group "${GROUP_NAME}" --location "${LOCATION}" --subscription "${SUBSCRIPTION_ID}"
-
-export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${GROUP_NAME}" --query 'principalId' -otsv)"
-export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${GROUP_NAME} --query 'clientId' -o tsv)
-```
-3. Create ACR and enable acrpull role to the workload identity.
-```shell
-az acr create --name "${ACR_NAME}" \
-  --resource-group "${GROUP_NAME}" \
-  --sku Standard
-
-# AAD application:
-az role assignment create \
-  --assignee ${IDENTITY_CLIENT_ID} \
-  --role acrpull \
-  --scope subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${GROUP_NAME}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
-
-# User-assigend managed identity:
-az role assignment create \
-  --assignee-object-id ${IDENTITY_OBJECT_ID} \
-  --role acrpull \
-  --scope subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${GROUP_NAME}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}
-```
-
-4. Create OIDC enabled AKS cluster by following the [doc](https://learn.microsoft.com/en-us/azure/aks/cluster-configuration#oidc-issuer) or below steps:
-```shell
-# Install the aks-preview extension
-az extension add --name aks-preview
-
-# Register the 'EnableWorkloadIdentityPreview' feature flag
-az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
-az provider register --namespace Microsoft.ContainerService
-
-az aks create \
-    --resource-group "${GROUP_NAME}" \
-    --name "${AKS_NAME}" \
-    --node-vm-size Standard_DS3_v2 \
-    --node-count 1 \
-    --generate-ssh-keys \
-    --enable-workload-identity \
-    --attach-acr ${ACR_NAME} \
-    --enable-oidc-issuer
-
-# Connect to the AKS cluster:
-az aks get-credentials --resource-group ${GROUP_NAME} --name ${AKS_NAME}
-
-export AKS_OIDC_ISSUER="$(az aks show -n ${AKS_NAME} -g ${GROUP_NAME} --query "oidcIssuerProfile.issuerUrl" -otsv)"
-```
-
-5. [optional] Install Mutating Admission Webhook onto AKS cluster by following steps [here](https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html) if you didn't add the `--enable-workload-identity` option while creating the AKS cluster.
-
-6. Establish federated identity credential. On AZ CLI `${RATIFY_NAMESPACE}` is where you deploy Ratify:
-
-Using AAD application:
-```shell
-cat <<EOF > body.json
-{
-  "name": "kubernetes-federated-credential",
-  "issuer": "${AKS_OIDC_ISSUER}",
-  "subject": "system:serviceaccount:${RATIFY_NAMESPACE}:ratify-admin",
-  "description": "Kubernetes service account federated credential",
-  "audiences": [
-    "api://AzureADTokenExchange"
-  ]
-}
-EOF
-
-az rest --method POST --uri "https://graph.microsoft.com/beta/applications/${APPLICATION_OBJECT_ID}/federatedIdentityCredentials" --body @body.json
-```
-
-Using user-assigned managed identity:
-```shell
-# Establish feerated identity credentials between the managed identity, the service account issuer and the subject.
-az identity federated-credential create \
-  --name ratify-federated-credential \
-  --identity-name "${IDENTITY_NAME}" \
-  --resource-group "${GROUP_NAME}" \
-  --issuer "${AKS_OIDC_ISSUER}" \
-  --subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin"
-```
-
-7. Create an Azure Key Vault and set an access policy.
-```shell
-az keyvault create \
-  --resource-group ${GROUP_NAME} \
-  --name ${KEYVAULT_NAME}
-
-export VAULT_URI=$(az keyvault show --name ${KEYVAULT_NAME} --resource-group ${GROUP_NAME} --query "properties.vaultUri" -otsv)
-
-# Import your own private key and certificates. You can import it on the portal as well.
-az keyvault certificate import \
-  --vault-name ${KEYVAULT_NAME} \
-  -n <Certificate Name> \
-  -f /path/to/certificate
-
-# Grant permission to access the certificate.
-az keyvault set-policy --name ${KEYVAULT_NAME} \
-  --certificate-permissions get \
-  --object-id ${IDENTITY_OBJECT_ID}
-```
-
-8. Deploy from local helm chart. Follow the [Quick Start](https://github.com/deislabs/ratify/blob/main/README.md#quick-start) to deploy Gatekeeper and Ratify.
-
-Notes: add below options while installing Ratify
-```shell
---set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID} \
---set akvCertConfig.enabled=true \
---set akvCertConfig.vaultURI=${VAULT_URI} \
---set akvCertConfig.cert1Name=${NOTATION_PEM_NAME} \
---set akvCertConfig.tenantId=${TENANT_ID}
-```
-
 ### 3. Kubernetes Secrets
 
 Ratify resolves registry credentials from [Docker Config Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets) in the cluster. Ratify considers kubernetes secrets in two ways:
diff --git a/pkg/certificateprovider/azurekeyvault/provider.go b/pkg/certificateprovider/azurekeyvault/provider.go
index fbdb8d3..2a4a643 100644
--- a/pkg/certificateprovider/azurekeyvault/provider.go
+++ b/pkg/certificateprovider/azurekeyvault/provider.go
@@ -20,6 +20,7 @@ package azurekeyvault
 import (
 	"context"
 	"crypto/x509"
+	"encoding/base64"
 	"encoding/pem"
 	"fmt"
 	"reflect"
@@ -29,6 +30,7 @@ import (
 	"github.com/deislabs/ratify/pkg/certificateprovider"
 	"github.com/deislabs/ratify/pkg/certificateprovider/azurekeyvault/types"
 	"github.com/deislabs/ratify/pkg/metrics"
+	"golang.org/x/crypto/pkcs12"
 
 	kv "github.com/Azure/azure-sdk-for-go/services/keyvault/v7.1/keyvault"
 	"github.com/Azure/go-autorest/autorest/azure"
@@ -37,13 +39,10 @@ import (
 	"gopkg.in/yaml.v2"
 )
 
-type keyvaultObject struct {
-	content string
-	version string
-}
-
 const (
-	providerName string = "azurekeyvault"
+	providerName      string = "azurekeyvault"
+	PKCS12ContentType string = "application/x-pkcs12"
+	PEMContentType    string = "application/x-pem-file"
 )
 
 type akvCertProvider struct{}
@@ -59,6 +58,7 @@ func Create() certificateprovider.CertificateProvider {
 }
 
 // returns an array of certificates based on certificate properties defined in attrib map
+// get certificate retrieve the entire cert chain using getSecret API call
 func (s *akvCertProvider) GetCertificates(ctx context.Context, attrib map[string]string) ([]*x509.Certificate, certificateprovider.CertificatesStatus, error) {
 	keyvaultURI := types.GetKeyVaultURI(attrib)
 	cloudName := types.GetCloudName(attrib)
@@ -98,30 +98,27 @@ func (s *akvCertProvider) GetCertificates(ctx context.Context, attrib map[string
 
 	certs := []*x509.Certificate{}
 	certsStatus := []map[string]string{}
-	lastRefreshed := time.Now().Format(time.RFC3339)
-
 	for _, keyVaultCert := range keyVaultCerts {
-		logrus.Debugf("fetching object from key vault, certName %v,  keyvault %v", keyVaultCert.CertificateName, keyvaultURI)
+		logrus.Debugf("fetching secret from key vault, certName %v,  keyvault %v", keyVaultCert.CertificateName, keyvaultURI)
 
 		// fetch the object from Key Vault
+		// GetSecret is required so we can fetch the entire cert chain. See issue https://github.com/deislabs/ratify/issues/695 for details
 		startTime := time.Now()
-		keyvaultResult, err := getCertificate(ctx, kvClient, keyvaultURI, keyVaultCert)
+		secretBundle, err := kvClient.GetSecret(ctx, keyvaultURI, keyVaultCert.CertificateName, keyVaultCert.CertificateVersion)
+
 		if err != nil {
-			return nil, nil, err
+			return nil, nil, fmt.Errorf("failed to get secret objectName:%s, objectVersion:%s, error: %w", keyVaultCert.CertificateName, keyVaultCert.CertificateVersion, err)
 		}
-		metrics.ReportAKVCertificateDuration(ctx, time.Since(startTime).Milliseconds(), keyVaultCert.CertificateName)
 
-		// convert bytes to x509
-		decodedCerts, err := certificateprovider.DecodeCertificates([]byte(keyvaultResult.content))
+		certResult, certProperty, err := getCertsFromSecretBundle(secretBundle, keyVaultCert.CertificateName)
+
 		if err != nil {
-			return nil, nil, fmt.Errorf("failed to decode certificate %s, error: %w", keyVaultCert.CertificateName, err)
+			return nil, nil, fmt.Errorf("failed to get certificates from secret bundle:%w", err)
 		}
 
-		certs = append(certs, decodedCerts...)
-		logrus.Debugf("cert '%v', version '%v' added", keyVaultCert.CertificateName, keyvaultResult.version)
-
-		certProperty := getCertStatusProperty(keyVaultCert.CertificateName, keyvaultResult.version, lastRefreshed)
-		certsStatus = append(certsStatus, certProperty)
+		metrics.ReportAKVCertificateDuration(ctx, time.Since(startTime).Milliseconds(), keyVaultCert.CertificateName)
+		certs = append(certs, certResult...)
+		certsStatus = append(certsStatus, certProperty...)
 	}
 
 	return certs, getCertStatusMap(certsStatus), nil
@@ -222,27 +219,75 @@ func initializeKvClient(ctx context.Context, keyVaultEndpoint, tenantID, clientI
 	return &kvClient, nil
 }
 
-// getCertificate retrieves the certificate from the vault
-func getCertificate(ctx context.Context, kvClient *kv.BaseClient, vaultURL string, kvObject types.KeyVaultCertificate) (keyvaultObject, error) {
-	certbundle, err := kvClient.GetCertificate(ctx, vaultURL, kvObject.CertificateName, kvObject.CertificateVersion)
-	if err != nil {
-		return keyvaultObject{}, fmt.Errorf("failed to get certificate objectName:%s, objectVersion:%s, error: %w", kvObject.CertificateName, kvObject.CertificateVersion, err)
+// Parse the secret bundle and return an array of certificates
+// In a certificate chain scenario, all certificates from root to leaf will be returned
+func getCertsFromSecretBundle(secretBundle kv.SecretBundle, certName string) ([]*x509.Certificate, []map[string]string, error) {
+	if secretBundle.ContentType == nil || secretBundle.Value == nil || secretBundle.ID == nil {
+		return nil, nil, errors.Errorf("found invalid secret bundle for certificate  %s, contentType, value, and id must not be nil", certName)
 	}
-	if certbundle.Cer == nil {
-		return keyvaultObject{}, errors.Errorf("certificate value is nil")
+
+	version := getObjectVersion(*secretBundle.ID)
+
+	// This aligns with notation akv implementation
+	// akv plugin supports both PKCS12 and PEM. https://github.com/Azure/notation-azure-kv/blob/558e7345ef8318783530de6a7a0a8420b9214ba8/Notation.Plugin.AzureKeyVault/KeyVault/KeyVaultClient.cs#L192
+	if *secretBundle.ContentType != PKCS12ContentType &&
+		*secretBundle.ContentType != PEMContentType {
+		return nil, nil, errors.Errorf("certificate %s version %s, unsupported secret content type %s, supported type are %s and %s", certName, version, *secretBundle.ContentType, PKCS12ContentType, PEMContentType)
 	}
-	if certbundle.ID == nil {
-		return keyvaultObject{}, errors.Errorf("certificate id is nil")
+
+	results := []*x509.Certificate{}
+	certsStatus := []map[string]string{}
+	lastRefreshed := time.Now().Format(time.RFC3339)
+
+	data := []byte(*secretBundle.Value)
+
+	if *secretBundle.ContentType == PKCS12ContentType {
+		p12, err := base64.StdEncoding.DecodeString(*secretBundle.Value)
+		if err != nil {
+			return nil, nil, fmt.Errorf("azure keyvault certificate provider: failed to decode PKCS12 Value. Certificate %s, version %s, error: %w", certName, version, err)
+		}
+
+		blocks, err := pkcs12.ToPEM(p12, "")
+		if err != nil {
+			return nil, nil, fmt.Errorf("azure keyvault certificate provider: failed to convert PKCS12 Value to PEM. Certificate %s, version %s, error: %w", certName, version, err)
+		}
+
+		var pemData []byte
+		for _, b := range blocks {
+			pemData = append(pemData, pem.EncodeToMemory(b)...)
+		}
+		data = pemData
 	}
-	version := getObjectVersion(*certbundle.ID)
 
-	certBlock := &pem.Block{
-		Type:  types.CertificateType,
-		Bytes: *certbundle.Cer,
+	block, rest := pem.Decode(data)
+
+	for block != nil {
+		switch block.Type {
+		case "PRIVATE KEY":
+			logrus.Warnf("azure keyvault certificate provider: certificate %s, version %s private key skipped. Please see doc to learn how to create a new certificate in keyvault with non exportable keys. https://learn.microsoft.com/en-us/azure/key-vault/certificates/how-to-export-certificate?tabs=azure-cli#exportable-and-non-exportable-keys", certName, version)
+		case "CERTIFICATE":
+			var pemData []byte
+			pemData = append(pemData, pem.EncodeToMemory(block)...)
+			decodedCerts, err := certificateprovider.DecodeCertificates(pemData)
+			if err != nil {
+				return nil, nil, fmt.Errorf("azure keyvault certificate provider: failed to decode Certificate %s, version %s, error: %w", certName, version, err)
+			}
+			for _, cert := range decodedCerts {
+				results = append(results, cert)
+				certProperty := getCertStatusProperty(certName, version, lastRefreshed)
+				certsStatus = append(certsStatus, certProperty)
+			}
+		default:
+			logrus.Warnf("certificate '%s', version '%s': azure keyvault certificate provider detected unknown block type %s", certName, version, block.Type)
+		}
+
+		block, rest = pem.Decode(rest)
+		if block == nil && len(rest) > 0 {
+			return nil, nil, errors.Errorf("certificate '%s', version '%s': azure keyvault certificate provider error, block is nil and remaining block to parse > 0", certName, version)
+		}
 	}
-	var pemData []byte
-	pemData = append(pemData, pem.EncodeToMemory(certBlock)...)
-	return keyvaultObject{content: string(pemData), version: version}, nil
+	logrus.Debugf("azurekeyvault certprovider getCertsFromSecretBundle: %v certificates parsed, Certificate '%s', version '%s'", len(results), certName, version)
+	return results, certsStatus, nil
 }
 
 // getObjectVersion parses the id to retrieve the version
diff --git a/scripts/create-azure-resources.sh b/scripts/create-azure-resources.sh
index 8526545..b72f044 100755
--- a/scripts/create-azure-resources.sh
+++ b/scripts/create-azure-resources.sh
@@ -96,7 +96,7 @@ create_akv() {
   echo "AKV '${KEYVAULT_NAME}' is created"
 
   # Grant permissions to access the certificate.
-  az keyvault set-policy --name ${KEYVAULT_NAME} --certificate-permissions get --object-id ${USER_ASSIGNED_IDENTITY_OBJECT_ID}
+  az keyvault set-policy --name ${KEYVAULT_NAME} --secret-permissions get --object-id ${USER_ASSIGNED_IDENTITY_OBJECT_ID}
 }
 
 main() {
@@ -104,7 +104,7 @@ main() {
   # might take around 20 minutes to register
   timeout --foreground 1200 bash -c register_feature
 
-  az group create --name "${GROUP_NAME}" --location "${LOCATION}" >/dev/null
+  az group create --name "${GROUP_NAME}" --tags "ratifye2e" --location "${LOCATION}" >/dev/null
 
   create_user_managed_identity
   create_akv


Test patch:
diff --git a/pkg/certificateprovider/azurekeyvault/provider_test.go b/pkg/certificateprovider/azurekeyvault/provider_test.go
index 5d0bf8d..3888b1b 100644
--- a/pkg/certificateprovider/azurekeyvault/provider_test.go
+++ b/pkg/certificateprovider/azurekeyvault/provider_test.go
@@ -24,6 +24,7 @@ import (
 	"testing"
 	"time"
 
+	kv "github.com/Azure/azure-sdk-for-go/services/keyvault/v7.1/keyvault"
 	"github.com/Azure/go-autorest/autorest/azure"
 	"github.com/deislabs/ratify/pkg/certificateprovider/azurekeyvault/types"
 	"github.com/sirupsen/logrus"
@@ -273,6 +274,71 @@ func TestGetKeyvaultRequestObj(t *testing.T) {
 	assert.Equal(t, "testversion", result[0].CertificateVersion)
 }
 
+func Test(t *testing.T) {
+	cases := []struct {
+		desc        string
+		value       string
+		contentType string
+		id          string
+		expectedErr bool
+	}{
+		{
+			desc:        "Pem Content Type",
+			value:       "-----BEGIN CERTIFICATE-----\nMIIC8TCCAdmgAwIBAgIUaNrwbhs/I1ecqUYdzD2xuAVNdmowDQYJKoZIhvcNAQEL\nBQAwKjEPMA0GA1UECgwGUmF0aWZ5MRcwFQYDVQQDDA5SYXRpZnkgUm9vdCBDQTAe\nFw0yMzA2MjEwMTIyMzdaFw0yNDA2MjAwMTIyMzdaMBkxFzAVBgNVBAMMDnJhdGlm\neS5kZWZhdWx0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtskG1BUt\n4Fw2lbm53KbwZb1hnLmWdwRotZyznhhk/yrUDcq3uF6klwpk/E2IKfUKIo6doHSk\nXaEZXR68UtXygvA4wdg7xZ6kKpXy0gu+RxGE6CGtDHTyDDzITu+NBjo21ZSsyGpQ\nJeIKftUCHdwdygKf0CdJx8A29GBRpHGCmJadmt7tTzOnYjmbuPVLeqJo/Ex9qXcG\nZbxoxnxr5NCocFeKx+EbLo+k/KjdFB2PKnhgzxAaMMMP6eXPr8l5AlzkC83EmPvN\ntveuaBbamdlFkD+53TZeZlxt3GIdq93Iw/UpbQ/pvhbrztMT+UVEkm15sShfX8Xn\nL2st5A4n0V+66QIDAQABoyAwHjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIH\ngDANBgkqhkiG9w0BAQsFAAOCAQEAGpOqozyfDSBjoTepsRroxxcZ4sq65gw45Bme\nm36BS6FG0WHIg3cMy6KIIBefTDSKrPkKNTtuF25AeGn9jM+26cnfDM78ZH0+Lnn7\n7hs0MA64WMPQaWs9/+89aM9NADV9vp2zdG4xMi6B7DruvKWyhJaNoRqK/qP6LdSQ\nw8M+21sAHvXgrRkQtJlVOzVhgwt36NOb1hzRlQiZB+nhv2Wbw7fbtAaADk3JAumf\nvM+YdPS1KfAFaYefm4yFd+9/C0KOkHico3LTbELO5hG0Mo/EYvtjM+Fljb42EweF\n3nAx1GSPe5Tn8p3h6RyJW5HIKozEKyfDuLS0ccB/nqT3oNjcTw==\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIDRTCCAi2gAwIBAgIUcC33VfaMhOnsl7avNTRVQozoVtUwDQYJKoZIhvcNAQEL\nBQAwKjEPMA0GA1UECgwGUmF0aWZ5MRcwFQYDVQQDDA5SYXRpZnkgUm9vdCBDQTAe\nFw0yMzA2MjEwMTIyMzZaFw0yMzA2MjIwMTIyMzZaMCoxDzANBgNVBAoMBlJhdGlm\neTEXMBUGA1UEAwwOUmF0aWZ5IFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB\nDwAwggEKAoIBAQDDFhDnyPrVDZaeRu6Tbg1a/iTwus+IuX+h8aKhKS1yHz4EF/Lz\nxCy7lNSQ9srGMMVumWuNom/ydIphff6PejZM1jFKPU6OQR/0JX5epcVIjbKa562T\nDguUxJ+h5V3EIyM4RqOWQ2g/xZo86x5TzyNJXiVdHHRvmDvUNwPpMeDjr/EHVAni\n5YQObxkJRiiZ7XOa5zz3YztVm8sSZAwPWroY1HIfvtP+KHpiNDIKSymmuJkH4SEr\nJn++iqN8na18a9DFBPTTrLPe3CxATGrMfosCMZ6LP3iFLLc/FaSpwcnugWdewsUK\nYs+sUY7jFWR7x7/1nyFWyRrQviM4f4TY+K7NAgMBAAGjYzBhMB0GA1UdDgQWBBQH\nYePW7QPP2p1utr3r6gqzEkKs+DAfBgNVHSMEGDAWgBQHYePW7QPP2p1utr3r6gqz\nEkKs+DAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwICBDANBgkqhkiG9w0B\nAQsFAAOCAQEAjKp4vx3bFaKVhAbQeTsDjWJgmXLK2vLgt74MiUwSF6t0wehlfszE\nIcJagGJsvs5wKFf91bnwiqwPjmpse/thPNBAxh1uEoh81tOklv0BN790vsVpq3t+\ncnUvWPiCZdRlAiGGFtRmKk3Keq4sM6UdiUki9s+wnxypHVb4wIpVxu5R271Lnp5I\n+rb2EQ48iblt4XZPczf/5QJdTgbItjBNbuO8WVPOqUIhCiFuAQziLtNUq3p81dHO\nQ2BPgmaitCpIUYHVYighLauBGCH8xOFzj4a4KbOxKdxyJTd0La/vRCKaUtJX67Lc\nfQYVR9HXQZ0YlmwPcmIG5v7wBfcW34NUvA==\n-----END CERTIFICATE-----\n",
+			contentType: "application/x-pem-file",
+			id:          "https://notarycerts.vault.azure.net/secrets/testCert6212/431ad135165741dcb95a46cf3e6686fb",
+			expectedErr: false,
+		},
+		{
+			desc:        "PKCS12 Content Type",
+			value:       "MIIKwAIBAzCCCnwGCSqGSIb3DQEHAaCCCm0EggppMIIKZTCCBhYGCSqGSIb3DQEHAaCCBgcEggYDMIIF/zCCBfsGCyqGSIb3DQEMCgECoIIE/jCCBPowHAYKKoZIhvcNAQwBAzAOBAhT2weR+ffbdgICB9AEggTY/fKh5zG3I4/5Xz2t8F0+FR8jyPUt98wZbGChS0e2u6ksaNm/GUT5oCmizPnTCLzGmi01nD6fZDsN6GuW3b70q8lkexACQyvkVwhdBhEVloOFpShBeWk+bycRMFO6F4aUJDgxzEzo9PaWK4xAq4V+g9pUo8opEzn73pxT664rEsvhrCVxBbWamVLJyQwQ6jkpcWDRKSNy46Pd/G4nqlE/Urf/N3VnmTDqqA8jHcACggPzmo3YfssiDabFgxztfHcQFZiTsCv6RcvmQ3e0yzGukQ7TuwnXmuiXYo+rAynK8aIrcgD4Csx8o4KKXyDjZhbODLdzQ701+B1MK8W269vwrtX2ukufHW1M55fxsLfqxbFYpblI3pj7oG9KYNlUG3Flc7GKgyQPETKxFxXsi9ZIUYZbWeMpXOG5v6Q/0YC9jDvWChlWqF+38UIQeFY/0aEFK9W2uYkVUvT4X9E8QrpuXL+5X1q1d5OKx1dWsLIAfFg2o4ZK1HpFrmRh4ptBElcrd623AcDPA/XSUcKQOdcJW8bnjmQt/+tHmF2a7QFYaLT3gH+V88sfG94aO7ArESaXFrWRw18FwzJVUprGE5kVfNpQcmJ4ls8gg/3c1T48vvSJYpeHcl9ShbfKPQj7KI9mn8sxeg8GLz3wM7fWN9/wK1/Z+NLLk0s2BtkM42acUh+2p2bLJwgKoA7rwv7pOytpi2oVUp+LSm3nyOnhYY/ZiO1yy3NXZ8qNzrzrns+RBp2/UM3jm5Cx+G1FLjxsO+twFUATS+numH93MvBF+YFlVcKxs082s7bkDuUyqAlZstPjlR8/dGobqAXKG8Fq3QLYXP95C4PzMzq61R7AHLi7Ojzl6hCK3kBD0aLmDy7D/p4tOkbhAJylyfX4lSA0zGTnobHVcNDzOhDWY3L+VzYuKQVPyqPKRwPYpfc/I97SUqtpz5Fx8D3tR6lHZ0BG2QDqPF6Rlx7S+oJlHwkfFzhsbYpi72zT7IV1/LV56d1/TOFVvqzX440j3zTh3upi+jQoIMVGLyu8ZtQw12pz8EdBenbiS3rkGHJLu1y0m0UiYzyowQrD4SogrsmSOR3x+pmGCj8QTKscEbmypTqMFXtIJqPt+mlS/B0x5ezeEC9NctYo21S5spmAV+X9HX2KN29kdRaBg+2AhMXWRklRt9DXZj2yd82RVsm9eL/dVkx6LvMksSqHHVy9/G2lWOIJy4d+i5hQ1QCeckmfot/udcR8vOwaJxc+gH8UlZpiNhix+xRi3rdqxJ26pEX9oYHjSTb8gZL3kbjHHtd0KyN1CTHhfSP/0d61ttYWhMp8umi1rV9pSV5rbyqbcKK0Q4NBUwAD7ZIOO7euh7m42r1/fjjhlxsmgO6KLXew5uIC/Di7I34rTBQLPfApg5PSgGGUxs2Vv6pg3Y8gqFajxt+b6uIodZo5LUWqhJxwFPgGc/N1aKe+hz+nEG7pD1AxX4OVMcc2r1y1TlQc8m06IjBSGhLXnp+JoL1UurEvQolR+xG+bs9YKgmzDgbxx1wajxfBsCDpYxhPO2VWMcV1J3MOzUcAAZjoV6AQq1V2+ggY5Cv33Khszqyk6jPjHvsQf0lJqhsByh3/wGll3DnOLzqy4o6OV/hJ8Jhv4mzhZRyEXbDqpZYQavt8VCB78zGB6TATBgkqhkiG9w0BCRUxBgQEAQAAADBXBgkqhkiG9w0BCRQxSh5IAGUAZgAyADQAZABhAGUANAAtAGQAYwBlADQALQA0AGIAMgBjAC0AOABjADEAMgAtAGYAYgBmAGIANAAzADAAZAA4ADIANwAwMHkGCSsGAQQBgjcRATFsHmoATQBpAGMAcgBvAHMAbwBmAHQAIABFAG4AaABhAG4AYwBlAGQAIABSAFMAQQAgAGEAbgBkACAAQQBFAFMAIABDAHIAeQBwAHQAbwBnAHIAYQBwAGgAaQBjACAAUAByAG8AdgBpAGQAZQByMIIERwYJKoZIhvcNAQcGoIIEODCCBDQCAQAwggQtBgkqhkiG9w0BBwEwHAYKKoZIhvcNAQwBAzAOBAimXLppRwdpdQICB9CAggQAv5+xRbONQxXaSgWoKOGeN/8CX3tzP0c0Mr4bC420v/IXZuUpaUplt4IBHRazdDRtMfcfb1pQig32j6aYnftUO7J62qwea7UT2t3+JYLye/lJ/EFeF++yqzXge5QQaK3s1E2YgSuSWdTNk4VaPZghA/7ar5UGluWac/112Uhdfn65ime2ysJvd5BHzZFFNy5TqrVN/POzGYM+NdhYtFV9Uy/v2/6zvr9Un4Ns6KhwSHyG4VL3dM2f9FFvW4sjErkWnkxeRLSGdzVPoWF8vO15V0/C6HIV6ug7WPoRODgnTdmWPDctyY+rjy//0jhA45AhIb2TIjdLjNi4RtP4uEGZ5WE8A61QZbJlp/nYKFggpEOqfQMOCYDEo5RhmZ3tEN9m/gLlFKxVswb/VjxHL0fHSRCA+2fmC/RuXw+ZspUFJEW7+SPM0GSq6trz6zYtCD8iVR+OgMY3CdGS5TRudArQLkcwL9vJm9IuAHW5IgvC25zGzM0BdPYylyws7XfMBmClXxBkWAd6WhjN+F9YR62Shk77Jj4rX/7460UzdWW4spZZnSPF/gAzHqUzYkTNJFqYCT3BDbYextG2cLaXB2H2CLwHlQIPGGhMBh/GpqYKCr726vBKlODhMAaZBrV6KzwXDVw75c04BWqRTEQ3xlvXsqP2CmzkHoF+WiOrl7eNs2RJhD/Ul7DN5GUVpanjBvPSxB04d/AXX3Rn4hrZWxtxjLVpQpZedjXA03kmjj/8tIQ3Fs0rAgqT+CZxpvplrdD3uWxWTH8xqAJHTXoNyFhnwv8oBkmkqw6AxoaHs+yFwS8vw2tO1aj1ky6HYxKQkt3U/rTiHSCUUPegvmBsk+obbuRG5r0gMasfXyU41sBq4kFjP+YcpqyyyFI1wKRY2Sgio8Rf6pd6NjcwE7IrTJywUVaLdaKOHR+AaY50I+UB1DApflYv32cN07XoiazZYu3uARD4PQEatWUps96rvJ6i2vhC0q2+qru+kpM89OEKO1uKPCBMy3m3g/cWofg/yGk62dbNWQu4WnOo0G+Cdg5UBwRRpg1dL4/JNur2F7LzuG4eQ2HAQhuZkaKcuhEFbGdCaqEWnM7uPdpEKmh5shKUtaHnq2sRQfAj/oprRhOv+XiFV79bjYUKSvUJ8ZE1W463mc53ygNKp12D1D2u/WSwrtc1DHvnNS3Sgu2X2SOIcQplssTGRpOpjN+guUOSQCeXmpo9gqCrkG1dpDnMDNb5Km/+kurqEH6ebG1iZ+xUItX7EXAymCMWpNgvY2Fuw9cK0xUaYS1SyNStSJgd3udB3o/mxuFd0sP28ojmloIBCroC5Cm0zgCg3+l/TeaCmLL/6VwI6yKr2bBG03gq4IYX+zA7MB8wBwYFKw4DAhoEFHBrDFC1fmAxcvGwsyS/Tl46Ox2eBBTWbe5YACqUwXIPT/K3bixCBGNytQICB9A=",
+			contentType: "application/x-pkcs12",
+			id:          "https://notarycerts.vault.azure.net/secrets/testCert6212/431ad135165741dcb95a46cf3e6686fb",
+			expectedErr: false,
+		},
+		{
+			desc:        "Invalid PKCS12 Content",
+			value:       "IKwAIBAzCCCnwGCSqGSIb3DQEHAaCCCm0EggppMIIKZTCCBhYGCSqGSIb3DQEHAaCCBgcEggYDMIIF/zCCBfsGCyqGSIb3DQEMCgECoIIE/jCCBPowHAYKKoZIhvcNAQwBAzAOBAhT2weR+ffbdgICB9AEggTY/fKh5zG3I4/5Xz2t8F0+FR8jyPUt98wZbGChS0e2u6ksaNm/GUT5oCmizPnTCLzGmi01nD6fZDsN6GuW3b70q8lkexACQyvkVwhdBhEVloOFpShBeWk+bycRMFO6F4aUJDgxzEzo9PaWK4xAq4V+g9pUo8opEzn73pxT664rEsvhrCVxBbWamVLJyQwQ6jkpcWDRKSNy46Pd/G4nqlE/Urf/N3VnmTDqqA8jHcACggPzmo3YfssiDabFgxztfHcQFZiTsCv6RcvmQ3e0yzGukQ7TuwnXmuiXYo+rAynK8aIrcgD4Csx8o4KKXyDjZhbODLdzQ701+B1MK8W269vwrtX2ukufHW1M55fxsLfqxbFYpblI3pj7oG9KYNlUG3Flc7GKgyQPETKxFxXsi9ZIUYZbWeMpXOG5v6Q/0YC9jDvWChlWqF+38UIQeFY/0aEFK9W2uYkVUvT4X9E8QrpuXL+5X1q1d5OKx1dWsLIAfFg2o4ZK1HpFrmRh4ptBElcrd623AcDPA/XSUcKQOdcJW8bnjmQt/+tHmF2a7QFYaLT3gH+V88sfG94aO7ArESaXFrWRw18FwzJVUprGE5kVfNpQcmJ4ls8gg/3c1T48vvSJYpeHcl9ShbfKPQj7KI9mn8sxeg8GLz3wM7fWN9/wK1/Z+NLLk0s2BtkM42acUh+2p2bLJwgKoA7rwv7pOytpi2oVUp+LSm3nyOnhYY/ZiO1yy3NXZ8qNzrzrns+RBp2/UM3jm5Cx+G1FLjxsO+twFUATS+numH93MvBF+YFlVcKxs082s7bkDuUyqAlZstPjlR8/dGobqAXKG8Fq3QLYXP95C4PzMzq61R7AHLi7Ojzl6hCK3kBD0aLmDy7D/p4tOkbhAJylyfX4lSA0zGTnobHVcNDzOhDWY3L+VzYuKQVPyqPKRwPYpfc/I97SUqtpz5Fx8D3tR6lHZ0BG2QDqPF6Rlx7S+oJlHwkfFzhsbYpi72zT7IV1/LV56d1/TOFVvqzX440j3zTh3upi+jQoIMVGLyu8ZtQw12pz8EdBenbiS3rkGHJLu1y0m0UiYzyowQrD4SogrsmSOR3x+pmGCj8QTKscEbmypTqMFXtIJqPt+mlS/B0x5ezeEC9NctYo21S5spmAV+X9HX2KN29kdRaBg+2AhMXWRklRt9DXZj2yd82RVsm9eL/dVkx6LvMksSqHHVy9/G2lWOIJy4d+i5hQ1QCeckmfot/udcR8vOwaJxc+gH8UlZpiNhix+xRi3rdqxJ26pEX9oYHjSTb8gZL3kbjHHtd0KyN1CTHhfSP/0d61ttYWhMp8umi1rV9pSV5rbyqbcKK0Q4NBUwAD7ZIOO7euh7m42r1/fjjhlxsmgO6KLXew5uIC/Di7I34rTBQLPfApg5PSgGGUxs2Vv6pg3Y8gqFajxt+b6uIodZo5LUWqhJxwFPgGc/N1aKe+hz+nEG7pD1AxX4OVMcc2r1y1TlQc8m06IjBSGhLXnp+JoL1UurEvQolR+xG+bs9YKgmzDgbxx1wajxfBsCDpYxhPO2VWMcV1J3MOzUcAAZjoV6AQq1V2+ggY5Cv33Khszqyk6jPjHvsQf0lJqhsByh3/wGll3DnOLzqy4o6OV/hJ8Jhv4mzhZRyEXbDqpZYQavt8VCB78zGB6TATBgkqhkiG9w0BCRUxBgQEAQAAADBXBgkqhkiG9w0BCRQxSh5IAGUAZgAyADQAZABhAGUANAAtAGQAYwBlADQALQA0AGIAMgBjAC0AOABjADEAMgAtAGYAYgBmAGIANAAzADAAZAA4ADIANwAwMHkGCSsGAQQBgjcRATFsHmoATQBpAGMAcgBvAHMAbwBmAHQAIABFAG4AaABhAG4AYwBlAGQAIABSAFMAQQAgAGEAbgBkACAAQQBFAFMAIABDAHIAeQBwAHQAbwBnAHIAYQBwAGgAaQBjACAAUAByAG8AdgBpAGQAZQByMIIERwYJKoZIhvcNAQcGoIIEODCCBDQCAQAwggQtBgkqhkiG9w0BBwEwHAYKKoZIhvcNAQwBAzAOBAimXLppRwdpdQICB9CAggQAv5+xRbONQxXaSgWoKOGeN/8CX3tzP0c0Mr4bC420v/IXZuUpaUplt4IBHRazdDRtMfcfb1pQig32j6aYnftUO7J62qwea7UT2t3+JYLye/lJ/EFeF++yqzXge5QQaK3s1E2YgSuSWdTNk4VaPZghA/7ar5UGluWac/112Uhdfn65ime2ysJvd5BHzZFFNy5TqrVN/POzGYM+NdhYtFV9Uy/v2/6zvr9Un4Ns6KhwSHyG4VL3dM2f9FFvW4sjErkWnkxeRLSGdzVPoWF8vO15V0/C6HIV6ug7WPoRODgnTdmWPDctyY+rjy//0jhA45AhIb2TIjdLjNi4RtP4uEGZ5WE8A61QZbJlp/nYKFggpEOqfQMOCYDEo5RhmZ3tEN9m/gLlFKxVswb/VjxHL0fHSRCA+2fmC/RuXw+ZspUFJEW7+SPM0GSq6trz6zYtCD8iVR+OgMY3CdGS5TRudArQLkcwL9vJm9IuAHW5IgvC25zGzM0BdPYylyws7XfMBmClXxBkWAd6WhjN+F9YR62Shk77Jj4rX/7460UzdWW4spZZnSPF/gAzHqUzYkTNJFqYCT3BDbYextG2cLaXB2H2CLwHlQIPGGhMBh/GpqYKCr726vBKlODhMAaZBrV6KzwXDVw75c04BWqRTEQ3xlvXsqP2CmzkHoF+WiOrl7eNs2RJhD/Ul7DN5GUVpanjBvPSxB04d/AXX3Rn4hrZWxtxjLVpQpZedjXA03kmjj/8tIQ3Fs0rAgqT+CZxpvplrdD3uWxWTH8xqAJHTXoNyFhnwv8oBkmkqw6AxoaHs+yFwS8vw2tO1aj1ky6HYxKQkt3U/rTiHSCUUPegvmBsk+obbuRG5r0gMasfXyU41sBq4kFjP+YcpqyyyFI1wKRY2Sgio8Rf6pd6NjcwE7IrTJywUVaLdaKOHR+AaY50I+UB1DApflYv32cN07XoiazZYu3uARD4PQEatWUps96rvJ6i2vhC0q2+qru+kpM89OEKO1uKPCBMy3m3g/cWofg/yGk62dbNWQu4WnOo0G+Cdg5UBwRRpg1dL4/JNur2F7LzuG4eQ2HAQhuZkaKcuhEFbGdCaqEWnM7uPdpEKmh5shKUtaHnq2sRQfAj/oprRhOv+XiFV79bjYUKSvUJ8ZE1W463mc53ygNKp12D1D2u/WSwrtc1DHvnNS3Sgu2X2SOIcQplssTGRpOpjN+guUOSQCeXmpo9gqCrkG1dpDnMDNb5Km/+kurqEH6ebG1iZ+xUItX7EXAymCMWpNgvY2Fuw9cK0xUaYS1SyNStSJgd3udB3o/mxuFd0sP28ojmloIBCroC5Cm0zgCg3+l/TeaCmLL/6VwI6yKr2bBG03gq4IYX+zA7MB8wBwYFKw4DAhoEFHBrDFC1fmAxcvGwsyS/Tl46Ox2eBBTWbe5YACqUwXIPT/K3bixCBGNytQICB9A=",
+			contentType: "application/x-pkcs12",
+			id:          "https://notarycerts.vault.azure.net/secrets/testCert6212/431ad135165741dcb95a46cf3e6686fb",
+			expectedErr: true,
+		},
+		{
+			desc:        "Secret Text File",
+			value:       "text",
+			contentType: "text",
+			id:          "https://notarycerts.vault.azure.net/secrets/testCert6212/431ad135165741dcb95a46cf3e6686fb",
+			expectedErr: true,
+		},
+		{
+			desc:        "Test empty",
+			value:       "",
+			contentType: "",
+			id:          "",
+			expectedErr: true,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.desc, func(t *testing.T) {
+			testdata := kv.SecretBundle{
+				Value:       &tc.value,
+				ID:          &tc.id,
+				ContentType: &tc.contentType,
+			}
+
+			certs, status, err := getCertsFromSecretBundle(testdata, "certName")
+			if tc.expectedErr {
+				assert.NotNil(t, err)
+				assert.Nil(t, certs)
+				assert.Nil(t, status)
+			} else {
+				assert.Nil(t, err)
+			}
+		})
+	}
+}
+
 func TestGetKeyvaultRequestObj_error(t *testing.T) {
 	cases := []struct {
 		desc        string
diff --git a/scripts/azure-ci-test-cleanup.sh b/scripts/azure-ci-test-cleanup.sh
new file mode 100755
index 0000000..4895b92
--- /dev/null
+++ b/scripts/azure-ci-test-cleanup.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+##--------------------------------------------------------------------
+#
+# Copyright The Ratify Authors.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+##--------------------------------------------------------------------
+
+export AZURE_SUBSCRIPTION_ID=${1}
+
+cleanup() {
+  az account set -s ${AZURE_SUBSCRIPTION_ID}
+  echo "Deleting resource group with ratifye2e tag"
+   
+  # Get list of resources with specific tag
+  resource_list=`az group list --tag ratifye2e`
+
+  # Get number of resources to be deleted
+  num_resources=`echo $resource_list | jq length`
+  echo 'Found' $num_resources 'resources'
+
+  # Delete resources
+  for((i=0; i<$num_resources; i++)) 
+  do
+      # Get $i th resource name and type
+     name=`echo $resource_list | jq .[$i].name | tr -d '"'`
+     type=`echo $resource_list | jq .[$i].type | tr -d '"'`
+      
+      echo 'Deleting following resource ...'
+      echo 'resource name': $name
+      echo 'resource type': $type
+      
+      # Delete $i th resource
+      az group delete --name ${name} -y --no-wait
+       
+  done
+}
+
+cleanup
\ No newline at end of file
diff --git a/scripts/azure-ci-test.sh b/scripts/azure-ci-test.sh
index d414149..697f093 100755
--- a/scripts/azure-ci-test.sh
+++ b/scripts/azure-ci-test.sh
@@ -33,6 +33,8 @@ TENANT_ID=$3
 export RATIFY_NAMESPACE=${4:-gatekeeper-system}
 CERT_DIR=${5:-"~/ratify/certs"}
 export NOTATION_PEM_NAME="notation"
+export NOTATION_CHAIN_PEM_NAME="notationchain"
+
 TAG="test${SUFFIX}"
 REGISTRY="${ACR_NAME}.azurecr.io"
 
@@ -72,6 +74,7 @@ deploy_ratify() {
     --set akvCertConfig.enabled=true \
     --set akvCertConfig.vaultURI=${VAULT_URI} \
     --set akvCertConfig.cert1Name=${NOTATION_PEM_NAME} \
+    --set akvCertConfig.cert2Name=${NOTATION_CHAIN_PEM_NAME} \
     --set akvCertConfig.tenantId=${TENANT_ID} \
     --set oras.authProviders.azureWorkloadIdentityEnabled=true \
     --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID} \
@@ -84,15 +87,28 @@ deploy_ratify() {
   kubectl apply -f https://deislabs.github.io/ratify/library/default/samples/constraint.yaml
 }
 
-upload_cert_to_akv() {
+upload_cert_to_akv() { 
   rm -f notation.pem
   cat ~/.config/notation/localkeys/ratify-bats-test.key >>notation.pem
   cat ~/.config/notation/localkeys/ratify-bats-test.crt >>notation.pem
 
+  echo "uploading notation.pem"
   az keyvault certificate import \
     --vault-name ${KEYVAULT_NAME} \
     -n ${NOTATION_PEM_NAME} \
     -f notation.pem
+
+  rm -f notationchain.pem
+  
+  cat .staging/notation/leaf-test/leaf.key >>notationchain.pem
+  cat .staging/notation/leaf-test/leaf.crt >>notationchain.pem   
+
+  echo "uploading notationchain.pem"
+  az keyvault certificate import \
+    --vault-name ${KEYVAULT_NAME} \
+    -n ${NOTATION_CHAIN_PEM_NAME} \
+    -f notationchain.pem \
+    -p @./test/bats/tests/config/akvpolicy.json
 }
 
 save_logs() {
diff --git a/test/bats/azure-test.bats b/test/bats/azure-test.bats
index d2f07fd..7583009 100644
--- a/test/bats/azure-test.bats
+++ b/test/bats/azure-test.bats
@@ -45,9 +45,13 @@ SLEEP_TIME=1
     assert_success
     run kubectl apply -f ./library/default/samples/constraint.yaml
     assert_success
+   
+    # verify that the image can be run with a root cert, root verification cert should have been configured on deployment
+    run kubectl run demo-leaf --namespace default --image=${TEST_REGISTRY}/notation:leafSigned
+    assert_success
 
-    # add the root certificate as an inline certificate store
-    cat ~/.config/notation/truststore/x509/ca/leaf-test/root.crt | sed 's/^/      /g' >>./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
+    # add the leaf certificate as an inline certificate store
+    cat ~/.config/notation/truststore/x509/ca/leaf-test/leaf.crt | sed 's/^/      /g' >>./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
     run kubectl apply -f ./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
     assert_success
     sed -i '9,$d' ./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
@@ -56,16 +60,6 @@ SLEEP_TIME=1
     run kubectl apply -f ./test/bats/tests/config/config_v1beta1_verifier_notation.yaml
     assert_success
 
-    # verify that the image can be run with a root cert
-    run kubectl run demo-leaf --namespace default --image=${TEST_REGISTRY}/notation:leafSigned
-    assert_success
-
-    # add the root certificate as an inline certificate store
-    cat ~/.config/notation/truststore/x509/ca/leaf-test/leaf.crt | sed 's/^/      /g' >>./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
-    run kubectl apply -f ./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
-    assert_success
-    sed -i '9,$d' ./test/bats/tests/config/config_v1beta1_certstore_inline.yaml
-
     # wait for the httpserver cache to be invalidated
     sleep 15
     # verify that the image cannot be run with a leaf cert
diff --git a/test/bats/tests/config/akvpolicy.json b/test/bats/tests/config/akvpolicy.json
new file mode 100644
index 0000000..da9b5db
--- /dev/null
+++ b/test/bats/tests/config/akvpolicy.json
@@ -0,0 +1,45 @@
+{
+  "issuerParameters": {
+    "certificateTransparency": null,
+    "name": "Self"
+  },
+  "keyProperties": {
+    "curve": null,
+    "exportable": false,
+    "keySize": 2048,
+    "keyType": "RSA",
+    "reuseKey": true
+  },
+  "lifetimeActions": [
+    {
+      "action": {
+        "actionType": "AutoRenew"
+      },
+      "trigger": {
+        "daysBeforeExpiry": 90,
+        "lifetimePercentage": null
+      }
+    }
+  ],
+  "secretProperties": {
+    "contentType": "application/x-pem-file"
+  },
+  "x509CertificateProperties": {
+    "ekus": [],
+    "keyUsage": [
+      "cRLSign",
+      "dataEncipherment",
+      "digitalSignature",
+      "keyEncipherment",
+      "keyAgreement",
+      "keyCertSign"
+    ],
+    "subject": "CN=ratify.default",
+    "subjectAlternativeNames": {
+      "dnsNames": null,
+      "emails": null,
+      "upns": null
+    },
+    "validityInMonths": 12
+  }
+}
\ No newline at end of file