Project: CyberArk Conjur (Provisioning / Key Management)
File: TROUBLESHOOTING.md

- - - - - This guide presents some tips and guidelines for troubleshooting the functionality of on a cluster. This section presents some tips and guidelines for troubleshooting specifically on a cluster that has been deployed via the . The intended audience for this section of the guide is anyone who encounters issues when deploying Kubernetes applications that make use of Conjur Kubernetes authentication brokers or clients such as the following to authenticate with Conjur: sidecar container as either a sidecar or init container This section of the guide assumes that you have: access to your Kubernetes cluster. <details> <summary>Click to expand installation examples.</summary> ```sh-session curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.19.0/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin ``` ```sh-session brew install kubernetes-cli ``` </details> client version 3 or newer. <details> <summary>Click to expand installation examples.</summary> ```sh-session mkdir -p ~/temp/helm-v3.3.1 cd ~/temp/helm-v3.3.1 helmtarfile=helm-v3.3.1-linux-amd64.tar.gz curl https://get.helm.sh/"$helmtarfile" --output "$helmtarfile" tar -zxvf "$helmtarfile" sudo mv linux-amd64/helm /usr/local/bin ``` ```sh-session brew install helm ``` </details> access to your server. If you don't have this set up already, see the section below. Before proceeding to the step-by-step guide to verifying your Conjur authentication, here are a few tools and "how-tos" that might come in handy while following the troubleshooting steps. In some cases, it may be helpful to create a Conjur CLI pod in your Kubernetes cluster, and create a `conjur` command alias that executes commands via that Conjur CLI pod. For example, you may be exploring Conjur Open Source and Kubernetes authentication on a or or cluster, and you prefer not to install a software load balancer such as . <details> <summary>Click to see an example of how to deploy a Conjur CLI pod.</summary> ```sh-session HELM_RELEASE=conjur-oss CONJUR_NAMESPACE=conjur-oss CLI_IMAGE=cyberark/conjur-cli:5-latest echo " apiVersion: apps/v1 kind: Deployment metadata: name: conjur-cli labels: app: conjur-cli spec: replicas: 1 selector: matchLabels: app: conjur-cli template: metadata: name: conjur-cli labels: app: conjur-cli spec: serviceAccountName: conjur-oss containers: name: conjur-cli image: $CLI_IMAGE imagePullPolicy: Always command: ["sleep"] args: ["infinity"] " | kubectl create -n "$CONJUR_NAMESPACE" -f - CONJURPOD="$(kubectl get pods -n $CONJURNAMESPACE -l app=conjur-oss \ -o jsonpath='{.items[0].metadata.name}')" CONJURACCOUNT="$(kubectl exec -n $CONJURNAMESPACE $CONJUR_POD -c conjur-oss -- printenv \ | grep CONJUR_ACCOUNT \ | sed 's/.*=//')" ADMINPASSWORD="$(kubectl exec -n $CONJURNAMESPACE $CONJUR_POD -c conjur-oss \ -- conjurctl role retrieve-key $CONJUR_ACCOUNT:user:admin | tail -1)" export CLIPOD="$(kubectl get pods -n $CONJURNAMESPACE -l app=conjur-cli \ -o jsonpath='{.items[0].metadata.name}')" CONJURURL="https://conjur-oss.$CONJURNAMESPACE.svc.cluster.local" kubectl exec -n $CONJURNAMESPACE $CLIPOD -- bash -c "yes yes | conjur init -a $CONJURACCOUNT -u $CONJURURL" kubectl exec -n $CONJURNAMESPACE $CLIPOD -- conjur authn login -u admin -p $ADMIN_PASSWORD alias conjur="kubectl exec -n conjur-oss $CLI_POD -- conjur" ``` </details> Sometimes it is helpful to be able to run the `curl` command from inside the Kubernetes cluster. <details> <summary>Click to see an example of how to deploy a Conjur CLI pod.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss echo " apiVersion: v1 kind: Pod metadata: name: pod-curl labels: name: pod-curl spec: containers: name: alpine-curl image: wiremind/docker-alpine-curl-ca-certificates imagePullPolicy: Always command: ["sh", "-c", "tail -f /dev/null"] volumeMounts: name: conjur-ca-cert mountPath: /usr/local/share/ca-certificates/conjur-ca-cert.crt subPath: conjur-ca-cert.crt readOnly: false volumes: name: conjur-ca-cert secret: secretName: conjur-oss-conjur-ssl-ca-cert items: key: tls.crt path: conjur-ca-cert.crt " | kubectl create -n "$CONJUR_NAMESPACE" -f - kubectl wait --for=condition=ready pod -n $CONJUR_NAMESPACE pod-curl --timeout 300s kubectl exec -it -n $CONJUR_NAMESPACE pod-curl -- update-ca-certificates alias pod-curl="kubectl exec -n $CONJUR_NAMESPACE pod-curl -- curl" ``` To test out curl access from within the Kubernetes cluster, try curling the `conjur-oss` service. For example: ``` $ pod-curl -s

| grep "Your Conjur" <p class="font-large">Your Conjur server is running!</p> $ ``` </details> In the troubleshooting steps below, it sometimes helps to get more detailed information by setting the log level to `debug` for the Conjur server before reproducing an error. <details> <summary>Click to see an example of how to enable Conjur debug logging.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss helm upgrade \ -n "$CONJUR_NAMESPACE" \ --reuse-values \ --set logLevel=debug \ "$HELM_RELEASE" \ ./conjur-oss ``` </details> <details> <summary>Click to see example of how to show Conjur server logs.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss pod_name=$(kubectl get pods \ -n "$CONJUR_NAMESPACE" \ -l "app=conjur-oss,release=$HELM_RELEASE" \ -o jsonpath="{.items[0].metadata.name}") kubectl logs -n "$CONJURNAMESPACE" "$podname" conjur-oss ``` </details> <details> <summary>Click to see example of how to show Conjur NGINX container logs.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss pod_name=$(kubectl get pods \ -n "$CONJUR_NAMESPACE" \ -l "app=conjur-oss,release=$HELM_RELEASE" \ -o jsonpath="{.items[0].metadata.name}") kubectl logs -n "$CONJURNAMESPACE" "$podname" conjur-oss-nginx ``` </details> <details> <summary>Click to see example of how to show Conjur Postgres pod logs.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss pod_name=$(kubectl get pods \ -n "$CONJUR_NAMESPACE" \ -l "app=conjur-oss-postgres,release=$HELM_RELEASE" \ -o jsonpath="{.items[0].metadata.name}") kubectl logs -n "$CONJURNAMESPACE" "$podname" conjur-oss-nginx ``` </details> Follow the steps below to perform a systematic, step-by-step evaluation of your Conjur authentication configuration. Checking the running state of the Conjur deployment. <details> <summary>Click to see how to check the running state of the Conjur deployment.</summary> Check the state of the Conjur deployment as follows: ```sh-session $ kubectl get pods -n conjur-oss -l release=conjur-oss NAME READY STATUS RESTARTS AGE conjur-oss-5cb86bf558-vrr4r 2/2 Running 0 26m conjur-oss-postgres-0 1/1 Running 0 26m $ ``` </details> If the Conjur server pod and the Postgres backend pod are not in the `Running` state, or their containers are not ready (2/2 for the server and 1/1 for the Postgresql pod), then: Check the for warnings or errors. Check the for warnings or errors. , and then delete the Conjur Open Source server pod to force a pod recreate, and check the again for warnings or errors. Check that the Conjur server is up and running. <details> <summary>Click to see how to check that the Conjur server is up.</summary> The easiest way to check Conjur status is to run a Helm test. The Conjur server is up if a Helm test reports a `Succeeded` phase: ```sh-session $ CONJUR_NAMESPACE=conjur-oss $ HELM_RELEASE=conjur-oss $ helm test -n "$CONJURNAMESPACE" "$HELMRELEASE" | grep "Phase:" Phase: Succeeded $ ``` </details> If the Conjur server check fails, check the for warnings or errors. Verify that the Conjur Kubernetes Authenticator plugin is enabled in Conjur. <details> <summary>Click to see how to check that the Kubernetes Authenticator is enabled.</summary> The list of enabled Conjur authenticators can be read from the Conjur authenticators secret. Here is an example of how to read the enabled authenticators from this secret: ```sh-session $ CONJUR_NAMESPACE=conjur-oss $ HELM_RELEASE=conjur-oss $ authenticators="$(kubectl get secret \ -n "$CONJUR_NAMESPACE" \ "$HELM_RELEASE-conjur-authenticators" \ --template={{.data.key}} \ | base64 -d \ | sed 's/,/\n /g')" $ echo "Enabled Conjur authenticators:"; echo " $authenticators" Enabled Conjur authenticators: authn authn-k8s/my-authenticator-id $ ``` In the above example, there are two Conjur authenticators that are enabled: authn: The default Conjur authenticator authn-k8s/my-authenticator-id: The Conjur Kubernetes authenticator with an authenticator ID of `my-authenticator-id`. </details> If you do not see the Conjur Kubernetes Authenticator (`authn-k8s`) enabled, you can enable it with Helm upgrade. <details> <summary>Click to see how to enable the Kubernetes Authenticator with Helm upgrade.</summary> ```sh-session $ CONJUR_NAMESPACE=conjur-oss $ HELM_RELEASE=conjur-oss $ AUTHENTICATOR_ID="my-authenticator-id" $ helm upgrade \ -n "$CONJUR_NAMESPACE" \ --reuse-values \ --set authenticators="authn\,authn-k8s/$AUTHENTICATOR_ID" \ --wait \ --timeout 300s \ "$HELM_RELEASE" \

``` </details> Verify that a Conjur Authentication `ClusterRole` exists. <details> <summary>Click to see how to check for an authentication `ClusterRole`.</summary> For example: ```sh-session $ CONJUR_NAMESPACE=conjur-oss $ HELM_RELEASE=conjur-oss $ kubectl get clusterrole -n "$CONJUR_NAMESPACE" -l app=conjur-oss -o name clusterrole.rbac.authorization.k8s.io/conjur-oss-conjur-authenticator $ ``` </details> If you don't see a Conjur authentication `ClusterRole`, make sure that the chart value for `rbac.create` is set to true. <details> <summary>Click to see how to check the current value of `rbac.create`.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss helm get values --all -n "$CONJURNAMESPACE" "$HELMRELEASE" ``` <details> Verify that either a `ClusterRoleBinding` or `RoleBinding` has been created for Conjur authentication. *_NOTE: Using a `RoleBinding` is preferred over using a `ClusterRoleBinding` for Conjur authentication since `RoleBindings` limit the scope of permissions granted to the Kubernetes `Namespace` in which it exists._* <details> <summary>Click to see how to check for a `ClusterRoleBinding`.</summary> ```sh-session HELM_RELEASE=conjur-oss crb="$(kubectl get clusterrolebinding | grep ClusterRole\/$HELM_RELEASE-conjur-authenticator | awk '{print $1}')" if [ -z "$crb" ]; then echo "No ClusterRoleBinding found for Conjur authentication!" else cluster_role="$(kubectl get clusterrolebinding $crb -o jsonpath='{.roleRef.name}')" subjects="$(kubectl get clusterrolebinding $crb -o jsonpath='{.subjects}')" echo "Found ClusterRoleBinding: $crb" echo " It is bound to ClusterRole: $cluster_role" echo " It applies to subjects: $subjects" fi ``` There is a properly configure `ClusterRoleBinding` if the above commands find a `ClusterRoleBinding`, and both of the following are true: It is bound to the `ClusterRole` found in the previous step, and... It applies to the Conjur `ServiceAccount` named `conjur-oss` Here is a sample successful output: ```sh-session Found ClusterRoleBinding: conjur-oss-conjur-authenticator It is bound to ClusterRole: conjur-oss-conjur-authenticator It applies to subjects: [map[kind:ServiceAccount name:conjur-oss namespace:conjur-oss]] ``` </details> <details> <summary>Click to see how to check for a `RoleBinding` in the application namespace.</summary> ```sh-session APP_NAMESPACE=app-test HELM_RELEASE=conjur-oss rb="$(kubectl get rolebinding -n $APPNAMESPACE | grep ClusterRole\/$HELMRELEASE-conjur-authenticator | awk '{print $1}')" if [ -z "$rb" ]; then echo "No RoleBinding found in namespace $APP_NAMESPACE for Conjur authentication!" else clusterrole="$(kubectl get rolebinding -n $APPNAMESPACE $rb -o jsonpath='{.roleRef.name}')" subjects="$(kubectl get rolebinding -n $APP_NAMESPACE $rb -o jsonpath='{.subjects}')" echo "Found RoleBinding: $crb" echo " It is bound to ClusterRole: $cluster_role" echo " It applies to subjects: $subjects" fi ``` There is a properly configure `RoleBinding` if the above commands find a `RoleBinding`, and both of the following are true: It is bound to the `ClusterRole` found in the previous step, and... It applies to the Conjur `ServiceAccount` named `conjur-oss` Here is a sample successful output: ```sh-session Found RoleBinding: conjur-oss-conjur-authenticator It is bound to ClusterRole: conjur-oss-conjur-authenticator It applies to subjects: [map[kind:ServiceAccount name:conjur-oss namespace:conjur-oss]] ``` </details> If neither a `ClusterRoleBinding` nor a `RoleBinding` has been created for Kubernetes authentication, expand the text block below to see an example of how to create a `RoleBinding` for Kubernetes authentication. <details> <summary>Click to see how to create a Conjur authentication `RoleBinding`.</summary> ```sh-session APP_NAMESPACE=app-test CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: conjur-authenticator-role-binding namespace: $APP_NAMESPACE subjects: kind: ServiceAccount name: conjur-oss namespace: $CONJUR_NAMESPACE roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: $HELM_RELEASE-conjur-authenticator EOF ``` </details> Check the authenticator container's Conjur SSL CA certificate. The Conjur authenticator container (whether it's Secretless Broker sidecar, Kubernetes Authenticator sidecar, or Kubernetes Authenticator init container) that is included in your application pod needs to be configured with Conjur's CA certificate in order to connect with Conur. <details> <summary>Click to see how to check the authenticator container's Conjur CA cert.</summary> To display the authenticator's Conjur CA certificate: ```sh-session APP_NAMESPACE="app-test" APPPODLABEL="app=test-app-secretless" AUTHN_CONTAINER="secretless" APPPODNAME="$(kubectl get pod \ -n $APP_NAMESPACE \ -l $APPPODLABEL \ -o

kubectl exec \ -n $APP_NAMESPACE \ $APPPODNAME \ -c $AUTHN_CONTAINER \ -- printenv CONJURSSLCERTIFICATE ``` This should be compared with the actual Conjur CA certificate, which can be read from the Conjur server's NGINX container: ```sh-session CONJUR_NAMESPACE=conjur-oss HELM_RELEASE=conjur-oss CONJURPOD="$(kubectl get pods -n $CONJURNAMESPACE -l app=conjur-oss \ -o jsonpath='{.items[0].metadata.name}')" kubectl exec -n "$CONJURNAMESPACE" "$CONJURPOD" -c conjur-oss-nginx \ -- cat /opt/conjur/etc/ssl/cert/tls.crt ``` </details> Check the Kubernetes authenticator container's `CONJUR_ACCOUNT` setting. <details> <summary>Click to see how to read the authn container's `CONJUR_ACCOUNT` setting.</summary> To display the Conjur account configured for the application's authenticator container: ```sh-session APP_NAMESPACE="app-test" APPPODLABEL="app=test-app-secretless" AUTHN_CONTAINER="secretless" APPPODNAME="$(kubectl get pod \ -n $APP_NAMESPACE \ -l $APPPODLABEL \ -o jsonpath='{.items[0].metadata.name}')" kubectl exec \ -n $APP_NAMESPACE \ $APPPODNAME \ -c $AUTHN_CONTAINER \ -- printenv CONJUR_ACCOUNT ``` </details> The Kubernetes authenticator container's `CONJUR_ACCOUNT` setting shouldmatch the Conjur account with which the Conjur server was initialized. <details> <summary>Click to see how to read the Conjur server's account.</summary> ```sh-session CONJUR_NAMESPACE=conjur-oss CONJURPOD="$(kubectl get pods -n $CONJURNAMESPACE -l app=conjur-oss \ -o jsonpath='{.items[0].metadata.name}')" kubectl exec -n "$CONJURNAMESPACE" "$CONJURPOD" -c conjur-oss \ -- printenv CONJUR_ACCOUNT ``` </details> Check the authenticator container's `CONJURAPPLIANCEURL` setting. The Conjur authenticator container (whether it's Secretless Broker sidecar, Kubernetes Authenticator sidecar, or Kubernetes Authenticator init container) that is included in your application pod needs to be configured with the correct `CONJURAPPLIANCEURL` environment setting in order to be able to connect with Conjur. <details> <summary>Click to see how to read the authn container's `CONJURAPPLIANCEURL` setting.</summary> To display the `CONJURAPPLIANCEURL` configured for the application's authenticator container: ```sh-session APP_NAMESPACE="app-test" APPPODLABEL="app=test-app-secretless" AUTHN_CONTAINER="secretless" APPPODNAME="$(kubectl get pod \ -n $APP_NAMESPACE \ -l $APPPODLABEL \ -o jsonpath='{.items[0].metadata.name}')" CONJURAPPLIANCEURL="$(kubectl exec \ -n $APP_NAMESPACE \ $APPPODNAME \ -c $AUTHN_CONTAINER \ -- printenv CONJURAPPLIANCEURL)" echo "CONJURAPPLIANCEURL: $CONJURAPPLIANCEURL" ``` </details> Confirm that the `CONJURAPPLIANCEURL` is correct by curling the Conjur server. <details> <summary>Click to see how to curl the Conjur server using `CONJURAPPLIANCEURL`.</summary> The CONJURAPPLIANCEURL setting can be either: A Kubernetes cluster internal address. This will be of the form: ``` https://<Conjur-service-name>.<Conjur-namespace>.svc.cluster.local ``` An address that is accessible from outside of the Kubernetes cluster. To test an internal (intra-cluster) Conjur address: Create a `pod-curl` pod along with a `pod-curl` command alias as decribed in the [Creating a Pod for Curling From Inside the Cluster](#creating-a-pod-for-curling-from-inside-the-cluster] section above. Curl the Conjur URL using the `pod-curl` alias, e.g.: ```sh-session $ pod-curl -s "$CONJURAPPLIANCEURL" | grep "Conjur server" <p class="font-large">Your Conjur server is running!</p> $ ``` To test an external Conjur address, use `curl` directly: ```sh-session curl -s "$CONJURAPPLIANCEURL" | grep "Conjur server" ``` </details> Check the authenticator container's `CONJURAUTHNURL` setting. The authenticator container uses the URL contained in its `CONJURAUTHNURL` environment when authenticating itself (i.e. logging in) to Conjur. <details> <summary>Click to see how to read the authn container's `CONJURAUTHNURL` setting.</summary> To display the `CONJURAUTHNURL` configured for the application's authenticator container: ```sh-session APP_NAMESPACE="app-test" APPPODLABEL="app=test-app-secretless" AUTHN_CONTAINER="secretless" APPPODNAME="$(kubectl get pod \ -n $APP_NAMESPACE \ -l $APPPODLABEL \ -o jsonpath='{.items[0].metadata.name}')" CONJURAUTHNURL="$(kubectl exec \ -n $APP_NAMESPACE \ $APPPODNAME \ -c $AUTHN_CONTAINER \ -- printenv CONJURAUTHNURL)" echo "CONJURAUTHNURL: $CONJURAUTHNURL" ``` </details> If correctly configured, this should be of the form: ``` https://<CONJURAPPLIANCEURL>/authn-k8s/<AUTHENTICATOR-ID> ``` Where: <CONJURAPPLIANCEURL> should match the value of the `CONJURAPPLIANCEURL` environment variable that was retrieved in the Step 8. <AUTHENTICATOR-ID> should match the authenticator ID that was read from the Conjur authenticators secret in Step 3. Check the authenticator container's `CONJURAUTHNLOGIN` setting. The authenticator container's `CONJURAUTHNLOGIN` environment setting configures the Conjur host that the authenticator container uses to authenticate the application with

<details> <summary>Click to see how to read the authn container's `CONJURAUTHNLOGIN` setting.</summary> To display the `CONJURAUTHNLOGIN` configured for the application's authenticator container: ```sh-session APP_NAMESPACE="app-test" APPPODLABEL="app=test-app-secretless" AUTHN_CONTAINER="secretless" APPPODNAME="$(kubectl get pod \ -n $APP_NAMESPACE \ -l $APPPODLABEL \ -o jsonpath='{.items[0].metadata.name}')" CONJURAUTHNLOGIN="$(kubectl exec \ -n $APP_NAMESPACE \ $APPPODNAME \ -c $AUTHN_CONTAINER \ -- printenv CONJURAUTHNLOGIN)" echo "CONJURAUTHNLOGIN: $CONJURAUTHNLOGIN" ``` </details> If correctly configured, this should be of the form: ``` host/conjur/authn-k8s/<AUTHENTICATOR-ID/<APP-LAYER>/<APPLICATION-NAME ``` Where: <AUTHENTICATOR-ID> should match the authenticator ID that was read from the Conjur authenticators secret in Step 3. <APP-LAYER> is a Conjur layer, which is a group of entities for granting permissions <APPLICATION-NAME> is a string to uniquely identify this application Here is an example: ``` host/conjur/authn-k8s/my-authenticator-id/apps/test-app-secretless ``` Confirm that Conjur policy is configured with the proper Application Identity The plugin that is running on the Conjur server makes use of several forms of to positively identify the application. Typically, the application identities that are used include the following: Application's `Namespace` name Application's `ServiceAccount` name Application's `Deployment` name Application's Authenticator (sidecar or init) container name If all of the above resource names match what has been specified in Conjur policy, then the application is permitted to access secrets as dictated by that policy. <details> <summary>Click to see how to read Application Identity annotations in Conjur Policy.</summary> If you performed the checks in the previous step (Step 10), then you should have a `CONJURAUTHNLOGIN` environment variable that is set to something like this: ``` host/conjur/authn-k8s/<AUTHENTICATOR-ID/<APP-LAYER>/<APPLICATION-NAME ``` The Conjur host resource that corresponds to this `CONJURAUTHNLOGIN` should looks something like this: ``` <CONJUR-ACCOUNT>:host:conjur/authn-k8s/<AUTHENTICATOR-ID/<APP-LAYER>/<APPLICATION-NAME ``` So based on the `CONJURAUTHNLOGIN`, we can display the corresponding Conjur host defined in authentication policy: ```sh-session CONJUR_ACCOUNT="myConjurAccount" HOSTRESOURCE="$CONJURACCOUNT:$(echo $CONJURAUTHNLOGIN | sed 's/\//:/')" conjur show $HOST_RESOURCE ``` </details> The output from the commands in the expandable text block above should show a Conjur host definition that looks similar to the following. <details> <summary>Click to see example Conjur authentication host definition.</summary> ``` $ conjur show $HOST_RESOURCE { "created_at": "2020-11-05T20:43:20.039+00:00", "id": "myConjurAccount:host:conjur/authn-k8s/my-authenticator-id/apps/test-app-secretless", "owner": "myConjurAccount:policy:conjur/authn-k8s/my-authenticator-id/apps", "policy": "myConjurAccount:policy:root", "permissions": [ ], "annotations": [ { "name": "authn-k8s/namespace", "value": "app-test", "policy": "myConjurAccount:policy:root" }, { "name": "authn-k8s/service-account", "value": "test-app-secretless", "policy": "myConjurAccount:policy:root" }, { "name": "authn-k8s/deployment", "value": "test-app-secretless", "policy": "myConjurAccount:policy:root" }, { "name": "authn-k8s/authentication-container-name", "value": "secretless", "policy": "myConjurAccount:policy:root" }, { "name": "kubernetes", "value": "true", "policy": "myConjurAccount:policy:root" } ], "restricted_to": [ ] } ``` </details> Based on the output from the commands in the expandable text blocks above, you should confirm that the `value` fields in the annotations array match the application's: `Namespace` name `ServiceAccount` name `Deployment` name Authentication container name respectively. If any of these annotation values do not match the corresponding Kubernetes resource name, then check the Conjur authentication policy that you loaded for this application, make corrections, and reload the policy. Check that a Conjur Webserver is configured for authenticating the application. <details> <summary>Click to see how to read Webserver configuration in Conjur.</summary> Using the `AUTHENTICATOR-ID` that was read in Step 3, we can find all Conjur Webservers that have been created for this authenticator ID, e.g.: ```sh-session $ conjur list -k webservice -s $AUTHENTICATOR_ID [ "myConjurAccount:webservice:conjur/authn-k8s/my-authenticator-id" ] $ ``` We can now look at permissions that are associated with that Webserver resource. For example: ```sh-session $ conjur show myConjurAccount:webservice:conjur/authn-k8s/my-authenticator-id { "created_at":

"id": "myConjurAccount:webservice:conjur/authn-k8s/my-authenticator-id", "owner": "myConjurAccount:policy:conjur/authn-k8s/my-authenticator-id", "policy": "myConjurAccount:policy:root", "permissions": [ { "privilege": "read", "role": "myConjurAccount:layer:conjur/authn-k8s/my-authenticator-id/users", "policy": "myConjurAccount:policy:root" }, { "privilege": "authenticate", "role": "myConjurAccount:layer:conjur/authn-k8s/my-authenticator-id/users", "policy": "myConjurAccount:policy:root" } ], "annotations": [ { "name": "description", "value": "authn service for cluster", "policy": "myConjurAccount:policy:root" } ] } $ ``` </details> The Webserver resource should have `read` and `authenticate` privileges for users associated with the authenticator ID. If this is not the case, then check the Conjur authentication policy that you loaded for this application, make corrections, and reload the policy. ``` conjur list -k host -s my-authenticator-id conjur show myConjurAccount:host:conjur/authn-k8s/my-authenticator-id/apps/test-app-secretless conjur role members myConjurAccount:layer:conjur/authn-k8s/my-authenticator-id/apps conjur list -k webservice -s my-authenticator-id --inspect ``` Authenticator container logs show a CAKC029E or CAKC029 error: ``` ERROR: 2020/10/28 19:30:30 authenticator.go:133: CAKC029E Received invalid response to certificate signing request. Reason: status code 401, ``` Conjur server logs (with debug logging enabled) show `cannot get resource "pods" in API group` error: ``` [origin=127.0.0.1] [request_id=21acd925-d8e5-4d4d-bbd9-12d54cab1202] [tid=51] Authentication Error: #<Kubeclient::HttpError: HTTP status code 403, pods "test-app-summon-sidecar-7c45f76f6c-krzsb" is forbidden: User "system:serviceaccount:conjur-oss:conjur-oss" cannot get resource "pods" in API group "" in the namespace "app-test" for GET https://10.96.0.1/api/v1/namespaces/app-test/pods/test-app-summon-sidecar-7c45f76f6c-krzsb> ``` Kubernetes RBAC (`ClusterRole`, `ClusterRoleBinding`, `RoleBinding`) is not configured properly for Kubernetes authentication. `ServiceAccount` name that is specified in the `ClusterRoleBinding` or `RoleBinding` for Kubernetes authentication does not match the Kubernetes `ServiceAccount` that the Conjur server pod is using. Do a Helm upgrade to set the `rbac.create` Helm chart value to `true`. Create a `RoleBinding` (preferred) for Kubernetes authentication (see Step 5 above). Modify the `ServiceAccount` name that is specified in the authentication `ClusterRoleBinding` or `RoleBinding` so that it matches the name of the `ServiceAccount` that the Conjur server pod is using. Authenticator container logs show a `CAKC016 Failed to authenticate` error Conjur server logs (with debug logging enabled) show `CONJ00004E 'authn-k8s/XXX' is not enabled` error The Kubernetes authenticator is not enabled. Use Helm upgrade to enable the Conjur Kubernetes authenticator as described in Step 3 above. Authenticator container logs show a CAKC028E error, e.g.: ``` ERROR: 2020/10/28 18:04:14 authenticator.go:128: CAKC028E Failed to send https login request or response. Reason: Post https://not-conjur-oss.conjur-oss.svc.cluster.local/authn-k8s/my-authenticator-id/injectclientcert: dial tcp: lookup not-conjur-oss.conjur-oss.svc.cluster.local: no such host ``` The `CONJURAPPLIANCEURL` environment variable is set incorrectly in the the authentication container's manifest. Correct the `CONJURAPPLIANCEURL` environment variable in the the authentication container's manifest. Authenticator container logs show a CAKC028 error, e.g.: ``` ERROR: 2020/10/28 17:21:35 authenticator.go:128: CAKC028E Failed to send https login request or response. Reason: Post https://another-conjur-oss.conjur-oss.svc.cluster.local/authn-k8s/my-authenticator-id/injectclientcert: x509: certificate is valid for conjur.myorg.com, conjur-oss, conjur-oss.conjur-oss, conjur-oss.conjur-oss.svc, conjur-oss.conjur-oss.svc.cluster.local, not another-conjur-oss.conjur-oss.svc.cluster.local ``` The URL specified in the `CONJURAUTHNURL` environment variable for the authentication container is correctly resolved to an IP, so that the container can connect with Conjur; however, the URL doesn't match any of the subject alternative names in the Conjur's SSL certificate. This can happen, for example, if you are directly using an IP to connect with Conjur. Modify the `CONJURAUTHNURL` environment variable for the authentication container so that it matches the Subject alternative names that are specified in the Conjur servers SSL certificate. Register the corresponding domain name with DNS if not done already. Authentication container logs show `CAKC029E Received invalid response to certificate signing request` error. Conjur server logs (with debug logging enabled) show `service_account` or `deployment` is `not found in namespace`. For example: ``` [origin=127.0.0.1] [requestid=bdca411b-c43b-4518-a1fc-5066dc277758] [tid=39] Authentication Error: #<Errors::Authentication::AuthnK8s::K8sResourceNotFound: CONJ00026E Kubernetes serviceaccount test-app-secretless not found in namespace app-test> ``` The Conjur policy that was used to configure application identity does not use the correct values for the application's `ServiceAccount` name or `Deployment` name. (See Step 7 above for how to check this). The Conjur policy should be corrected so that it contains the proper