Project: Pinniped (Provisioning / Key Management)
File: concierge-and-supervisor-demo.md

title: Learn to use Pinniped for federated authentication to Kubernetes clusters description: See how the Pinniped Supervisor streamlines login to multiple Kubernetes clusters. cascade: layout: docs menu: docs: name: Concierge with Supervisor parent: tutorials weight: 1 There are many benefits to using the Pinniped Supervisor, Concierge, and CLI components together to provide Kubernetes authentication. It's easy to bring your own OIDC, LDAP, or Active Directory identity provider to act as the source of user identities. A user's identity in the external identity provider becomes their identity in Kubernetes. All other aspects of Kubernetes that are sensitive to identity, such as authorization policies and audit logging, are then based on the user identities from your identity provider. You can bring identities from your own identity provider into many types of Kubernetes clusters in a consistent way. This includes clusters from various vendors run on-prem, and clusters provided as a cloud service by various popular cloud companies. Kubeconfig files will not contain any specific user identity or credentials, so they can be safely shared. Deep integration with `kubectl` means that when a user runs `kubectl` commands, they will be interactively prompted to log in using their own unique identity from your identity provider. Users will be prompted by `kubectl` to interactively authenticate only once per day, and then will be able to use multiple clusters for the rest of the day without being asked to authenticate again. All credentials are short-lived, and refreshed often. Additionally, frequent checks are made against your identity provider to ensure that the user should continue to have access to the Kubernetes clusters. For example, within minutes of locking an Active Directory account, that user will lose access to Kubernetes clusters, even if they were already logged in. A user can safely be granted high levels of authorization on a cluster, if needed. Even if they abuse their privilege by capturing the credentials sent by other users to the cluster, they will not be able to use the captured credentials to access other clusters, because all credentials sent to clusters are uniquely scoped to each individual cluster. Pinniped will not interfere with a cluster's original vendor-specific authentication system. The original admin-level kubeconfig from a cluster can be privately kept by the cluster's creator for bootstrapping and break-glass access purposes. Pinniped is open source and will never be tied to any one vendor's authentication system. As Pinniped improves in the future, all your Kubernetes clusters can benefit, regardless of which vendor provided the clusters. The code is available on GitHub for any expert to audit, and for any community member to contribute. This tutorial will show: A detailed example of how to install and configure a Supervisor with ingress, DNS, TLS, and an external identity provider How to install the Concierge onto multiple workload clusters and configure them all to trust identities from the Supervisor How an admin can create and distribute kubeconfig files for the workload clusters How a developer or devops user can authenticate with kubectl using their identity from the external identity provider, and how they can securely access all workload clusters for the rest of the day without needing to authenticate again This tutorial is intended to be a step-by-step example of installing and configuring the Pinniped components to provide a multi-cluster federated authentication

It will show every command needed to replicate the same setup to allow the reader to follow the same steps themselves. A single Pinniped Supervisor can provide authentication for any number of Kubernetes clusters. In a typical deployment: A single Supervisor is deployed on a special cluster where app developers and devops users have no access. App developers and devops users should have no access at least to the resources in the Supervisor's namespace, but usually have no access to the whole cluster. For this tutorial, let's call this cluster the "supervisor cluster". App developers and devops users can then use their identities provided by the Supervisor to log in to many clusters where they can manage their apps. For this tutorial, let's call these clusters the "workload clusters". The Pinniped Concierge component is installed into each workload cluster and is configured to trust the single Supervisor. The Concierge acts as an in-cluster agent to provide authentication services. There are many ways to install and configure Pinniped. To make the steps of this tutorial as specific as possible, we had to make some choices. The choices made for this tutorial were: The Pinniped Supervisor can draw user identities from OIDC identity providers, Active Directory providers (via LDAP), and generic LDAP providers. In this tutorial we will use Okta as an OIDC identity provider. Okta offers a free developer account, so any reader should be able to sign up for an Okta account if they would like to try these steps themselves. The Pinniped Supervisor can be installed on any type of Kubernetes cluster. In this tutorial we will demonstrate the installation process for GKE because any reader should be able to sign up for a Google Cloud account if they would like to try these steps themselves. We will use separate supervisor and workload clusters. The Pinniped Supervisor needs working ingress. There are many ways to configure ingress for apps running on Kubernetes clusters, as described in the . For this tutorial we will use a LoadBalancer Service with a public IP address. This is a simple setup which allows us to terminate TLS inside the Supervisor app, keeping the connection secure all the way into the Supervisor app's pods. A corporate installation of the Supervisor might keep it behind the corporate firewall, but for this tutorial a public IP also allows your desktop (and anyone on the internet) to access the Supervisor's endpoints. The HTTPS endpoints of a properly configured Supervisor are generally safe to expose publicly, as long as you are not concerned with denial of service attacks (or have some external protection against such attacks). Although it is possible to configure the Supervisor's FederationDomain to use an IP address, it is better to use a DNS name. There are many ways to manage DNS. For this tutorial, we will use Google Cloud's service to register a new hostname for the Supervisor app's load balancer's public IP address. We won't describe how to prepare Cloud DNS to manage DNS for the parent domain in this tutorial. This typically involves setting up Cloud DNS's servers as the list of DNS servers for your domain within your domain registrar. We'll assume that this has already been

For web-based login flows as used by OIDC identity providers, the Pinniped Supervisor needs TLS certificates that are trusted by the end users' web browsers. There are many ways to create TLS certificates. There are also several ways to configure the TLS certificates on the Supervisor, as described in the . For this tutorial we will use with , because any reader could use these services if they would like to try these steps themselves. The Pinniped Concierge can be installed in many types of Kubernetes clusters, as described in . In this tutorial we will use GKE clusters as our workload clusters, for the same reasons that we are using GKE for the supervisor cluster. It is worth noting that a Supervisor running on GKE can provide authentication for workload clusters of any supported Kubernetes type, not only for GKE workload clusters. GKE and Google Cloud DNS can be managed in the Google Cloud Console web UI, or via the gcloud CLI. For this tutorial, we will use the so we can be as specific as possible. However, the same steps could be performed via the UI instead. This tutorial assumes that you have already authenticated with the gcloud CLI as a user who has permission to run all the gcloud commands used below. Pinniped provides authentication, not authorization. Inside Kubernetes, a user authenticated via Pinniped will have a username and may also have a list of group names. These usernames and group names can be used to create authorization policies using any Kubernetes authorization system, usually using . The details of the steps shown in this tutorial would be different if any of the above choices were made differently, however the general concepts at each step would still apply. If you have not already done so, . On macOS or Linux, you can do this using Homebrew: ```sh brew install vmware-tanzu/pinniped/pinniped-cli ``` On other platforms, see the for more details. For the rest of this tutorial, let's assume that your Google Cloud project name and your preferred Google Cloud zone name are set as environment variables. ```sh PROJECT="my-gcp-project-name" ZONE="us-central1-c" ``` Let's create one supervisor cluster and two workload clusters. There are many options to consider here, but for this tutorial we will use only the most basic options. ```sh gcloud container clusters create "demo-supervisor-cluster" \ --project "$PROJECT" --zone "$ZONE" gcloud container clusters create "demo-workload-cluster1" \ --project "$PROJECT" --zone "$ZONE" gcloud container clusters create "demo-workload-cluster2" \ --project "$PROJECT" --zone "$ZONE" ``` Most of the following installation and configuration steps are performed using the cluster's admin kubeconfig. Let's download those kubeconfig files now. ```sh KUBECONFIG="supervisor-admin.yaml" gcloud container clusters get-credentials \ "demo-supervisor-cluster" --project "$PROJECT" --zone "$ZONE" KUBECONFIG="workload1-admin.yaml" gcloud container clusters get-credentials \ "demo-workload-cluster1" --project "$PROJECT" --zone "$ZONE" KUBECONFIG="workload2-admin.yaml" gcloud container clusters get-credentials \ "demo-workload-cluster2" --project "$PROJECT" --zone "$ZONE" ``` The Pinniped maintainers own the pinniped.dev domain and have already set it up for use with Google Cloud DNS, so for this tutorial we will call our Supervisor server `demo-supervisor.pinniped.dev`. There are several installation options described in the . For this tutorial, we will install the latest version using the `kubectl` CLI. ```sh kubectl apply \ -f https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-supervisor.yaml \ --kubeconfig supervisor-admin.yaml ``` There are several options for exposing the Supervisor's endpoints outside the cluster, which are described in the . For this tutorial, we will use a public

Create a LoadBalancer to expose the Supervisor's endpoints to the public, being careful to only expose the HTTPS endpoint (not the HTTP endpoint). ```sh cat <<EOF | kubectl create --kubeconfig supervisor-admin.yaml -f - apiVersion: v1 kind: Service metadata: name: pinniped-supervisor-loadbalancer namespace: pinniped-supervisor spec: type: LoadBalancer selector: app: pinniped-supervisor ports: protocol: TCP port: 443 targetPort: 8443 # 8443 is the TLS port. EOF ``` Check for an IP using the following command. The value returned is the public IP of you LoadBalancer, which will be used in the steps below. It may take a little time for the LoadBalancer to be assigned a public IP, and this command will have empty output until then. ```sh kubectl get service pinniped-supervisor-loadbalancer \ -o jsonpath='{.status.loadBalancer.ingress[*].ip}' \ --namespace pinniped-supervisor --kubeconfig supervisor-admin.yaml ``` Install cert-manager. ```sh kubectl apply \ -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml \ --kubeconfig supervisor-admin.yaml ``` Create a GCP service account for cert manager to be able to manage to Google Cloud DNS. cert-manager will need this as part of its process to prove to Let's Encrypt that we own the domain. ```sh gcloud iam service-accounts create demo-dns-solver \ --display-name "demo-dns-solver" --project "$PROJECT" gcloud projects add-iam-policy-binding "$PROJECT" \ --member "serviceAccount:demo-dns-solver@$PROJECT.iam.gserviceaccount.com" \ --role roles/dns.admin --condition=None ``` Create and download a key for the new service account, and then put it into a Secret on the cluster. Be careful with this key as it allows full control over the DNS of your Cloud DNS zones. ```sh gcloud iam service-accounts keys create demo-dns-solver-key.json \ --iam-account "demo-dns-solver@$PROJECT.iam.gserviceaccount.com" \ --project "$PROJECT" kubectl create secret generic demo-dns-solver-svc-acct \ --namespace pinniped-supervisor --from-file=demo-dns-solver-key.json \ --kubeconfig supervisor-admin.yaml ``` Configure cert-manager to use Let's Encrypt. ```sh MY_EMAIL="someone@example.com" cat <<EOF | kubectl create --kubeconfig supervisor-admin.yaml -f - apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: demo-issuer namespace: pinniped-supervisor spec: acme: email: "$MY_EMAIL" server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: demo-issuer-account-key solvers: dns01: cloudDNS: project: "$PROJECT" serviceAccountSecretRef: name: demo-dns-solver-svc-acct key: demo-dns-solver-key.json EOF ``` Create a record in Cloud DNS for the public IP of the LoadBalancer created above. Of course, you would replace these sample argument values with your actual public IP address, DNS zone name, and domain. ```sh PUBLIC_IP="1.2.3.4" DNS_ZONE="pinniped-dev" DNS_NAME="demo-supervisor.pinniped.dev" gcloud dns record-sets transaction start \ --zone="$DNS_ZONE" --project "$PROJECT" gcloud dns record-sets transaction add "$PUBLIC_IP" \ --name="$DNS_NAME." --ttl="300" --type="A" \ --zone="$DNS_ZONE" --project "$PROJECT" gcloud dns record-sets transaction execute \ --zone="$DNS_ZONE" --project "$PROJECT" ``` This will take a few moments to move from status "pending" to status "done". Using the change ID that was output from the previous command (e.g. "87"), you can check the status with this command. ```sh gcloud dns record-sets changes describe "87" \ --zone "$DNS_ZONE" --project "$PROJECT" ``` ```sh cat <<EOF | kubectl create --kubeconfig supervisor-admin.yaml -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: supervisor-tls-cert-request namespace: pinniped-supervisor spec: secretName: supervisor-tls-cert issuerRef: name: demo-issuer dnsNames: "$DNS_NAME" EOF ``` Wait for the Secret to get created. This may take a few minutes. Use the following command to see if it exists. ```sh kubectl get secret supervisor-tls-cert \ --namespace pinniped-supervisor --kubeconfig supervisor-admin.yaml ``` The Supervisor should be configured to have a , which, under the hood: Acts as an OIDC provider to the Pinniped CLI, creating a consistent interface for the CLI to use regardless of which protocol the Supervisor is using to talk to the external identity provider Also acts as an OIDC provider to the workload cluster's Concierge component, which will receive JWT tokens from the CLI and cryptographically validate that they were issued by the Supervisor Create the

```sh cat <<EOF | kubectl create --kubeconfig supervisor-admin.yaml -f - apiVersion: config.supervisor.pinniped.dev/v1alpha1 kind: FederationDomain metadata: name: demo-federation-domain namespace: pinniped-supervisor spec: issuer: "https://$DNS_NAME/demo-issuer" tls: secretName: supervisor-tls-cert EOF ``` Check that the DNS, certificate, and FederationDomain are all working together by trying to fetch one of its endpoints. If it works it should return a nice json-formatted discovery response. Note that it may take a little time for the new DNS entry created above to propagate to your machine. ```sh curl "https://${DNS_NAME}/demo-issuer/.well-known/openid-configuration" ``` In this tutorial we are using Okta as an OIDC identity provider. Refer to the for examples of using other identity providers. The Pinniped Supervisor app will be a client of Okta. The general steps required to create and configure a client in Okta are: Sign up for Okta if you don't already have an account. They offer a free developer account. Login to the admin UI of your account. Create a test user with an email and a password. It does not need to be a real email address for the purposes of this tutorial. Create an app in the Okta UI. For more information about creating an app in the Okta UI, see the . Make sure that the test user is assigned to the app in the app's "Assignments" tab. Add the FederationDomain's callback endpoint to the "Sign-in redirect URIs" list on the app in the UI. The callback endpoint is the FederationDomain's issuer URL plus `/callback`, e.g. `https://demo-supervisor.pinniped.dev/demo-issuer/callback`. Get the app's "Okta Domain", "Client ID", and "Client secret" from the UI for use in the next step. Create an and a Secret. ```sh cat <<EOF | kubectl create --kubeconfig supervisor-admin.yaml -f - apiVersion: idp.supervisor.pinniped.dev/v1alpha1 kind: OIDCIdentityProvider metadata: namespace: pinniped-supervisor name: okta spec: issuer: https://dev-123456.okta.com/oauth2/default authorizationConfig: additionalScopes: [groups, email, offline_access] claims: username: email groups: groups client: secretName: okta-client-credentials apiVersion: v1 kind: Secret metadata: namespace: pinniped-supervisor name: okta-client-credentials type: secrets.pinniped.dev/oidc-client stringData: clientID: "0oa45dekegIzOlvB17x9" clientSecret: "<redacted>" EOF ``` To check that the connection to Okta is working, look at the status conditions and status phase of the resource. It should be in phase "Ready". ```sh kubectl get OIDCIdentityProvider okta \ --namespace pinniped-supervisor --kubeconfig supervisor-admin.yaml -o yaml ``` There are several installation options described in the . For this tutorial, we will install the latest version using the `kubectl` CLI. ```sh kubectl apply -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-crds.yaml" \ --kubeconfig workload1-admin.yaml kubectl apply -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-resources.yaml" \ --kubeconfig workload1-admin.yaml kubectl apply -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-crds.yaml" \ --kubeconfig workload2-admin.yaml kubectl apply -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-resources.yaml" \ --kubeconfig workload2-admin.yaml ``` Configure the Concierge on the first workload cluster to trust the Supervisor's FederationDomain for authentication by creating a . ```sh cat <<EOF | kubectl create --kubeconfig workload1-admin.yaml -f - apiVersion: authentication.concierge.pinniped.dev/v1alpha1 kind: JWTAuthenticator metadata: name: demo-supervisor-jwt-authenticator spec: issuer: "https://$DNS_NAME/demo-issuer" audience: workload1-ed9de33c370981f61e9c EOF ``` Apply a similar configuration in the other workload cluster with a different `audience` value. ```sh cat <<EOF | kubectl create --kubeconfig workload2-admin.yaml -f - apiVersion: authentication.concierge.pinniped.dev/v1alpha1 kind: JWTAuthenticator metadata: name: demo-supervisor-jwt-authenticator spec: issuer: "https://$DNS_NAME/demo-issuer" audience: workload2-86af71b821afe8d9caf4 EOF ``` For this tutorial, we will keep the Kubernetes RBAC configuration simple. We'll use a contrived example of RBAC policies to avoid getting into RBAC policy design discussions. If one of your Okta users has the email address

then you could allow that user to things in a new namespace in one workload cluster, and most things in the other workload cluster, with the following commands. ```sh kubectl create namespace "dev" \ --kubeconfig workload1-admin.yaml cat <<EOF | kubectl create --kubeconfig workload1-admin.yaml -f - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: developer-can-edit-dev-ns namespace: dev subjects: kind: User name: walrus@example.com apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: edit apiGroup: rbac.authorization.k8s.io EOF kubectl create clusterrolebinding developer-can-view \ --clusterrole view \ --user walrus@example.com \ --kubeconfig workload2-admin.yaml ``` RBAC rules can be defined for your users using their usernames and/or their group memberships. As the cluster admin, create kubeconfig files for the workload clusters that can be used by the developer and devops users. These commands should be run using the admin kubeconfigs of the workload clusters, and they will output the new Pinniped-compatible kubeconfigs for the workload clusters. The `--kubeconfig` and `--kubeconfig-context` options, along with the `KUBECONFIG` environment variable, can help you specify how the command should find the admin kubeconfig for the cluster. The new Pinniped-compatible kubeconfig will be printed to stdout, so in these examples we will redirect that to a file. ```sh pinniped get kubeconfig \ --kubeconfig workload1-admin.yaml > workload1-developer.yaml pinniped get kubeconfig \ --kubeconfig workload2-admin.yaml > workload2-developer.yaml ``` These new kubeconfig files may be distributed to the app developers and devops users who will be using these workload clusters. They do not contain any particular identity or credential. As the cluster creator, do not share the admin kubeconfig files with your workload cluster users. Save the admin kubeconfig files somewhere private and secure for your own future use. See the for other available optional parameters. The `kubectl` CLI . If you wanted to distribute one kubeconfig file instead of one per cluster, you could choose to merge the Pinniped-compatible kubeconfig files. ```sh KUBECONFIG="workload1-developer.yaml:workload2-developer.yaml" kubectl \ config view --flatten -o yaml > all-workload-clusters-developer.yaml ``` The developer who uses the combined kubeconfig file will need to use the standard `kubectl` methods to choose their current context. For clarity, the steps shown below will continue to use the separate kubeconfig files. A developer or devops user who would like to use the workload clusters may do so using kubectl with the kubeconfig files provided to them by the cluster admin in the previous step. The kubeconfig files tell kubectl how to invoke the Pinniped CLI as a plugin to aid in authentication. First, the user will need to install the Pinniped CLI at the same full path where it is referenced inside the kubeconfig file. Or, they can adjust the full path to the Pinniped CLI inside their own copy of the kubeconfig file, to make it match where they have locally installed the Pinniped CLI. Then the developer can run any kubectl command using a kubeconfig file that was provided to them by the cluster admin. For example, let's run a command against the first workload cluster. ```sh kubectl get namespaces --kubeconfig workload1-developer.yaml ``` The first time this command is run, it will open their default web browser and redirect them to Okta for login. After successfully logging in to Okta, for example as the user `walrus@example.com`, the kubectl command will continue and will try to list the namespaces. The user's identity in Kubernetes (username and group memberships) came from Okta, through Pinniped. Oops! This results in an RBAC error similar to `Error from server (Forbidden): namespaces is forbidden: User

cannot list resource "namespaces" in API group "" at the cluster scope`. Recall that in the first workload cluster, the user only has RBAC permissions in the `dev` namespace. Let's try again, but this time we will list something in the `dev` namespace. ```sh kubectl get serviceaccounts --namespace dev \ --kubeconfig workload1-developer.yaml ``` This will successfully list the default service account in the `dev` namespace. That same developer user can access all other workload clusters in a similar fashion. For example, let's run a command against the second workload cluster. Recall that the developer is allowed to read everthing in the second workload cluster. ```sh kubectl get namespaces --kubeconfig workload2-developer.yaml ``` This time, the command will list namespaces immediately. Even though you are accessing a different cluster, the web browser will not open again. You do not need to interactively sign in again for the rest of the day to access any workload cluster within the same FederationDomain. Behind the scenes, Pinniped is performing token refreshes and token exchanges on behalf of the user to create a short-lived, cluster-scoped token to access this new workload cluster using the same identity from Okta. Note that users can use any of kubectl's supported means of providing kubeconfig information to kubectl. They are not limited to only using the `--kubeconfig` flag. For example, they could set the `KUBECONFIG` environment variable instead. For more information about logging in to workload clusters, see the . Not sure what identity you're using on the cluster? Pinniped has a convenient feature to help out with that. ```sh pinniped whoami --kubeconfig workload2-developer.yaml ``` The output will include your username and group names, and will look similar to the following output. ``` Current cluster info: Name: gkeyourprojectus-central1-cdemo-workload-cluster2-pinniped URL: https://1.2.3.4 Current user info: Username: walrus@example.com Groups: Everyone, developers, system:authenticated ``` This tutorial showed: A detailed example of how to install and configure a Supervisor with ingress, DNS, TLS, and an external identity provider How to install the Concierge onto multiple workload clusters and configure them all to trust identities from the Supervisor How an admin can create and distribute kubeconfig files for the workload clusters How a developer or devops user can authenticate with kubectl using their identity from the external identity provider, and how they can securely access all workload clusters for the rest of the day without needing to authenticate again If you would like to delete the resources created in this tutorial, you can use the following commands. ```sh kubectl delete \ -f "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-supervisor.yaml" \ --kubeconfig supervisor-admin.yaml \ --ignore-not-found kubectl delete -f \ "https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml" \ --kubeconfig supervisor-admin.yaml kubectl delete -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-resources.yaml" \ --kubeconfig workload1-admin.yaml kubectl delete -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-crds.yaml" \ --kubeconfig workload1-admin.yaml kubectl delete -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-resources.yaml" \ --kubeconfig workload2-admin.yaml kubectl delete -f \ "https://get.pinniped.dev/{{< latestversion >}}/install-pinniped-concierge-crds.yaml" \ --kubeconfig workload2-admin.yaml gcloud container clusters delete "demo-supervisor-cluster" \ --project "$PROJECT" --zone "$ZONE" --quiet gcloud container clusters delete "demo-workload-cluster1" \ --project "$PROJECT" --zone "$ZONE" --quiet gcloud container clusters delete "demo-workload-cluster2" \ --project "$PROJECT" --zone "$ZONE" --quiet gcloud dns record-sets transaction start \ --zone="$DNS_ZONE" --project "$PROJECT" gcloud dns record-sets transaction remove "$PUBLIC_IP" \ --name="$DNS_NAME." --ttl="300" --type="A" \ --zone="$DNS_ZONE" --project "$PROJECT" gcloud dns record-sets transaction execute \ --zone="$DNS_ZONE" --project "$PROJECT" gcloud projects remove-iam-policy-binding "$PROJECT" \ --member "serviceAccount:demo-dns-solver@$PROJECT.iam.gserviceaccount.com" \ --role roles/dns.admin --condition=None gcloud iam service-accounts delete \ "demo-dns-solver@$PROJECT.iam.gserviceaccount.com" \ --project "$PROJECT" --quiet ```