Repository: kubernetes/kops

Issue #5706: kubelet-api is Forbidden to logs pods
Created: 2018-08-27T09:51:34Z
1. What kops version are you running? The command kops version, will display
this information.
Version 1.10.0
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:17:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T09:56:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
3. What cloud provider are you using?
aws
4. What commands did you run?  What is the simplest way to reproduce this issue?
kops update cluster
5. What happened after the commands executed?
After enable https://github.com/kubernetes/kops/blob/master/docs/cluster_spec.md#bootstrap-tokens and https://github.com/kubernetes/kops/blob/master/docs/node_authorization.md
we got
Error from server (Forbidden): Forbidden (user=kubelet-api, verb=get, resource=nodes, subresource=proxy) ( pods/log etcd-server-events-xxxxxx)


apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  name: test
spec:
  api:
    loadBalancer:
      type: Internal
  channel: stable
  authorization:
    rbac: {}
  cloudLabels:
    env: infra
  cloudProvider: aws
  configBase: s3://bucket/test
  dnsZone: dreamteam.internal
  docker:
    storage: overlay2
    storageOpts:
    - overlay2.override_kernel_check=true
    version: 17.03.2
    liveRestore: true
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
      encryptedVolume: true
    - instanceGroup: master-us-east-1b
      name: b
      encryptedVolume: true
    - instanceGroup: master-us-east-1c
      name: c
      encryptedVolume: true
    name: main
    version: 3.2.18
    enableEtcdTLS: true
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
      encryptedVolume: true
    - instanceGroup: master-us-east-1b
      name: b
      encryptedVolume: true
    - instanceGroup: master-us-east-1c
      name: c
      encryptedVolume: true
    name: events
    version: 3.2.18
    enableEtcdTLS: true
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    authorizationMode: Node,RBAC
    authorizationRbacSuperUser: admin
    enableBootstrapTokenAuth: true
    runtimeConfig:
      rbac.authorization.k8s.io/v1: "true"
      authentication.k8s.io/v1beta1: "true"
  kubelet:
    anonymousAuth: false
    authorizationMode: Webhook
    authenticationTokenWebhook: true
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.10.7
  masterInternalName: api.internal.test
  masterPublicName: api.test
  networkCIDR: xxxxxxxxxxxxxx
  networkID: xxxxxxxxxxxxxx
  networking:
     cilium: {}
  kubeDNS:
    provider: CoreDNS
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: us-east-1a
    type: Private
    zone: us-east-1a
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: us-east-1b
    type: Private
    zone: us-east-1b
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: us-east-1c
    type: Private
    zone: us-east-1c
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: utility-us-east-1a
    type: Utility
    zone: us-east-1a
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: utility-us-east-1b
    type: Utility
    zone: us-east-1b
  - cidr: xxxxxxxxxxxxxx
    id: xxxxxxxxxxxxxx
    name: utility-us-east-1c
    type: Utility
    zone: us-east-1c
  topology:
    dns:
      type: Private
    masters: private
    nodes: private
  nodeAuthorization:
     nodeAuthorizer: {}

Duplicate issue #6280: Unable to access pod logs when using Webhook authorizationMode for Kubelet in cluster with RBAC
Created: 2018-12-31T05:28:55Z
1. What kops version are you running?
Version 1.10.0
2. What Kubernetes version are you running?
Both client (kubectl) and server are running v1.11.6
3. What cloud provider are you using?
AWS
4. What commands did you run?  What is the simplest way to reproduce this issue?
We are trying to disable anonymous auth on our Kubelet's in a cluster with RBAC authorization and have added the following to our cluster spec:
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook

We updated our the cluster (kops update cluster --yes), but did not roll the update out the entire cluster due to the risk of breakage. We manually terminated a node and the auto-scaling group brought up a new node with anonymous auth disabled.
5. What happened after the commands executed?
We are now unable to access logs on those pods, even after creating the ClusterRoleBindings outlined here:
#5176 (comment)
When running kubectl logs <pod_name> we get the following error:
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <pod_name> )`

6. What did you expect to happen?
We expect to be able to access logs on the new node without any issues.
7. Please provide your cluster manifest.
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-03-17T16:47:08Z
  name: <REDACTED>
spec:
  api:
    loadBalancer:
      additionalSecurityGroups:
      - <REDACTED>
      idleTimeoutSeconds: 600
      type: Internal
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: <REDACTED>
  etcdClusters:
  - etcdMembers:
    - encryptedVolume: false
      instanceGroup: master-<REDACTED>
      name: a
    name: main
  - etcdMembers:
    - encryptedVolume: false
      instanceGroup: master-<REDACTED>
      name: a
    name: events
  iam:
    legacy: true
  kubeAPIServer:
    admissionControl:
    - AlwaysPullImages
    - DenyEscalatingExec
    runtimeConfig:
      batch/v2alpha1: "true"

  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook

  kubernetesApiAccess:
  - <REDACTED>
  kubernetesVersion: 1.11.6
  masterInternalName: <REDACTED>
  masterPublicName: <REDACTED>
  networkCIDR: <REDACTED>
  networking:
    weave: {}
  nonMasqueradeCIDR: <REDACTED>
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: <REDACTED>
    name: <REDACTED>
    type: Private
    zone: <REDACTED>
  - cidr: <REDACTED>
    name: <REDACTED>
    type: Utility
    zone: <REDACTED>
  topology:
    bastion:
      bastionPublicName: <REDACTED>
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-03-17T16:47:08Z
  labels:
    kops.k8s.io/cluster: <REDACTED>
  name: <REDACTED>
spec:
  additionalSecurityGroups:
  - <REDACTED>
  image: <REDACTED>
  machineType: m5.large
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - <REDACTED>

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-03-17T16:47:08Z
  labels:
    kops.k8s.io/cluster: <REDACTED>
  name: nodes
spec:
  additionalSecurityGroups:
  - <REDACTED>
  image: <REDACTED>
  machineType: m5.large
  maxSize: <REDACTED>
  minSize: <REDACTED>
  role: Node
  subnets:
  - <REDACTED>

8. Please run the commands with most verbose logging by adding the -v 10 flag.
$ kubectl -v 10 logs <pod_name>
I1230 23:14:31.672355    9524 round_trippers.go:405] GET https://<REDACTED>/api/v1/namespaces/default/pods/<pod_name>/log 401 Unauthorized in 108 milliseconds
I1230 23:14:31.672384    9524 round_trippers.go:411] Response Headers:
I1230 23:14:31.672390    9524 round_trippers.go:414]     Date: Mon, 31 Dec 2018 05:14:31 GMT
I1230 23:14:31.672396    9524 round_trippers.go:414]     Content-Type: application/json
I1230 23:14:31.672402    9524 round_trippers.go:414]     Content-Length: 286
I1230 23:14:31.672441    9524 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server has asked for the client to provide credentials ( pods/log <pod_name>)","reason":"Unauthorized","details":{"name":"<pod_name>","kind":"pods/log"},"code":401}
I1230 23:14:31.673605    9524 helpers.go:198] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials ( pods/log <pod_name>)",
  "reason": "Unauthorized",
  "details": {
    "name": "<pod_name>",
    "kind": "pods/log"
  },
  "code": 401
}]
F1230 23:14:31.673646    9524 helpers.go:116] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <pod_name>))

9. Anything else do we need to know?
The master node is not running with the updated kubelet configuration.
It seems like running RBAC with Webhook Kubelet auth is still under development according to #5176. I tried creating the ClusterRoleBindings described in #5176 (comment), but it did not fix the issue.
This older issue #3891 mentions specifying things relating to certificate files. Is there something we have to do there?

Closing comment (2019-03-19T08:59:59Z): I think this is a straight duplicate of #5706
Should we close this one?