Change:
} } <ins> if rawResponseCode != http.StatusSwitchingProtocols { // If the backend did not upgrade the request, finish echoing the response from the backend to the client and return, closing the connection. klog.V(6).Infof("Proxy upgrade error, status code %d", rawResponseCode) _, err := io.Copy(requestHijackedConn, backendConn) if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { klog.Errorf("Error proxying data from backend to client: %v", err) } // Indicate we handled the request return true } </ins> // Proxy the connection. This is bidirectional, so we need a goroutine // to copy in each direction. Once one side of the connection exits, we // exit the function which performs cleanup and in the process closes

Discussion:
(9.8, critical) With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection. Thanks to Darren Shepherd for reporting this problem. CVE-2018- is fixed in the following Kubernetes releases: Kubernetes API server Kubernetes v1.0.x-1.9.x Kubernetes v1.10.0-1.10.10 (fixed in ) Kubernetes v1.11.0-1.11.4 (fixed in ) Kubernetes v1.12.0-1.12.2 (fixed in ) Note: If you are using binaries or packages provided by a distributor (not the ones provided in the open source release artifacts), you should contact them to determine what versions resolve this CVE. Distributors may choose to provide support for older releases beyond the ones maintained by the open source project. Clusters = 1.6.x that run aggregated API servers (like the metrics server) that are directly accessible from the Kubernetes API server’s network. If there are aggregated API servers configured in a cluster, the following command will return the names of the associated APIService objects (if no names are listed, or the kube-apiserver is an older version that does not have the apiservices API, then the cluster has no aggregated API servers configured): Clusters = 1.0.x that grant pod exec/attach/portforward permissions to users that are not expected to have full access to kubelet APIs An API call to any aggregated API server endpoint can be escalated to perform any API request against that aggregated API server, as long as that aggregated API server is directly accessible from the Kubernetes API server’s network. allows all users (authenticated and unauthenticated) to perform discovery API calls that allow this escalation against any aggregated API servers configured in the cluster. A pod exec/attach/portforward API call can be escalated to perform any API request against the kubelet API on the node specified in the pod spec (e.g. listing all pods on the node, running arbitrary commands inside those pods, and obtaining the command output). Pod exec/attach/portforward permissions are included in the admin/edit RBAC roles intended for namespace-constrained users. This section lists possible mitigations to use prior to upgrading.

Note that many of the mitigations are likely to be disruptive, and upgrading to a fixed version is strongly recommended. suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) disable anonymous requests by passing to the kube-apiserver (note that this may disrupt load balancer or kubelet health checks of the kube-apiserver, and breaks setup flows) remove all anonymous access to all aggregated APIs (including discovery permissions granted by the ) suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) remove all access to all aggregated APIs (including discovery permissions granted by the ) from users that should not have full access to the aggregated APIs (note that this may disrupt users and controllers that make use of discovery information to map API types to URLs) Remove pod exec/attach/portforward permissions from users that should not have full access to the kubelet API There is no simple way to detect whether this vulnerability has been used. Because the unauthorized requests are made over an established connection, they do not appear in the Kubernetes API server audit logs or server log. The requests do appear in the kubelet or aggregated API server logs, but are indistinguishable from correctly authorized and proxied requests via the Kubernetes API server.
, why doesn't kubelet's webhook authorization help here? Shouldn't it reject subsequent requests on still open TCP connection from APIServer?
Because the kube-apiserver <-kubelet connection was established with the kube-apiserver's TLS credentials, which are broadly authorized against the kubelet API. The kubelet would authorize the kube-apiserver to make that request, and allow it.
Hmm, I misunderstood how webhook authorization works then, I thought that no requests are trusted in that mode and even for /proxy requests, initiator's credentials are passed by APIServer to kubelet, which then comes back to apiserver for SubjectAccessReview. So I thought scheme is like following: Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) But seems it is not the case :(
the credentials authorized by the kubelet in that case are the apiserver's
Thanks for the fix and the detailed summary! Just for clarification - this vulnerability can't be exploited by anonymous users from outside of the cluster, as long as we disable , right?

I'm asking because my users tend to open up K8S API endpoint to the Internet, occasionally but not always restricting accesses to the endpoint at L4 by the source IP.
Correct, escalations start with a single authorized request through kube-apiserver to the backend. Preventing that (either by disallowing anonymous requests with --anonymous-auth=false or by removing the default discovery permissions granted to anonymous users) mitigate the anonymous escalation. Be aware that can disrupt normal operations of a cluster, including load balancer and kubelet detection of kube-apiserver health (via anonymous requests to /healthz), and kubeadm join flows that depend on public access to a config map containing cluster information in the namespace.
Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) Interesting, are there any drawbacks of passing initiator's credentials all the way to kubelet? Would it allow to eliminate one more "superuser" in the form of APIServer?
Exposing initiator's credentials broadly to the cluster is a non-goal. Additionally, many API callers have non-forwardable credentials (TLS client certificates). The apiserver could forward information about the initiator to the kubelet, but that information would have to be trusted (e.g. initiator info in headers, verified by mTLS connection proving the apiserver provided the info). Interestingly, that is exactly the scenario exercised by kube-apiserver -aggregated API server connections, where the kube-apiserver acts as an authenticating proxy, and the aggregated API server trusts the identity assertions sent to it by the kube-apiserver. not really, as long as kube-apiserver is responsible for terminating and asserting authentication information to backends.
- I'm having a hard time understanding the full extent of this. If the API is in a private network (no public exposure), with RBAC enabled, but still around, would it still be vulnerable to this?
The first step in the escalation is a properly authorized request to the kube-apiserver that gets proxied to a backend server. A malicious request must be able to reach the kube-apiserver for the vulnerability to be exploited.
That's what I thought, thanks so much for taking the time to answer so quickly!
Thanks for the clarification !
Hi, I want to know why the issue isn't fixed in 1.9.x?

We have a lot of clusters running the 1.9.x k8s.
Pardon stupid question, but is Heapster considered aggregated API server same as metrics server?
The Kubernetes project supports the most recent three minor releases with bug and security fixes. The currently supported releases are 1.10.x, 1.11.x, and 1.12.x. If you are running builds provided by a distributor (not the binaries or packages provided by the Kubernetes project), contact their support to determine which of their versions are vulnerable. Some distributors may apply fixes to versions prior to the open-source-supported versions. No, Heapster is not an aggregated API server. Aggregated API servers register objects with non-nil fields, and requests to endpoints on the kube-apiserver are proxied to the appropriate aggregated server.
Are there any detail procedures to reproduce the issue? So we can verify that if we have fixed it on 1.9.x. Thanks a lot.
How to reproduce the issue, so we can check if the kubernetes version we're using is affected by this issue? An example script/command shows how gets all pods information on that node can be helpful.
Is it enough to just update the API Server to a fixed version, or should all the components (scheduler, kubelet, etc.) be updated to the fixed Kubernetes version to be secure?
, fix is in the apiserver codebase, so it should be enough to update just it
Does this affect ELB/ALB/Ingress controllers in any way? I mean if K8s API is hidden in a private network but load balancers for services (not for K8s API) are wide open, is it possible to use this vulnerability?
If the request cannot reach the API servers, the exploit cannot work. Even if the request can reach the API servers, but not have the permission to establish backend/proxy connection, it is also fine. But if a user in your cluster can use kubectl "freely", it means that they can connect to K8s API server, and the CVE may be applied here. Or, if the services that run on your cluster has a role that allows pod exec/attach/port-forward, it can proxy the request from the service to the API server.

Or if one of your services are API server extension, and your users can interact with it directly, then he/she can craft the exploiting request
Could someone give an example of an aggregated API endpoint that is usually enabled by default?
the metrics server is an aggregated API server, so is often a valid endpoint
What is the safest way to "suspend use of aggregated API servers" and to "remove all anonymous access to all aggregated APIs" () ? Is it sufficient to move aside the proxy-client-key-file mentioned here: ?
Don't laugh at this question. is Kubernetes version 1.7.11 affected? I would assume so. But if anyone can confirm this will help me out considerably. Thanks James Sorry I won't add another comment, I'll edit this one to thank you for your reply below, I mis-read the post on affected versions, I did not see 1.0.x-1.9.x, Sorry for wasting your time with the question.
I would say Yes, as Kubernetes v1.0.x-1.9.x 1.7.x falls within that range.
Sorry to bug you, but it would be a huge help to get an answer to my questions in .
I know the official docs on upgrading state to to from 1.7 to 1.8 then to 1.9 and then to 1.10, Would anyone know of a way to from 1.7.11 to 1.10.11. If not we will just figure out the steps and do the 3 separate upgrades. If something similar has been done before we'd like to hear about it as we have a few customers on 1.7.11 and all need to get upgraded. We will of course do our own trial and error in a lab and try and script much of it.
I don't understand why this leads to security issues. what is the "In default configurations"?
Without knowledge of your particular cluster and use, that is difficult to say. The safest way to suspend use of aggregated API servers might be to leave their objects registered, and scale down the deployments backing those objects, so that no aggregated server instances are available to receive requests.

Some potential disruptions that would cause: If portions of the cluster depend on an aggregated server (like HPA depending on metrics-server), this would break those dependencies aggregated servers that cannot be reached cause the garbage collector and namespace cleanup controllers to wait to perform some operations until the unavailable server can be reached again. This could prevent deletion of namespaces, and prevent foreground deletion of objects until the servers are made available again Removing anonymous access would either consist of disabling anonymous requests entirely with (see issue description for possible problems with that approach) or removing authorization permissions from anonymous users (specifically the user and group). If using , that includes removing the anonymous subjects from the and on restart. Keep in mind that preventing anonymous access still leaves the cluster vulnerable to escalation by authenticated users. Moving the would just prevent your kube-apiserver from starting Skip-version upgrading is not supported. A low-privilege request (like an anonymous call to a discovery API listing available resource types, like ) can be escalated to make any API call (including reading and writing API objects) to the aggregated server (like ). The default configuration refers to the discovery permissions granted to all users by .
on a 1.9.5 version i have it as --anonymous-auth=false and i deny all unauthenticated requests
In our Kubernetes cluster (Azure ACS, Kubernetes version# 1.8.4), we are using kubeconfig in order to establish proxy connection with kubernetes cluster. This is limited to only certain operations team members. It means in order to set connection with kubernetes api server, certificate from kubeconfig is used. Also, all the apis hosted on this cluster are exposed via azure api management service. In that case, is our kubernetes cluster still applicable to above vulnerability?
you mentioned "Aggregated API servers register APIService objects with non-nil spec.service fields" ... so is this a good query to see if someone has one of those in their environment?
This would work without jq: Added to the description
Gravitational has created a vulnerability test utility:
Under "affected configurations", it seems to suggest that if you run the given kubectl command and have no output, you do not have any exploitable endpoints for the aggregation api portion of this vulnerability. However, in "vulnerability impact" immediately below it states "In default configurations, all users (authenticated and unauthenticated) are allowed to perform discovery API calls that allow this escalation.".

Does this actually more precisely mean that the ability to discover vulnerable api calls that might exist is enough access to perform the exploit, if you happen to have exploitable endpoints? Asking because we appear to have no output from the kubectl command, but I can anonymously call "get --raw /apis" with . Unsure if this cluster is currently exploitable or not via this mechanism.
- clarified the "Affected configurations" and "Vulnerability impact" sections
thanks
Great thanks to !
there are two tests circulating for configurations that are vulnerable to the aggregated API exploits: the test you to the issue description, and which posted above. As mentioned in this issue: their results aren't always the same. In fact, we have clusters that are not affected according to your test but are affected according to . Could you clarify whether a negative from your test plus a positive from Gravitational's test indicates that we are not exposed?
the gravitational tests for ... however, if there are NO aggregated apis (as shown by 's command) then the cluster is still ok (hopefully can confirm this). You are still vulnerable to the other code path so you will have to review the RBAC permissions.
If there are no objects that route to a service (the check), that means no aggregated servers are configured in the cluster, and the anonymous escalation via aggregated discovery API requests is not possible. That said, if you are still running a vulnerable version and later add an aggregated API server, it would become exploitable.
Thanks very much for the clarity.
What about Kubernetes <1.5.x that no have "apiservices" daemon running/installed? And Could you please explain "user", which user do you mean. User that have shell access without root-access to kube-masters (what i think) and/or user from outside through an hosted application (ex. apache server on running pod-API-Server) and/or user that have general access to kube machines but without shell access. Thanks in advance
The APIService API was alpha in 1.6.x, and beta and enabled by default in 1.7.x+. kube-apiserver releases without an APIService API do not have the ability to proxy to aggregated API servers, so anonymous escalation via aggregated discovery API requests is not possible on those versions. Any actor that can send a network request to the kube-apiserver.

Escalation paths for this vulnerability begin with a network request to the kube-apiserver, which must first pass authorization, then be proxied to a backend (either to a kubelet in the case of pod exec/attach/portforward, or to an aggregated API server in the case of aggregated APIs).
The pod exec/attach/portforward escalation path applies back to 1.0.x
Fixed in for
Updated the description with a link to the document and recording of the post-mortem for this incident. Links to action items are in the document and reference this issue.
/label official-cve-feed (Related to )