Penetration Test Report (TrailOfBits): Kubernetes Final Report

Kubernetes
Security Assessment
May 31, 2019
Prepared For:
Kubernetes Security WG | ​Kubernetes
Prepared By:
Stefan Edwards | T​ rail of Bits
stefan.edwards@trailofbits.com
Dominik Czarnota | T​ rail of Bits
dominik.czarnota@trailofbits.com
Robert Tonic | T​ rail of Bits
robert.tonic@trailofbits.com
Ben Perez | ​Trail of Bits
benjamin.perez@trailofbits.com
Executive Summary
Project Dashboard
Engagement Goals
Methodology
Coverage
Recommendation Summary
General Issues
Networking
Cryptography
Authentication
Authorization
Secrets Management
Multi-tenancy isolation
Findings Summary
1. hostPath PersistentVolumes enable PodSecurityPolicy bypass
2. Kubernetes does not facilitate certificate revocation
3. HTTPS connections are not authenticated
4. TOCTOU when moving PID to manager’s cgroup via kubelet
5. Bearer tokens are revealed in logs
6. Seccomp is disabled by default
7. Pervasive world-accessible file permissions
8. Environment variables expose sensitive data
9. Use of InsecureIgnoreHostKey in SSH connections
10. Use of InsecureSkipVerify and other TLS weaknesses
11. Kubeadm performs potentially-dangerous reset operations
12. Overflows when using strconv.Atoi and downcasting the result
13. kubelet can cause an Out Of Memory error with a malicious manifest
14. Kubectl can cause an Out Of Memory error with a malicious Pod specification
15. Improper fetching of PIDs allows incorrect cgroup movement
16. Directory traversal of host logs running kube-apiserver and kubelet
17. Encryption recommendations not in accordance with best practices
18. Adding credentials to containers by default is unsafe
19. kubelet liveness probes can be used to enumerate host network
20. Hard-coded credential paths
21. Log rotation is not atomic
Kubernetes Product Assessment | 1
22. Arbitrary file paths without bounding
23. Unsafe JSON Construction
24. kubelet crash due to improperly handled errors
25. Legacy tokens do not expire
26. CoreDNS leaks internal cluster information across namespaces
27. Services use questionable default functions
28. Incorrect docker daemon process name in container manager
29. Use standard formats everywhere
30. Superficial health check provide false sense of safety
31. Hardcoded use of insecure gRPC transport
32. Incorrect handling of Retry-After
33. Incorrect isKernelPid check
34. Kubelet supports insecure TLS ciphersuites
A. Vulnerability Classifications
B. strconv.Atoi result conversion may cause integer overflows
C. Proof of Concept Exploit for TOB-K8S-038
D. Proof of Concept Exploit for TOB-K8S-022
E. Proof of Concept Exploit for TOB-K8S-024
F. Further detail regarding TOB-K8S-021
G. Fault injection testing of Kubernetes with KRF
Testing components of Kubernetes
Testing containerized applications
Testing cluster operations
H. Documentation changes for cryptographic best practices
Kubernetes Product Assessment | 2
Executive Summary
Between October and December of 2018, the Kubernetes Security Working Group (SWG)
opened a Request For Proposal (RFP) to select a vendor to assess the core Kubernetes
project. The SWG engaged with Trail of Bits to review the security of the open source
Kubernetes system in February, 2019 with the assessment taking place between March
11th, 2019 and May 10th, 2019. The assessment consisted of 12 person-weeks of effort
with four engineers working from the v​ 1.13.4 ​release from the Kubernetes GitHub
repository.
The first week of the assessment was spent investigating environments in which to perform
live testing. Both Kops and Kubespray were tested as they appeared to be the most likely
candidates for testing that included both a base configuration and a deployer. Guidance
from the Kubernetes SWG suggested that Kubespray may be the better of the two, since it
matches the general direction of the SWG (namely, using kubeadm). This allowed us to
deploy Kubernetes both locally and in a cloud environment with consistent configuration
across both. Initial code review was also performed, leveraging both manual and
automated analysis. Finally, threat modeling activities also began, with asset creation and
meeting coordination being the primary focus.
The second week’s focus was on manual code review, allowing the assessment team to
become familiarized with the Kubernetes codebase. This review resulted in ten issues,
ranging in severity from Medium to Informational. Most notably, findings ​TOB-K8S-001:
Bearer tokens revealed in logs​ and ​TOB-K8S-005: Environment variables expose sensitive
data​ represent areas within the code base that leaked secrets to the wider cluster
environment, either through Unix environment variables or cluster logs. Further findings
related to file permissions and paths were identified, detailed in T​ OB-K8S-004: Pervasive
world-accessible file permissions​, T​ OB-K8S-006: Hard-coded credential paths,​ and
TOB-K8S-008: Arbitrary file paths without bounding.​ Finally, T​ OB-K8S-002: Seccomp is
disabled by default​ and ​TOB-K8S-007: Log rotation is not atomic ​were identified as areas
that could be strengthened with architectural changes.
The third week was spent continuing manual code review, resulting in five issues, ranging
in severity from Medium to Informational across multiple control families, mainly related to
authentication and authorization. Issues regarding lack of cryptographic key verifications
were identified in ​TOB-K8S-012: Use of InsecureIgnoreHostKey in SSH connections ​and
TOB-K8S-013: Use of InsecureSkipVerify and other TLS weaknesses.​ Additional findings
related to dangerous shell operations and data construction, as well as data validation.
TOB-K8S-017: Kubeadm performs potentially dangerous reset operations​ details how
Kubeadm offloads command execution to an external shell instead of directly invoking
commands, and ​TOB-K8S-014: Overflows when using strconv.Atoi and downcasting the
result​, T​ OB-K8S-015: Unsafe JSON construction,​ and ​TOB-K8S-016: Use standard formats
everywhere ​detail data validation and construction issues.
The fourth week was spent performing dynamic live testing of a Kubernetes cluster,
resulting in six issues, ranging in severity from High to Informational. Notable findings from
Kubernetes Product Assessment | 3
week four were generally related to denial-of-service attack vectors and process cgroup
manipulation. T​ OB-K8S-019: Kubelet can cause an Out Of Memory error with a malicious
manifest ​and ​TOB-K8S-020: Kubectl can cause an Out Of Memory with a malicious Pod
specification ​both detail denial-of-service attack vectors. ​TOB-K8S-022: Improper fetching of
PIDs allows incorrect cgroup-limited movement ​and ​TOB-K8S-023: TOCTOU when moving
pid to manager’s cgroup via kubelet​ both detail methods to manipulate the kubelet to
escalate access to privileged resources.
The fifth week was spent on code review assisted with live testing, resulting in two findings:
one Medium severity, the other Informational. An issue with reading arbitrary files in
/var/log of any node host was documented in ​TOB-K8S-026: Directory traversal of host
​
logs running kube-apiserver and kubelet.​ The second finding ​TOB-K8S-27: Incorrect
isKernelPid check​ related to further investigation of PID and cgroup related issues found
during the fourth week.
The sixth week was spent reviewing cryptography-related areas of Kubernetes, as well as
further dynamic live testing, resulting in seven issues ranging in severity from High to
Informational. Most notably, an issue related to exposing secrets to all containers by
default was documented in T​ OB-K8S-031: Adding credentials to containers by default is
unsafe.​ Another finding, connected to unauthenticated component communication, was
reported in T​ OB-K8S-034: HTTPS connections not authenticated.​ Further findings related to
certificates revocation and Kubernetes documentation encryption recommendation were
detailed in T​ OB-K8S-028: Kubernetes does not facilitate certificate revocation​ and
TOB-K8S-029: Encryption recommendations not