Penetration Test Report (Cure53): Jaeger Cure53 20190504

Dr.-Ing. Mario Heiderich, Cure53
Bielefelder Str. 14
D 10709 Berlin
cure53.de · mario@cure53.de
Pentest-Report CNCF Jaeger 04.-05.2019
Cure53, Dr.-Ing. M. Heiderich, M. Wege, MSc. N. Krein, MSc. D. Weißer, J. Larsson,
BSc. J. Hector
Index
Introduction
Scope
Test Methodology
Part 1. Manual Code Auditing
Part 2. Code-Assisted Penetration Testing
Focus Areas and General Recommendations
Input Manipulation and Code Injection Attacks
Logic Bugs, ACL & Integrity Tests, Privilege Escalation
Security aspects of the general Kubernetes deployment
Programmatic DoS Attacks, Memory Exhaustion
Web-User Interface Security
Service Robustness & Proneness to Crashes
Miscellaneous Issues
JT-01-001 Collector: Missing authentication on data submission (Info)
JT-01-002 UI: DoS on Elasticsearch due to large input values (Info)
JT-01-003 K8s: Missing defense-in-depth procedures (Info)
Conclusions
Cure53, Berlin · 05/14/19 1/12
Dr.-Ing. Mario Heiderich, Cure53
Bielefelder Str. 14
D 10709 Berlin
cure53.de · mario@cure53.de
Introduction
“As on-the-ground microservice practitioners are quickly realizing, the majority of
operational problems that arise when moving to a distributed architecture are ultimately
grounded in two areas: networking and observability. It is simply an orders of magnitude
larger problem to network and debug a set of intertwined distributed services versus a
single monolithic application.”
From https://www.jaegertracing.io/
This report documents the findings of a security assessment targeting the CNCF Jaeger
software, a distributed Tracing System released as open source by Uber. Carried out by
Cure53 in spring 2019, this project entailed a source code audit and a penetration test.
As for the resources, six members of the Cure53 took part in the project. The allocated
time budget was eighteen days and these were all dedicated to project tasks and
objectives in late April and early May of 2019. The targeted release - equating to the
scope of this examination - was CNCF Jaeger 1.11.0. It can be noted that the source
code was taken from the publicly available Github repository, as the project has an open
source character. Consequently, the chosen methodology was white-box and paired a
comprehensive code audit with a classic penetration testing against a system put
forward by the maintainers of the scoped Jaeger version.
Besides the actual software, several of the available clients were in scope as well. Note
that given the time available to the testers, not all clients could be audited. In addition,
the Kubernetes-related code and configuration files were also inspected. All in all,
Cure53 did not manage to find any notable threats. Only a very small array of three
general weaknesses with negligible risk potential could be distinguished. Nevertheless,
even though the threat model for CNCF Jaeger is rather “generous” and not many items
could be pointed out as “risks”, experience has shown that attackers, once a system like
CNCF Jaeger is exposed to the public Internet, benefit from missing security controls.
Therefore, Cure53 found it crucial to also focus on hardening strategies that can make
the software even better and safer.
The next section in this report will elaborate on the scope details, list the release version
and repositories in scope. Next, the report will detail the chosen methodology and
describes various testing approaches, tasks and strategies to illuminate the span and
scope of the coverage reached by this Cure53 assessment. This means that a checklist
of the analyzed items is included to document the baseline of this evaluation of the
CNCF Jaeger security posture. The ensuing section is linked with the above statements
about the hardening needs, as the report furnishes on general advice and hardening
recommendation across the areas possibly prone to attacks. A detailed discussion of the
Cure53, Berlin · 05/14/19 2/12
Dr.-Ing. Mario Heiderich, Cure53
Bielefelder Str. 14
D 10709 Berlin
cure53.de · mario@cure53.de
aforementioned three findings follows. Last but not least, the report closes with a
conclusion, in which Cure53 summarizes their overall impressions about the CNCF
Jaeger’s general security posture.
Scope
• CNCF Jaeger
◦ In scope was the core system for CNCF Jaeger (1.11.0)
▪ https://github.com/jaegertracing/jaeger/releases
◦ Also investigated were the relevant client libraries (a selection done by Cure53)
▪ https://github.com/jaegertracing/jaeger-client-python
▪ https://github.com/jaegertracing/jaeger-client-node
▪ https://github.com/jaegertracing/jaeger-client-go
▪ https://github.com/jaegertracing/jaeger-client-cpp
◦ Kubernetes-related code and config
▪ https://github.com/jaegertracing/jaeger-kubernetes
◦ Cure53 was given access to a dedicated setup for the purpose of completing the
penetration test
Test Methodology
The following paragraphs describe the testing methodology used during the audit of the
Jaeger codebase and its related client-libraries. The test was divided into two phases,
each fulfilling different goals. In the first phase, the focus was on manual source code
reviews needed to spot insecure code patterns. In this realm, issues around memory
corruption, information leakage or similar flaws can be found. During the second phase,
it was evaluated whether the stated security goals and premise can, in fact, withstand
real-life attack scenarios.
Part 1. Manual Code Auditing
This section lists the steps that were undertaken during the first phase of the audit
against the Jager software compound. It describes the key aspects of the manual code
audit. Since no major issues were spotted, the list portraits the thoroughness of the
penetration test and attests a high quality of the project.
• The Jaeger codebase was checked for potentially vulnerable sinks in the
locations were user-input is parsed and handled.
• Client-libraries were audited to check how a connection is established to either
the agent daemon or the collector. The polling of the sampling strategy and
submitting of trace data were examined.
Cure53, Berlin · 05/14/19 3/12
Dr.-Ing. Mario Heiderich, Cure53
Bielefelder Str. 14
D 10709 Berlin
cure53.de · mario@cure53.de
• Additionally, the code for creating spans and tags was audited to ensure that the
data is stored in a safe manner. Flaws in this area could otherwise lead to
injection-based attacks.
• Client libraries were also checked for OS interactions (e.g. logging or binary
execution), however, the logging was either to null or console. No further OS
interactions were found.
• Special attention was given to the C++ client library (due to the nature of the C++
language). Dynamic memory allocations, unsafe use of typical C APIs and
possible integer overflows were checked. However, the strict use of modern C++
constructs prevents common pitfalls on the CNCF Jaeger scope.
• HTTP handler functions of the collector and agent daemon were audited for
logical flaws and general handling of untrusted input. Additionally, it was checked
whether proper ACL checks were in place. More information on this subject can
be found in JT-01-001.
Part 2. Code-Assisted Penetration Testing
The following list documents the distinguishable steps taken during the second part of
the test. A code-assisted penetration test was executed against the pre-configured
Jaeger cluster provided by the development team. Since only a few miscellaneous
issues were found during the first part of the audit, this additional approach was used to
ensure maximum coverage of the originally defined attack surface.
• The admin web interface - which allows to view and submit traces - was tested
for common, web-related server and client issues like CSRF, XSS and injections.
Besides a minor DoS issue, no problems with the handling of user-input have
been found.
• The accepted Content-Types were evaluated for possibly exploitable avenues.
The gzip Content-Type was examined in relation to the so-called gzip-bombs but
no such weaknesses could be identified.
• The JSON