Penetration Test Report (TrailOfBits): argo threat model

Argo
Threat Model
February 25, 2021
Prepared For:
Edward Lee | I​ ntuit Chris Aniszczyk | L​ inux Foundation
edward_lee@intuit.com caniszczyk@linuxfoundation.org
Alex Collins | ​Intuit Jesse Suen | I​ ntuit
alex_collins@intuit.com jesse_suen@intuit.com
Alexander Matyushentsev | I​ ntuit Henrik Blixt | I​ ntuit
alexander_matyushentsev@intuit.com henrik_blixt@intuit.com
Prepared By:
David Pokora | T​ rail of Bits Brian Glas | T​ rail of Bits
david.pokora@trailofbits.com brian.glas@trailofbits.com
Alex Useche | ​Trail of Bits Mike Martel | T​ rail of Bits
alex.useche@trailofbits.com mike.martel@trailofbits.com
Changelog:
March 19, 2021: Revised report delivered
Introduction
The Cloud Native Computing Foundation (CNCF) tasked Trail of Bits with conducting a
component-focused threat model of the Argo CD, Argo Workflows, Argo Rollouts, and Argo
Events systems. This threat model reviewed Argo components across six control families:
● Networking
● Cryptography
● Authentication
● Authorization
● Secrets Management
● Multi-tenancy
The Argo project consisted of four discrete services: Argo CD, Argo Workflows, Argo
Rollouts, and Argo Events. While there may be optional connections between some
components (e.g., Argo Events and Argo Workflows), each service operates independently
and was therefore modeled and evaluated separately within the report. All four services
rely on Kubernetes for their core functionality. Argo CD and Argo Workflows consist of a
series of Kubernetes pods, services, and deployments that support Continuous
Deployment pipelines and job orchestration workflows, respectively. Argo Rollouts extends
Kubernetes deployment capabilities through a Kubernetes controller and a series of CRDs
to provide functionality such as canary deployments. Argo Events leverages a series of
Kubernetes resources to provide event-driven dependency management capabilities.
Rather than providing the capabilities of each project as a service, the evaluated
components are deployed by their users (typically Kubernetes operators) in their own
infrastructure. As such, our evaluation considered how the design of each component
could introduce threats to the confidentiality, integrity, and availability of Argo customers’
data, infrastructure, and reputation. We also considered the direct risk to Argo’s reputation
that could result from attacks on clients’ Argo infrastructure.
The above control families were chosen due to system sensitivities in those areas. Trail of
Bits also considered general confidentiality, integrity, and availability concerns.
Trail of Bits’ assessment of the four Argo services resulted in 22 findings ranging from
informational to high severity.
© 2021 Trail of Bits Argo Threat Model | 1
Key Findings
Throughout the review, Trail of Bits focused on revealing concerns in the abovementioned
control families while considering confidentiality, integrity, and availability.
Generally, Trail of Bits found that while Argo services often support various risk or threat
prevention methods, these methods are frequently under-documented or are provided on
an opt-in basis, rather than by default.
Examples of this in Argo CD include an insufficient guide regarding secrets management
(T​ OB-ARGO-TM03)​ , disabled (by default) metrics/logging for some components
(T​ OB-ARGO-TM05)​ , and insufficient documentation regarding best practices for logging
desirable information such as IP addresses in the event of an attack (T​ M-ARGO-TM07​).
Similarly, in Argo Workflows, we noted a lack of authentication by default (T​ M-ARGO-TM11​)
as well as TLS disabled by default (T​ M-ARGO-TM15​) and similar concerns regarding IP
logging documentation (​TM-ARGO-TM17)​ .
It would be beneficial to implement component isolation as an added layer of security
where possible. During the remote threat modeling sessions, Trail of Bits noted that event
sources and sensors in Argo Events isolate their secrets and network traffic from each
other; however, Argo CD and Workflows do not enforce any ingress/egress rules for
components throughout the system (T​ OB-ARGO-TM02​, ​TM-ARGO-TM10)​ . We would suggest
reorienting how trust zones exist within the system.
Report Position
The Argo services provide a wide range of capabilities for end users, with various external
connections and integrations. We conducted a best-effort review of each service within the
given time constraints and also attempted to review all previously mentioned security
controls while considering general confidentiality, availability, and integrity concerns. This
report catalogs many of the key takeaways from the Rapid Risk Assessment process.
This report analyzes components, trust zones, data flows, threat actors, controls, and
findings of the Argo threat model. It represents a fixed point-in-time assessment that may
not account for future iterations or changes implemented during the engagement. For
example, the e​ ventbus-default-metrics component was removed from Argo Events in
​
between drafts of this report but is still referenced in this report.
Trail of Bits operated under the assumption that information shared during the daily
remote meetings held between February 16 and February 19, 2021 accurately described
© 2021 Trail of Bits Argo Threat Model | 2
the current state of the infrastructure. Additionally, documentation from that period was
leveraged from the following pages:
● https://argo-cd.readthedocs.io/en/stable/
● https://argoproj.github.io/argo-workflows/
● https://argoproj.github.io/argo-rollouts/
● https://argoproj.github.io/argo-events/
© 2021 Trail of Bits Argo Threat Model | 3
Introduction 1
Key Findings 1
Report Position 2
Methodology 6
Components 7
Argo CD 7
Argo Workflows 8
Argo Rollouts 9
Argo Events 9
Trust Zones 11
Argo CD 11
Argo Workflows 11
Argo Rollouts 12
Argo Events 12
Trust Zone Connections 14
Argo CD 14
Argo Workflows 15
Argo Rollouts 16
Argo Events 16
Threat Actors 18
Argo CD 19
Argo Workflows 22
Argo Rollouts 24
Argo Events 25
Dataflow 27
Argo CD 27
Argo Workflows 28
Argo Rollouts 29
Argo Events 30
Findings Summary 31
Argo CD Findings 33
1. Redis communications are unencrypted 34
2. Insufficient default network access controls between pods 35
3. Insufficient guidance on secure storage of secrets 36
4. Admin JWTs never expire 37
© 2021 Trail of Bits Argo Threat Model | 4
5. gRPC metrics are turned off by default 38
6. Manual password reset process does not enforce password complexity requirements
39
7. Documentation does not make recommendations for IP address logging, will not
occur by default 40
8. JSON Web Token-based authentication 41
9. Secrets used in one component are not isolated from others 42
Argo Workflows Findings 43
10. Insufficient default network access controls between pods 44
11. Lack of authentication rate limiting 45
12. API does not require authentication by default 46
14. User authentication is not logged if user isn’t mapped to RBAC rules 47
15. TLS is not enabled by default 48
16. Weak database password for base deployments of Argo Workflows 49
17. Documentation does not make recommendations for IP address logging, will not
occur by default 50
18. JSON Web Encryption-based authentication 51
Argo Events Findings 52
19. Insufficient logging of dropped events 53
20. Undocumented potential race condition in Event Bus 54
21. Event Bus contains plaintext events 55
22. Events cannot be optionally rate limited 56
Argo Rollouts Findings 57
A. RRA Template 58
Overview 58
Service Notes 58
How does the service work? 58
Are there any subcomponents or shared boundaries? 58
What communication protocols does it use? 58
Where does it store data? 58
What is the most sensitive data it stores? 58
How is that data stored? 58
Data Dictionary 58
Control Families 58
Threat Scenarios 59
Networking 60
Cryptography 60
Secrets Management 60
© 2021 Trail of Bits Argo Threat Model | 5
Authentication 60
Authorization 60
Multi-tenancy Isolation 60
Summary 60
Recommendations 60
© 2021 Trail of Bits Argo Thr