Acto: Automatic End-to-End Testing for Operation
Correctness of Cloud System Management
Jiawei Tyler Gu

Xudong Sun

Wentao Zhang

University of Illinois
Urbana-Champaign, IL, USA
jiaweig3@illinois.edu

University of Illinois
Urbana-Champaign, IL, USA
xudongs3@illinois.edu

University of Illinois
Urbana-Champaign, IL, USA
wentaoz5@illinois.edu

Yuxuan Jiang

Chen Wang

Mandana Vaziri

University of Illinois
Urbana-Champaign, IL, USA
yuxuanj9@illinois.edu

IBM Research
Yorktown Heights, NY, USA
Chen.Wang1@ibm.com

IBM Research
Yorktown Heights, NY, USAmvaziri@us.ibm.com

Owolabi Legunsen

Tianyin Xu

Cornell University
Ithaca, NY, USA
legunsen@cornell.edu

University of Illinois
Urbana-Champaign, IL, USA
tyxu@illinois.edu

Abstract
Cloud systems are increasingly being managed by operation programs termed operators, which automate tedious,human-based operations. Operators of modern management
platforms like Kubernetes, Twine, and ECS implement declarative interfaces based on the state-reconciliation principle. An
operation declares a desired system state and the operator
automatically reconciles the system to that declared state.
Operator correctness is critical, given the impacts on system operations—bugs in operator code put systems in undesired or error states, with severe consequences. However,validating operator correctness is challenging due to theenormous system-state space and complex operation interface. A correct operator must not only satisfy correctness
properties of its own code, but it must also maintain managed systems in desired states. Unfortunately, end-to-end
testing of operators significantly falls short.
We present Acto, the first automatic end-to-end testing
technique for cloud system operators. Acto uses a statecentric approach to test an operator together with a managed
system. Acto continuously instructs an operator to reconcilea system to different states and checks if the system successfully reaches those desired states. Acto models operations as
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must
be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
SOSP ’23, October 23–26, 2023, Koblenz, Germany© 2023 Copyright held by the owner/author(s). Publication rights licensed
to ACM.
ACM ISBN 979-8-4007-0229-7/23/10. . . $15.00
https://doi.org/10.1145/3600006.3613161

state transitions and systematically realizes state-transition
sequences to exercise supported operations in different scenarios. Acto’s oracles automatically check whether a system’s state is as desired. To date, Acto has helped find 56
serious new bugs (42 were confirmed and 30 have been fixed)
in eleven Kubernetes operators with few false alarms.
CCS Concepts: • Computer systems organization → Distributed architectures; Reliability; • Software and its
engineering → Software testing and debugging.
Keywords: Kubernetes, operation, system management, cloud,reliability, operation correctness, operator
ACM Reference Format:
Jiawei Tyler Gu, Xudong Sun, Wentao Zhang, Yuxuan Jiang, Chen
Wang, Mandana Vaziri, Owolabi Legunsen, and Tianyin Xu. 2023.
Acto: Automatic End-to-End Testing for Operation Correctnessof Cloud System Management. In ACM SIGOPS 29th Symposium
on Operating Systems Principles (SOSP ’23), October 23–26, 2023,
Koblenz, Germany. ACM, New York, NY, USA, 17 pages. https://doi.
org/10.1145/3600006.3613161

1

Introduction

Cloud systems are growing in scale and demand beyond what
human-based operation can reliably, continuously, and efficiently manage. Modern cloud systems are increasingly being managed by operation programs, termed operators [2, 47],
that automate labor-intensive operations. Operators of cloud
management platforms like Kubernetes [39], Twine [83], andECS [67] implement declarative interfaces based on state reconciliation. An operation declares the desired system state
and the operator automatically reconciles the system fromits current state to the declared state. This “cloud-native” operator pattern simplifies operations and improves efficiency.
The cloud-native operator pattern has led to a thriving
ecosystem of high-quality, reusable operator code [55, 56, 61,

# Desired State# Desired Statereplicas:
3 # <- 5

replicas:
5 # <- 3

affinity:
unsatisfiable_rule

affinity:satisfiable_rule
Rollback

ZooKeeper

2

Figure 1. A safety bug in ZooKeeperOp, a ZooKeeper
operator, detected by our tool, Acto [30]. The bug manifests when the operator first scales down and then scales up
ZooKeeper. Newly created pods fall into crash loops.

75, 78], many of which are used in production [49, 50, 77, 85,89]. Taking Kubernetes as an example: most cloud systems
today have operators to manage them atop the Kubernetes
platform. These operators automate important managementtasks like software upgrades, configuration updates, and autoscaling. Even for the same cloud system, multiple different
operators are developed by commercial vendors and opensource communities, to support different operation practices
and deployment environments.
The rapid development and deployment of operators maketheir quality assurance a pressing need—operation correctness
is critical to system reliability [51, 70]. A buggy operator
can impair correctly implemented systems in production.
Compared with human operator mistakes—major causesof system failures [38, 51, 68–70]—bugs in operators havemore magnified impacts due to the nature of automation andwidespread software reuse. In fact, buggy operators caused
many recent production incidents [41, 42, 52, 59, 60, 65, 86].
Figure 1 shows a safety bug that our technique detects in a
Kubernetes operator for managing ZooKeeper. When scaling
down a ZooKeeper cluster, the operator only removes thepods, but not the data volumes attached to those pods. If theoperator later scales up the ZooKeeper cluster, newly created pods will try to reuse old volumes. Due to membershipinconsistencies between new pods and old volumes, the new
ZooKeeper nodes fail to start. As a result, the ZooKeeper
cluster cannot scale up and is thus vulnerable to overloads.
Figure 2 shows a liveness bug that our technique detects
in an operator for managing TiDB. To update the affinity
rule of a TiDB pod [1], the operator must stop the pod andreassign it with a new affinity rule. But, if the new affinity
rule is not satisfiable, the pod cannot be reassigned. In thiscase, the buggy operator waits forever for the assignment
to complete. To make matters worse, the operator cannot be
restored by resetting the affinity rule, because the operator
does not carry out new operations before the ongoing one
completes, to avoid race conditions.
These two bugs are among a myriad of operator bugs that
affect operation correctness. Compared with the management platform (e.g., Kubernetes) and the managed system

TiDB
Vols Pods

1

ZooKeeper

Scale up

# Desired State

Scale downVols Pods

ZooKeeper

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

Reassign

SOSP ’23, October 23–26, 2023, Koblenz, Germany# Desired State

TiDB
1

Waiting forever
TiDB

2Figure 2. A liveness bug in TiDBOp, a TiDB operator,
detected by Acto [32]. If a declared affinity rule cannot
be satisfied, TiDBOp enters an infinite waiting loop and the
pod will never be assigned. TiDBOp cannot be recovered byrolling back with a satisfiable affinity rule.

(e.g., ZooKeeper and TiDB), operator code is often muchless tested. For example, we find that existing operators rely
mostly on unit tests which cannot check operation correctness end to end, i.e., if an operator reconciles the managed
system to desired states. Some operators include a few endto-end (e2e) tests but only cover small parts of the enormous
system state space and the complex operations exposed bydeclarative interfaces (see §3).
We seek a practical testing technique that can test cloudsystem operators end to end and can be readily applied to
any types of operators for managing different systems. Unfortunately, existing automated test generation techniques
like fuzzing [66] or symbolic execution [40] cannot effectively test operators end to end, since they neither model
the semantics of operations nor reason about system states.
In particular, operator bugs do not commonly manifest as
crashes but they drive systems into undesired states (§6.1).
Technique. This paper presents Acto, the first automatic
technique and tool for end-to-end testing of cloud system
operators. Acto automatically generates end-to-end tests tocheck three operation correctness requirements: the operator(1) always reconciles the managed system to desired states,(2) performs managed system recovery from undesired or
error states by rolling back to a previous good state, and(3) should be resilient to misoperations (i.e., operation errors)
by preventing them from driving the system into error states.
Acto is state centric. It models an operation as a pair ofcurrent system state and a declaration of the desired state. A
correct operation enables a state transition from the current
state to a new state that satisfies the declaration. Within thisstate-transition model, bugs in operators manifest as (1) undesired transitions in which the new state implicitly violates
the declaration, or (2) failure to recover from error states. In
addition to bugs, vulnerable code in operators would allow
misoperations to cause transitions into explicit error states.
To systematically explore state transitions under different
scenarios, Acto generates state declarations that cover all
system properties exposed by the operation interface (e.g.,Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

ated state declarations semantically meaningful, Acto automatically infers the semantics of properties and predicates.
Acto ensures that generated declarations are syntacticallyvalid and that they can exercise operators in diverse ways.
To validate operation correctness under different system
states, Acto executes the e2e tests in a test campaign, where
the operator is continuously tested under a sequence of operations. Each operation reconciles the system to a new statefrom which the subsequent operation starts. Acto’s test campaigns leverage the level-triggering principle [57]: a correct
operator must reconcile the system to the desired state regardless of the start state and must recover from error states.
Acto’s oracles check for errors that (1) manifest in explicit
forms, such as unexpected exceptions and panic signals, and(2) only manifest implicitly as mismatches between the reconciled system state and the declared desired state. To detect
implicit mismatches after state reconciliation, Acto checks
for consistency in state views of the operator and the underlying management platform (e.g., Kubernetes); inconsistencies
indicate bugs. Acto also employs a differential oracle atop
state objects from different state transitions to the same end
states, taking advantage of the interpretability and uniformity of state objects in modern management platforms.
Key results. We implemented Acto for Kubernetes operators. It works in two modes: a blackbox mode (Acto-■) that
only requires the operator’s interface specification (custom
resource definition of Kubernetes operators) and a whitebox
mode (Acto-□) that additionally takes the operator’s sourcecode for semantic inference and predicate analysis.
We evaluated Acto on eleven popular Kubernetes operators of various kinds. Acto found 56 new operator bugs in
total, among which 42 have been confirmed and 30 havebeen fixed. Acto also found six bugs in Kubernetes and in the
Go runtime that affected multiple operators (all have been
confirmed or fixed). The detected bugs lead to severe safety
and liveness issues, affecting not only the operators, but also
the reliability and security of the managed systems. Lastly,
Acto finds many vulnerabilities to misoperations. Acto tests
all these operators within eight hours (a nightly run) on acluster of eight machines; five of eleven operators only need
one machine. Acto has few false positives: Acto-□ reports
no false alarm and Acto-■ has a 0.19% false alarm rate.
Contributions. This paper makes four main contributions:• We present the first fully automatic end-to-end testing
technique that checks operation correctness for cloud system operators using a state-centric approach.
• We develop Acto, a practical tool that uses the proposed
technique to automatically test unmodified Kubernetes
operators and can detect many kinds of bugs.
• Acto has already helped improve the quality of elevenpopular Kubernetes operators by finding bugs that were
fixed by developers. Acto can be run nightly.

kubectl

Upscale ZooKeeper# ZooKeeper CR
replicas:
3 # <- 2
image:
zookeeper:2.4
persistence:storage: 20Gi
reclaim: Delete...

ZooKeeper Operator

replicas and affinity in Figures 1 and 2). To make gener-

SOSP ’23, October 23–26, 2023, Koblenz, Germany

State
Objects

Kubernetes
Core

Pod

Statefulset
Controller

Vol

VolumeController

...

...

Managed
ZooKeeper
Pods:

Vols:

...

Figure 3. Scaling up a ZooKeeper system (from 2 to 3
replicas) with a new desired-state declaration (CR).
• Acto is released as an open-source project and is hosted
at https://github.com/xlab-uiuc/acto, where the sosp-ae
branch includes detailed instructions on reproducing theresults in this paper.

2

Background

Operation programs (i.e., operators) for modern cloud management platforms like Kubernetes [39], Twine [83], andECS [67] follow a declarative, state-reconciliation design pattern. An operation declares a desired system state and theoperator automatically reconciles the system to the declaredstate. This design pattern simplifies system management operations by removing the need to write ad hoc, imperative
scripts for different one-off tasks. The pattern also makes
system management declarative and intent driven. We givea brief overview of the pattern, using Kubernetes [39] as arepresentative example.
Declarative operation interface. In Kubernetes, operators
expose a declarative interface in the form of custom resources(CRs) [3]. A CR defines a system resource and its properties
that can be modified to manage that resource. A state declaration specifies property values in a CR. Figure 3 shows anexample of desired-state declarations for ZooKeeper; it specifies primitive properties like replicas and image, and composite properties like persistence which has sub-properties.
A ZooKeeper operator reconciles a managed ZooKeeper cluster to satisfy the declared state. Management operations are
expressed by changing one or more property values in a CR.
Kubernetes operators maintain CR definitions in the OpenAPISchema format [13], which defines constraints on each
CR property (e.g., data type and data range). Operations thatchange a CR are first validated against the specification by
the API servers, before being forwarded to the operator.
Operator design pattern. Kubernetes operators follow
the state-reconciliation pattern of modern cloud management platforms and control planes, such as Kubernetes, Borg,
Omega, Twine, and ECS [39, 67, 76, 83, 87, 90]. An operator
continuously reconciles the managed system from its current state to a newly declared desired state, if the current
state does not match the declared state. The management

SOSP ’23, October 23–26, 2023, Koblenz, Germany

platforms maintain their current system states in a collection of state objects in strongly consistent datastores (e.g.,
etcd [7]). Every entity in the system, such as a pod, a volume,
and a stateful set (representing a stateful system), has a corresponding state object. State objects have uniform APIs andconsistent data schema, making them highly interpretableand extensible [39].
Figure 3 shows how a ZooKeeper operator scales up a
managed ZooKeeper cluster. A user declares the desired
state of the ZooKeeper cluster by submitting a new CR thatchanges the replicas property from 2 to 3 via the Kubernetes
client (kubectl). The operator processing the desired-state
declaration first confirms that the current number of replicas
in the ZooKeeper cluster is different from 3—only two pod
objects for replicas currently exist in etcd. To reconcile to thedesired state, the operator notifies Kubernetes to increase thestateful-set count for replicas. To do so, Kubernetes creates a
new pod and a new volume. State reconciliation stops when
the desired state with three replicas is reached.
Operation correctness. We define three correctness requirements for operations: the operator (1) always reconciles
the managed system to valid, reachable desired states, regardless of its current or previous states; (2) can recover the
managed system from implicit or explicit error states byrolling back to a previous good state; and (3) should prevent
misoperations from driving the managed system into error
states. Figure 1 violates the first requirement and Figure 2
violates the second requirement. In this paper, we treat root
causes of violations to the first two requirements bugs andreport them to developers. We refer to root causes of violations of the third requirement as misoperation vulnerabilities,
which are known to be serious issues [37, 38, 51, 68–70, 93].
We discuss systematic mitigations for misoperation vulnerabilities with developers.
Operation correctness is hard to achieve. Operator developers face the twin fundamental challenges of (1) anticipating relevant system states to explore in the enormous state
space, and (2) correctly reconciling the managed systems
from all the different start states.

3

Motivating Study

To understand the kinds of test cases (i.e., tests) that operator developers write and the limitations of their current
testing practices, we study 50 open-source Kubernetes operator projects from GitHub and their tests.
Finding 1. Most operators that we study rely on unit tests,
which cannot validate operation correctness. Only 34% of thesestudied operators have a few end-to-end tests.
Checking if a managed system reaches desired states isbeyond the scope of unit tests, each of which checks a method
in operator code. Such checks need end-to-end (e2e) tests [10]to validate operation correctness of the managed systems.

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

Table 1. Properties covered by existing e2e tests andcharacteristics of tests that trigger multiple operations.
Operator# Properties
Tested Total

Tests with multiple operations% (#)# Ops (Avg)

KnativeOp
PCN/MongoOp
RabbitMQOp
ZooKeeperOp

8 (2.15%) 372
70 (1.27%) 5495
19 (1.43%) 1332
13 (1.47%) 886

14.29% (1/7)
38.71% (12/31)
25.00% (2/8)
75.00% (6/8)

6
2.58
2.5
2

Typically, an e2e test first causes an operator to carry out
an operation, for example, to deploy, scale, or reconfigure
the managed system. Then, the e2e test checks if the operation succeeded by means of assertions that compare thereconciled managed system state with the expected state.
However, only 17 (34%) of 50 operators include e2e tests, andthose manually written e2e tests are few, with a median ofsix e2e tests per operator.
We focus the rest of our study on the effectiveness of existing e2e tests, since we address operation correctness. We
study four operators from the 50 and their e2e tests: KnativeOp, PCN/MongoOp, RabbitMQOp, and ZooKeeperOp.
These operators are developed either by official teams of the
managed systems, or by companies that sell services built
around the managed systems. These four operators contain
7–31 e2e tests; PCN/MongoOp relies only on e2e tests (nounit tests). Table 4 provides more data about these operators.
Finding 2. Existing e2e tests cover only 1.27–2.15% of supported properties exposed by the operation interface. Also, most
tested operations start from the default initial state.
Table 1 shows that existing e2e tests change very few
properties when testing operation correctness in these four
Kubernetes operators that we study. We find that some operators’ e2e tests do not check basic operations, e.g., backend
migration in RabbitMQOp. Also, few e2e tests check operations in multiple configurations, e.g., deploying ZooKeeper
with persistent and ephemeral storage. Acto efficiently helps
test more operations in multiple configurations.
Operators are long-running processes that continuously
monitor and reconcile managed systems from any state to thedesired states. So, operations should be tested from different
start states. Consider scaling: given a desired number of
replicas, triggering a scale-up or a scale-down procedure
depends on the current state. Table 1 (third column) showsthat the few e2e tests that check multiple operations only
check 2.97 operations on average, a small number comparedto how operators work in practice. Most tests trigger only
one operation from the default initial state.
Finding 3. State-based assertions in existing e2e tests cover
only 0.24–10.90% of managed systems’ state-object fields.
Given the enormous state space, developers likely find it
tedious to write assertions on many state-object fields. Table 2 shows a breakdown of three kinds of assertions that

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

Table 2. Three types of assertions in existing e2e tests.
Operator# Assertions
Env. State Behav. Total# State Objects
Asserted Total

KnativeOp
PCN/MongoOp
RabbitMQOp
ZooKeeperOp

18
2
26
62

14 (0.93%)
329 (10.90%)
12 (0.42%)
7 (0.24%)

32
209
19
54

0
177
29
0

50
388
74
116

1506
3017
2852
2934

we observe in existing e2e tests. These tests check (1) the environment (e.g., can operators request Kubernetes services?);(2) system states—is the managed system reconciled to thedesired state?; and (3) managed system behavior. Assertions
on the environment check that operators run in compatible
settings; they do not validate operation correctness. Stateand system-behavior assertions could validate operation correctness. But, in our study, these kinds of assertions either
only check a small part of the system state or only check theavailability of system services.
Finding 4. The few assertions on system behavior are basic
and mostly check service availability.
KnativeOp and ZooKeeperOp tests have no assertion on
system behavior. In PCN/MongoOp and RabbitMQOp, suchassertions only check that the managed system responds
to read/write requests from clients. We find a few assertions on system-specific behavior: (1) 36 of 177 assertions in
PCN/MongoOp check backup availability; and (2) only oneof 77 RabbitMQOp assertions checks membership list size.
Implications. Our study shows that current manual testingof operation correctness is significantly limited, even for
popular operators with many GitHub stars (see Table 4, §6).
Our results suggest that manually writing end-to-end (e2e)tests is tedious and inadequate. So, automatic e2e testingof operation correctness is desirable. We believe that suchautomatic testing is viable and can be done effectively byleveraging the declarative, state-reconciliation pattern ofmodern cloud system operators.

Acto generates minimized e2e test code for every alarmthat it raises. These generated tests can help developers toreliably reproduce a bug or a vulnerability, without rerunning the entire test campaign. That is, generated e2e tests
only run operations that are necessary to set up the state forreproducing a bug or a vulnerability. Developers can include
the generated e2e test in their regression test suite.
Acto is automatic—it tests unmodified operators and requires no manual annotation, instrumentation, or assertion.
The test inputs that Acto automatically generates are operations, which drive the operator under test to reconcile themanaged system to declared desired states. Acto ensures thatgenerated operations are syntactically valid and represent
various scenarios by analyzing the constraints and semantics of properties exposed by an operator’s interface. Acto
dynamically computes the desired state for triggering thenext operation based on the current state.
Acto’s test oracles check if the system state after an operation matches the desired state. Automatic test oracle generation is a hard problem in general. Acto’s test oracles areenabled by a key opportunity in modern cloud management
platforms based on state reconciliation like Kubernetes: they
maintain the system states in uniform, interpretable state
objects that can be systematically queried and analyzed.
Usage. Acto works in two modes: a blackbox mode (Acto■) and a whitebox mode (Acto-□). Acto-■ takes two inputs:
1) a manifest for building and deploying the target operator,and 2) the specification of state declaration provided by theoperator interface (e.g., the custom resource definition of
Kubernetes operators). Both inputs are abundant in mature
operator projects; they are widely used for operator development and deployment. Finding these inputs is straightforward. Acto-□ requires an additional input: the operator’ssource code for static program analysis. Acto outputs test
failures, debugging information for root cause analysis, andminimized test code that reproduces detected failures.
4.1

4Technique

Acto is a state-centric testing technique. It tests operation
correctness by performing end-to-end (e2e) testing of cloudnative operators together with the managed systems. To doso, Acto continuously generates new operations during atest campaign. Then, Acto’s oracles check if the operator
always correctly reconciles the system from each current
state to the desired state, or raises an alarm otherwise.
Acto detects bugs when requirements of operation correctness (§2) are violated. Such bugs include those that (1) cause
an operator not to reconcile the system to desired states,(2) crash the operator or the system, and (3) prevent the
managed system from recovering from an error state. Acto
also detects vulnerabilities to misoperations that can drive
the systems into explicit error states.

SOSP ’23, October 23–26, 2023, Koblenz, Germany

Operation Model

Acto models an operation as a pair, (𝑆 𝑐 , 𝐷), where 𝑆 𝑐 denotes
a current system state and 𝐷 is a declaration of a valid desired
state. 𝐷 is constrained by the operation interface specification (e.g., a CR definition in Kubernetes). If successful, an𝐷
operation triggers a state transition, 𝑆 𝑐 −→ 𝑆 𝐷 , where 𝑆 𝐷
satisfies 𝐷, i.e., 𝑆 𝐷 |= 𝐷. 𝐷 often only specifies a (small) partof the system state. So, there are multiple possible system
states that can satisfy 𝐷, and, in practice, only a small partof 𝑆 needs to be examined to check if 𝑆 𝐷 |= 𝐷.
If an operation fails (e.g., due to bugs in operator code),the system enters an error state, 𝑆 𝑒 ̸ |= 𝐷, i.e., 𝑆 𝑒 does notsatisfy the desired state. When 𝑆 𝑒 ̸ |= 𝐷, the operator shouldbe able to rollback the state from 𝑆 𝑒 with a state transition𝐷𝑖 −1
𝑆 𝑒 −−−→ 𝑆 𝑐 , where 𝐷𝑖 −1 is the desired-state declaration that
previously triggered a transition to 𝑆 𝑐 .

SOSP ’23, October 23–26, 2023, Koblenz, Germany

𝑆!

𝐷!

𝑆"

𝑆!

𝐷#

(a) Single operation

𝐷%𝑆#

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

…

𝑆$

𝐷!

𝑆"

…

(b) Operation sequence
𝐷!"#

…

𝑆!

𝐷!"#

𝑆!"#

𝐷!

𝑆"%

𝐷!$#

𝑆!$#

…

(c) Operation sequence with error-state recovery
𝐷!"#

𝑆!

𝐷#

𝑆#

𝐷%
𝐷%

𝑆$

…
𝐷!"#

𝑆!"#

𝐷!

𝑆"%

𝐷!$#

𝑆!$#

…

𝐷!$#

(d) Acto’s test exploration strategy

Figure 4. State transitions of different test strategies.

The fundamental challenge in testing operators is the prohibitive cost of testing all elements in the Cartesian product˘ where 𝑆 𝐶 is the set of all possible validof 𝑆 = 𝑆 𝐶 ∪ 𝑆 𝐸 and 𝐷,𝑐system states (𝑆 ∈ 𝑆 𝐶 ), 𝑆 𝐸 is the set of all possible error
states (𝑆 𝑒 ∈ 𝑆 𝐸 ), and 𝐷˘ is the set of all possible declarations
˘ There can be a large number of valof desired state (𝐷 ∈ 𝐷).
ues for different properties that constitute the system state.
Exhaustive testing could be prohibitively expensive and any
practical testing approach can only exercise a part of the
˘state space, i.e., 𝑆 × 𝐷.
4.2

Test Strategy

Acto systematically explores the state space using the following three test strategies (Figures 4a–c).
Single operation. Acto generates a declaration of a desired
state 𝐷, triggers an operation to reconcile the current system
state 𝑆 𝑐 to the desired system state 𝑆 𝐷 , and checks whether
𝑆 𝐷 |= 𝐷. The single operation is applied to the initial system
state 𝑆 𝑐 = 𝑆 0 (starting from a non-initial state requires more
operations). This simple single-operation strategy is similar
to the current testing practices discussed in §3; it is easy to
implement and reason about. The key challenge is how to˘
explore an effective and representative subset of 𝐷.
Operation sequence. Acto extends single operations into
a test campaign, which consists of a sequence of operations. Test campaigns overcome the limitation of the singleoperation strategy, which must always start from the initialstate 𝑆 𝑐 = 𝑆 0 . As discussed in §3, it is important to test
whether an operator can reconcile the system to desired
states from different, non-initial start states. Reaching an
end state from different start states increases the chance of
invoking different procedures in the operator code. In a test
campaign, earlier operations take the system to new states
which become the start states for subsequent operations.
Acto generates a test campaign by chaining the expected end

states {𝑆𝑖 } from the single-operation strategy, and generating a new 𝐷𝑖 after each successful reconciliation, as shownin Figure 4b. The result is a sequence of state transitions,
𝐷1
𝐷2
𝐷𝑖
𝐷𝑖+1
𝑆 0 −−→ 𝑆 1 −−→ ... −−→ 𝑆𝑖 −−−→ ...; Acto checks whether each
𝑆𝑖 |= 𝐷𝑖 , where i ≠ 0.
Error-state recovery. The operation-sequence strategy
does not test whether or not an operator correctly restores asystem from implicit or explicit error states. If the system isin an error state 𝑆 𝑒 , the operator is responsible for recoveringfrom 𝑆 𝑒 by reconciling the system from 𝑆 𝑒 back to the priorhealthy state 𝑆𝑖 −1 . The subsequent operations start from𝐷𝑖+1
𝑆𝑖 −1 , such as in the transition, 𝑆𝑖 −1 −−−→ 𝑆𝑖+1 , in Figure 4c.
Error states can be reached because of operator bugs thatreconcile the system to a state 𝑆 𝑒 ̸ |= 𝐷, or misoperations—semantic errors in 𝐷 that escape syntactic validation against
the interface specification.
Acto combines these three test exploration strategies (Figures 4a–c) to realize the state transition sequences in one
test campaign, as shown in Figure 4d.

5Design and Implementation

This section describes the main components of Acto and how
we implement them. These components embody Acto’s statecentric testing technique (§4); they generate declarations ofdesired system states, execute test campaigns, and checkreconciled states using automated test oracles.
5.1

Realizing State Transitions

During a test campaign (Figure 4d), Acto automatically generates a new state declaration 𝐷𝑖+1 based on the current
𝐷𝑖+1
system state 𝑆𝑖 to realize a state transition, 𝑆𝑖 −−−→ 𝑆𝑖+1 . Test
campaigns start from the initial state 𝑆 0 . Acto triggers statetransitions with the goals to: (1) cover all properties exposedby the operation interface, and (2) exercise representative
operation scenarios based on property semantics.
Acto systematically exercises all the properties that are
defined in the operation interface. Each new 𝐷𝑖+1 changes
one property in the current state 𝑆𝑖 and any other properties
that are needed to satisfy predicates on property relationships (§5.2.4). Specifically, Acto selects a previously untested
property and uses it to declare a new desired state. The end
state after one transition, becomes the start state for the
next transition (Figure 4b). All state declarations collectively
change every property at least once during a test campaign.
Acto tests different scenarios based on the semantics of thechanged properties. (Acto automatically infers these semantics, §5.2.2). Table 3 gives a few such scenarios. For example,
Acto tests the scale-up-and-scale-down and the scale-downand-scale-up sequences if a property represents the numberof replicas. Acto also tests different pod assignments thattrigger the operator to re-configure or re-deploy managed
systems differently. This scenario-driven approach allows
Acto to focus on a small number of representative states,

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

Property

Scenarios

Replicas

Scale up and then down; scale down and then up;
upscale over system resource limit.
Place all pods on one node; spread pods to different nodes;
set unsatisfiable affinity rules.
Expand storage volumes; shrink storage volumes;
request more storage than is available in a cluster.
Switch between normal and privileged roles.

Affinity
StorageAccessinstead of the very large set of all possible property values.
We implement scenarios as plugins that can be extended orcustomized; users of Acto can add more plugins.
In addition to valid operation scenarios, Acto also generates misoperations, each of which triggers a state transition
to an error state, 𝑆 𝑒 . For example, Acto generates misoperations that (1) scale the replicas beyond the total number ofavailable physical resources, and (2) set unsatisfiable affinity
rules (Table 3). Acto uses misoperations to check if an operator (1) is resilient to operation errors, and (2) can recoverfrom undesired or error states. Acto’s oracles (§5.3) check theformer (is the system in a state 𝑆 𝑒 ?). Acto checks the latterby rolling back 𝑆 𝑒 to the most recent healthy state. Misoperations that declare semantically erroneous states could escapeconstraint validation (see §5.2.1). A correct operator should
not carry out an erroneous operation or at least should be
able to recover from operation failures.
5.2

Generating State Declarations

˘ that are
Acto generates desired-state declarations, 𝐷 ∈ 𝐷,
syntactically valid (§5.2.1), resemble real-world scenarios(§5.2.2, §5.2.3), and satisfy predicates on property relationships (§5.2.4). Such desired states improve the effectivenessand efficiency of Acto’s state space exploration. End-to-end
tests are expensive, so a 𝐷 that does not satisfy these conditions has a low chance to find bugs. We next discuss how
Acto generates 𝐷 to satisfy these conditions.
5.2.1 Extracting Constraints. The operation interface
specification defines syntactic validity constraints on state
declarations. For example, Kubernetes’ OpenAPISchema specification defines constraints on all supported properties. Acto
uses these constraints to ensure that all property values indeclared desired states are syntactically valid. (Invalid declarations would likely be directly rejected by the API servers
before reaching the operator.) For composite properties, Acto
uses composite constraints like required properties and also
derives constraints from the sub-properties. For primitive
properties, Acto uses constraints like the type, min/max values (for numeric types), length (for string type), regularexpression patterns, etc.

Operator Interface Specification

Kubernetes Core Resource# Cassandra CRD
cassandraDataVolumeClaimSpec:
accessModes:...
dataSource:...
resource:...
...
size: int...

# K8s StatefulSetSpec
VolumeClaimTemplates:
accessModes:...
dataSource:...
resource:...
...
replicas: int...

Operator Code

Table 3. Examples of built-in scenarios of Acto to generate new state declarations and trigger state transitions.
Scenarios are created based on property semantics inferredby Acto and they can be extended or customized.

SOSP ’23, October 23–26, 2023, Koblenz, Germany// pkg/reconciliation/construct_statefulset.go

desiredSts := &appsv1.StatefulSet {spec: &appsv1.StatefulSetSpec {replicas:replicas: cr.spec.size
cr.spec.size
...
} ...
} ...

rc.Client.Update(rc.Ctx, desiredSts)

Figure 5. Semantic analysis maps the properties in theoperation interface to the properties of a Kubernetes
core resource.
5.2.2 Inferring Property Semantics. To exercise different scenarios (§5.1), Acto changes properties based on theirsemantics. Acto infers the semantics of a property in the interface specification by mapping it to a set of resource types
in the Kubernetes core APIs. Such mapping is feasible because many operations for property changes are eventuallydelegated to Kubernetes core services.
Inferring semantics from property structure (Acto-■).
Acto exploits the insight that property structure is effective
for mapping to properties in the Kubernetes core resource
specification. Specifically, all Kubernetes core resource types
have unique structures. Figure 5 exemplifies how Acto infers semantics from the property structure: CassOp has a
cassandraDataVolumeClaimSpec property with the same structure as the VolumeClaimTemplates property in Kubernetes’
StatefulSet resource. Therefore, Acto infers the semantics of
cassandraDataVolumeClaimSpec using a structural mapping.
Inferring semantics from source code (Acto-□). Acto-■
cannot use property structure to map primitive properties(e.g., integer). Also, naming conventions can be ambiguousor unreliable. For example, the integer size property in Figure 5 maps to replicas in Kubernetes’ StatefulSet. To map
primitive properties, Acto-□ analyzes operator code. The
idea is to track the data flow of the property value in theoperator code and analyze how the values are used. If a property value is passed to a Kubernetes API or assigned to a
Kubernetes resource object, Acto-□ maps the property to a
Kubernetes object that stores its value, as shown in Figure 5.
Acto-□ implements a static taint analysis to track property
values. The initial taints are pointers and references to thedesired-state declaration (e.g., cr.spec in Figure 5) and thetaints are propagated via data-flow dependencies. The analysis is field sensitive—to track each primitive (sub-)property
in the declaration—, inter-procedural and context sensitive.

SOSP ’23, October 23–26, 2023, Koblenz, Germany

5.2.3 Generating Property Values. To generate values
for properties with inferred semantics, Acto currently implements 57 property-specific generators based on Kubernetes
resource semantics. Most of these properties are composite.
The generators focus on high-level semantics to exercise
different scenarios (Table 3). Each generator creates property
values to realize a scenario. We find that most properties
exposed by operation interfaces (83% on average in our evaluated operators) can be mapped to Kubernetes resources.
Acto’s generators are invoked at runtime. Some generators
read environment and runtime information to inform value
generation (e.g., an unsatisfiable affinity rule).
For properties whose semantics Acto cannot infer, Actomutates current values based on their data types while satisfying syntactic constraints (§5.2.1). Acto only mutates primitive sub-properties of composite properties. Acto’s mutation
ensures syntactic validity but does not guarantee semantic
meaningfulness. Mutated values that are not semanticallymeaningful help check for vulnerabilities to misoperations.
Our manual inspection during Acto evaluation (§6) showsthat 80+% of mutations are semantically meaningful.
5.2.4 Satisfying Predicates. The values that Acto generates should satisfy predicates, in the form of property dependencies, for changed property values to trigger state transitions. For example, an operation that changes a backup policy
only triggers a state transition if backup is also turned on.
But, dependencies among properties are often not specified,so Acto automatically infers them.
Inferring dependencies from naming convention (Acto■). Property names that are exposed by the operation interface provide hints from which dependencies can be inferred. In Kubernetes, dependencies can be identified byfeature toggles—each composite property has a Boolean subproperty whose name contains “enabled”. For example, operations that change PCN/MongoOp’s backup policy must also
set Backup.Enabled to True. Acto-■ infers dependencies oneach property that uses this convention based on a breadthfirst search that iteratively collects feature toggles. We findthis simple heuristic to be effective—it captures 98.05% ofcontrol dependencies that we find. Not all dependencies are
identifiable from feature toggles, but we only find a small
number of other subtle dependencies.
Inferring dependencies using control-flow analysis(Acto-□). Acto-□ analyzes control-flow relationships amongprogram variables in operator code to detect dependencies
among property values that do not follow the “*enabled*”naming convention. This analysis is similar to those used forfinding dependencies among program inputs [43, 93].
dep
Property 𝑝 2 depends on property 𝑝 1 , i.e., 𝑝 1 ←−−𝑝 2 , if 𝑝 2
is only used when 𝑝 1 satisfies a predicate. Acto-□ searches
dep
for control dependencies, (𝑝 1, 𝜑, 𝑐) ←−−𝑝 2 , where 𝑐 is somevalue and 𝜑 is a predicate, e.g., an arithmetic, logic, string, orobject comparison. Specifically, if a predicate 𝜑 dominates a

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

sink statement of property 𝑝 2 and 𝜑 is not postdominated by
the sink, then there is a control-flow dependency between 𝜑and 𝑝 2 , i.e., 𝑝 2 is used only when 𝜑 is True. Sinks consume
property values, e.g., a call to an external API. Further, if
𝜑 is determined by comparing the value of 𝑝 1 with 𝑐, thendep
Acto-□ records a control dependency, (𝑝 1, 𝜑, 𝑐) ←−−𝑝 2 . If 𝑝 2
has multiple sinks, Acto-□ reports a control dependency,dep(𝑝 1, 𝜑, 𝑐) ←−−𝑝 2 , iff all sinks of 𝑝 2 depend on (𝑝 1, 𝜑, 𝑐).
5.3

Test Oracles

Acto’s oracles check whether the state to which the managed
system is reconciled matches the specified desired state. If
there is a match, Acto reports the operation as successful.
Otherwise, Acto signals an alarm that the user can inspect
to find bugs or vulnerabilities to misoperations.
The complexity of Acto’s oracles depends on whether
mismatches between reconciled and desired states manifest
explicitly or implicitly. Acto implements oracles to check forstate mismatches that manifest as explicit error states, such as
exceptions, error codes, and timeouts. These oracles 1) scanan operator’s error log for unexpected exceptions, e.g., thepanic signal in Go; 2) check runtime status of the managed
system (recorded in state objects); and 3) check whether anoperation returns an error code or fails to complete on time.
Acto’s oracles that check for explicit errors are insufficient:
many operator bugs manifest as implicit-state mismatcheswith no explicit symptoms. To find such bugs, Acto also implements oracles to check if 𝑆𝑖 |= 𝐷𝑖 for each state transition
𝐷𝑖
𝑆𝑖 −1 −−→ 𝑆𝑖 . Checking 𝑆𝑖 |= 𝐷𝑖 is challenging. First, 𝑆𝑖 and 𝐷𝑖
are represented differently: 𝐷𝑖 is a specification [3] and 𝑆𝑖
is embodied in state objects [17]. Second, satisfiability (|=)
is domain-specific; its semantics may not be obvious. Acto
uses two types of oracles to detect implicit-state mismatch:• Consistency oracle (§5.3.1). Acto checks whether 𝑆𝑖 |= 𝐷𝑖
from the operator and the management platform (e.g.,
Kubernetes) views. A buggy operator’s view may show
𝑆𝑖 |= 𝐷𝑖 while the management view shows 𝑆𝑖 ̸ |= 𝐷𝑖 . Suchview inconsistencies likely indicate the presence of bugs.
• Differential oracle (§5.3.2). This oracle leverages the leveltriggering principle [57] that operators should follow: thesame desired state should be reached from different start𝐷𝑖𝐷𝑖
states. So, for each transition pair, 𝑆𝑖 −1 −−→ 𝑆𝑖 and 𝑆 0 −−→ 𝑆𝑖′ ,Acto checks whether 𝑆𝑖 and 𝑆𝑖′ match after state reconciliation based on 𝐷𝑖 . This differential oracle also checks
whether the operator can recover from an error state, 𝑆 𝑒 , bychecking whether the system state after a rollback matches
𝑆𝑖 −1 , the preceding state before the error.
In addition to the automated built-in oracles, Acto also has aninterface to allow users to add custom oracles, e.g., domainspecific oracles to check managed systems.
5.3.1 Consistency Oracle. Some bugs occur if an operator stops reconciliation because the system is in state 𝑆𝑖 |= 𝐷Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

𝑆!"## Redis CR:
redisFollower:
pdb:
minAvailable: 2...

...
PodDisruptionBudget:
spec:
redis-follower:(pdb) null...

𝐷!

SOSP ’23, October 23–26, 2023, Koblenz, Germany𝐷!

The views of the pdb
property are inconsistent𝑆!

...
PodDisruptionBudget:
spec:
redis-follower:(pdb) null...

# Knative CR
ingress:
contour:enabled: false...

𝑆!"# ...

𝑆!

ContourPod: ...

𝑆$...
(no ContourPod)

𝑆!%

ContourPod exists

in 𝑆! but not in 𝑆!%...
ContourPod: ...
...
(no ContourPod)

Figure 6. An OCK/RedisOp bug detected by Acto’s consistency oracle [25]. The PodDisruptionBudget state object
has a null pdb, inconsistent with the pdb declared in 𝐷𝑖 .

Figure 7. A KnativeOp bug that is detected by Acto’s differential oracle [22]. Contour continues to manage ingress
after an operation explicitly disables it.

in the operator’s view, but 𝑆𝑖 ̸ |= 𝐷 in the management platform’s view. To detect such bugs, Acto additionally checks
whether the management platform’s view matches 𝐷, basedon the platform’s description of the reconciled state. In Kubernetes, the platform’s view is encoded in spec sections ofstate objects, which are jointly maintained by all running
controllers and operators.
𝐷𝑖
For each transition 𝑆𝑖 −1 −−→ 𝑆𝑖 , Acto attempts to match
each property 𝑝 (specified in 𝐷𝑖 ) to the corresponding spec
fields in the state objects. If a match is found, it indicates thatthe management platform agrees with the operator. Otherwise, Acto raises an alarm.
Figure 6 shows a bug detected by the consistency oracle.
OCK/RedisOp should reconcile the system to a declared statewith a pdb property for Redis followers (to ensure that replicas are available during managed disruptions [16]). But, theproperty in 𝐷𝑖 is not consistent with Kubernetes’ view of Redis followers, in which there is no pdb. The root cause is that
OCK/RedisOp was missing code to support pdb for followers,
risking the Redis availability during transient disruptions.
Such bugs are common due to the operation interface complexity, especially as software evolves [36].
Acto uses property structure analysis (§5.2.2) to infer correspondences between fields in the spec section of state objects
and a declared property. A declared property could match
fields in multiple state objects, but not every matched field isrelevant to the property. For example, PodDisruptionBudget
objects that are not used by Redis followers could also define
pdb. Acto uses the insight that state object changes occur insmall increments, because Acto changes a few properties at atime. So, Acto only matches a specified property to changed
fields. Acto raises an alarm if a matched field’s value is different from the declared property’s value, or if a propertychange does not cause any change to matched field values.

Figure 7 shows a bug detected by the differential oracle.
There, the Boolean KnativeOp property contour.enabled enables or disables Contour, an ingress controller. But, a KnativeOp bug makes it impossible to disable Contour after it
is enabled. The consistency oracle does not detect this bug:
it is hard to automatically map the Boolean property to theexistence of a Contour pod. The differential oracle detects
the bug because a Contour pod appears in 𝑆𝑖 , but not in 𝑆𝑖′ .
Comparison with a second transition that starts from initial state 𝑆 0 results from Acto’s exploration strategy (Figure 4d). Our choice of 𝑆 0 is justified by the fact that 𝑆 0 isalways a good state and it is used frequently in manually
written e2e tests (§3). Conceptually, Acto can compare witha second transition that starts from any good state.
Note that reporting alarms for any difference in the state
objects of 𝑆𝑖 and 𝑆𝑖′ would be brittle and lead to false positives, because execution-specific values like timestamps, IP
addresses, and ports may change nondeterministically. Acto
excludes execution-specific fields when comparing state objects. Acto automatically labels those fields by (1) running
𝐷1
the transition 𝑆 0 −−→ 𝑆 1 multiple times as a calibration andlabeling fields with values varying across runs, and (2) run𝐷𝑖ning 𝑆 0 −−→ 𝑆𝑖 multiple times, iff the differential oracle fires
an alarm on 𝑆𝑖 , to ensure relevant fields are deterministic.

5.3.2 Differential Oracle. The differential oracle does notcheck against 𝐷𝑖 ; it checks that an operator 1) reconciles tothe matching desired states from different states 𝑆𝑖 −1 and
𝑆 0 , and 2) recovers from (implicit or explicit) error state 𝑆 𝑒
to state 𝑆𝑖 −1 . Acto rolls back to 𝑆𝑖 −1 to continue explorationfrom a known good state.

5.4

Reproduction and Debugging

Acto generates minimized e2e test code for every alarm thatit raises. When a test fails (the system is in an error state 𝑆 𝑒 ),Acto records failure information (e.g., a dump of the error
state, log messages, and system status). Then, Acto rolls backto a valid state 𝑆𝑖 −1 and continues the test campaign.
To generate test code, Acto minimizes the operation sequence that reached 𝑆 𝑒 to only two operations, (𝑆 0, 𝐷𝑖 −1 )and (𝑆𝑖 −1, 𝐷𝑖 ). Here, (𝑆 0, 𝐷𝑖 −1 ) reconciles the system state to𝑆𝑖 −1 . Acto outputs the minimized sequence as an executablefunction that developers can include in their regression test
suite after fixing the bug. In our experience, the recorded
failure information suffices to effectively locate root causesof test failures. Since the minimized test code reliably reproduces the bug, interactive debuggers [4, 6] can also be used.
Acto users can suppress alarms by writing annotations.

SOSP ’23, October 23–26, 2023, Koblenz, Germany

5.5

Implementation

We implement Acto for Kubernetes operators. Acto-■ has
12,100 lines of Python code. Roughly 9,000 of those lines
implement generic test logic (e.g., input generation, test execution, and oracles). Kubernetes-specific semantic inference
and value generation take ∼2K lines. If new Kubernetes resources are introduced in the future, we will need to extend
Acto to add new value generators for the associated properties (§5.2.3). The remaining lines of Acto-■ code implement
utilities: environment setup, state analysis, etc. Acto-□ isbuilt on top of Acto-■ using an additional 5,700 lines of Go
code for program analysis. We currently support operators
written in Go, the most popular language among operators.
Acto runs tests on virtualized Kubernetes clusters. It supports
three backends, Kind [9], Minikube [12], and K3d [8].
Static analysis in Acto-□. We use ssa [15] which provides intra-procedural static single-assignment (SSA) representation. We use pointer [14] for alias analysis, which
implements the Andersen-style point-to analysis [33].
State convergence. Acto applies test oracles only after
the system state converges. Convergence time ranges fromone second to 10 minutes, so setting a fixed timer would beunreliable. Acto uses a reset timer to check for convergence—
it resets the timer when it observes a system event, until noevent occurs and the timer times out. We conservatively set
the timer to three times the system restart time.
Test parallelization. To speed up testing, Acto partitions
operation sequences, [(𝑆 0, 𝐷 1 ), (𝑆 1, 𝐷 2 ), ..., (𝑆𝑥 , 𝐷𝑥+1 )], into
multiple tests and runs them in parallel. To run three partitions of this sequence in parallel, Acto creates three tests cor𝐷𝑖
𝐷1
𝐷2
𝐷𝑖
𝐷𝑖+1
responding to 1) 𝑆 0 −−→ 𝑆 1 −−→, ..., −−→ 𝑆𝑖 , 2) 𝑆 0 −−→ 𝑆𝑖 −−−→
𝐷𝑛
𝐷𝑛
𝐷𝑛+1
𝐷𝑥
, ..., −−→ 𝑆𝑛 , and 3) 𝑆 0 −−→ 𝑆𝑛 −−−→, ..., −−→ 𝑆𝑥 . If 𝑆𝑖 is an error
state, it is “rolled back” based on 𝐷𝑖 −1 . Acto can run multiple
test partitions on one machine, each in a virtualized Kubernetes cluster with a separate namespace. This approach saves
time as test runs wait for convergence. Acto keeps container
file systems in memory to reduce the image loading time.

6

Evaluation

Acto’s premise is that fully automatic end-to-end correctness
testing for unmodified operators is viable and effective. We
answer three research questions: (1) Can Acto effectively
find new bugs in real-world operators? (2) How efficient isActo? (3) Are Acto’s signaled alarms trustworthy?
We apply Acto to eleven popular open-source Kubernetes
operators which manage nine cloud systems (Table 4). Allevaluated operators are developed by the official teams of the
managed systems, or by companies that sell services built
around the managed systems. Test suites in the evaluatedoperators have similar characteristics as those in §3.
Our main evaluation results are summarized as follows:• Acto finds 56 new bugs in eleven operators; 42 bugs in theoperators have been confirmed; 30 have been fixed. Acto

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

Table 4. The Kubernetes operators that we evaluate.
Operator

SystemDev.

CassOpCassandra
K8ssandra
CockroachOp CockroachDB Official
KnativeOp
Knative
Official
OCK/RedisOp Redis
OCK
OFC/MongoOp MongoDB
Official
PCN/MongoOp MongoDB
Percona
RabbitMQOp RabbitMQ
Official
SAH/RedisOp Redis
Spotahome
TiDBOpTiDB
Official
XtraDBOp
XtraDB
Percona
ZooKeeperOp ZooKeeper
Pravega# Stars LOC # E2E Tests
148
238
157
531
977
268
669
1303
1130
448
332

23.1K
17.4K
16.3K
2.5K
17.1K
15.0K
14.7K
10.5K
132.8K
15.5K
5.5K

48
21
7
0
62
31
8
1
131
37
8

also finds six bugs in Kubernetes and in the Go runtime
that affect multiple operators; all were confirmed or fixed.
• Acto’s test campaigns take less than eight hours per operator on a cluster of eight machines (a nightly run). Five of
eleven operators only need one machine.
• Acto generates few false positives: Acto-□ reports no false
alarms and Acto-■ has a very low false alarm rate: 0.19%.
6.1

Finding New Bugs and Vulnerabilities

Acto finds previously unknown bugs in all evaluated operators, 56 bugs in total (Table 5). We reported all these bugs.
So far, 42 were confirmed and 30 have been fixed. No bug
report was rejected. Acto-□ found all 56 bugs. Acto-■ missed
one bug, due to not being able to infer the semantics of aprimitive property that is needed to generate a scenario.
Acto generates e2e tests to reproduce all 56 bugs that it
detects; developers can add these e2e tests to their regression
test suite (§5.4). In fact, for six bug fixes, developers added
regression tests that perform the same state transition generated by Acto. Our experience tells that the generated e2etests are invaluable for debugging and validating bug fixes.
Many bugs detected by Acto have severe consequences:managed-system failures, reliability issues, and security issues (Table 6). Estimating the likelihood of encountering
each bug “in the field” is hard—the data for such estimation
is not publicly available. However, a bug detected by Acto
was also encountered by a real user after we reported it [19].
Also, some previously reported bugs are similar to those that
Acto detects (e.g., [20]). Note that the evaluated operators are
popular open-source projects (GitHub “#Stars” in Table 4),suggesting that operator correctness is hard to achieve.
Acto also finds six bugs in Kubernetes and in the Go runtime that affect multiple operators. These bugs cause wrong
or imprecise quantity conversions [26], incompatibility between declarations and API-server validation [18], crashes
due to Go’s generated shared object [21], etc. All these six
bugs were confirmed or fixed after we reported them.
Acto also detects 630 misoperation vulnerabilities (§6.1.2).
Each vulnerability corresponds to a unique misoperationthat drives the managed system into an error state.

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

Table 5. New bugs detected by Acto-□ (Acto-■) in theevaluated operators. Acto also detected six new bugs in
Kubernetes and Go runtime that affect multiple operators.

Table 6. Consequences of the 56 detected bugs in Table 5. One bug can have multiple consequences.
Consequence

Operator

Undesired
Error State
RecoveryState
System Operator Failure

Total

CassOpCockroachOp
KnativeOp
OCK/RedisOp
OFC/MongoOp
PCN/MongoOp
RabbitMQOp
SAH/RedisOp
TiDBOp
XtraDBOp
ZooKeeperOp

2
3
1
4
3
4
3
2
2
4
4

0
0
0
0
1
0
0
1
1
0
1 (0)

0
2
2
3
2
0
0
0
0
1
0

2
0
0
1
2
1
0
1
1
1
1

4
5
3
8
8
5
3
4
4
6
6 (5)

Total

32

4 (3)

10

10

56 (55)6.1.1 Bugs Detected by Acto. Acto detects bugs that violate the first two operation correctness requirements: (1)driving managed systems into undesired or error states, or(2) failing to recover from error states.
Undesired state. Acto found 32 bugs, where an operator
does not reach the desired state, but neither the operator
nor the managed system reports errors explicitly. The consequences of these bugs are latent and hard to observe (e.g., security vulnerabilities). These bugs have different root causes
in code, but a common theme is that the operator stops reconciliation before the desired state is reached. We showed
two such bugs in Figures 6 and 7. These bugs show the importance of modeling operations as state transitions and testing
different state transitions to the same declared states (§4.1).
Error state. Acto found 14 bugs that result in runtime errors
or crashes of the managed system or the operator. Amongthese, four bugs caused runtime errors in the managed systems (such as the one in Figure 1). In another example [31],when testing TiDBOp, Acto generates a valid operation thatturns on binlog to replicate data using the TiDB binlog. However, TiDB binlog requires a pump cluster to record and sortbinlogs, which is not set up by TiDBOp. So, TiDBOp restarts
TiDB nodes to load the new configuration, but the replicas
crash because of the missing pump cluster.
Acto also found ten bugs that caused operator failure.
For example, CockroachOp crashed due to an “index-outof-range” error when parsing a valid state declaration generated by Acto [27]. The crash brought down the webhook
service [5] that the operator uses to validate declarations. After restart, CockroachOp crashed again due to the offending
declaration and it got stuck in a crash-then-restart loop.
Recovery failure. Acto detected ten bugs that lead to serious liveness issues (e.g., permanent operator failures) thatcan neither be addressed by restarting the operator nor byissuing new operations (such as the one in Figure 2). Acto

SOSP ’23, October 23–26, 2023, Koblenz, Germany

ExampleSystem failure
MongoDB is down and cannot recover [23]Reliability issue Redis is not protected by disruption budget [25]
Security issue
CockroachDB uses outdated secrets [29]Resource issue
Redis runs with no resource guarantee [24]Operation outage CockroachOp crashes and cannot recover [27]Misconfiguration Ingress controller cannot be disabled [22]# Bugs
5
15
2
9
18
15

detected these bugs by testing rollback operations with thedifferential oracle. Our investigation reveals a common coding practice: operators perform new operations only after
the system is in a stable state. This practice is a double-edged
sword: it prevents bugs caused by racing operations andreduces risks during upgrade, but it makes failure recovery
difficult, because it also blocks rollback operations if thesystem is in an error state.
6.1.2 Misoperation Vulnerabilities Detected by Acto.
Acto-□ detects 630 misoperation vulnerabilities that violate
the third operation correctness requirement (Acto-■ detects
616 of these 630). Each vulnerability corresponds to a uniqueproperty. Acto detects these vulnerabilities by generating declared states with unsatisfiable affinity rules, misconfigured
security contexts, unavailable resources, etc. (Table 3). All
these vulnerabilities can lead to severe consequences including entire system failure, partial service failures, and reliability issues. In practice, the triggering misoperations could
result from human mistakes or wrong policies. These results
show that operator developers do not anticipate and defend
well against misoperations, which are frequently reported
as major causes of system failures [37, 38, 51, 68–70, 93].
We actively discuss with developers on potential mitigation (e.g., by more rigorous early checks). In practice, some ofthese vulnerabilities are difficult to prevent. The reason liesin the challenges of encoding sufficient domain knowledge inoperators to check the semantics of requested operations. For
example, it is hard to replicate Kubernetes core scheduler’s
complex logic [82]. Checking some misoperations requires
knowledge of managed systems. State rollback can be an effective mitigation strategy, but it does not always work—over
35% of 630 misoperation vulnerabilities cannot be mitigatedby rollbacks due to the recovery-failure bugs in §6.1.1.
6.1.3 Effectiveness of Different Oracles. Acto’s consistency and differential oracles catch 43 of the 56 bugs (Table 7).
The consistency oracle detects 23 bugs by matching and comparing properties in state declarations to the spec sections
in state objects (§5.3.1). The differential oracle catches ten
more bugs that are triggered during normal state transitions.
It also catches all ten recovery-failure bugs during rollback
state transitions. The regular error checks detect 14 bugs

SOSP ’23, October 23–26, 2023, Koblenz, Germany

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

Table 7. Breakdown of the number of bugs detected by
the oracles. Same bug can be detected by multiple oracles.
Test Oracle
Consistency oracle
Differential oracle for normal state transition
Differential oracle for rollback state transition
Regular error check (e.g., exception, error code)# Bugs (Percentage)
23 (41.07%)
25 (44.64%)
10 (17.86%)
14 (25.00%)

by checking process status of the operator and runtime status of the managed system (recorded in the state objects).
Compared with state-based assertions in existing tests that
only cover 0.24%–10.9% of state-object fields (Table 2), Acto’s
oracles systematically check all related fields. For example,
the differential oracle compares all state-object fields thatare deterministic (71.4%–80.5% of all fields across evaluatedoperators) through different transitions to the same end state.
6.1.4 Coverage. Acto achieves 100% property coveragefor every operator—Acto generates at least one operationfor each property (§5.1). Acto’s effectiveness over manuallywritten tests (§3) comes from its ability to cover more properties and their values, and more transitions from different
states (including error states). In 38 of 56 detected bugs, therelated property is uncovered by existing tests. Relevant properties for the other 18 bugs are covered, but these bugs elude
existing tests because a revealing transition is not exercised.
For example, in CassOp, existing tests check that labels [11]are correctly added to pods, but Acto detects a bug [28] thatcan only be triggered when pod labels are deleted.
6.1.5 Bug Fixes. We reported all 56 bugs that Acto finds tothe developers of the respective operators; 42 have been confirmed and 30 of those have been fixed. Developers typically
fix these reported bugs by improving reconciliation logic forthe bug-triggering transitions generated by Acto, and adding
validation logic before reconciling on each state declaration
to prevent error conditions. Fixing bugs in failure-recovery
logic usually requires more effort, because it needs domain
knowledge to differentiate permanent error states from transient unstable states. For example, the bug in Figure 2 hasbeen confirmed, but the developers cannot easily fix it because the operator cannot reliably detect liveness violations—the pod migration will never succeed in the future—by observing the current state.
6.1.6 Tradeoffs between Acto-■ and Acto-□. We expect Acto-□ to be more beneficial than Acto-■ for operatorsthat heavily use primitive-typed properties or do not follow
naming conventions for property dependencies. In the evaluated operators, most properties have composite type withclear structure features and they follow naming conventions.
Hence, the benefit of Acto-□ over Acto-■ is small in our evaluation. Note that Acto-□ is language specific—it currently
only supports operators written in Go. Acto-■ is language

Table 8. Acto-□ test campaign time per operator.
Operator
CassOpCockroachOp
KnativeOp
OCK/RedisOp
OFC/MongoOp
PCN/MongoOp
RabbitMQOp
SAH/RedisOp
TiDBOp
XtraDBOp
ZooKeeperOp

Testing Time (Machine Hours)# Ops # Workers
Generation Execution Total
0.02
0.02
0.04
0.02
0.01
0.04
0.03
0.02
0.03
0.03
0.02

10.39
6.08
6.25
9.72
5.73
26.55
4.69
7.92
16.08
57.48
8.54

10.41
6.10
6.29
9.75
5.74
26.58
4.72
7.94
16.11
57.51
8.55

568
371
774
597
434
1749
394
718
824
1950
740

16
16
16
16
16
12
16
16
12
8
16

agnostic and can apply to operators written in languages
other than Go, and proprietary, close-sourced operators.
6.2

Test Efficiency

Table 8 shows machine hours Acto-□’s test campaigns takeper operator and the number of operations in each test campaign (“#Ops”). The longest campaign (XtraDBOp) had 1,950
operations. Acto stops generating operations when a campaign covers all properties and corresponding scenarios.
All experiments are run on Cloudlab [48] Clemson c6420
machines with 2 Intel Xeon Gold 6142 CPUs (16 cores) and
376 GB of memory, with Ubuntu 20.04 LTS. Campaign times
vary from 4.72 to 57.51 hours across operators. Using eightmachines, test campaigns for all operators finish in less thaneight hours. So, Acto-□ can be run nightly.
Acto’s efficiency comes from test parallelization (§5.5). By
default, Acto spawns 16 parallel workers to run tests on eachmachine. But, parallelism can be reduced if the operator or
the managed system requires more resources (e.g., memory).
Semantic analysis for composite properties (§5.2.2) drastically reduces the number of operations in test campaigns andallows Acto to focus on high-level semantics of composite
properties to exercise representative scenarios, rather thanmutating fine-grained primitive (sub-)properties.
Acto-■ takes 8.47% less time on average than Acto-□ because it generates, on average, 48 fewer test operations per
operator than Acto-□. The reason is that Acto-■ cannot infer semantics for a few primitive properties and thus cannot
generate operations for several scenarios; it only mutatescurrent values within the constraints of a property (§5.2.3).
6.3

False Positives

Acto’s alarms have a low false positive rate. Acto-□ reports
no false alarm. Every test failure during the test campaigns
points to either a bug in the operator code or a misoperation
vulnerability. In total, Acto-□ reports 2243 test failures: 738
test failures are caused by the 56 bugs in the operator and six
bugs in Kubernetes and Go runtime, and 1505 test failures arecaused by 630 misoperation vulnerabilities. Fixing one bug

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

or vulnerability may resolve multiple test failures. We areautomating alarm clustering based on fault localization [72,88], but it is now beyond the scope of testing.
Acto-■ reports four false alarms in total. It reports 2071
test failures in total; among them, 653 test failures are caused
by 55 bugs in operators and six bugs in Kubernetes or Go;
1414 test failures are caused by 616 misoperation vulnerabilities. Therefore, the overall false positive rate of Acto-■
is 0.19%, or 4 out of 2071 alarms. All four false alarms arecaused by unsatisfied predicates when Acto-■ changes properties. As discussed in §5.2.4, Acto-■ is unable to infer dependencies that do not follow the naming convention. For
example, in ZooKeeperOp, the property, ephemeral, depends
on a predicate: another property, storageType, must also be
set to “ephemeral”. Hence, Acto-■ fails to satisfy the predicate when changing the ephemeral property, but it expects astate change and raises a false alarm. These dependencies arecaptured by Acto-□ through control-flow analysis (§5.2.4).
6.4

Implications and Discussion

We reflect on our experience on finding root causes of detected bugs and vulnerabilities, and discuss implications.
Operation coverage. It is nontrivial to validate operators
under the declarative model. A key challenge is to reach desired states from many different start states (including error
states). We observe that operators invoke different imperative procedures, based on how a declared state differs fromthe current state. However, it can be tedious and error-prone
to cover all such conditions. In fact, most bugs that Actofinds do not manifest when performing operations from theinitial state 𝑆 0 . Operations from the initial state are likely
already validated by developers manually or by writing tests.
Modeling and testing diverse state transitions are critical tovalidating operation correctness (§4.1). Declarative programming [82] may make operator testing less error-prone.
As for testing, Acto uses property coverage to drive statetransitions in the test campaigns (§5.1). The rationale is toachieve high coverage of desired states, as state transitions
are triggered by changing property values via the operationinterface. Traditional coverage metrics like code coverage
are insufficient because they are not concerned with system
states: tests that are adequate for the code in one state may
not be adequate in a different state. Code coverage may nothelp test all properties either, e.g., an operator that is missing
code to handle transition-triggering property changes could
have high code coverage. Acto can find bugs due to missing
code if the end state does not match the desired state.
Reducing risks. Operations can pose new reliability risks tomanaged systems—what happens if an operation fails duringexecution? An operation can span a series of procedures. For
example, we observe that existing Kubernetes operators commonly implement reconfiguration operations in two stages:

SOSP ’23, October 23–26, 2023, Koblenz, Germany(1) stopping the current running node (with the old configuration); then (2) starting a new node (with new configuration).
In such implementations, failure in either stage is risky. First,
such a failure could leave the operator in intermediate states
which are nontrivial to recover from [80, 81]. Acto’s results
show that recovery failures are common (§6.1.1). Second, insuch implementations, the first step can open a small window of downtime (e.g., due to stopping the current node).
That downtime would be magnified if a new node fails tostart. So, it is safer to turn down the old node after the newnode starts successfully. But, in practice, this safe start ordercan be hard to implement, due to the semantic requirementsof the managed system and version incompatibility of thechanges [64, 96]. For example, a ZooKeeper cluster cannot
have two leaders at the same time, to avoid a split brain. So, areconfiguration operation must first stop the old leader node
before starting the new one to avoid a split brain. System
support for speculative execution or emulation can help.
Closing the knowledge gaps. Operations must also respect the constraints of the managed system. Otherwise, anoperation can harm the managed system. The TiDBOp bug
described in §6.1.1 is one example. Also, many vulnerabilities
to misoperations that Acto detects are rooted in the essential
cross-system interaction challenge [84]—it is hard for an
operator to comprehensively check a requested operation’s
semantic validity if the semantics are not defined inside theoperator code but in the managed system or the underlying
management framework (e.g., Kubernetes). One potential
solution is to replicate the validity checks of the relevant components in the operator. (Prior work showed the promiseof automatically extracting configuration checks [92].) In
essence, the knowledge gap lies in the fact that operator developers may not be the managed-system developers, or they
may not be aware of subtle, complex constraints. Since operation correctness should be a first-class concern in reliable
system design, a rigorous interface between the operator
and the managed systems is needed to close these gaps.

7

Limitations and Future Work

Acto is a first step towards thorough validation of operation
correctness for modern cloud systems. Like any testing technique, Acto is incomplete and it can miss bugs. Acto does
not cover all possible system states and transitions; doing
so is prohibitively expensive. For example, Acto generates
property values to cover a few representative scenarios. This
design aims to balance efficiency and coverage—it covers
each property at least once and it exercises diverse scenarios
based on the semantics of operations. The results are promising, but there is plenty of room for future work to improve
Acto’s state-space exploration and its efficiency.
Acto has other soundness and completeness issues. First,
the predicate analysis of Acto-■ is incomplete, resulting infalse alarms; Acto-□’s control-flow analysis only captures

SOSP ’23, October 23–26, 2023, Koblenz, Germany

predicates that manifest as control-flow dependencies (wedid not observe any other kind). Moreover, Acto’s automatedoracles do not incorporate domain knowledge about managed systems and they rely only on state objects managed
by the platform. Hence, the current oracles may not capturecomplex and subtle failure states that are not reflected in state
objects, such as loss of writes, linearizability violations, andgray or partial failures [34, 54, 58, 63]. We design Acto as anextensible and “push-button” testing utility for unmodified
operators, while also enabling users to add domain-specificoracles that have stronger managed-system observability.
The state-centric testing principle that Acto leverages may
apply to generic distributed systems, to capture issues related to operation assumptions. The challenges would be
to automatically validate system states and to synthesize
state transitions for arbitrary systems. Systems with clearlydefined protocols or models may be more amenable.
Some types of bugs can only be triggered by external faults
like node failures and network delays [44, 80]; a reliable operator must tolerate common faults. Acto does not target
those fault-tolerance bugs. Our prior work, Sieve [80], detects those bugs by injecting faults and checking operator
safety and liveness. We plan to integrate Sieve with Acto:(1) Sieve’s inputs are end-to-end tests; Acto could be used
to systematically generate these tests to make fault injection more comprehensive, and (2) Sieve can generate diverse
error states for Acto to test operator recovery. The key challenge is to efficiently navigate the combination of the input
operation space and the fault space (each space is very large).
Lastly, Acto currently focuses on testing individual operators. But, a system may be managed by multiple operatorsin practice. So, operation correctness could be violated byconflicting operations from different operators. We plan toextend Acto to test interdependent operators together. A key
challenge will be to address a larger state space and to reason
about state transitions in interleaving operation schedules.

8

Related Work

Prior work identified operation errors as major causes of
production failures [35, 37, 38, 51, 53, 68–71]; they result
mostly from human mistakes. As human-based operations
are increasingly being replaced by automated operation programs, the correctness of those programs is critical. Acto is a
first step towards automatic testing of operation correctness.
We believe that Acto’s ideas can apply beyond Kubernetes to other cloud platforms like Twine [83], ECS [67],and Borg [90]. These platforms also adopt declarative, statereconciliation patterns for operators or controllers, as a resultof many design iterations [39] and discussions [46, 74].
DCM [82] uses declarative programming to synthesize
cluster managers based on constraint solving; the idea canpotentially be extended for custom operators. However, mostoperators are currently written in imperative code.

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu

Acto is complementary to prior work on software deployment [45, 62, 73, 91, 96] and configuration [64, 79, 92–95].
Acto checks programs that perform those operations rather
than the correctness of code or configuration changes.
Acto can potentially be enhanced with ideas from symbolic execution [40] and fuzzing [66]. But, naïve applicationof these techniques is unlikely to yield benefits. For example,
without reasoning about state transition, techniques only
guided by code coverage will be insufficient (§6.4).
Sieve [80] is a closely related testing technique. It finds
bugs in Kubernetes controllers that are triggered by external faults like node failures, network delays, etc. Operators
are custom controllers for managing systems atop the Kubernetes platform. Acto is fundamentally different from, but
complementary to Sieve. In essence, Sieve is a fault injector
that checks fault tolerance, while Acto is an end-to-end test
generator that checks functional correctness. Sieve cannot
find the bugs Acto detects, because it assumes that the operator works correctly without faults. Sieve detects bugs bycomparing operator executions with and without injectedfaults. Sieve does not report errors in any fault-free reference execution. More importantly, Sieve takes test workloads
as input—those test workloads are currently written manually, but it is challenging and costly for developers to write
comprehensive test workloads (see §3). Acto automaticallygenerates test workloads (i.e., “test campaigns” in Acto’s
terminology). Conversely, Acto cannot directly detect bugs
that Sieve finds, because Acto does not inject external faults.
We discuss potential Acto and Sieve integration in §7.

9Concluding Remarks

With the rapidly growing practice of automating operations
and deploying operators in production, operator correctness
has become a critical component of cloud system reliability.
This paper presents Acto, an automatic technique for testing
cloud-native operators end to end with the managed systems.
We show that Acto’s state-centric approach enables effective
and practical end-to-end testing that is readily applicable toexisting operators and complements the significant inadequacy of manually written tests. Our goal now is to makeActo a common utility in developing and testing operators,
towards correct automation of cloud system operations.

Acknowledgement
We thank our shepherd, Rebecca Isaacs, and the anonymous
reviewers for their insightful comments. We thank Paolo
Dettori, Kai-Hsun (Kevin) Chen, Kuan-Yin Chen, Kunle Li,
Muhammad Taha, Jiajun (Kashun) Cheng, and Thrivikraman
Varadharajan for their participation and contributions to theActo project. We thank all the Kubernetes operator developers who engaged with us and reviewed our reports. This
work was funded by an IIDAI grant and NSF grants CNS2130560, CNS-2145295, CCF-2019277, and CCF-2045596.

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management

References[1] Assigning Pods to Nodes. https://kubernetes.io/docs/concepts/
scheduling-eviction/assign-pod-node/.
[2] Cloud Native Computing Foundation Operator White Paper.
https://www.cncf.io/wp-content/uploads/2021/07/CNCF_Operator_
WhitePaper.pdf.
https://kubernetes.io/docs/concepts/extend[3] Custom Resources.
kubernetes/api-extension/custom-resources/.
[4] Debugging Go Code with GDB. https://go.dev/doc/gdb.
[5] Dynamic Admission Control. https://kubernetes.io/docs/reference/
access-authn-authz/extensible-admission-controllers/.
[6] Ephemeral Containers.
https://kubernetes.io/docs/concepts/
workloads/pods/ephemeral-containers/.
[7] etcd. https://etcd.io/.
[8] K3d. https://github.com/k3d-io/k3d.
[9] Kind. https://kind.sigs.k8s.io/.
[10] Kubernetes End-to-end Testing for Everyone. https://kubernetes.io/
blog/2019/03/22/kubernetes-end-to-end-testing-for-everyone/.
[11] Labels and Selectors. https://kubernetes.io/docs/concepts/overview/
working-with-objects/labels/.
[12] Minikube. https://minikube.sigs.k8s.io/.
[13] OpenAPI Specification. https://swagger.io/specification/#schemaobject.
[14] Package pointer. https://pkg.go.dev/golang.org/x/tools/go/pointer.
[15] Package ssa. https://pkg.go.dev/golang.org/x/tools/go/ssa.
[16] Specifying a Disruption Budget for your Application. https://kubernetes.io/docs/tasks/run-application/configure-pdb/.
[17] Understanding Kubernetes Objects. https://kubernetes.io/docs/
concepts/overview/working-with-objects/kubernetes-objects/.
[18] Automatically generated regex validation for Quantity does not matchthe validation used by unmarshalerDecoder. https://github.com/
kubernetes-sigs/controller-tools/issues/665, 2022.
[19] Cassandra operator becomes partially inoperable if replaceNodes has awrong pod name (issue comment). https://github.com/k8ssandra/cassoperator/issues/315#issuecomment-1090149844, 2022.
[20] CLOUDP-116155 Initial bootup with arbiters. https://github.com/
mongodb/mongodb-kubernetes-operator/pull/1024, 2022.
[21] cmd/cgo: allow cgo to pass strings or []bytes bigger than 1«30. https://go-review.googlesource.com/c/go/+/418557, 2022.
[22] Contour pod is not deleted when disabled by user. https://github.com/
knative/operator/pull/1176, 2022.
[23] Mongodb system is down and unable to recover when the featureCompatibilityVersion is not specified and changed to an invalid
value. https://github.com/mongodb/mongodb-kubernetes-operator/
pull/1118, 2022.
[24] Redis does not run with resource request/limit set by cr.spec.resources.
https://github.com/OT-CONTAINER-KIT/redis-operator/issues/290,
2022.
[25] Specifying the field redisFollower.pdb does not have any effect. https://github.com/OT-CONTAINER-KIT/redis-operator/pull/301, 2022.
[26] The number conversion of Value() of type Quantity is incorrect. https://github.com/kubernetes/kubernetes/issues/110653, 2022.
[27] The operator crashes if the image name does not contain colon. https://github.com/cockroachdb/cockroach-operator/pull/922, 2022.
[28] Unable to remove the additional labels on the seed service throughCR. https://github.com/k8ssandra/cass-operator/pull/344, 2022.
[29] Updating the field spec.ingress.sql.tls.secretName is not reflected in
the sql ingress object. https://github.com/cockroachdb/cockroachoperator/issues/920, 2022.
[30] Zookeeper pod keeps crashing when scaling down and up. https://github.com/pravega/zookeeper-operator/pull/526, 2022.
[31] TiDB crash loop when enabling binlog. https://github.com/pingcap/
tidb-operator/issues/4945, 2023.

SOSP ’23, October 23–26, 2023, Koblenz, Germany[32] TiDB operator unable to recover an unhealthy cluster even withmanual revert. https://github.com/pingcap/tidb-operator/issues/4946,
2023.
[33] Andersen, L. O. Program Analysis and Specialization for the C Programming Language. PhD thesis, DIKU, University of Copenhagen,May 1994.
[34] Arpaci-Dusseau, R. H., and Arpaci-Dusseau, A. C. Fail-Stutter Fault
Tolerance. In Proceedings of the 8th Workshop on Hot Topics in Operating
Systems (HotOS-VIII) (May 2001).
[35] Barroso, L. A., Hölzle, U., and Ranganathan, P. The Datacenter asa Computer: Designing Warehouse-Scale Machines, 3 ed. Morgan and
Claypool Publishers, 2018.
[36] Behrang, F., Cohen, M. B., and Orso, A. Users Beware: Preference
Inconsistencies Ahead. In Proceedings of the 10th Joint Meeting of
the European Software Engineering Conference and the ACM SIGSOFT
Symposium on the Foundations of Software Engineering (ESEC/FSE’15)(Aug. 2015).
[37] Bianchini, R., Martin, R. P., Nagaraja, K., Nguyen, T. D., andOliveira, F. Human-Aware Computer System Design. In Proceedingsof the 10th Workshop on Hot Topics in Operating Systems (HotOS-X)(June 2005).
[38] Brown, A. B., and Patterson, D. A. Undo for Operators: Building
an Undoable E-mail Store. In Proceedings of the 2003 USENIX Annual
Technical Conference (ATC’03) (June 2003).
[39] Burns, B., Grant, B., Oppenheimer, D., Brewer, E., and Wilkes, J.
Borg, Omega, and Kubernetes. Communications of the ACM 59, 5 (May2016), 50–57.
[40] Cadar, C., and Sen, K. Symbolic Execution For Software Testing:Three Decades Later. Communications of the ACM 56, 2 (Feb. 2013),
82–90.
[41] Cebula, M., and Sherrod, B. 10 Weird Ways to Blow Up Your Kubernetes. In KubeCon North America (Nov. 2019).
[42] Chekrygin, I. Keep the Space Shuttle Flying: Writing Robust Operators. In KubeCon Europe (May 2019).
[43] Chen, Q., Wang, T., Legunsen, O., Li, S., and Xu, T. Understanding
and Discovering Software Configuration Dependencies in Cloud andDatacenter Systems. In Proceedings of the 2020 ACM Joint European
Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering (ESEC/FSE’20) (Nov. 2020).
[44] Chen, Y., Sun, X., Nath, S., Yang, Z., and Xu, T. Push-Button Reliability Testing for Cloud-Backed Applications with Rainmaker. In
Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI’23) (Apr. 2023).
[45] Crameri, O., Knežević, N., Kostić, D., Bianchini, R., andZwaenepoel:, W. Staged Deployment in Mirage, an Integrated Software Upgrade Testing and Distribution System. In Proceedings of the
21st Symposium on Operating System Principles (SOSP’07) (Oct. 2007).
[46] DeTreville, J. Making System Configuration More Declarative. In
Proceedings of the 10th Workshop on Hot Topics in Operating Systems(HotOS-X) (June 2005).
[47] Dobies, J., and Wood, J. Kubernetes Operators: Automating the Container Orchestration Platform. O’Reilly Media, Inc., 2020.
[48] Duplyakin, D., Ricci, R., Maricq, A., Wong, G., Duerig, J., Eide, E.,
Stoller, L., Hibler, M., Johnson, D., Webb, K., Akella, A., Wang,
K., Ricart, G., Landweber, L., Elliott, C., Zink, M., Cecchet, E.,
Kar, S., and Mishra, P. The Design and Operation of CloudLab. In
Proceedings of the 2019 USENIX Annual Technical Conference (ATC’19)(July 2019).
[49] Flemström, D., and Buck, A.
Fleet Management at
Spotify (Part 2): The Path to Declarative Infrastructure.
https://engineering.atspotify.com/2023/05/fleet-managementat-spotify-part-2-the-path-to-declarative-infrastructure/, May 2023.
Shopify Engineering Blog.

SOSP ’23, October 23–26, 2023, Koblenz, Germany[50] Gao, L., and Menon, R. Scaling Apache Spark on Kubernetes atLyft. https://www.youtube.com/watch?v=PPtrY_XxYBE, Apr. 2019.
Spark+AI Summit.
[51] Gray, J. Why Do Computers Stop and What Can Be Done About It?
Tandem Technical Report 85.7 (June 1985).
[52] Guilloux, S. Writing a Kubernetes Operator: the Hard Parts. In
KubeCon North America (Nov. 2019).
[53] Gunawi, H. S., Hao, M., Suminto, R. O., Laksono, A., Satria, A. D.,
Adityatama, J., and Eliazar, K. J. Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages. In Proceedings ofthe 7th ACM Symposium on Cloud Computing (SOCC’16) (Oct. 2016).
[54] Gunawi, H. S., Suminto, R. O., Sears, R., Golliher, C., Sundararaman, S., Lin, X., Emami, T., Sheng, W., Bidokhti, N., McCaffrey,C., Srinivasan, D., Panda, B., Baptist, A., Grider, G., Fields, P. M.,
Harms, K., Ross, R. B., Jacobson, A., Ricci, R., Webb, K., Alvaro, P.,
Runesha, H. B., Hao, M., and Li, H. Fail-Slow at Scale: Evidence ofHardware Performance Faults in Large Production Systems. In Proceedings of the 16th USENIX Conference on File and Storage Technologies(FAST’18) (Feb. 2018).
[55] Haase, S. How an Operator Becomes the Hero of the Edge. In OperatorCon (May 2019).
AWS, Google, Microsoft, Red Hat’s New Reg[56] Hall, C.
istry to Act as Clearing House for Kubernetes Operators.
https://www.datacenterknowledge.com/open-source/aws-googlemicrosoft-red-hats-new-registry-act-clearing-house-kubernetesoperators, Mar. 2019.
[57] Hockin, T. Kubernetes: Edge vs. Level Triggered Logic. https://speakerdeck.com/thockin/edge-vs-level-triggered-logic, June 2017.
[58] Huang, P., Guo, C., Zhou, L., Lorch, J. R., Dang, Y., Chintalapati, M.,
and Yao, R. Gray Failure: The Achilles’ Heel of Cloud-Scale Systems.
In Proceedings of the 16th Workshop on Hot Topics in Operating Systems(HotOS-XVI) (May 2017).
[59] Kumar, H., and Šafránek, J. Storage on Kubernetes - Learning From
Failures. In KubeCon North America (Nov. 2019).
[60] Lagresle, M. Moving to Kubernetes: the Bad and the Ugly. In ContainerDays (June 2019).
[61] Lander, R.
Kubernetes Operators: Should You Use Them?
https://tanzu.vmware.com/developer/blog/kubernetes-operatorsshould-you-use-them/, July 2021. VMware Blog.
[62] Li, Z., Cheng, Q., Hsieh, K., Dang, Y., Huang, P., Singh, P., Yang,
X., Lin, Q., Wu, Y., Levy, S., and Chintalapati, M. Gandalf: AnIntelligent, End-To-End Analytics Service for Safe Deployment inLarge-Scale Cloud Infrastructure. In Proceedings of the 17th USENIX
Symposium on Networked Systems Design and Implementation (NSDI’20)(Feb. 2020).
[63] Lou, C., Huang, P., and Smith, S. Understanding, Detecting and
Localizing Partial Failures in Large System Software. In Proceedings ofthe 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI’20) (Feb. 2020).
[64] Ma, S., Zhou, F., Bond, M. D., and Wang, Y. Finding HeterogeneousUnsafe Configuration Parameters in Cloud Systems. In Proceedings ofthe 16th ACM European Conference on Computer Systems (EuroSys’21)(Apr. 2021).
[65] Madhu, C. Preventing Controller Sprawl From Taking Down Your
Cluster. In KubeCon North America (Oct. 2022).
[66] Manes, V. J., Han, H., Han, C., Cha, S. K., Egele, M., Schwartz,
E. J., and Woo, M. The Art, Science, and Engineering of Fuzzing: A
Survey. IEEE Transactions on Software Engineering 47, 11 (Nov. 2021),
2312–2331.
[67] Melissaris, T., Nabar, K., Radut, R., Rehmtulla, S., Shi, A., Chandrashekar, S., and Papapanagiotou, I. Elastic Cloud Services: Scaling Snowflake’s Control Plane. In Proceedings of the 13th ACM Symposium on Cloud Computing (SOCC’22) (Nov. 2022).

J. T. Gu, X. Sun, W. Zhang, Y. Jiang, C. Wang, M. Vaziri, O. Legunsen, and T. Xu[68] Nagaraja, K., Oliveira, F., Bianchini, R., Martin, R. P., and Nguyen,
T. D. Understanding and Dealing with Operator Mistakes in Internet
Services. In Proceedings of the 6th USENIX Conference on Operating
Systems Design and Implementation (OSDI’04) (Dec. 2004).
[69] Oliveira, F., Tjang, A., Bianchini, R., Martin, R. P., and Nguyen,
T. D. Barricade: Defending Systems Against Operator Mistakes. In
Proceedings of the 5th European Conference on Computer Systems (EuroSys’10) (Apr. 2010).
[70] Oppenheimer, D., Ganapathi, A., and Patterson, D. A. Why Do
Internet Services Fail, and What Can Be Done About It? In Proceedingsof the 4th USENIX Symposium on Internet Technologies and Systems(USITS’03) (Mar. 2003).
[71] Patterson, D., Brown, A., Broadwell, P., Candea, G., Chen, M.,
Cutler, J., Enriqez, P., Fox, A., Kiciman, E., Merzbacher, M., Oppenheimer, D., Sastry, N., Tetzlaff, W., Traupman, J., and Treuhaft,
N. Recovery-Oriented Computing (ROC): Motivation, Definition, Techniques, and Case Studies. Tech. Rep. UCB//CSD-02-1175, Universityof California Berkeley, Mar. 2002.
[72] Pham, V.-T., Khurana, S., Roy, S., and Roychoudhury, A. Bucketing
Failing Tests via Symbolic Analysis. In Proceedings of the 20th International Conference on Fundamental Approaches to Software Engineering(FASE’17) (Apr. 2017).
[73] Pina, L., Andronidis, A., Hicks, M., and Cadar, C. MVEDSUA:
Higher Availability Dynamic Software Updates via Multi-Version Execution. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems(ASPLOS’19) (Apr. 2019).
[74] Rajagopalan, S., Williams, D., Jamjoom, H., and Warfield, A. Escape Capsule: Explicit State is Robust and Scalable. In Proceedingsof the 16th Workshop on Hot Topics in Operating Systems (HotOS-XIV)(May 2013).
[75] Ratis, P. Lessons Learned using the Operator Pattern to build a
Kubernetes Platform. In USENIX SREcon (Oct. 2021).
[76] Schwarzkopf, M., Konwinski, A., Abd-El-Malek, M., and Wilkes, J.
Omega: Flexible, Scalable Schedulers for Large Compute Clusters. In
Proceedings of the 8th ACM European Conference on Computer Systems(EuroSys’13) (Apr. 2013).
[77] Shen, Z., Shang, R., and Bedi, R. How eBay Leverages Kubernetes,
Helm Charts and Jenkins Pipelines to Deliver High-Quality Software. https://tech.ebayinc.com/engineering/how-ebay-leverageskubernetes-helm-charts-and-jenkins-pipelines-to-deliver-highquality-software/, 2021. eBay Tech Blog.
[78] Sosa, C., and Bhatia, P. Application management made easier with
Kubernetes Operators on GCP Marketplace. https://cloud.google.com/
blog/products/containers-kubernetes/application-managementmade-easier-with-kubernete-operators-on-gcp-marketplace, May
2019. Google Cloud Blog.
[79] Sun, X., Cheng, R., Chen, J., Ang, E., Legunsen, O., and Xu, T. Testing
Configuration Changes in Context to Prevent Production Failures. In
Proceedings of the 14th USENIX Symposium on Operating Systems Design
and Implementation (OSDI’20) (Nov. 2020).
[80] Sun, X., Luo, W., Gu, J. T., Ganesan, A., Alagappan, R., Gasch, M.,
Suresh, L., and Xu, T. Automatic Reliability Testing for Cluster
Management Controllers. In Proceedings of the 16th USENIX Symposium
on Operating Systems Design and Implementation (OSDI’22) (July 2022).
[81] Sun, X., Suresh, L., Ganesan, A., Alagappan, R., Gasch, M., Tang, L.,
and Xu, T. Reasoning about modern datacenter infrastructures using
partial histories. In Proceedings of the 18th Workshop on Hot Topics inOperating Systems (HotOS-XVIII) (May 2021).
[82] Suresh, L., ao Loff, J., Kalim, F., Jyothi, S. A., Narodytska, N.,
Ryzhyk, L., Gamage, S., Oki, B., Jain, P., and Gasch, M. Building Scalable and Flexible Cluster Managers Using Declarative Programming.
In Proceedings of the 14th USENIX Symposium on Operating Systems
Design and Implementation (OSDI’20) (Nov. 2020).

Acto: Automatic End-to-End Testing for Operation Correctness of Cloud System Management[83] Tang, C., Yu, K., Veeraraghavan, K., Kaldor, J., Michelson, S.,
Kooburat, T., Anbudurai, A., Clark, M., Gogia, K., Cheng, L.,
Christensen, B., Gartrell, A., Khutornenko, M., Kulkarni, S.,
Pawlowski, M., Pelkonen, T., Rodrigues, A., Tibrewal, R., Venkatesan, V., and Zhang, P. Twine: A Unified Cluster Management System
for Shared Infrastructure. In Proceedings of the 14th USENIX Conference
on Operating Systems Design and Implementation (OSDI’20) (Nov. 2020).
[84] Tang, L., Bhandari, C., Zhang, Y., Karanika, A., Ji, S., Gupta, I., andXu, T. Fail through the Cracks: Cross-System Interaction Failures inModern Cloud Systems. In Proceedings of the 18th European Conference
on Computer Systems (EuroSys’23) (May 2023).
[85] Tang, Z., Li, X., and Guo, F. Demystifying Kubernetes as a service– How Alibaba cloud manages 10,000s of Kubernetes clusters.
https://www.cncf.io/blog/2019/12/12/demystifying-kubernetes-asa-service-how-does-alibaba-cloud-manage-10000s-of-kubernetesclusters/, Dec. 2019. Cloud Native Computing Foundation Blog.
[86] Templeton, G., and Davidson, S. How a Couple of Characters (andGitOps) Brought Down Our Site. In KubeCon Europe (May 2022).
[87] Tirmazi, M., Barker, A., Deng, N., Haqe, M. E., Qin, Z. G., Hand, S.,
Harchol-Balter, M., and Wilkes, J. Borg: The Next Generation. In
Proceedings of the 15th ACM European Conference on Computer Systems(EuroSys’20) (Apr. 2020).
[88] van Tonder, R., Kotheimer, J., and Goues, C. L. Semantic Crash Bucketing. In Proceedings of the 33rd ACM/IEEE International Conference on
Automated Software Engineering (ASE’18) (Sept. 2018).
[89] Vasudevan, J. Azure Service Operators – A Kubernetes native wayof Deploying Azure Resources. https://devblogs.microsoft.com/cse/
2021/11/11/azure-service-operators-a-kubernetes-native-way-of-

SOSP ’23, October 23–26, 2023, Koblenz, Germany

deploying-azure-resources/, Nov. 2021. Microsoft Developer Blogs.
[90] Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E.,
and Wilkes, J. Large-Scale Cluster Management at Google with Borg.
In Proceedings of the 10th European Conference on Computer Systems(EuroSys’15) (Apr. 2015).
[91] Wang, S., Lian, X., Marinov, D., and Xu, T. Test Selection for Unified
Regression Testing. In Proceedings of the 45th IEEE/ACM International
Conference on Software Engineering (ICSE’23) (May 2023).
[92] Xu, T., Jin, X., Huang, P., Zhou, Y., Lu, S., Jin, L., and Pasupathy, S.
Early Detection of Configuration Errors to Reduce Failure Damage.
In Proceedings of the 12th USENIX Symposium on Operating Systems
Design and Implementation (OSDI’16) (Nov. 2016).
[93] Xu, T., Zhang, J., Huang, P., Zheng, J., Sheng, T., Yuan, D., Zhou,
Y., and Pasupathy, S. Do Not Blame Users for Misconfigurations.
In Proceedings of the 24th Symposium on Operating System Principles(SOSP’13) (Nov. 2013).
[94] Xu, T., and Zhou, Y. Systems Approaches to Tackling Configuration
Errors: A Survey. ACM Computing Surveys (CSUR) 47, 4 (July 2015).
[95] Zhang, J., Renganarayana, L., Zhang, X., Ge, N., Bala, V., Xu, T., andZhou, Y. EnCore: Exploiting System Environment and Correlation
Information for Misconfiguration Detection. In Proceedings of the
19th International Conference on Architecture Support for Programming
Languages and Operating Systems (ASPLOS’14) (Mar. 2014).
[96] Zhang, Y., Yang, J., Jin, Z., Sethi, U., Rodrigues, K., Lu, S., andYuan, D. Understanding and Detecting Software Upgrade Failures inDistributed Systems. In Proceedings of the 28th ACM Symposium on
Operating Systems Principles (SOSP’21) (Oct. 2021).