AUTOMATIC ROLLBACK OF INJECTED FAULTS IN SYSTEM TESTING

A fault injection system injects one or more faults into software or hardware components of a target system and automatically rolls back the injected faults upon incident creation at the target system. In one embodiment, modules of the fault injection system identify or receive one or more fault injection parameters for a target component, inject one or more faults into the target component according to the parameters, and monitor performance of the component and the target system to assess target system performance under adverse conditions. Responsive to detecting that an incident has been created at the target system, the fault injection system implements countermeasures to automatically roll back the injected faults and updates the incident to notify the target system of the fault injection testing.

BACKGROUND

1. Technical Field

The subject matter described relates generally to computer system testing and, in particular, to automatic rollback of injected faults upon incident creation.

2. Background Information

Chaos engineering is the discipline of deliberately injecting faults into a software or hardware system to discover its behavior under adverse conditions. Fault injection is in some cases performed without the explicit consent of the system owners. This may be advisable in a situation where the fault will definitely occur in n days and be irreversible at that time. Therefore, after notifying the system owners (who may or may not see the notification), a user introduces the fault early, together will a rollback mechanism, to mitigate the risk of the fault occurring later without the possibility of rollback.

In such a scenario, the system owner should be made aware of the outage so that they have knowledge of the upcoming irreversible fault and can fix the system such that it will be resilient to the fault when it occurs. But one also needs to ensure that the outage is automatically reversed as soon as the system owner is aware of it so that the impact of the outage is the minimum required to alert the system owner and no more.

Typically, faults are rolled back based on monitoring the system under test using predetermined SLIs (Service Level Indicators) against defined SLOs (Service Level Objectives). However, in a large organization, this may be difficult to do systematically. Different systems may use different monitoring tools. Additionally, some systems have monitoring that is overly noisy (many false alarms, which the owners have learned to ignore), or inadequate (an owner may not be aware of an outage until notified by a third-party, such as a customer).

SUMMARY

The above and other problems are addressed by a computer system that automatically rolls back injected faults upon creation of an “incident” at the affected system. A fault identification module of a fault injection system receives or identifies one or more fault injection parameters for testing of a target software or hardware component at a target system. In various embodiments, the fault injection parameters include, for example, a fault type, a fault location (e.g., in a specified system, subsystem, or link between systems), a fault amplitude, whether the fault should be injected on demand or occur at a specified activation time or upon occurrence of an activation condition, a predicted impact on target component or system behavior, and the like. The fault injection system may perform testing in an operational system to test an active component under adverse conditions.

A fault injection module injects the one or more faults into the identified component of the target system according to the fault injection parameters and notifies a system monitoring module of the fault injection system, which tracks execution of the fault injection commands and, in some embodiments, performs data collection and analysis to assess the performance of the component under test and the target system. The system monitoring module also monitors the target system to detect the creation of an incident related to the one or more injected faults, e.g., via an application programming interface (API) that facilitates data exchange between the fault injection system and the target system. Responsive to detecting that an incident has been created, the fault injection module automatically implements one or more countermeasures to rollback the injected fault. In some embodiments, the system monitoring module notifies the target system of the fault injection and, optionally, an assessment of the effected target system components, allowing target system operators to identify and correct system defects or security vulnerabilities before anomalies occur without the possibility of rollback.

DETAILED DESCRIPTION

Reference will now be made to several embodiments, examples of which are illustrated in the accompanying figures. Wherever practicable, similar or like reference numbers are used in the figures to indicate similar or like functionality. Where similar elements are identified by a reference number followed by a letter, a reference to the number alone in the description that follows may refer to all such elements, any one such element, or any combination of such elements. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods may be employed without departing from the principles described.

FIG.1illustrates one embodiment of a networked computing environment100suitable for providing automatic rollback of injected faults. In the embodiment shown inFIG.1, the networked computing environment100includes a fault injection system110, one or more target systems120A-N, and one or more user devices130A-N, all connected via a network140. In other embodiments, the networked computing environment100contains different or additional elements. In addition, the functions may be distributed among the elements in a different manner than described. Moreover, while three target systems120are shown inFIG.1in order to simply and clarify the description, in other embodiments, the networked computing environment many include many target subsystems120.

The fault injection system110is a network system configured to inject faults into software or hardware components of one or more target systems120and to automatically roll back the one or more injected faults upon creation of an incident at one or more of the impacted target systems120. In one embodiment, the fault injection system110injects the faults in fully operational systems (i.e., after the component under test has been deployed). After injecting one or more faults into a component of a target system120, the fault injection system110monitors the system120under test to determine when the target system120detects the presence of the fault(s) and implements one or more countermeasures to automatically roll back the injected fault(s) upon creation of an incident at the target system120. In one example, the injected fault is a blocking control to the target system production environment (e.g., removal of log4j from all hosts by a specified date). The fault injection system rolls back the blocking control responsive to detecting that an incident has been created at the target system such that system operators who might otherwise be unaware of the system's dependency on log4j discover the dependency in a relatively safe manner and plan to remove it.

The modules and corresponding functionalities of the fault injection system110are described in more detail below with reference toFIG.2. Moreover, while the embodiment ofFIG.1illustrates the fault injection system110as a separate component of the networked computing environment100that interacts with the one or more target system120through the network140, in other embodiments, the fault injection system110is a subsystem of the one or more target systems120.

The target systems120A-N are one or more systems having software or hardware components available for testing during normal operation of a deployed component. For example, a system operator may use the fault injection system110to subject an application to various types of faults at specified frequencies or during specified testing intervals. Injecting faults into an operational system might allow an operator to identify and resolve behaviors that may not occur in staging or prototype environments. Example software and hardware faults that may be injected include disconnecting a data center, powering down a machine, revoking an access key, unplugging a network cable, removing a software package from a package repository, and the like.

Each target system120has an incident creation and management process with which the target system120detects and logs anomalies, errors, and other deviations from expected behavior in its software and hardware components. These anomalies may cause a disruption to one or more components, impact internal or external users of the target system120(e.g., employees or customers), and require immediate response and resolution. Types of anomalies may vary significantly in terms of scope, duration, impact, and severity and may be detected in different ways, for example, via firewalls, antivirus software, intrusion detection and prevention systems that monitor system and network behavior, user reporting, etc.

Responsive to detecting the occurrence of an anomaly, the target system120creates an incident (also referred to as an incident log or an incident ticket) to document the anomaly and initial findings. Incident creation may be automatic or manual, and information included in the incident log may include one or more of an identification of an anomaly, an initial classification, an indication of infected system(s), timestamps, network logs, a source of the anomaly, an initial assessment of the impact on systems and internal and external users, response measures, previous incidents associated with the infected component or system, and the like. In one embodiment, an incident is automatically created at the target system120by running, on the target system, a probe configured to execute on the target system120at a specified interval to collect telemetry data describing operations of the target system120. Responsive to the telemetry data failing one or more tests, the target system120automatically creates the incident.

The user device130is a computing device capable of receiving user input as well as transmitting to or receiving data from the fault injection system110via the network120. For example, a user device130can be a desktop or a laptop computer, a smartphone, tablet, or another suitable device. Each user device130may have a screen for displaying content (e.g., documents, images, videos, or other content items) or receiving user input (e.g., a touchscreen). User devices130are configured to communicate via the network140.

The network140may include any combination of local area and wide area networks employing wired or wireless communication links. In one embodiment, network140uses standard communications technologies and protocols. For example, network140includes communication links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, 5G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols used for communicating via the network140include multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged over the network140may be represented using any format, such as hypertext markup language (HTML) or extensible markup language (XML). Some or all of the communication links of the network140may be encrypted.

FIG.2illustrates one embodiment of the fault injection system110. In the embodiment shown, the fault injection system110includes a fault identification module210, a fault injection module220, a system monitoring module230, a reporting module240, and a fault library250. In other embodiments, the fault injection system110has alternative configurations than shown inFIG.2, including different, fewer, or additional components.

The fault identification module210determines fault injection parameters for one or more software or hardware components of a target system120. Fault injection parameters include one or more target components for testing, one or more types of faults, fault amplitude, a fault activation time or condition, a fault location (e.g., in a specified system, subsystem, or function or in a link between systems), a predicted impact on system behavior, and the like. In one embodiment, the fault identification module210selects a fault for injection based on, for example, the type of component under test or the results of previous testing of the target component or other related components of the target system120, the results of previous testing of similar components at other target systems120(which may be retrieved from the fault library250), a predicted impact on the target system120, whether the fault will affect internal or external users of the target system120, a status of the target component, and the like. Alternatively, one or more of the fault injection parameters are specified by a user of the target system120, e.g., via the user device130. For example, a user may specify a specific software or hardware component for testing but not a testing interval or fault type or may select all of the fault injection parameters for a test.

Responsive to identifying or receiving fault injection parameters for the target component, the fault identification module210sends the parameters to the fault injection module220for implementation. The parameters are also be sent to the fault library250for storage in association with the one or more target component(s) under test and the target system120. In one embodiment, where the target components are software programs, the fault injection module220uses one or more of the following software implemented fault injection (SWIFI) techniques to inject faults into components of the target system120:

The fault injection module210uses a software trigger to inject a fault into the target component. Triggers may be time-based (e.g., when a timer reaches a specified time, an interrupt is generated and the interrupt handler associated injects the fault) or interrupt-based (hardware exceptions and software trap mechanisms are used to generate an interrupt at a specific location in the program code or on a particular event within the system, for instance, access to a specific memory location). The fault injection module210may use a number of different techniques to insert faults into a target system120, including corruption of memory space (e.g., RAM, processor registers, or I/O map), system call interposition techniques (e.g., by intercepting operating system calls made by user-level software and injecting faults into them), and network-level fault injection (e.g., corruption, loss, or reordering of network packets at the network interface).

Still further, in some embodiments, the fault injection module210injects one or more faults into the target software component using commercial fault injection tools, such as Xception, beSTORM, Holodeck, Exhaustif, or the Mu Service Analyzer or research tools, such as Xception, Grid-FIT, MODIFI, Ferrari, FTAPE, DOCTOR, and the like.

In one embodiment, where the target components for testing comprise hardware elements of the target system120, such as a central processing unit (CPU) or device memory, the fault injection module120uses one or more of the following hardware-implemented fault injection techniques:

Pin-level injection: The fault injection module120makes direct physical contact with circuit pins at the target component, producing voltage or current changes. Electrical currents and voltages at the pins may be altered using active probes (current added via probes attached to the pins) or via socket insertion (socket inserted between the target hardware component and its circuit board).

Injection without contact: The fault injection module120has no direct physical contact with the target hardware component. Instead, the fault is injected using an external source, such as heavy-ion radiation (an ion passes through the depletion region of the target hardware component and generates current) or electromagnetic interference (the target hardware component is placed in or near an electromagnetic field).

The system monitoring module230tracks execution of the fault injection commands and performs data collection, processing, and analysis to assess the performance of the component(s) under test and the target system120. Activated faults may cause one or more errors to occur in the target component and can lead to component or target system120failure, e.g., a service outage. For example, where the target component is a central processing unit (CPU) of the target system120, the injected faults may cause skipped or repeated CPU instructions, incorrect evaluation of CPU instructions, or corrupt reads from memory devices and can simultaneously impact all stages of a CPU pipeline.

The system monitoring module230monitors the target system120to detect the creation of an incident related to the one or more injected faults. In one embodiment, the system monitoring module230detects incident creation via an application programming interface (API) that facilitates data exchange between the fault injection system110and the target system120. Alternatively, an incident response subsystem at the target system120can be configured to automatically notify the fault injection system110upon creation of an incident associated with a component under test.

Upon detecting the creation of an incident report for the target component, the system monitoring module230instructs the fault injection module220to implement countermeasures to automatically roll back the one or more injected faults. In one embodiment, the incident is automatically updated to indicate that the anomaly may have been caused by the injected fault such that the target system operator may investigate and remediate the cause of the anomaly. Depending on the type of fault(s) injected, countermeasures may include removal of added code, reversal of code alterations, reconnecting a data center, powering on a machine, reconnecting a network cable, re-enabling an access key, re-instating a software package from a package repository, and the like.

In one embodiment, one or more rollback countermeasures are defined before a fault is injected (e.g., based on the fault type, fault location, target system, or one or more other types of fault parameters) and tested in a pre-production environment to ensure that the countermeasures correctly and completely roll back a corresponding fault. Each countermeasure may be associated with a validation script (also tested in pre-production), which validates that the fault rollback is performed correctly.

The target component(s) behavior under the adverse conditions associated with the injected fault(s) may be logged and stored in the fault library250. In addition to the fault injection parameters discussed above, the system monitoring module230may collect and analyze data regarding the effect of the fault(s) on the target component behavior and the behavior of other software or hardware components of the target system120, the impact on external systems or users, the time between fault injection and incident creation at the target system120. In one example, the injected fault may manifest in the affected system as skipped or repeated CPU instructions, incorrect evaluation of CPU instructions, or corrupt reads from memory devices and can cause the target component to execute unintended or insecure code paths. In another example, where the fault is a simulated expiration of a TLS certificate, the effect of the fault may be a system or website outage and a warning to users attempting to reach that the target system or website is not secure.

Responsive to the fault injection module220implementing the rollback mechanisms, the reporting module240notifies the target system120of the fault injection and, optionally, an assessment of the effected target system components. In one embodiment, the system monitoring module230generates a fault injection report based on the collected data regarding performance of the component under test and the target system120and sends the report to the reporting module240for relay to an operator of the target system120(e.g., to a user device130of the target system120who has provided instructions to conduct the fault injection testing and specified one or more fault injection parameters). The fault injection report may identify, for example, the type of injected fault(s), the target component(s), timestamps associated with fault injection and incident creation, implemented countermeasure(s), and the effects of the injected fault(s) on the target component and other components of the target system120, including an indication of impacted internal or external users of the target system120. The fault injection testing and associated report thus allows users of the target system120to identify and correct system defects and security vulnerabilities before anomalies occur without the possibility of rollback.

In some embodiments, the same target software or hardware component is tested more than once by injecting the same fault or a different fault, or a different combination of faults. For example, where the target component comprises software, the fault injection system110may perform both code modification and insertion in different tests or may perform a first type of runtime injection during a first test and a second type during a second test. In another example, target component code may be modified and code added during a single fault injection test.

The fault library250stores data regarding previous testing, including indications of previously tested software and hardware components associated with target systems120, associated injected faults and countermeasures, performance of the target components and other components of the target systems120in response to the testing, time to incident creation and fault rollback, and the like.

EXAMPLE METHODS

FIG.3illustrates an example method300for automatic rollback of an injected fault, according to one embodiment. The steps ofFIG.3are illustrated from the perspective of the fault injection system110performing the method300. However, some or all of the steps may be performed by other entities or components. In addition, some embodiments may perform the steps in parallel, perform the steps in different orders, or perform different steps. Moreover, while the embodiment described herein contemplates injection of a single fault into a single component of a target system120, in other embodiments, multiple faults are simultaneously injected into a single component or multiple hardware or software target system components.

In the embodiment shown inFIG.3, the method300begins with the fault identification module210identifying305fault injection parameters for testing a software or hardware component of a target system110. Fault injection parameters may include, for example, an identification of a target component, a fault type, a fault location, a fault amplitude, a fault activation time or condition, a predicted impact on the behavior of the target component or target system120, and the like.

In various embodiments, the one or more fault injection parameters based on user input via the user device130to the fault injection system110or are determined by the fault identification module210based, e.g., an identified or suspected vulnerability at the target system120, previous testing of the target component or related components of the target system120, previous fault injection testing of the target component or similar components at other target systems120, and the like.

The fault identification module210sends the fault identification parameters to the fault injection module220, which injects310the fault into the predetermined location of the target component based on the parameters. In one example, where the parameters include a fault activation condition, the fault injection module220monitors the target system120for occurrence of the condition and automatically injects the fault in response to determining that the condition has occurred. Methods of fault injection are based on the type of component under test and may include one or more of the techniques discussed above with respect toFIG.2.

The fault injection module220notifies the system monitoring module230upon fault injection to allow the system monitoring module230to monitor315performance of the target component and target system120under test. In one embodiment, the monitoring includes data logging and analysis of the target system's behavior under adverse conditions, including target system120failure points, service outages, impacts on one or more related software or hardware components of the target system120, attempted remediation efforts, and the like.

The system monitoring module230detects creation of an incident at the target system120and notifies the fault injection module220, which automatically implements325one or more countermeasures to roll back the injected fault. For example, where the fault is powering off of a server or other machine, the countermeasure may be powering on the affected machine. In another example, where the fault is disabling network connectivity, the fault injection module220reenables connectivity. In still another example, where the injected fault is modification of source code, the fault injection module220reverts the affected code to its original state.

At330, the reporting module240notifies an operator of the target system120n(e.g., via the user device130) of the injected fault and, optionally, sends a fault injection report summarizing the fault injection test and collected data, enabling operators of the target system120to correct component or system vulnerabilities and mitigate the risk of the fault occurring later without the possibility of rollback.

Computing System Architecture

The entities shown inFIG.1are implemented using one or more computers.FIG.4is an example architecture of a computing device, according to an embodiment. AlthoughFIG.4depicts a high-level block diagram illustrating physical components of a computer used as part or all of one or more entities described herein, in accordance with an embodiment, a computer may have additional, fewer, or variations of the components provided inFIG.4. AlthoughFIG.4depicts a computer400, the figure is intended as functional description of the various features that may be present in computer systems than as a structural schematic of the implementations described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated.

Illustrated inFIG.4are at least one processor402coupled to a chipset404. Also coupled to the chipset404are a memory406, a storage device408, a keyboard410, a graphics adapter412, a pointing device414, and a network adapter416. A display418is coupled to the graphics adapter412. In one embodiment, the functionality of the chipset404is provided by a memory controller hub420and an I/O controller hub422. In another embodiment, the memory406is coupled directly to the processor402instead of the chipset404. In some embodiments, the computer400includes one or more communication buses for interconnecting these components. The one or more communication buses optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.

The storage device408is any non-transitory computer-readable storage medium, such as a hard drive, compact disk read-only memory (CD-ROM), DVD, or a solid-state memory device or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Such a storage device408can also be referred to as persistent memory. The pointing device414may be a mouse, track ball, or other type of pointing device, and is used in combination with the keyboard410to input data into the computer400. The graphics adapter412displays images and other information on the display418. The network adapter416couples the computer400to a local or wide area network.

The memory406holds instructions and data used by the processor402. The memory406can be non-persistent memory, examples of which include high-speed random-access memory, such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory.

As is known in the art, a computer400can have different and/or other components than those shown inFIG.4. In addition, the computer400can lack certain illustrated components. In one embodiment, a computer400acting as a server may lack a keyboard410, pointing device414, graphics adapter412, and/or display418. Moreover, the storage device408can be local and/or remote from the computer400(such as embodied within a storage area network (SAN)).

As is known in the art, the computer400is adapted to execute computer program modules for providing functionality described herein. As used herein, the term “module” refers to computer program logic utilized to provide the specified functionality. Thus, a module can be implemented in hardware, firmware, and/or software. In one embodiment, program modules are stored on the storage device408, loaded into the memory406, and executed by the processor402.

Some portions of above description describe the embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the computing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs comprising instructions for execution by a processor or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of functional operations as modules, without loss of generality.

As used herein, any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. Similarly, use of “a” or “an” preceding an element or component is done merely for convenience. This description should be understood to mean that one or more of the element or component is present unless it is obvious that it is meant otherwise.

Where values are described as “approximate” or “substantially” (or their derivatives), such values should be construed as accurate +/-10% unless another meaning is apparent from the context. From example, “approximately ten” should be understood to mean “in a range from nine to eleven.”

While particular embodiments and applications have been illustrated and described, it is to be understood that the described subject matter is not limited to the precise construction and components disclosed. The scope of protection should be limited only by any claims that ultimately issue.