Applying impairments to nodes of a distributed computing environment using a software operator

Impairments can be applied to nodes of a distributed computing environment using a software operator. For example, a system can receive, by a controller of a distributed computing environment executing a network-impairment operator, a custom resource defining a reduced-performance configuration for a worker node of the distributed computing environment. The system can deploy the reduced-performance configuration to the worker node for a predetermined period of time. Subsequent to the predetermined period of time passing, the system can remove the reduced-performance configuration from the worker node.

TECHNICAL FIELD

The present disclosure relates generally to software operators in distributed computing environments. More specifically, but not by way of limitation, this disclosure relates to applying impairments to nodes of a distributed computing environment using a software operator.

BACKGROUND

To help automate the deployment, scaling, and management of software resources such as applications and micro-services, some distributed computing environments may include container orchestration platforms. Container orchestration platforms can help manage containers to reduce the workload on users. One example of a container orchestration platform is Kubernetes. Distributed computing environments running Kubernetes can be referred to as Kubernetes environments.

Kubernetes environments can include software operators (“operators”) for automating various repeatable tasks, such as deployment, scaling, and backup of software resources. In the context of Kubernetes, an operator is a software extension that can manage an assigned software resource, such as a stateful application. Once deployed, operators can create, configure, and manage instances of their assigned software resources on behalf of a user in a declarative way.

DETAILED DESCRIPTION

It may be beneficial to simulate remote worker nodes and effects of placing nodes of a distributed computing environment over a wide-area network (WAN) for evaluating a performance of the nodes and the distributed computing environment as a whole. But, typically nodes a user is attempting to test are located in a single server rack, or are virtualized nodes on a same piece of hardware. Therefore, the nodes may not experience network conditions similar to that of being remote from each other or over a WAN. The user can manually apply impairments, but these impairments are limited to that which the user is permitted to configure through command-line operations. The user is typically limited to applying impairments to one node at a time, so simulating multiple remote worker nodes may involve manually performing the same command-line operations multiple times for each of the nodes. Thus, the types of impairments that can be applied to the nodes may be limited and the process of simulating and evaluating multiple remote nodes can be time-intensive for the user.

Some examples of the present disclosure can overcome one or more of the abovementioned problems by providing a system that uses a software operator to apply impairments to nodes of a distributed computing environment. A controller of a distributed computing environment that executes a network-impairment operator can receive a custom resource defining a reduced-performance configuration for a worker node of the distributed computing environment. The reduced-performance configuration can correspond to the impairments that are to be applied to the worker node. For example, the impairments can include network latency, bandwidth, packet loss, link flapping, packet duplication, or packet reordering impairments. The system can deploy the reduced-performance configuration to the worker node for a predetermined period of time. Subsequent to the predetermined period of time passing, the system can remove the reduced-performance configuration from the worker node. In some examples, the custom resource can specify that the reduced-performance configuration is to be applied to multiple worker nodes. As a result, once a user creates the custom resource, impairments can be automatically applied to nodes without additional interaction by the user. The time involved in applying impairments can be reduced and the types of impairments that can be applied to the nodes can be increased.

One particular example can involve a computing cluster that is running Kubernetes as a container orchestration platform. The computing cluster can include a controller executing a network-impairment operation. The controller can receive a custom-resource specifying that five worker nodes of the computing cluster are to be subjected to fifty milliseconds of ingress and egress network latency for sixty seconds. The controller can generate a daemonset that generates pods including a reduced-performance configuration of the fifty milliseconds of ingress and egress network latency. The five worker nodes can then create the respective pod with the reduced-performance configuration. During the sixty seconds, a ping between one of the five worker nodes and a management node of the computing cluster can take one-hundred milliseconds. Additionally, a ping between two of the five worker nodes can take two-hundred milliseconds. An impact of the added network latency can be evaluated by the controller during the sixty seconds. As a result, a user only needs to create the custom resource, and the operator can apply the reduced-performance configuration to multiple nodes at once. This can significantly reduce time involved in testing an impairment of nodes of a computing cluster, as well as expand the impairments that can be applied to the nodes.

FIG. 1shows a block diagram of an example of a system100for applying impairments to nodes of a distributed computing environment110according to some aspects of the present disclosure. The distributed computing environment110may be a computing cluster, such as a Kubernetes cluster, formed from multiple nodes (e.g., servers or other computing devices) in communication with one another over one or more networks.FIG. 1includes a control plane120with a management node122, a controller124, and worker nodes140a-c. The controller124may execute on a node of the distributed computing environment110. The control plane120, the controller124, and the worker nodes140a-cmay communicate with one another to collectively perform tasks in the distributed computing environment110. For example, the controller124, management node122, and the worker nodes140a-ccan communicate with one another to perform distributed data processing or other distributed projects in the distributed computing environment.

The control plane120can be a layer of the computing cluster that exposes an application programming interface (API) and interfaces to define, deploy, and manage containers. For example, the control plane120can manage the worker nodes140a-cby scheduling pods to run on the worker nodes140a-c. In some examples, multiple management nodes can make up the control plane120. The management node122can include a controller124that executes a network-impairment operator126. Operators can enhance behavior of a computing cluster without modifying the code of the cluster by linking controllers to one or more custom resources. The network-impairment operator126can be linked with a custom resource104.

The custom resource104may be a yet another markup language (YAML) file defined by a user that specifies actions the network-impairment operator126is to perform. For example, the custom resource104can define a reduced-performance configuration134that the network-impairment operator126is to apply to one or more of the worker nodes140a-c. The reduced-performance configuration134can specify performance statistics, such as network latency specifications, bandwidth specifications, a link flapping specification, a packet loss specification, a packet duplication specification, a packet reordering specification, or a combination thereof, along with an indication of which of the worker nodes140a-care to receive the reduced-performance configuration134. The reduced-performance configuration134may be designed to mimic placing one or more of the worker nodes140a-cover a WAN so that the user can evaluate a performance of the worker nodes140a-cunder such conditions without physically separating the worker nodes140a-cfrom each other, the controller124, or the control plane120. That is, the worker nodes140a-c, the controller124, and the control plane120may all be located in a common geographic location, such as in one or more server racks in the same building.

It may be beneficial to test a large number of worker nodes being subjected to the reduced-performance configuration134at the same time, so a number of worker nodes specified in the custom resource104that are to receive the reduced-performance configuration134may be greater than a threshold. For example, the threshold may be two worker nodes, five worker nodes, ten worker nodes, fifty worker nodes, or any other suitable number of worker nodes of the distributed computing environment110.

The controller124can receive the custom resource104from a user device102subsequent to the user creating the custom resource104and causing the custom resource104to be sent to the distributed computing environment110. Along with the reduced-performance configuration134and the specification of which of the worker nodes140a-care to receive the reduced-performance configuration134, the custom resource104can also specify a predetermined period of time for which the specified worker node(s) are to be subjected to the reduced-performance configuration134. Specifying the predetermined period of time can ensure that the distributed computing environment110is not subjected to sub-optimal conditions indefinitely so that the distributed computing environment110is able to recover. The custom resource104may specify a start time and an end time for applying the reduced-performance configuration134to the worker nodes140a-c, or the custom resource104may specify a duration for applying the custom resource104to the worker nodes140a-c.

The controller124can deploy the reduced-performance configuration134to the specified worker nodes for the predetermined period of time. For example, the custom resource104can indicate that each of the worker nodes140a-cis to receive the reduced-performance configuration134for five minutes, so the controller124can deploy the reduced-performance configuration134to the worker nodes140a-cfor five minutes. To deploy the reduced-performance configuration134to the worker nodes140a-c, the network-impairment operator126can generate a daemonset130, which ensures that a copy of a pod runs across the specified worker nodes in the distributed computing environment110. The daemonset130can generate pods132a-cthat include the reduced-performance configuration134and send the pods132a-cto the worker nodes140a-c. As illustrated inFIG. 1, the pod132acan be run on the worker node140a, the pod132bcan be run on the worker node140b, and the pod132ccan be run on the worker node140c.

Each of the worker nodes140a-ccan include a network interface card (NIC) that applies egress components of the reduced-performance configuration134to the worker nodes140a-c. Egress components can affect traffic going to the worker nodes140a-c. For example, the reduced-performance configuration134can apply an egress network latency specification, an egress bandwidth specification, an egress packet loss specification, or a combination thereof for the worker nodes140a-c. As one particular example, the reduced-performance configuration134can apply one-hundred milliseconds of egress network latency and one-thousand kilobits-per-second of egress bandwidth for traffic going to the worker nodes140a-c.

Ingress components of the reduced-performance configuration134can be applied by an intermediate function block150of the distributed computing environment110. Ingress components can affect traffic coming from the worker nodes140a-c. For example, the reduced-performance configuration134can apply an ingress network latency specification, an ingress bandwidth specification, an ingress packet loss specification, or a combination thereof for the worker nodes140a-c. As one particular example, the reduced-performance configuration134can apply ten milliseconds of ingress network latency and 5% of ingress packet loss for traffic coming from the worker nodes140a-c. Ingress traffic152from the worker nodes140a-ccan be routed to the intermediate function block150that then can apply the ingress components of the reduced-performance configuration134to the ingress traffic152.

FIG. 2illustrates a block diagram of an example of applying network latency of a reduced-performance configuration to worker nodes240a-bof a distributed computing environment according to some aspects of the present disclosure. The ingress latency for the worker nodes240a-bis ten milliseconds and the egress latency for the worker nodes240a-bis one-hundred milliseconds. A management node222in communication with the worker nodes240a-bis not subjected to impairments of the reduced-performance configuration.

Based on the reduced-performance configuration, traffic going from the worker node240ato the management node222takes one-hundred milliseconds and traffic going from the management node222to the worker node240atakes ten milliseconds. A round-trip ping between the worker node240aand the management node222takes one-hundred ten milliseconds, since the ping is subjected to the ten milliseconds of ingress latency and the one-hundred milliseconds of egress latency.

Additionally, traffic going from the worker node240ato the worker node240btakes one-hundred ten milliseconds and traffic going from the worker node240bto the worker node240atakes one-hundred ten milliseconds, since the traffic going either way is subjected to the ten milliseconds of ingress latency and the one-hundred milliseconds of egress latency. A round-trip ping between the worker node240aand the worker node240btakes two-hundred twenty milliseconds, since the ping is subjected to the ten milliseconds of ingress latency and the one-hundred milliseconds of egress latency at each of the worker nodes240a-b.

While the reduced-performance configuration134is applied to one or more of the worker nodes140a-c, the controller124can evaluate a performance of the one or more worker nodes140a-c. The controller124may evaluate whether an eviction policy for a pod on a worker node experiencing the reduced-performance configuration134is executed, how taints and tolerations are applied to the worker node experiencing the reduced-performance configuration134, or a scalability of having a number of worker nodes subjected to the reduced-performance configuration134. The evaluation may additionally involve determining that a worker node becomes “NotReady”, which may result in the pod on the worker node being evicted and rescheduled to another worker node. The evaluation can allow the controller124to determine how configurations of the worker nodes140a-ccan be adjusted for better performance in the distributed computing environment110. For example, if the reduced-performance configuration134is applied to the worker node140a, the controller124may determine that the worker node140ais evicted during the predetermined period of time. The controller124can then determine or indicate to the user device102that worker nodes of the distributed computing environment110, or a similar distributed computing environment, should not have a configuration similar to the reduced-performance configuration134because the worker nodes may be evicted.

In some examples, the controller124can configure the daemonset130to automatically remove the pods132a-cfrom the worker nodes140a-csubsequent to the predetermined period of time passing so that the distributed computing environment110can recover from the reduced-performance configuration134. In addition, the controller124may generate the daemonset130to deploy the reduced-performance configuration134to the worker nodes140a-cfor a first predetermined period of time and to deploy another reduced-performance configuration to some or all of the worker nodes140a-cfor a second predetermined period of time subsequent to the first predetermined period of time. To do this, the start time for the other reduced-performance configuration can be subsequent, such as a few seconds after, the end time for the initial reduced-performance configuration. The daemonset130can remove the reduced-performance configuration134from the worker nodes140a-cprior to applying the other reduced-performance configuration.

While the example shown inFIG. 1has a certain number and arrangement of components, these are merely illustrative. Other examples can include more components, fewer components, or a different arrangement of the components shown inFIG. 1. For instance, while the example ofFIG. 1includes three worker nodes, other examples can include a smaller or larger number of worker nodes that can be subjected to a reduced-performance configuration. Additionally, although only the worker nodes inFIG. 1are described as receiving the reduced-performance configuration, management nodes of the control plane may additionally or alternatively receive a reduced-performance configuration in other examples.

FIG. 3shows a block diagram of another example of a system300for applying impairments to nodes of a distributed computing environment310according to some aspects of the present disclosure. The system300includes a processor302communicatively coupled with a memory device304. The processor302may be part of a node that executes the controller124inFIG. 1. The processor302can include one processing device or multiple processing devices. Non-limiting examples of the processor302include a Field-Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), a microprocessor, etc. The processor302can execute instructions306stored in the memory device304to perform operations. In some examples, the instructions306can include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, such as C, C++, C#, etc.

The memory device304can include one memory device or multiple memory devices. The memory device304can be non-volatile and may include any type of memory device that retains stored information when powered off. Non-limiting examples of the memory device304include electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of non-volatile memory. In some examples, at least some of the memory device can include a computer-readable medium from which the processor302can read instructions306. A computer-readable medium can include electronic, optical, magnetic, or other storage devices capable of providing the processor302with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include magnetic disk(s), memory chip(s), ROM, random-access memory (RAM), an ASIC, a configured processor, optical storage, or any other medium from which a computer processor can read the instructions306.

The processor302can execute the instructions306to perform operations. For example, the processor302can receive, by a controller324of the distributed computing environment310executing a network-impairment operator326, a custom resource330defining a reduced-performance configuration334for a worker node340of the distributed computing environment310. The reduced-performance configuration334can include a network latency specification, a packet loss specification, a link flapping specification, a bandwidth specification, a packet duplication specification, a packet reordering specification, or a combination thereof for the worker node340. The processor302can deploy the reduced-performance configuration334to the worker node340for a predetermined period of time308. The predetermined period of time308can be specified by the custom resource330. The controller324can generate a daemonset for generating a pod with the reduced-performance configuration334on the worker node340. While the worker node340is subjected to the reduced-performance configuration334, the controller324can evaluate a performance of the worker node340. Subsequent to the predetermined period of time308passing, the processor302can remove the reduced-performance configuration334from the worker node340. The daemonset can automatically remove the pod with the reduced-performance configuration334from the worker node340once the predetermined period of time308passes. As a result, a user can generate a custom resource, and the custom resource can automatically be used by a network-impairment operator to apply impairments to nodes of a distributed computing environment without the user manually configuring the worker node and other worker nodes the user wants to test.

FIG. 4shows a flow chart of an example of a process for applying impairments to nodes of a distributed computing environment according to some aspects of the present disclosure. In some examples, the processor302can implement some or all of the steps shown inFIG. 4. Other examples can include more steps, fewer steps, different steps, or a different combination of steps than are shown inFIG. 4. The steps ofFIG. 4are discussed below with reference to the components discussed above in relation toFIG. 3.

In block402, the processor302receives, by a controller324of a distributed computing environment310executing a network-impairment operator326, a custom resource330defining a reduced-performance configuration334for a worker node340of the distributed computing environment310. The custom resource330may be a YAML file created by a user that specifies actions the network-impairment operator326is to perform. The reduced-performance configuration334can specify network latency specifications, bandwidth specifications, a link flapping specification, a packet loss specification, a packet duplication specification, a packet reordering specification, or a combination thereof that are to be applied to the worker node340. The reduced-performance configuration334can also indicate that the worker node340, and optionally additional worker nodes of the distributed computing environment310, is to receive the reduced-performance configuration334. The reduced-performance configuration334can mimic placing the worker node340over a WAN without physically separating the worker node340from the controller324. As such, the worker node340and the controller124can be may be located in a common geographic location.

In block404, the processor302deploys the reduced-performance configuration334to the worker node340for a predetermined period of time308. The predetermined period of time308can be specified in the custom resource330, either with a start time and an end time for applying the reduced-performance configuration334to the worker node340, or by specifying a duration of time that the reduced-performance configuration334is to be applied to the worker node340. The processor302can generate a daemonset that can generate a pod including the reduced-performance configuration334. The daemonset can create the configuration of the pod for the worker node340to apply the reduced-performance configuration334to the worker node340. Egress components of the reduced-performance configuration334can be applied by a network interface card of the worker node340. Alternatively, ingress components of the reduced-performance configuration334can be applied by an intermediate function block of the distributed computing environment310.

In block406, the processor302, subsequent to the predetermined period of time308passing, removes the reduced-performance configuration334from the worker node340. The processor302can configure the daemonset to automatically remove the pod from the worker node340subsequent to the predetermined period of time308passing. Removing the reduced-performance configuration334can allow the distributed computing environment310to recover. Prior to removing the reduced-performance configuration334, the processor302can evaluate a performance of the worker node340while subjected to the reduced-performance configuration334.