Self-healing job executor pool

Aspects of the present disclosure relate to a self-healing job executor pool. A server detects that a job executing on an executor failed. The server determines, based on at least one factor from a predetermined set of executor-related factors, that the job executing on the executor failed due to a state of the executor. The server adjusts, in response to determining that the job executing on the executor failed due to the state of the executor, the state of the executor to a known good state, where the known good state is selected from a stored set of known good states.

TECHNICAL FIELD

The subject matter disclosed herein relates to job execution, where a job may include, among other things, software code for testing. In particular, example embodiments may relate to a self-healing job executor pool.

BACKGROUND

Software developers may submit jobs that include software code for testing for execution at an executor within an executor pool. In some cases, a job may fail. The failure may be caused due to a problem related to the job or due to a state of the executor. If the problem is related to the job, the software developer may make changes to the software code of the job. However, if the problem is related to the executor, the executor with the problem may need to be identified and repaired, for example, by a technician visiting the executor pool. As the foregoing illustrates, new approaches may be desirable for identifying whether a job execution failure is due to a problem with the executor or a problem with the job, and for repairing the executor if the failure is due to the problem with the executor.

SUMMARY

In one innovative aspect, the disclosed subject matter can be embodied in a method. The method includes detecting that a job executing on an executor failed. The method includes determining, based on at least one factor from a predetermined set of executor-related factors, that the job executing on the executor failed due to a state of the executor. The method includes adjusting, in response to determining that the job executing on the executor failed due to the state of the executor, the state of the executor to a known good state, wherein the known good state is selected from a stored set of known good states.

In one innovative aspect, the disclosed subject matter can be embodied in a non-transitory computer-readable medium including instructions. The instructions include code to detect that a job executing on an executor failed. The instructions include code to determine, based on at least one factor from a predetermined set of executor-related factors, that the job executing on the executor failed due to a state of the executor. The instructions include code to adjust, in response to determining that the job executing on the executor failed due to the state of the executor, the state of the executor to a known good state, wherein the known good state is selected from a stored set of known good states.

In one innovative aspect, the disclosed subject matter can be embodied in a system. The system includes one or more processors and a memory. The memory includes instructions to detect that a job executing on an executor failed. The memory includes instructions to determine, based on at least one factor from a predetermined set of executor-related factors, that the job executing on the executor failed due to a state of the executor. The memory includes instructions to adjust, in response to determining that the job executing on the executor failed due to the state of the executor, the state of the executor to a known good state, wherein the known good state is selected from a stored set of known good states.

DETAILED DESCRIPTION

Reference will now be made in detail to specific example embodiments for carrying out the inventive subject matter. Examples of these specific embodiments are illustrated in the accompanying drawings, and specific details are set forth in the following description in order to provide a thorough understanding of the subject matter. It will be understood that these examples are not intended to limit the scope of the claims to the illustrated embodiments. On the contrary, they are intended to cover such alternatives, modifications, and equivalents as may be included within the scope of the disclosure. Examples merely typify possible variations. Unless explicitly stated otherwise, components and functions are optional and may be combined or subdivided, and operations may vary in sequence or be combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of example embodiments. It will be evident to one skilled in the art, however, that the present subject matter may be practiced without these specific details.

As noted above, according to some implementations, a software developer working at a client computing device submits a job, which includes software code, for execution or testing at an executor within an executor pool. The executor pool may include many (e.g., millions) executors, some of which may have problems that prevent them from executing code correctly. Multiple different developers may submit many (e.g., hundreds of thousands) jobs each day. This causes executor failure to become more and more common, causing software developers to worry about their code not running correctly when, in fact, there are no programming errors in their code and the problems lie with the executor.

After receiving a job submission, execution of the job is requested at the executor. A control server detects that the job failed. The control server determines whether the job failed due to a state of the executor or due to a problem related to the job. Problems related to the executor may include physical damage to the hardware of the executor, programming bugs in the source code of the executor, or the executor being in a state in which it cannot execute the job. Problems related to the job may include programming bugs in the software of the job. If the job failed due to a problem related to the job, the control server provides, to the client computing device of the software developer, a notification that the job failed due to a problem related to the job. If the job failed due to a problem with the executor, the control server causes execution of the job at a different executor within the executor pool. The control server removes the executor that failed from the executor pool and quarantines the executor. The control server adjusts the state of the executor to a known good state, from which the executor can execute jobs without experiencing executor-related problems. After adjusting the state of the executor to the known good state, the control server returns the executor from quarantine to the executor pool.

The job submission may be received at the executor from any source. In one example, the job submission is received from a developer's client computing device. In another example, the job submission is received from a central orchestrator in response to other events, such as developer requests, the submission of completed code, etc.

Implementations of the subject technology may provide advantages. For example, according to some implementations, if a job fails for an executor-related problem, the job may be executed at another executor. The software developer submitting the job may experience additional latency due to the job being submitted to the second executor but, otherwise, may not experience any negative effects due to the executor-related problem. In addition, some implementations of the subject technology provide for adjustment of an executor to a known good state in response to an executor-related problem. This adjustment may be automatic, removing the need for a technician to visit the executor and, thereby, saving time and money.

FIG. 1is a diagram of an example system100in which a job may be executed. As shown, the system100includes a client computing device110, a control server120, and an executor pool130connected to one another via a network140. The executor pool130includes multiple executors135.1-3. The network140may include one or more of the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a cellular network, a WiFi network, a virtual private network (VPN), a public network, a wired network, a wireless network, and the like.

The client computing device110may include one or more of a laptop computer, a desktop computer, a mobile phone, a tablet computer, a personal digital assistant (PDA), a digital music player, and the like. The client computing device110may include an application (or multiple applications), such as a web browser or a special purpose application, for communicating with the control server120and the executor pool130. Using the application, a user of the client computing device110, who may be a software developer, may create a job and request execution of the job at an executor135.k(where k is a number between 1 and 3) within the executor pool130.

The control server120stores data or instructions. The control server may be programmed to monitor the executors135.1-3in the executor pool130to detect that a job executing at an executor135.khas failed and to respond to the failure. More details of the operation of the control server120are discussed in conjunction withFIG. 2andFIG. 3. While the control server120is illustrated herein as being a single machine, the control server120may be implemented either as a single machine or as multiple machines. For example, the control server120may be a server farm including multiple machines where processing tasks are split between the multiple machines.

The executor pool130includes multiple executors135.1-3. While three executors135.1-3are illustrated, the subject technology may be implemented with any number of executors within the executor pool130. For example, the executor pool130may include millions of executors. Each executor135.kmay include processing hardware and a memory. The processing hardware may be programmed to receive, via the network140, a job from the client computing device110, to execute the job, and to provide, via the network140, a result of the execution of the job to the client computing device110. The executor pool130is illustrated as including executors135.1-3that are proximate to one another. However, the executors135.1-3may be located in distant geographic locations and connected to one another over the network140. As used herein, the phrase “executor pool” does not imply that the executors135.1-3of the executor pool130are in the same physical place or geographic location, but only implies that multiple executors are pooled for access via client computing device(s) and control server(s). Each executor135.kmay include any machine capable of executing a job. For example, the executor135.kmay be a physical machine, a virtual machine, or a combination of physical or virtual resources.

The subject technology is illustrated inFIG. 1as being implemented in conjunction with a single client computing device110, control server120, and executor pool130connected via a single network140. However, the subject technology may be implemented in conjunction with one or more client computing devices, control servers, executor pools, or networks. Some implementations may include multiple client computing devices (of multiple software developers) connected with multiple control servers and executor pools over the Internet and various private networks.

Furthermore,FIG. 1illustrates the client computing device110, control server120, and executor pool130being separate and distinct from one another. However, in some implementations, these different machines can be combined. For example, the control server120may reside within the executor pool130or a single machine may serve as both the client computing device110and the control server120.

FIG. 2is a block diagram of an example of the control server120ofFIG. 1. As shown, the control server120includes a processor205, a network interface210, and a memory215. The processor205executes machine instructions, which may be stored in the memory215. While a single processor205is illustrated, the control server120may include multiple processors arranged into multiple processing units (e.g., central processing unit (CPU), graphics processing unit (GPU), etc.). The processor205may include one or more processors. The network interface210allows the control server120to send and receive data via the network140. The network interface210may include one or more network interface cards (NICs). The memory215stores data or instructions. As shown, the memory215includes a detect failure module220, a determine failure reason module225, a repair executor module230, known good states235.1-n, and executor-related factors240.1-m.

The detect failure module220, when executed by the processor205, causes the processor205to detect that a job executing on an executor135.kfailed. The determine failure reason module225, when executed by the processor205, causes the processor205to determine whether the job failed due to a state of the executor135.kor due to a problem related to the job. The processor205may determine whether the job failed due to the state of the executor135.kbased on at least one factor from the executor-related factors240.1-r. The executor-related factors240.1-mmay include, among other things, an amount of time the job has been executing on the executor, an amount of code executed on the executor, an output from the executor, feedback from a machine requesting the job (e.g., client computing device110), a sequence of actions taken by the executor, and an environmental state of the executor. For example, the control server120may determine that the job failed due to the state of the executor135.kif the amount of time the job has been executing on the executor135.kis either below a lower threshold time (e.g. 1 second, 5 seconds, 10 seconds, etc.) or above an upper threshold time (e.g., 1 hour, 2 hours, etc.). The lower threshold time and upper threshold time may either be fixed amounts of time or may be variable amounts of time that depend on (e.g., are linearly proportional to) the size (e.g., in lines of code) of the job. If the processor205determines that the job failed due to the state of the executor135.k, the repair executor module230is invoked.

To avoid wrong assertion being added to executor-related factors240which might affect entire executor pool, certain thresholds could be enabled in the control server120. For example, the thresholds could be used to ensure that only X number of executors are identified with executor-related errors in a time frame of Y, where X could be a fixed number or a percentage of entire capacity of executor pool and Y could be a fixed number of minutes/hours or dynamically determined based on the frequency of executor-related errors. When these thresholds are reached, the control server120pauses some or all activities and notifies system administrators to scrutinize the results. The control server120resumes its activity automatically after the number of errors falls below the threshold. In some cases, this technique prevents activity of the control server120in a case where executor related errors are being identified excessively.

The repair executor module230, when executed by the processor205, causes the processor205to remove the executor135.kwith the problem from the executor pool130and to quarantine the executor135.k. As a result of the quarantine, the executor135.kmay not be able to accept additional jobs from client computing devices (e.g., client computing device110). The processor205executing the repair executor module230may adjust the executor135.kto one of the known good states235.1-nstored in the memory215of the control server120. The known good states235.1-nrepresent states of the executors135.1-3in the executor pool130when the executors were functioning normally and able to execute jobs without experiencing problems. Thus, adjusting the executor135.kto one of the known good states235.1-nmay cause the executor135.kto be able to execute jobs without experiencing problems. After adjusting the executor135.kto one of the known good states235.1-n, the processor205executing the repair executor module230returns the executor135.kfrom quarantine to the executor pool130.

FIG. 3is a flowchart illustrating a process300for handling a job execution failure. The process300may be implemented at the control server120.

The process300begins at step310, where the control server120detects that a job executing on an executor135.kfailed. For example, the control server120may monitor the executors135.1-3in the executor pool130for failures. Alternatively, the control server120may occasionally access an executor135.kthat is executing a job to determine whether it has failed or completed execution of the job successfully. The job may include software code for testing and may have been submitted to the executor pool130from a client computing device110of a software developer. The executor135.kmay be any physical or virtual device capable of executing software code.

At step320, the control server120determines whether the job failed due to a state of the executor135.kor due to a problem related to the job. Examples of states of the executor135.kthat could cause failure of the job include bugs in the source code of the executor135.k, the executor135.klacking access to processing, memory or network resources, contention for resources of the executor135.kwith other jobs, problems with the virtual machine, physical machine, or resources of the executor135.k, and the like. Examples of problems related to the job include bugs in the source code of the job.

The control server120may determine whether the job failed due to the state of the executor135.kbased on at least one factor from a predetermined set of executor-related factors240.1-m. The executor-related factors240.1-mmay include, among other things, an amount of time the job has been executing on the executor, an amount of code executed on the executor, an output from the executor, feedback from a machine requesting the job (e.g., client computing device110), a sequence of actions taken by the executor, and an environmental state of the executor. For example, the control server120may determine that the job failed due to the state of the executor135.kif the amount of time the job has been executing on the executor135.kis either below a lower threshold time (e.g. 1 second, 5 seconds, 10 seconds, etc.) or above an upper threshold time (e.g., 1 hour, 2 hours, etc.). The lower threshold time and upper threshold time may either be fixed amounts of time or may be variable amounts of time that depend on (e.g., are linearly proportional to) the size (e.g., in lines of code) of the job. If the control server120determines that the job failed due to the state of the executor135.k, the process300continues to step330. Otherwise, the process300continues to step325.

According to some examples, the output from the executor includes “no disk space available” or includes more complex information, such as a dynamic determination that completion is unlikely based on the progress of the job. In some examples, the output from the executor includes a job executor (e.g., implemented on a mobile phone) not being able to produce device emulators because the tooling for device emulators was uninstalled or corrupted, generating specific error messages.

According to some implementations, the feedback from the machine requesting the job includes oversight external to the job itself. For instance, the machine requesting the job measures a time difference between a time when the job was requested and a time when the result of the job was provided. If the time difference is below a minimum threshold or above a maximum threshold, the machine determines that the executor is in a bad state.

In some cases, an executor may lose power (or otherwise stop functioning) in the middle of a job. In these circumstances, the executor itself is unable to indicate that it is in a bad state. Thus, to respond to these circumstances, the machine requesting the job may determine that a maximum threshold time has expired or that the executor is not responsive (e.g., to a ping request). Upon determining that the maximum threshold time has expired or that the executor is not responsive, the machine requesting the job provides an output indicating that the executor is in a bad state, such that remediation may be performed.

At step325, upon determining that the job did not fail due to a state of the executor135.k, the control server120provides, to the client computing device110of the software developer, a notification that the job failed due to a problem related to the job. The notification may be a push notification or a notification transmitted via a messaging system, such as email. After step325, the process300ends.

At step330, upon determining that the job failed due to a problem with the executor135.k, the control server120may remove the executor135.kthat failed from the executor pool130and quarantine the executor135.kthat failed. While quarantined, the executor135.kmay be prevented from running jobs that are submitted from client computing device(s), such as client computing device110, to the executor pool130. Meanwhile, the control server120may execute the job that failed on the executor135.kon a different executor selected from among the executors135.1-3remaining in the executor pool130.

At step340, the control server120adjusts a state of the executor135.kto a known good state235.j(where j is a number between 1 and n). The known good state235.jmay be selected from a set of known good states235.1-n. The control server120may compute a difference (e.g., measured in an amount of software or hardware that needs to be modified) between the state of the executor135.kand each of the known good states235.1-n. The control server120may select the known good state235.jto which the state of the executor135.kis to be adjusted based on the difference being smaller than a threshold difference. For instance, the state235.jmay correspond to the state, from among the known good states235.1-n, that has the smallest difference from the state of the executor135.k.

At step350, after the executor135.khas been adjusted to the known good state235.j, the control server120returns the executor135.kfrom quarantine to the executor pool130. Upon return to the executor pool130, the executor135.kis able to receive job(s) from client computing device(s) and to execute the received job(s). After step350, the process300ends.

FIG. 4conceptually illustrates an electronic system400with which some implementations of the subject technology are implemented. For example, one or more of the client computing device110, the control server120, or the executors135.1-3in the executor pool130may be implemented using the arrangement of the electronic system400. The electronic system400can be a computer (e.g., a mobile phone, PDA) or any other sort of electronic device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system400includes a bus405, processor(s)410, a system memory415, a read-only memory (ROM)420, a permanent storage device425, an input device interface430, an output device interface435, and a network interface440.

The bus405collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system400. For instance, the bus405communicatively connects the processor(s)410with the read-only memory420, the system memory415, and the permanent storage device425.

From these various memory units, the processor(s)410retrieves instructions to execute and data to process in order to execute the processes of the subject technology. The processor(s) can include a single processor or a multi-core processor in different implementations.

The ROM420stores static data and instructions that are needed by the processor(s)410and other modules of the electronic system. The permanent storage device425, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system400is off. Some implementations of the subject technology use a mass-storage device (for example, a magnetic or optical disk and its corresponding disk drive) as the permanent storage device425.

Other implementations use a removable storage device (for example a floppy disk, flash disk, and the corresponding disk drive) as the permanent storage device425. Like the permanent storage device425, the system memory415is a read-and-write memory device. However, unlike storage device425, the system memory415is a volatile read-and-write memory, such as a random access memory (RAM). The system memory415stores some of the instructions and data that the processor needs at runtime. In some implementations, the processes of the subject technology are stored in the system memory415, the permanent storage device425, or the read-only memory420. For example, the various memory units include instructions for executing a job or repairing a failing executor in accordance with some implementations. From these various memory units, the processor(s)410retrieves instructions to execute and data to process in order to execute the processes of some implementations.

The bus405also connects to the input and output device interfaces430and435. The input device interface430enables the user to communicate information and select commands to the electronic system. Input devices used with input device interface430include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). Output device interface435enables, for example, the display of images generated by the electronic system400. Output devices used with output device interface435include, for example, printers and display devices (for example, cathode ray tubes (CRT) or liquid crystal displays (LCD)). Some implementations include devices that function as both input and output devices (for example, a touch screen).

Finally, as shown inFIG. 4, bus405also couples electronic system400to a network (not shown) through a network interface440. In this manner, the electronic system400can be a part of a network of computers (for example, a LAN, a WAN, or an Intranet) or a network of networks (for example, the Internet). Any or all components of electronic system400can be used in conjunction with the subject technology.

The above-described features and applications can be implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processor(s) (which may include, for example, one or more processors, cores of processors, or other processing units), they cause the processor(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, compact disk read-only memories (CD-ROMs), flash drives, RAM chips, hard drives, erasable programmable read only memories (EPROMs), and the like. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage or flash storage, for example, a solid-state drive, which can be read into memory for processing by a processor. Also, in some implementations, multiple software technologies can be implemented as sub-parts of a larger program while remaining distinct software technologies. In some implementations, multiple software technologies can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software technology described here is within the scope of the subject technology. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some aspects of the disclosed subject matter, a server transmits data (e.g., a hypertext markup language (HTML) page) to a client computing device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client computing device). Data generated at the client computing device (e.g., a result of the user interaction) can be received from the client computing device at the server.

A phrase, for example, an “aspect,” does not imply that the aspect is essential to the subject technology or that the aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or one or more configurations. A phrase, for example, an aspect, may refer to one or more aspects and vice versa. A phrase, for example, a “configuration,” does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or one or more configurations. A phrase, for example, a configuration, may refer to one or more configurations and vice versa.

In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended; that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” “third,” and so forth are used merely as labels, and are not intended to impose numerical requirements on their objects.