Fine grained failure detection in distributed computing

A client sends a request message to a process hosted by a remote server via a middleware service, wherein the request message specifies a procedure for the process to execute. The client waits a predetermined time period to receive a response message from the process. If no response message is received within the predetermined time period, the client probes the process to determine why no response message has been received, wherein said probing reveals thread level information about the process.

TECHNICAL FIELD

Embodiments of the present invention relate to distributed computing systems, and more specifically to determining statuses of processes within a distributed computing system.

BACKGROUND

Distributed computing systems include multiple services and/or applications that operate on different machines (computing devices) that are connected via a network. Some services or applications may rely on other services and/or applications to operate. However, machines, and services and applications that operate on the machines, may occasionally become unavailable (e.g., when a machine loses power, an application crashes, a network connection to the machine is lost, etc.).

In some distributed computing systems, to determine which machines, services and applications are operative at a given time, each machine in the distributed computing system can periodically transmit status inquiry messages, which are typically referred to as “are-you-alive messages” or “heartbeat messages.” The status inquiry message is a small control message that is generated and sent between machines or services on machines. A queried machine that receives the status inquiry message generates a status response message. The status response message is then sent back to the original querying machine that sent the status inquiry message. The querying machine can then receive the status response message, which provides confirmation that the queried machine and/or service is still active. Such status inquiry and status response messages may be continuously transmitted between machines within a distributed computing system at a specified frequency.

Conventional distributed computing systems can determine whether a machine or a service operating on a machine has failed. However, conventional distributed computing systems cannot detect failure at a fine grained level, such as failure of a container that houses a service or failure of individual threads within a service. Therefore, conventional distributed computing systems offer only course grained failure detection.

DETAILED DESCRIPTION

Described herein is a method and apparatus for determining statuses of fine grained components within a distributed computing system. In one embodiment, a client sends a request message to a process hosted by a remote server via a middleware service. The request message may specify a procedure for the process to execute, work to perform, etc. The client waits a predetermined time period to receive a response message from the process. If no response message is received within the predetermined time period, the client and/or the middleware service probes the process to determine why no response message has been received. By probing the process, the client and/or middleware service may determine thread level information about the process. For example, probing may reveal that a specific thread has failed, that a thread is still performing a requested operation, etc.

FIG. 1illustrates an exemplary distributed computing system100, in which embodiments of the present invention may operate. In one embodiment, the distributed computing system100includes a service oriented architecture (SOA). A service oriented architecture (SOA) is an information system architecture that organizes and uses distributed capabilities (services) for one or more applications. SOA provides a uniform means to offer, discover, interact with and use capabilities (services) distributed over a network. Through the SOA, applications may be designed that combine loosely coupled and interoperable services.

The distributed computing system100includes multiple machines (e.g., client machine105and server machine110) connected via a network115. The network115may be a public network (e.g., Internet), a private network (e.g., Ethernet or a local area Network (LAN)), or a combination thereof. In one embodiment, the network115includes an enterprise service bus (ESB). An ESB is an event-driven and standards-based messaging engine that provides services for more complex architectures. The ESB provides an infrastructure that links together services and clients to enable distributed applications and processes. The ESB may be implemented to facilitate an SOA. In one embodiment, the ESB is a single bus that logically interconnects all available services and clients. Alternatively, the ESB may include multiple buses, each of which may logically interconnect different services and/or clients.

Machines (e.g., client machine105and server machine110) may be desktop computers, laptop computers, servers, etc. Each of the machines105,110includes an operating system (e.g., first operating system120and second operating system125) that manages an allocation of resources of the machine (e.g., by allocating memory, prioritizing system requests, controlling input and output devices, managing file systems, facilitating networking, etc.). Examples of operating systems that may be included in machines105,110include Linux, UNIX, Windows®, OS X®, etc. Different machines may include different operating systems, and/or multiple machines may each include the same operating system. For example, first machine105and second machine110may each include Linux, or first machine105may include Linux and second machine110may include UNIX.

In one embodiment, first operating system120includes a client130and a client side middleware component150. Client120may be an application that runs on a machine, and that accesses services. For example, client130may initiate procedures on service160that cause service160to perform one or more operations, and may receive the results of those procedures. The client side middleware component150is described in greater detail below.

In one embodiment, second operating system125includes a service160and a server side middleware component155. The server side middleware component155is described in greater detail below. Service160is a discretely defined set of contiguous and autonomous functionality (e.g., business functionality, technical functionality, etc.) that operates on server machine110. Service160may represent a process, activity or other resource that can be accessed and used by other services or clients on network115. Service160may be independent of other services, and may be accessed without knowledge of its underlying platform implementation.

In an example for a business function of “managing orders,” service160may provide, for example, the functionality to create an order, fulfill an order, ship an order, invoice an order, cancel/update an order, etc. Service160may be autonomous from the other services that are used to manage orders, and may be remote from the other services and have different a platform implementation. However, the service160may be combined with other services and used by one or more applications to manage orders.

In one embodiment, service160includes multiple threads of execution (e.g., first thread180and second thread185). Each thread of execution (thread) may be assigned different commands, and may execute different procedures to perform different work. Each thread is only active while it is performing work, and otherwise resides in a thread pool. By using multiple threads, service160can perform two or more concurrent tasks. For example, on a multiprocessor system, multiple threads may perform their tasks simultaneously. This can allow service160to operate faster than it would operate if using only a single thread. In a further embodiment, first thread180is an active thread, and is associated with communication ports of server machine110and/or second operating system125. First thread180receives and dispatches messages, and is responsible for spawning additional threads to handle work requests (e.g., requests to execute procedures, perform operations, etc.) when required. In this embodiment, second thread185is a thread that has been spawned to handle a work request received from client130, and does not have access to communication ports.

In one embodiment, service160operates within a container140. Container140is a component (e.g., a software component) that encapsulates business logic (e.g., logic that performs core concerns of an application or service). In one embodiment, container140is an application server. An application server handles most or all of the business logic and/or data access of an application or service (e.g., service160). The application server enables applications and services to be assembled from components offered by the application server. Therefore, such applications and services may be assembled without a need to be programmed. This can simplify application development. An example of an application server is a Java Application Server (e.g., Java Platform Enterprise Edition (J2EE) Application Server).

Service160may store incoming messages in an input message buffer165and outgoing messages in an output message buffer170. The input message buffer165and output message buffer170may be maintained in a volatile memory (e.g., random access memory (RAM), a nonvolatile memory (e.g., nonvolatile random access memory (NVRAM), a hard disk drive, etc.), or a combination thereof. Contents of the incoming message buffer and outgoing message buffer may also be included in a transmission log195stored in data store185, which may be internal or externally connected with server machine110. Data store185may be a hard disk drive, optical drive, solid state memory and/or tape backup drive.

In one embodiment, second operating system125includes a process monitor145. Process monitor145monitors the activities of applications and services that are hosted by server machine110. Process monitor145may gather operating statistics of applications and/or services. Process monitor145may also monitor each application and/or service to determine a current functionality of the monitored applications and/or services. The process monitor145can monitor file system, registry, process and thread information.

To facilitate networking, each operating system120,125may include a middleware component (e.g., client side middleware component150and server side middleware component155) that facilitates the exchange of messages between the client machine105and the server machine110. The middleware components150,155are components of a middleware service135. The Middleware service135provides a layer of interconnection between processes, applications, services, etc. over network115. For example, the middleware service115may enable client130to interact with service160.

Examples of middleware services include remote procedure calls (RPC), message oriented middleware (MOM), object request brokers (ORB), enterprise service bus (ESB), etc. A remote procedure call (RPC) enables an application (e.g., client130) to cause a subroutine or procedure to execute in an address space that is different from an address space in which the application is running. For example, a remote procedure call could permit client130to cause service160to execute a procedure (e.g., to perform one or more operations). Message oriented middleware (MOM) is a client/server infrastructure that allows an application to be distributed over multiple machines, each of which may be running the same or different operating systems. Object request brokers (ORB) enable applications to make program calls between machines over a network. The most common implementation of an ORB is the common object request brokerage architecture (CORBA). Enterprise service buses (ESB) are described above.

In one embodiment, the client side middleware component150includes a first failure detection agent175, and the server side middleware component155includes a second failure detection agent178. Middleware service135may provide failure detection capabilities via one or both of first failure detection agent175and second failure detection agent178. In one embodiment, first failure detection agent175and second failure detection agent178perform both course grained failure detection and fine grained failure detection. Course grained failure detection may include detecting a status of server machine110, second operating system125and/or service160. Fine grained failure detection may include determining a status of container140, first thread180and/or second thread185. Fine grained failure detection may also include determining whether service160has received a request message, whether a thread within service160has processed the request message, whether the service160has sent a response message, etc.

First failure detection agent175and second failure detection agent178may operate independently or together. In one embodiment, some failure detection capabilities are provided by first failure detection agent175, while other failure detection capabilities are provided by second failure detection agent178. For example, some failure detection capabilities may only be performed by a failure detection agent that is external to a machine that hosts a process that is of concern, while other failure detection capabilities may only be provided by a failure detection agent that is hosted by the same machine that hosts the process that is of concern. Therefore, if service160is the process of concern, then first failure detection agent175may, for example, be able to detect whether server machine110and/or second operating system125are operable, while second failure detection agent178may not have such a capability. Alternatively, all failure detection capabilities may be provided by each failure detection agent.

The middleware service135may perform failure detection on behalf of client130. In one embodiment, failure detection is performed upon request from client130. Such a request may be received from client130if client130has failed to receive a response from service160after sending a request message to service160. Alternatively, middleware service135may automatically initiate failure detection. In one embodiment, failure detection is initiated a predetermined period of time after a message is sent from client130to service160.

In one embodiment, middleware service135is configured to probe a process to determine information about the process. Such probes may be generated and/or sent by failure detection agents175,178. Probes can be used to determine why client130has not yet received a response message from service160. Probes may be directed to container140, service160, and/or process monitor125. A probe directed to container140may request information regarding whether the container140is functioning properly and/or whether a process running within container (e.g., service160) is functioning properly. A probe directed to service160may request information regarding whether service160is still functioning properly and/or whether threads of service160are still functioning properly. A probe directed to process monitor145may request information regarding a status of container140, service160, first thread180and/or second thread185. The probe message may identify the process ID of the container140and/or service160, and may further identify the thread ID of the first thread and/or second thread. Based upon an outcome of the probe, middleware service135may elect to continue to wait for a response, retransmit the request message, cause service to retransmit a response message, terminate a thread spawned to perform operations for client, or perform other actions.

In one embodiment, middleware service135scans the input message buffer165(e.g., by sending a probe message to service160that causes it to examine the input message buffer165) to determine if a message from client130was ever received by service160. Likewise, the middleware service135may scan the output message buffer165(e.g., by sending a probe message to service160that causes it to examine the output message buffer170) to determine if the service160ever sent a response message to client130. This may enable middleware service135to determine whether any message delivery failures prevented client130from receiving a response message from service160. If, for example, it is determined that the service160never received the request message, middleware service135may resend the request message. If it is determined that service160sent a response message, but the response message was not received by client130, middleware service135may cause service160to resend the response message. In another embodiment, middleware service135probes the transaction log195to determine whether service160has sent or received specific messages.

In one embodiment, middleware service135generates two different levels of probes, each of which consists of one or more probe messages. First level probes relate to message delivery, and second level probes relate to thread liveness (whether a thread has failed, is in an infinite loop, is still performing work, etc.). These two probing mechanisms can be used separately or in parallel. For example, a first level probe and a second level probe may be sent concurrently. Alternatively, a second level probe may only be sent if no responses have been received after sending the first level probe. In one embodiment, second level probes are used after a predetermined number of first level probes have failed.

First level probes are used to determine whether there were any problems in message delivery. First level probe messages may be sent to service, or to another process that has access to service's160input message buffer165and output message buffer170. Alternatively, the first level probe message may be sent to data store185to scan transaction log195to determine whether the request message was received or a response message was sent.

First level probes may be used to determine whether a request message was received by the service and whether the service sent a response message. First level probes may be probe messages that include the original request message and/or that request information regarding the fate of the original request message. In one embodiment, the first level probe message includes instructions that cause the service160to check the input message buffer165and/or the output message buffer170to discern whether the request message was received or a response message was sent. If it is determined that the request message as not received, middleware service135may resend the request message. If it is determined that the response message was sent, the probe message may cause service160to resend the response message.

Second level probes are used to determine the liveness of individual threads within service160. In one embodiment, a second level probe includes a message that is sent to service160requesting information about individual threads of service160. The service160may then reply with information about the queried threads. In another embodiment, the second level probe includes a message sent to an external agent, such as process monitor145. In some instances, such as where service160is in an infinite loop, the service160may not be able to respond to probe messages. In such an instance, the external agent can be used to determine thread liveness information.

In one embodiment, in which a reliable message delivery protocol is implemented (e.g., TCP/IP), first level probes are not used. Such message delivery protocols guarantee that communication endpoints will be informed of any communication failures. This involves the protocol itself sending periodic low-level “are you alive” probe messages. As such, first level probe messages may not be necessary, as the information gathered by such probe messages can be implicit in the message delivery protocol. However, reliable message delivery protocols do not determine thread level information such as thread liveness. Therefore, second level probes may still be used to determine thread liveness. For example, where service160includes a single communication thread, the “are you alive” probe messages sent by the message delivery protocol would only determine whether the communication thread is alive. Second level probes would still be necessary to determine the liveness of other threads.

In conventional distributed systems, if no response message is received the request message would be resent numerous times, and the middleware service135and/or client130would wait a predetermined time after each delivery attempt. Only after repeated failures to receive a response would the conventional distributed system determine that the service160has failed. By using probe messages for fine grained failure detection, the middleware service135and/or client130may determine whether the service160is functional, whether it has received the request message, whether individual threads within the service160are functional etc. in a reduced time frame. This can also reduce the number of resend attempts, which minimizes network traffic.

In one embodiment, in which the middleware service is an RPC, probe messages are a specialized form of RPC with specific parameters. As such, a probe primitive may occur at the same level as the RPC (which includes a send primitive that is used to request the transmission of messages and a receive primitive that is used to collect messages). This allows components that generate the probe messages to sit directly on the message passing layer. In one embodiment, first failure detection agent175and second failure detection agent178sit on the message passing layer.

In one embodiment, the container140operates within a virtual machine (e.g., the Java Virtual Machine). In such an embodiment, middleware service135may also probe the virtual machine to determine whether the virtual machine has failed. The virtual machine may include multiple containers, each of which may be probed by middleware service135. Additionally, each virtual machine may include an additional operating system running within it, and the operating system may include multiple containers. Middleware service135may probe each of these components to determine whether they are operational. In one embodiment, middleware service135communicates with an additional process monitor within the virtual machine to determine status information of containers, services and/or threads that operate within the virtual machine.

In one embodiment, service160is configured to periodically inform client and/or middleware service of a current status of work being performed for client. Such periodic updates may identify operations that have already been performed, operations that are to be performed, and an estimated time to completion. Such periodic updates may also identify whether a response message has already been sent, whether a thread executing the procedure has failed, or other additional information. As long as periodic updates are received, there may be no need to send probe messages. If an expected periodic update is not received, middleware service135may then probe service160.

FIG. 2illustrates a flow diagram of one embodiment for a method200of performing fine grained failure detection. The method may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof. In one embodiment, method200is performed by a machine of distributed computing system100ofFIG. 1. In a further embodiment, the method200is performed by a middleware service135(e.g., by a failure detection agent(s)175,178of middleware service135) and/or a client130ofFIG. 1.

Referring toFIG. 2, at block205processing logic sends a request message to a process hosted by a remote server via a middleware service. For example, a client may send a request message to a service. At block210, processing logic determines whether a response message has been received within a predetermined time. Such a determination may be made, for example, by the client or by the middleware service. If a response message is received within the predetermined time, then the method ends. If no response message is received within the predetermined time, then the message proceeds to block215.

At block215, processing logic sends a first level probe to the process to determine whether there has been a problem in message delivery and/or to resolve any problems in message delivery. In one embodiment the first level probe is simply a resend of the original request message. In another embodiment, the first level probe is an explicit probe message sent to determine the fate of the original request message. Alternatively, the first level probe may include both an explicit probe message sent to determine the fate of the original message and a resend of the original request message. The explicit probe message may include instructions that cause a recipient to search through its input message buffer to determine whether the original request was received. The probe message may also cause the recipient to check an output message buffer to search for a reply message that corresponds to the request message. The first level probe message may cause the process to perform an appropriate action after searching the input buffer and/or output buffer. For example, if a response message is found in the output message, the probe message may cause the process to resend the response message.

At block220, processing logic sends a second level probe to the process or to an external agent (e.g., a process monitor) to determine the liveness of threads within the process. In one embodiment, a second level probe causes the process or external agent to check the status of a specific thread that was spawned to perform work identified in the request message. If the specific thread is not responsive, then the specific thread may be terminated and/or a new thread may be spawned to perform the work. The process may also send back a caution message, indicating that the original thread failed. The caution message may be useful, for example, in error checking. For example, if the thread became nonresponsive due to a programming bug, then the error may recur. If a caution message is repeatedly received when specific work is requested, this may indicate a programming bug.

If the process fails to respond to the first level probe message, the original request message, and/or a second level probe message sent to the process, then it may be either that the process has failed or that the threads have gone deaf (e.g., gone into an infinite loop). In such an occurrence, only an external agent can determine the reason that the service has failed to respond. If the external agent fails to respond, then it may be assumed that a machine and/or operating system hosting the process has failed. If the external agent determines that the process is not responding, then the external agent may terminate the process. The external agent may then notify processing logic that the process was terminated.

FIG. 3illustrates a flow diagram of another embodiment for a method300of performing fine grained failure detection. The method may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof. In one embodiment, method300is performed by a machine of distributed computing system100ofFIG. 1. In a further embodiment, the method300is performed by a middleware service135(e.g., by failure detection agent(s)175,178of middleware service135) and/or a client130ofFIG. 1.

Referring toFIG. 3, at block305processing logic sends a request message to a process hosted by a remote server via a middleware service. For example, a client may send a request message to a service. At block310, processing logic determines whether a response message has been received within a predetermined time. Such a determination may be made, for example, by the client or by the middleware service. If a response message is received within the predetermined time, then the method ends. If no response message is received within the predetermined time, then the message proceeds to block315.

At block315, processing logic searches an input message buffer of the process. In one embodiment, the search is performed by sending a probe message (e.g., a first level probe message) to the process. The probe message may cause the process to search its input message buffer. The process may then send a result of the search back to processing logic (e.g., back to middleware service and/or a client).

At block320, processing logic determines whether the request message was found in the input message buffer. If the request message was found in the input message buffer, this indicates that the request message was received by the process. If the request message was not found in the input message buffer, this indicates that an error occurred in transmission of the request message. If the request message was not found in the input message buffer, then the method proceeds to block325. Otherwise, the method proceeds to block330.

At block325, processing logic resends the request message. The method then returns to block310to wait for a response message.

At block330, processing logic searches an output message buffer of the process. In one embodiment, the search is performed by sending a probe message (e.g., a first level probe message) to the process. The probe message may cause the process to search its output message buffer. The process may then send a result of the search back to processing logic (e.g., back to middleware service and/or a client).

At block335, processing logic determines whether a response message was found in the output message buffer. If a response message was found in the output message buffer, this indicates that the response message was generated by the process. If the response message was not found in the input message buffer, this indicates that for some reason a response message has not yet been generated. If the response message was not found in the output message buffer, then the method proceeds to block345. Otherwise, the method proceeds to block340.

At block340, processing logic causes the process to resend the response message. In one embodiment, the probe message includes instructions that cause the process to resend the response message if the response message is found in the output message buffer. The method then returns to block310to wait for the response message.

At block345, processing logic checks the status of a thread that was spawned by the process to perform work in response to receiving the request message (e.g., to execute a procedure identified in the request message). In one embodiment the status of the thread is checked by sending a probe message (e.g., a second level probe message) to the process. In another embodiment, the status of the thread is checked by sending a probe message to an external agent such as a process monitor. The process or external agent may then reply with a message indicating a current status of the thread.

At block350, processing logic determines whether the thread has failed. If thread has failed, then the method proceeds to block355. If the threat has not failed, then the method proceeds to block360.

At block355, processing logic causes the process to respawn a thread to perform the work requested in the request message (e.g., to execute a procedure and/or perform one or more operations). In one embodiment, the probe message includes instructions that cause the process to respawn the thread. The method then returns to block310.

At block360, processing logic determines whether the thread is still performing work (e.g., still executing a procedure, performing an operation, etc.). If the thread is no longer performing work requested in the request message, then the method proceeds to block365and processing logic causes the thread to redo the work (e.g., to reexecute a procedure or operation). The method then returns to block310.

The most common reason that a response message is not received within a predetermined time period is that the sender has underestimated the time that it will take the receiving process to execute the work required. Therefore, if the thread is still performing work at block360, then the method may continue to block370and processing logic may wait an additional amount of time for the response message. The method then returns to block310. However, a thread may also still be performing work because the thread has gone into an infinite loop or has otherwise stopped responding. In such an instance the thread may never complete the work. Therefore, if the thread is still performing work, the method may continue to block375and processing logic may cause the thread to be terminated. The method then may continue to block380, and processing logic may cause the process to respawn the thread to perform the work. The method then returns to block310.

The computer system400may further include a network interface device408. The computer system400also may include a video display unit410(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device412(e.g., a keyboard), a cursor control device414(e.g., a mouse), and a signal generation device416(e.g., a speaker).

The secondary memory418may include a machine-readable storage medium (or more specifically a computer-readable storage medium)431on which is stored one or more sets of instructions (e.g., software422) embodying any one or more of the methodologies or functions described herein. The software422may also reside, completely or at least partially, within the main memory404and/or within the processing device402during execution thereof by the computer system400, the main memory404and the processing device402also constituting machine-readable storage media. The software422may further be transmitted or received over a network420via the network interface device408.

The machine-readable storage medium431may also be used to store middleware components (e.g., client side middleware component and/or server side middleware component) ofFIG. 1, and/or a software library containing methods that call the middleware components. While the machine-readable storage medium431is shown in an exemplary embodiment to be a single medium, the term “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.