Distributed task execution in different locations with dynamic formation of testing groups

Particular embodiments execute tasks to measure performance in a computing system. The method uses a master computing device and helper client computing devices. The helper client computing devices may be situated in a pool where the helper client computing devices are available to help a master computing device to perform a task. When the master computing device wants to perform a task, the master computing device may send a message to the pool requesting help with a task. Helper client computing devices can respond to the message when the helper clients are available to join in groups to process tasks. Once the master computing device configures a group with helper client computing devices that responded to the message, the master computing device and the helper client computing devices perform the task together.

BACKGROUND

Many companies use data centers to provide web services to users. Data centers include servers and other computing devices that can provide various services to client devices that users are using. In one example, the data centers host websites in which users can connect to a video delivery service to request videos for delivery to client devices.

The video delivery service would like to provide the best possible service to users. To make sure the service is working properly and has not suffered any degradation, the video delivery service may measure web services' responsiveness constantly to detect any performance degradation and/or breaches of users' service level agreement (SLA) latencies. Generally, data centers will perform their own tests individually. This, however, may not efficiently use the computing resources of each data center.

SUMMARY

In one embodiment, a method configures, in a first location, a helper client computing device with task code that all helper client computing devices can execute. The task code tests a video delivery system configured to deliver videos to video devices. The helper client computing device waits in a group of helper client computing devices for requests from a master computing device that are sent to the group of helper client computing devices in the first location for helping a master computing device in a second location with processing a task. The helper client computing device determines a request in which the helper client computing device can help with processing based on characteristics of the task and sends a response to the master computing device that the helper client computing device will help process the task. The helper client computing device receives an identifier for a first role in the task from the master computing device. The method configures the helper client computing device with the first role where the master computing device and the helper client computing device execute task code that is the same for the task. The helper client computing device determines portions of the task code associated with the first role for the helper client computing device and executes the portions of the task code to process the task to test the video delivery system. Also, the helper client computing device skips portions of the task code not associated with the first role. After finishing processing of the task, the helper client computing device rejoins the group of helper client computing devices that are available to process tasks.

In one embodiment, a non-transitory computer-readable storage medium contains instructions, that when executed, control a computer system to be configured for: configuring, in a first location, a helper client computing device with task code that all helper client computing devices can execute, wherein the task code tests a video delivery system configured to deliver videos to video devices; waiting, by the helper client computing device, in a group of helper client computing devices for requests from a master computing device that are sent to the group of helper client computing devices in the first location for helping a master computing device in a second location with processing a task; determining, by the helper client computing device, a request in which the helper client computing device can help with processing based on characteristics of the task; sending, by the helper client computing device, a response to the master computing device that the helper client computing device will help process the task; receiving, by the helper client computing device, an identifier for a first role in the task from the master computing device; configuring the helper client computing device with the first role, wherein the master computing device and the helper client computing device execute task code that is the same for the task; determining, by the helper client computing device, portions of the task code associated with the first role for the helper client computing device; executing, by the helper client computing device, the portions of the task code to process the task to test the video delivery system, wherein the helper client computing device skips portions of the task code not associated with the first role; and after finishing processing of the task, rejoining, by the helper client computing device, the group of helper client computing devices that are available to process tasks.

In one embodiment, an apparatus includes: one or more computer processors; and a non-transitory computer-readable storage medium comprising instructions, that when executed, control the one or more computer processors to be configured for: configuring, in a first location, a helper client computing device with task code that all helper client computing devices can execute, wherein the task code tests a video delivery system configured to deliver videos to video devices; waiting, by the helper client computing device, in a group of helper client computing devices for requests from a master computing device that are sent to the group of helper client computing devices in the first location for helping a master computing device in a second location with processing a task; determining, by the helper client computing device, a request in which the helper client computing device can help with processing based on characteristics of the task; sending, by the helper client computing device, a response to the master computing device that the helper client computing device will help process the task; receiving, by the helper client computing device, an identifier for a first role in the task from the master computing device; configuring the helper client computing device with the first role, wherein the master computing device and the helper client computing device execute task code that is the same for the task; determining, by the helper client computing device, portions of the task code associated with the first role for the helper client computing device; executing, by the helper client computing device, the portions of the task code to process the task to test the video delivery system, wherein the helper client computing device skips portions of the task code not associated with the first role; and after finishing processing of the task, rejoining, by the helper client computing device, the group of helper client computing devices that are available to process tasks.

DETAILED DESCRIPTION

Particular embodiments execute tasks to measure performance in a computing system. The method uses a master computing device and helper client computing devices. The helper client computing devices may be situated in a pool where the helper client computing devices are available to help a master computing device to perform a task. When the master computing device wants to perform a task, the master computing device may send a message to the pool requesting help with a task. Helper client computing devices can respond to the message when the helper clients are available to join in groups to process tasks. Once the master computing device configures a group with helper client computing devices that responded to the message, the master computing device and the helper client computing devices perform the task together. When a service (e.g., a video delivery service) includes multiple locations, such as data centers, particular embodiments allow a master computing device to perform the task using helper client computing devices in any location, such as in multiple locations. This allows efficient use of resources and also allows cross-location measurements to be made. Further, to allow the use of pools and the forming of groups, master computing device and helper client computing devices are specially configured so that the groups can be dynamically formed to execute tasks.

FIG. 1depicts a simplified system100including multiple locations that are used to perform tasks according to one embodiment. System100includes a first location102-1and a second location102-2, but more locations102may also be included. Locations102may be locations that include multiple computing devices, such as data centers. The data centers may internally link the computing devices (e.g., servers) via local area networks (LANs). Different data centers may then be externally linked via wide area networks (WANs). Generally, communications via the LAN (or internally within a data center) are faster than communications via the WAN. Locations are not restricted to data centers, however. For example, locations102may be any places in which computing devices are needed for performing tasks, such as a location including user client devices, an office including servers, etc.

Location102-1includes a pool103-1of master computing devices105-1and a pool105-1of helper client computing devices106-1. Location102-2similarly has a pool103-2of master computing devices104-2and a pool105-2of helper client computing devices106-2. It should be noted that master computing devices and helper client computing devices may be found on the same computing device, such as in different virtual machines (VMs). Further, a master computing device104may at another time become a helper client computing device106, and vice versa.

Master computing devices104-1may be computing devices that want to run a task. The master may initiate and coordinate the running of the task, which may be any set of operations that need to be performed. For example, a task may make a measurement of a service that is performed. Master computing devices104may be instantiated to form groups to run tasks constantly.

Helper client computing devices106may have multiple states. One state is an available state where helper client computing devices106are waiting to help a master computing device104with a task. Another state is an unavailable state when helper client computing devices106are currently helping a master computing device104with a task and do not have computing resources available to help another master computing device104with a task. Helper client computing devices106may transition between the available and unavailable status multiple times as tasks are processed and then finished.

Helper client computing devices106can help other master computing devices104and also other helper client computing devices to perform a task. Accordingly, one helper client computing device106may help on multiple different tasks with different master computing devices and helper client computing devices at different times. Depending on availability, different helper client computing devices106may join a group to process a task when requested.

Particular embodiments configure helper client computing devices106with logic such that helper client computing devices106can join any tasks that are requested from master computing devices104. In this way, master computing devices104can send a general request in which any helper client computing devices106can respond to join the group for the task. These helper client computing devices106can also be in different locations. This may provide a greater number of possible resources and also allow for tasks that can be tested in different locations102. By allowing any available helper client computing devices106to join any task, tasks can be completed across different locations. This allows geographical location to be used in creating tasks. For example, a master computing device104may specifically ask for a helper client computing device in a specific location102and a helper client computing device in that location may join the task if available. The task can then be written based on geographic location. Also, having the dynamic groups helps with failover situations. Conventionally, if one computing device was assigned to task A and happens to be offline/unreachable for a while, then task A is unable to run. By having dynamic groups some other helper client computing devices106can step in to take offline computing device's place. This is important as the system as a whole is measuring the responsiveness of the system under test, and being able to produce measurements under failure conditions/network outages is key.

The ability to join tasks dynamically allows for the more efficient use of resources. Helper client computing devices106may be available in the group for immediate execution of a task. Helper client computing devices106do not need to be specifically configured for the task. Rather, helper client computing devices106leverage pre-loaded logic that allows it to process the requested task. For example, as will be described below, helper client computing devices106have access to all task code in which code can be executed based on what role the helper client computing device is assigned.

Master Computing Device and Helper Client Computing Device Configuration

To allow helper client computing devices106to dynamically join groups to perform tasks, system100is pre-configured before the tasks are performed.FIG. 2depicts an example system for pre-configuring master computing devices104and helper client computing devices106according to one embodiment. A pre-configuration server202may pre-configure helper client computing devices106. In one example, pre-configuration server202loads task code204into helper client computing devices106. Task code204includes information about any tasks that can be performed by master computing devices104and helper client computing devices106. Task code204includes information such as the amount of helper client computing devices106needed for the task, the task to be executed, and the software code for the task.

Master computing devices104and helper client computing devices106are pre-configured with a shared library of system software code shown as master/helper system code206to run tasks. The system code208may include a list of tasks and when to run the tasks for master computing devices104. For helper client computing devices106, system code206allows them to join groups to perform tasks. Also, helper client computing devices106are configured to task code204when joining the group to determine how to perform the task.

In one embodiment, task code204and system code206may include code for both master computing devices104and helper client computing devices106together. This allows the same code to be distributed to both master computing devices104and helper client computing devices106. As will be discussed below, by using the same code, performing of the task in a distributed manner may be coordinated more easily. Task code204and system code206include both master computing devices104sections and helper client computing devices106sections.

One part of the pre-configuration is to configure helper client computing devices106with logic to switch between the available and unavailable status. Also, the pre-configuration does not specify which tasks will be performed by helper client computing devices106. Rather, helper client computing devices106perform tasks based on requests and which helper client computing devices106are available at that time. Helper client computing devices106may use system code206to determine attributes about themselves that will allow them to response to task requests. For example, helper client computing devices106load several constants from the system to determine what pool the helper client will use in the form of several system environment variables: “DC” contains the datacenter the client is located in, “ENV” contains the environment (Might be Production, Staging, Testing, Development, etc.), etc.

Group Formation

The first step to performing a task is for master computing device104to form a group of helper client computing devices106that will help in performing the task.FIG. 3shows a data flow to form the group of helper client computing devices according to one embodiment. When a master computing device104-1wants to perform a task, master computing device104-1may consult task code204to determine information about the task. The task may indicate whether or not helper client computing devices106are needed or not. If not, then master computing device104-1will perform the task and not form a group. When a task requires a group, master computing device104-1must dynamically form a group. The members of the group are not pre-configured and are based on availability of helper client computing devices106at the time.

In one example, the task may require the help of a helper client computing device106-2in location102-2. Further, although not shown in this data flow, the task may also require help of another helper client computing device106-1in the same location102-1as master computing device104-1.

At302, to obtain a helper client computing device in a different location, master computing device104-1sends a request for helper client computing devices to a pool105-2of helper client computing devices106-2. The request may include some parameters that are needed to join the group. For example, the parameters may specify characteristics that are needed, such as location, processing power, etc. Helper client computing devices106-2that are available in pool105-2can then respond to the request. In this case, helper client computing devices106-2with a state of available review the request. In one example, at304, a specific helper client #1responds to the request with a message to master computing device104-1. The response may indicate helper client #1is available to help. Also, other helper client computing devices106-2may also respond indicating their availability to help (not shown).

Once receiving responses from helper client computing devices106-2, at306, master computing device104-1can determine which helper client computing devices can be used to perform the task and also determine the role in which the helper client computing devices will play in performing the task. That is, a task may require multiple helper client computing devices to perform certain actions. By assigning the roles to each helper client computing device, the helper client will know which part of the task to perform. The role may be identified by an identifier, such as one role will be client #1and another role will be client #2. Once the roles are determined, at308, master computing device104-1sends an acknowledgement message and an identifier for the role of client #1to helper client computing device106-2.

Because helper client computing device106-2is located in a different location, master computing device104-1needs to determine the latency it will take to message with client #1. In this case, at310, master computing device104-1sends a request to test the latency that asks for an echo message to be sent. At312, when client #1receives the message, client #1sends an echo message. At314, master computing device104-1calculates the latency to determine how long the message took to reach client #1. This latency is then used to adjust any responses from client #1during performing of the test. This adjustment allows the task to be performed cross-location as the adjustment accounts for the latency that may occur to receive confirmation of when actions are performed in different locations. That is, using the latency of1second, master computing device104-1can determine that client #1completed action #1one second before receiving a message stating that action #1was completed.

At316, master computing device104-1then starts the task by sending a task start message along with a configuration. The configuration that is sent may identify which configuration should be used in performing a task. For example, the configuration may contain task level configurations such as # of times to perform a measurement in order to calculate an average. Additionally if a task requires some state to be shared by helper client computing devices106and master computing devices104, it may be sent within this configuration, such as a specific channel to use during a send/receive test.

FIG. 4shows an example of a group that has been configured to perform a task according to one embodiment. In this case, master computing device104-1, a client #1in second location102-2and a client #2in first location102-1are performing the task as a group. At402-1and402-2, respectively, client #1is configured with configuration #1in the task and client #2is configured with configuration #2. Also, at403, master computing device104-1is configured with a master configuration. These configurations may be used to perform different portions of the task.

Because helper client computing devices106and master computing devices104have a copy of task code204for all helper clients and masters, helper client computing devices106can look up the task in task code204. When configured with the client number in the task, helper client computing devices106then execute what sections are associated with their client number as will be discussed in more detail below.

The status of each of the clients has been moved from available to unavailable at404-1and404-2, respectively. In this case, clients #1and #2will not respond to other task requests because they are not available to help.

Task Execution

Each task may include software code that defines actions that need to be performed to execute the task.FIG. 5shows an example of software code for a task according to one embodiment. The software code shown is conceptual, and shows certain code blocks may be identified by configuration identifiers. For example, code blocks #1-#7may be code that needs to be executed by either master computing device104-1or helper client computing devices106. That is, actions A-G are performed by executing code blocks #1-#7. Each code block may be identified by a configuration role, such as code block #1is associated with the master computing device, code block #2is associated with client #1, code block #3with client #2, code block #4with the master computing device, code block #5with client #2, code block #6with the master computing device, and code block #7with client #1. In one embodiment, each master computing device and client in the task group may load the same code for execution. However, a specific master computing device or client only executes the code blocks that have been identified by its assigned configuration. That is, the master computing device executes code blocks #1, #4, and #6; client #1executes code blocks #2and #7; and client #2executes code blocks #3and #5.

By loading the same software code, helper client computing devices can automatically be configured with software code and dynamically join various tasks without being pre-configured to perform the tasks. Instead of a client #1being configured to just execute code blocks #2and #7, client #1may review all code blocks in the task, but just execute code blocks #2and #7. Various barriers may be set up such that client #1may execute its code blocks in the correct order. This is different than if client #1only had code blocks #2and #7to execute without any knowledge of what the master computing device and client #2are executing.

Because helper client computing devices106do not know what task code is going to be executed until they join groups, task code is written for all helper client computing devices106for readability, clarity, and efficiency. In order to allow helper client computing devices106to have access to the same code, helper client computing devices106only execute the task code relevant to the client's role. Without including all of the task code, coordination as to when clients can execute code is needed and which clients load which code.

FIG. 6shows a more detailed example of a client106-2, such as client #1, and a master computing device104-1, according to one embodiment. Both client #1and master computing device104-1include a computer processor602-1and a computer processor602-1602-2, respectively. An identity processor604-1and identity processor604-2, respectively, may determine the configuration being used. For example, master computing device104-1includes a master configuration403and client #1includes a client #1configuration402-1. Identity processors604-1and604-2determine the configuration and provides the identity to a code block analyzer610.

Code block analyzers610-1and610-2may determine the code for the task from a task code204. For example, task code204may include code for all tasks that may be initiated by master computing devices106-1.

Processor602-1may select the specific code for the correct task from task code204. Then, code block analyzer610-1may execute the code based on the configuration role assigned to it, such as the master configuration identifier. Also, code block analyzer610-2may execute the code based on the client configuration role assigned to it.

Master computing device106-1may retrieve the task from task code204. The task will indicate whether other helper clients are needed to perform the task. If other helper clients are needed, master computing device106-1may send the requests for other helper clients106-2as described above. If no other helper clients are needed, master computing device106-1may start to execute the task without any helper client computing devices106.

In one example, the task may want to measure how long an action taken in a location takes to perform. For example, master computing device106-1may want to know how long it takes for a helper client computing device106-2in location102-2to join a channel, such as a video channel or a communication channel. In this case, the system is testing is essentially a real-time messaging system. This messaging system is channel based, so a client would join a channel and will now receive all messages sent to that channel. A client can also send to any channel. In this case, master computing device106-1may join the channel. This may be performed by executing code block #1for master computing device106-1. Master computing device106-1may also start a timer to determine how long it takes client #1to join the channel. In a code block #2, client #1may also join the channel. After joining the channel, client #1sends an update to master computing device106-1that the channel has been joined. Master computing device106-1may stop a timer when the update is received. Then, because there may be latency when sending the update from location102-2to location102-1, master computing device106-1subtracts the latency from the time of the timer. Then, master computing device106-1now knows how long it took for client #1to join the channel. Other tests may be run such as basic application programming interface (API) calls, such as sending messages, leaving a channel, querying a channel, how long it takes to receive a presence update, or a channel message.

Once the task is finished, then client #1and client #2return to the pool of helper client computing devices. The helper client computing devices change their status and are now available to help with another task.

Particular embodiments may be used to test a messaging system being used with a video delivery system.FIG. 7depicts an example of testing the video delivery system according to one embodiment. In a location102-3, such as a user's home, a first video device702-1is messaging with a second video device702-2. For example, first video device702-1and second video device702-2may be using a video delivery service to play videos on second video device702-2(or on a television connected to second video device702-2). First video device702-1may be a smart phone or other computing device that is used to control second video device702-2, such as the smart phone is being used as a remote control for console.

To message with each other, first video device702-1and second video device702-2join a channel in a data center associated with the video delivery service. For example, first video device702-1and second video device702-2join a channel706being run on a server704in location102-2. The channel that is joined may be offered by a location (data center) that is closest to the video devices702, such as location102-2. In one embodiment, a channel can be joined by devices and messages sent to the channel are forwarded by server704to the devices that joined the channel. In one example, first video device702-1may cause second video device702-2to play a video. During the playing of the video, second video device702-2may send updates on the progress of the video to first video device702-1through channel706.

Master computing device104may want to test the response times for location102-2. To test that server704is running properly, master computing device104may test another channel708running on server704instead of the channel that video devices702joined. This may not affect the operation of a live channel being used by video devices. In this case, helper client computing device106may execute task code204to join channel708and master computing device104tests the responsiveness of the channel.

Because the video delivery service may use multiple locations102, particular embodiments provide a testing system to allow for helper client computing devices106to help perform tests in all locations102. Also, the testing system is flexible and efficient. Helper client computing devices106can join tasks when they are available, which uses resources more efficiently than if there were dedicated resources for tasks. Also, failover situations can be handled because other helper client computing devices106can join a task if some devices fail.

Particular embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in connection with the instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform a method described by particular embodiments. The computer system may include one or more computing devices. The instructions, when executed by one or more computer processors, may be configured to perform that which is described in particular embodiments.