Patent Publication Number: US-2022232069-A1

Title: Actor-and-data-grid-based distributed applications

Description:
TECHNICAL FIELD 
     The current document is directed to distributed-computer-systems and, in particular, to distributed applications with stateless-protocol interfaces, actor-based application nodes that access a distributed data grid, and automated load balancing. 
     BACKGROUND 
     During the past seven decades, electronic computing has evolved from primitive, vacuum-tube-based computer systems, initially developed during the 1940s, to modern electronic computing systems in which large numbers of multi-processor servers, work stations, and other individual computing systems are networked together with large-capacity data-storage devices and other electronic devices to produce geographically distributed computing systems with hundreds of thousands, millions, or more components that provide enormous computational bandwidths and data-storage capacities. These large, distributed computing systems are made possible by advances in computer networking, distributed operating systems and applications, data-storage appliances, computer hardware, and software technologies. However, despite all of these advances, the rapid increase in the size and complexity of computing systems has been accompanied by numerous scaling issues and technical challenges, including technical challenges associated with communications overheads encountered in parallelizing computational tasks among multiple processors, component failures, distributed-system management, and distribute-application management. As new distributed-computing technologies are developed, and as general hardware and software technologies continue to advance, the current trend towards ever-larger and more complex distributed computing systems appears likely to continue well into the future. 
     During the past 10 years, distributed applications based on micro services have become a prominent architecture for distributed-application development. Distributed service-oriented applications provide many advantages to distributed-application developers and to those who configure and manage distributed applications in cloud-computing facilities and other distributed-computational environments. However, distributed service-oriented applications have also presented many new problems and challenges, including problems and challenges associated with distributing service instances among the computational resources within a distributed-computing system and redistributing the service instances in order to respond to changes in distributed-application workloads. Distributed-application developers and users, as well as cloud-computing-facility administrators and managers, continue to seek improved methods for developing distributed service-oriented applications and improved architectures for distributed-service-oriented-applications to address these problems and challenges associated with traditional distributed service-oriented applications. 
     SUMMARY 
     The current document is directed to improved distributed service-oriented applications developed according to a new and improved architecture for developing distributed service-oriented applications. The new and improved architecture includes a stateless-communications-protocol interface to external users and clients, services implemented by actors that communicate using message passing, and a distributed data grid for persistent storage of data. Distributed service-oriented applications developed according to the new and improved architecture are referred to as “RAD-squared applications” (“RAD{circumflex over ( )}2 applications”). The acronym “RAD{circumflex over ( )}2” stands for “Rapid Application Development with REST-actor-data-grid” and the acronym “REST” stands for the Representational State Transfer (“REST”) protocol. Alternative stateless communications protocols can be used as alternatives to REST in RAD{circumflex over ( )}2 applications. Distributed service-oriented applications developed in conformance with the new and improved RAD{circumflex over ( )}2 architecture provides more efficient scaling operations and workload-balancing operations eliminating the need for application re-architecture under changing load conditions. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  provides a general architectural diagram for various types of computers. 
         FIG. 2  illustrates an Internet-connected distributed computing system. 
         FIG. 3  illustrates cloud computing. 
         FIG. 4  illustrates generalized hardware and software components of a general-purpose computer system, such as a general-purpose computer system having an architecture similar to that shown in  FIG. 1 . 
         FIGS. 5A-D  illustrate two types of virtual machine and virtual-machine execution environments. 
         FIG. 6  illustrates an OVF package. 
         FIG. 7  illustrates virtual data centers provided as an abstraction of underlying physical-data-center hardware components. 
         FIG. 8  illustrates virtual-machine components of a VI-management-server and physical servers of a physical data center above which a virtual-data-center interface is provided by the VI-management-server. 
         FIG. 9  illustrates a cloud-director level of abstraction. 
         FIG. 10  illustrates virtual-cloud-connector nodes (“VCC nodes”) and a VCC server, components of a distributed system that provides multi-cloud aggregation and that includes a cloud-connector server and cloud-connector nodes that cooperate to provide services that are distributed across multiple clouds. 
         FIG. 11  illustrates a distributed service-oriented application. 
         FIG. 12  illustrates a sequence of service calls that implement a particular distributed-service-oriented-application API call or entry point. 
         FIGS. 13A-C  provide a high-level illustration of the currently disclosed REST-actor-data-grid-based (“RAD{circumflex over ( )}2”) architecture for RAD{circumflex over ( )}2 applications. 
         FIG. 14  illustrates a RAD{circumflex over ( )}2-application node in greater detail. 
         FIG. 15  illustrates basic operations of an actor. 
         FIGS. 16A-B  illustrates logical data structures that represent the roles of message types and virtual actor addresses in a RAD{circumflex over ( )}2 application. 
         FIG. 17  indicates a motivation for workload-balancing. 
         FIGS. 18A-C  illustrate a workload-balancing example. 
         FIGS. 19A-E  illustrate a page-rank algorithm used to rank webpages based on the number of hyperlinks directed from the webpages to other webpages. 
         FIGS. 20A-C  and  21 A-F show a simple C++ implementation of the workload-estimation method for determining the percentages of a total workload handled by each actor, as in the example discussed above with reference to  FIGS. 18A-C . 
         FIG. 22  illustrates the workload-distribution results generated by applying the workload-distribution calculation shown in the C++ implementation discussed above with reference to  FIGS. 20A-21F  to the graph shown in  FIG. 18C . 
         FIGS. 23A-C  illustrate, using the example of  FIGS. 18A-C , how the workload distribution computed by the method illustrated by the implementation provided in  FIGS. 20A-21F  and shown in  FIG. 22  can be used to distribute the workload more evenly across RAD{circumflex over ( )}2-application nodes. 
         FIGS. 24A-B  provide control-flow diagrams that illustrate fine-grained workload-balancing within a RAD{circumflex over ( )}2 application. 
     
    
    
     DETAILED DESCRIPTION 
     The current document is directed to distributed service-oriented applications developed according to a new and improved architecture. In a first subsection, below, a detailed description of computer hardware, complex computational systems, and virtualization is provided with reference to  FIGS. 1-10 . An overview of distributed service-oriented applications is provided in a second subsection, with reference to  FIGS. 11-12 . The currently disclosed methods and systems are discussed with reference to  FIGS. 13A-2413  in a third and final subsection. 
     Computer Hardware, Complex Computational Systems, and Virtualization 
     The term “abstraction” is not, in any way, intended to mean or suggest an abstract idea or concept. Computational abstractions are tangible, physical interfaces that are implemented, ultimately, using physical computer hardware, data-storage devices, and communications systems. Instead, the term “abstraction” refers, in the current discussion, to a logical level of functionality encapsulated within one or more concrete, tangible, physically-implemented computer systems with defined interfaces through which electronically-encoded data is exchanged, process execution launched, and electronic services are provided. Interfaces may include graphical and textual data displayed on physical display devices as well as computer programs and routines that control physical computer processors to carry out various tasks and operations and that are invoked through electronically implemented application programming interfaces (“APIs”) and other electronically implemented interfaces. There is a tendency among those unfamiliar with modern technology and science to misinterpret the terms “abstract” and “abstraction,” when used to describe certain aspects of modern computing. For example, one frequently encounters assertions that, because a computational system is described in terms of abstractions, functional layers, and interfaces, the computational system is somehow different from a physical machine or device. Such allegations are unfounded. One only needs to disconnect a computer system or group of computer systems from their respective power supplies to appreciate the physical, machine nature of complex computer technologies. One also frequently encounters statements that characterize a computational technology as being “only software,” and thus not a machine or device. Software is essentially a sequence of encoded symbols, such as a printout of a computer program or digitally encoded computer instructions sequentially stored in a file on an optical disk or within an electromechanical mass-storage device. Software alone can do nothing. It is only when encoded computer instructions are loaded into an electronic memory within a computer system and executed on a physical processor that so-called “software implemented” functionality is provided. The digitally encoded computer instructions are an essential and physical control component of processor-controlled machines and devices, no less essential and physical than a cam-shaft control system in an internal-combustion engine. Multi-cloud aggregations, cloud-computing services, virtual-machine containers and virtual machines, communications interfaces, and many of the other topics discussed below are tangible, physical components of physical, electro-optical-mechanical computer systems. 
       FIG. 1  provides a general architectural diagram for various types of computers. The computer system contains one or multiple central processing units (“CPUs”)  102 - 105 , one or more electronic memories  108  interconnected with the CPUs by a CPU/memory-subsystem bus  110  or multiple busses, a first bridge  112  that interconnects the CPU/memory-subsystem bus  110  with additional busses  114  and  116 , or other types of high-speed interconnection media, including multiple, high-speed serial interconnects. These busses or serial interconnections, in turn, connect the CPUs and memory with specialized processors, such as a graphics processor  118 , and with one or more additional bridges  120 , which are interconnected with high-speed serial links or with multiple controllers  122 - 127 , such as controller  127 , that provide access to various different types of mass-storage devices  128 , electronic displays, input devices, and other such components, subcomponents, and computational resources. It should be noted that computer-readable data-storage devices include optical and electromagnetic disks, electronic memories, and other physical data-storage devices. Those familiar with modern science and technology appreciate that electromagnetic radiation and propagating signals do not store data for subsequent retrieval and can transiently “store” only a byte or less of information per mile, far less information than needed to encode even the simplest of routines. 
     Of course, there are many different types of computer-system architectures that differ from one another in the number of different memories, including different types of hierarchical cache memories, the number of processors and the connectivity of the processors with other system components, the number of internal communications busses and serial links, and in many other ways. However, computer systems generally execute stored programs by fetching instructions from memory and executing the instructions in one or more processors. Computer systems include general-purpose computer systems, such as personal computers (“PCs”), various types of servers and workstations, and higher-end mainframe computers, but may also include a plethora of various types of special-purpose computing devices, including data-storage systems, communications routers, network nodes, tablet computers, and mobile telephones. 
       FIG. 2  illustrates an Internet-connected distributed computing system. As communications and networking technologies have evolved in capability and accessibility, and as the computational bandwidths, data-storage capacities, and other capabilities and capacities of various types of computer systems have steadily and rapidly increased, much of modern computing now generally involves large distributed systems and computers interconnected by local networks, wide-area networks, wireless communications, and the Internet.  FIG. 2  shows a typical distributed system in which a large number of PCs  202 - 205 , a high-end distributed mainframe system  210  with a large data-storage system  212 , and a large computer center  214  with large numbers of rack-mounted servers or blade servers all interconnected through various communications and networking systems that together comprise the Internet  216 . Such distributed computing systems provide diverse arrays of functionalities. For example, a PC user sitting in a home office may access hundreds of millions of different web sites provided by hundreds of thousands of different web servers throughout the world and may access high-computational-bandwidth computing services from remote computer facilities for running complex computational tasks. 
     Until recently, computational services were generally provided by computer systems and data centers purchased, configured, managed, and maintained by service-provider organizations. For example, an e-commerce retailer generally purchased, configured, managed, and maintained a data center including numerous web servers, back-end computer systems, and data-storage systems for serving web pages to remote customers, receiving orders through the web-page interface, processing the orders, tracking completed orders, and other myriad different tasks associated with an e-commerce enterprise. 
       FIG. 3  illustrates cloud computing. In the recently developed cloud-computing paradigm, computing cycles and data-storage facilities are provided to organizations and individuals by cloud-computing providers. In addition, larger organizations may elect to establish private cloud-computing facilities in addition to, or instead of, subscribing to computing services provided by public cloud-computing service providers. In  FIG. 3 , a system administrator for an organization, using a PC  302 , accesses the organization&#39;s private cloud  304  through a local network  306  and private-cloud interface  308  and also accesses, through the Internet  310 , a public cloud  312  through a public-cloud services interface  314 . The administrator can, in either the case of the private cloud  304  or public cloud  312 , configure virtual computer systems and even entire virtual data centers and launch execution of application programs on the virtual computer systems and virtual data centers in order to carry out any of many different types of computational tasks. As one example, a small organization may configure and run a virtual data center within a public cloud that executes web servers to provide an e-commerce interface through the public cloud to remote customers of the organization, such as a user viewing the organization&#39;s e-commerce web pages on a remote user system  316 . 
     Cloud-computing facilities are intended to provide computational bandwidth and data-storage services much as utility companies provide electrical power and water to consumers. Cloud computing provides enormous advantages to small organizations without the resources to purchase, manage, and maintain in-house data centers. Such organizations can dynamically add and delete virtual computer systems from their virtual data centers within public clouds in order to track computational-bandwidth and data-storage needs, rather than purchasing sufficient computer systems within a physical data center to handle peak computational-bandwidth and data-storage demands. Moreover, small organizations can completely avoid the overhead of maintaining and managing physical computer systems, including hiring and periodically retraining information-technology specialists and continuously paying for operating-system and database-management-system upgrades. Furthermore, cloud-computing interfaces allow for easy and straightforward configuration of virtual computing facilities, flexibility in the types of applications and operating systems that can be configured, and other functionalities that are useful even for owners and administrators of private cloud-computing facilities used by a single organization. 
       FIG. 4  illustrates generalized hardware and software components of a general-purpose computer system, such as a general-purpose computer system having an architecture similar to that shown in  FIG. 1 . The computer system  400  is often considered to include three fundamental layers: (1) a hardware layer or level  402 ; (2) an operating-system layer or level  404 ; and (3) an application-program layer or level  406 . The hardware layer  402  includes one or more processors  408 , system memory  410 , various different types of input-output (“I/O”) devices  410  and  412 , and mass-storage devices  414 . Of course, the hardware level also includes many other components, including power supplies, internal communications links and busses, specialized integrated circuits, many different types of processor-controlled or microprocessor-controlled peripheral devices and controllers, and many other components. The operating system  404  interfaces to the hardware level  402  through a low-level operating system and hardware interface  416  generally comprising a set of non-privileged computer instructions  418 , a set of privileged computer instructions  420 , a set of non-privileged registers and memory addresses  422 , and a set of privileged registers and memory addresses  424 . In general, the operating system exposes non-privileged instructions, non-privileged registers, and non-privileged memory addresses  426  and a system-call interface  428  as an operating-system interface  430  to application programs  432 - 436  that execute within an execution environment provided to the application programs by the operating system. The operating system, alone, accesses the privileged instructions, privileged registers, and privileged memory addresses. By reserving access to privileged instructions, privileged registers, and privileged memory addresses, the operating system can ensure that application programs and other higher-level computational entities cannot interfere with one another&#39;s execution and cannot change the overall state of the computer system in ways that could deleteriously impact system operation. The operating system includes many internal components and modules, including a scheduler  442 , memory management  444 , a file system  446 , device drivers  448 , and many other components and modules. To a certain degree, modern operating systems provide numerous levels of abstraction above the hardware level, including virtual memory, which provides to each application program and other computational entities a separate, large, linear memory-address space that is mapped by the operating system to various electronic memories and mass-storage devices. The scheduler orchestrates interleaved execution of various different application programs and higher-level computational entities, providing to each application program a virtual, stand-alone system devoted entirely to the application program. From the application program&#39;s standpoint, the application program executes continuously without concern for the need to share processor resources and other system resources with other application programs and higher-level computational entities. The device drivers abstract details of hardware-component operation, allowing application programs to employ the system-call interface for transmitting and receiving data to and from communications networks, mass-storage devices, and other I/O devices and subsystems. The file system  436  facilitates abstraction of mass-storage-device and memory resources as a high-level, easy-to-access, file-system interface. Thus, the development and evolution of the operating system has resulted in the generation of a type of multi-faceted virtual execution environment for application programs and other higher-level computational entities. 
     While the execution environments provided by operating systems have proved to be an enormously successful level of abstraction within computer systems, the operating-system-provided level of abstraction is nonetheless associated with difficulties and challenges for developers and users of application programs and other higher-level computational entities. One difficulty arises from the fact that there are many different operating systems that run within various different types of computer hardware. In many cases, popular application programs and computational systems are developed to run on only a subset of the available operating systems and can therefore be executed within only a subset of the various different types of computer systems on which the operating systems are designed to run. Often, even when an application program or other computational system is ported to additional operating systems, the application program or other computational system can nonetheless run more efficiently on the operating systems for which the application program or other computational system was originally targeted. Another difficulty arises from the increasingly distributed nature of computer systems. Although distributed operating systems are the subject of considerable research and development efforts, many of the popular operating systems are designed primarily for execution on a single computer system. In many cases, it is difficult to move application programs, in real time, between the different computer systems of a distributed computing system for high-availability, fault-tolerance, and workload-balancing purposes. The problems are even greater in heterogeneous distributed computing systems which include different types of hardware and devices running different types of operating systems. Operating systems continue to evolve, as a result of which certain older application programs and other computational entities may be incompatible with more recent versions of operating systems for which they are targeted, creating compatibility issues that are particularly difficult to manage in large distributed systems. 
     For all of these reasons, a higher level of abstraction, referred to as the “virtual machine,” has been developed and evolved to further abstract computer hardware in order to address many difficulties and challenges associated with traditional computing systems, including the compatibility issues discussed above.  FIGS. 5A-D  illustrate several types of virtual machine and virtual-machine execution environments.  FIGS. 5A-B  use the same illustration conventions as used in  FIG. 4 . Figure SA shows a first type of virtualization. The computer system  500  in  FIG. 5A  includes the same hardware layer  502  as the hardware layer  402  shown in  FIG. 4 . However, rather than providing an operating system layer directly above the hardware layer, as in  FIG. 4 , the virtualized computing environment illustrated in  FIG. 5A  features a virtualization layer  504  that interfaces through a virtualization-layer/hardware-layer interface  506 , equivalent to interface  416  in  FIG. 4 , to the hardware. The virtualization layer provides a hardware-like interface  508  to a number of virtual machines, such as virtual machine  510 , executing above the virtualization layer in a virtual-machine layer  512 . Each virtual machine includes one or more application programs or other higher-level computational entities packaged together with an operating system, referred to as a “guest operating system,” such as application  514  and guest operating system  516  packaged together within virtual machine  510 . Each virtual machine is thus equivalent to the operating-system layer  404  and application-program layer  406  in the general-purpose computer system shown in  FIG. 4 . Each guest operating system within a virtual machine interfaces to the virtualization-layer interface  508  rather than to the actual hardware interface  506 . The virtualization layer partitions hardware resources into abstract virtual-hardware layers to which each guest operating system within a virtual machine interfaces. The guest operating systems within the virtual machines, in general, are unaware of the virtualization layer and operate as if they were directly accessing a true hardware interface. The virtualization layer ensures that each of the virtual machines currently executing within the virtual environment receive a fair allocation of underlying hardware resources and that all virtual machines receive sufficient resources to progress in execution. The virtualization-layer interface  508  may differ for different guest operating systems. For example, the virtualization layer is generally able to provide virtual hardware interfaces for a variety of different types of computer hardware. This allows, as one example, a virtual machine that includes a guest operating system designed for a particular computer architecture to run on hardware of a different architecture. The number of virtual machines need not be equal to the number of physical processors or even a multiple of the number of processors. 
     The virtualization layer includes a virtual-machine-monitor module  518  (“VMM”) that virtualizes physical processors in the hardware layer to create virtual processors on which each of the virtual machines executes. For execution efficiency, the virtualization layer attempts to allow virtual machines to directly execute non-privileged instructions and to directly access non-privileged registers and memory. However, when the guest operating system within a virtual machine accesses virtual privileged instructions, virtual privileged registers, and virtual privileged memory through the virtualization-layer interface  508 , the accesses result in execution of virtualization-layer code to simulate or emulate the privileged resources. The virtualization layer additionally includes a kernel module  520  that manages memory, communications, and data-storage machine resources on behalf of executing virtual machines (“VM kernel”). The VM kernel, for example, maintains shadow page tables on each virtual machine so that hardware-level virtual-memory facilities can be used to process memory accesses. The VM kernel additionally includes routines that implement virtual communications and data-storage devices as well as device drivers that directly control the operation of underlying hardware communications and data-storage devices. Similarly, the VM kernel virtualizes various other types of I/O devices, including keyboards, optical-disk drives, and other such devices. The virtualization layer essentially schedules execution of virtual machines much like an operating system schedules execution of application programs, so that the virtual machines each execute within a complete and fully functional virtual hardware layer. 
       FIG. 5B  illustrates a second type of virtualization. In  FIG. 5B , the computer system  540  includes the same hardware layer  542  and software layer  544  as the hardware layer  402  shown in  FIG. 4 . Several application programs  546  and  548  are shown running in the execution environment provided by the operating system. In addition, a virtualization layer  550  is also provided, in computer  540 , but, unlike the virtualization layer  504  discussed with reference to  FIG. 5A , virtualization layer  550  is layered above the operating system  544 , referred to as the “host OS,” and uses the operating system interface to access operating-system-provided functionality as well as the hardware. The virtualization layer  550  comprises primarily a VMM and a hardware-like interface  552 , similar to hardware-like interface  508  in  FIG. 5A . The virtualization-layer/hardware-layer interface  552 , equivalent to interface  416  in  FIG. 4 , provides an execution environment for a number of virtual machines  556 - 558 , each including one or more application programs or other higher-level computational entities packaged together with a guest operating system. 
     While the traditional virtual-machine-based virtualization layers, described with reference to  FIGS. 5A-B , have enjoyed widespread adoption and use in a variety of different environments, from personal computers to enormous distributed computing systems, traditional virtualization technologies are associated with computational overheads. While these computational overheads have been steadily decreased, over the years, and often represent ten percent or less of the total computational bandwidth consumed by an application running in a virtualized environment, traditional virtualization technologies nonetheless involve computational costs in return for the power and flexibility that they provide. Another approach to virtualization is referred to as operating-system-level virtualization (“OSL virtualization”).  FIG. 5C  illustrates the OSL-virtualization approach. In  FIG. 5C , as in previously discussed  FIG. 4 , an operating system  404  runs above the hardware  402  of a host computer. The operating system provides an interface for higher-level computational entities, the interface including a system-call interface  428  and exposure to the non-privileged instructions and memory addresses and registers  426  of the hardware layer  402 . However, unlike in  FIG. 5A , rather than applications running directly above the operating system, OSL virtualization involves an OS-level virtualization layer  560  that provides an operating-system interface  562 - 564  to each of one or more containers  566 - 568 . The containers, in turn, provide an execution environment for one or more applications, such as application  570  running within the execution environment provided by container  566 . The container can be thought of as a partition of the resources generally available to higher-level computational entities through the operating system interface  430 . While a traditional virtualization layer can simulate the hardware interface expected by any of many different operating systems, OSL virtualization essentially provides a secure partition of the execution environment provided by a particular operating system. As one example, OSL virtualization provides a file system to each container, but the file system provided to the container is essentially a view of a partition of the general file system provided by the underlying operating system. In essence. OSL virtualization uses operating-system features, such as name space support, to isolate each container from the remaining containers so that the applications executing within the execution environment provided by a container are isolated from applications executing within the execution environments provided by all other containers. As a result, a container can be booted up much faster than a virtual machine, since the container uses operating-system-kernel features that are already available within the host computer. Furthermore, the containers share computational bandwidth, memory, network bandwidth, and other computational resources provided by the operating system, without resource overhead allocated to virtual machines and virtualization layers. Again, however, OSL virtualization does not provide many desirable features of traditional virtualization. As mentioned above, OSL virtualization does not provide a way to run different types of operating systems for different groups of containers within the same host system, nor does OSL-virtualization provide for live migration of containers between host computers, as does traditional virtualization technologies. 
       FIG. 5D  illustrates an approach to combining the power and flexibility of traditional virtualization with the advantages of OSL virtualization.  FIG. 5D  shows a host computer similar to that shown in  FIG. 5A , discussed above. The host computer includes a hardware layer  502  and a virtualization layer  504  that provides a simulated hardware interface  508  to an operating system  572 . Unlike in  FIG. 5A , the operating system interfaces to an OSL-virtualization layer  574  that provides container execution environments  576 - 578  to multiple application programs. Running containers above a guest operating system within a virtualized host computer provides many of the advantages of traditional virtualization and OSL virtualization. Containers can be quickly booted in order to provide additional execution environments and associated resources to new applications. The resources available to the guest operating system are efficiently partitioned among the containers provided by the OSL-virtualization layer  574 . Many of the powerful and flexible features of the traditional virtualization technology can be applied to containers running above guest operating systems including live migration from one host computer to another, various types of high-availability and distributed resource sharing, and other such features. Containers provide share-based allocation of computational resources to groups of applications with guaranteed isolation of applications in one container from applications in the remaining containers executing above a guest operating system. Moreover, resource allocation can be modified at run time between containers. The traditional virtualization layer provides flexible and easy scaling and a simple approach to operating-system upgrades and patches. Thus, the use of OSL virtualization above traditional virtualization, as illustrated in  FIG. 5D , provides much of the advantages of both a traditional virtualization layer and the advantages of OSL virtualization. Note that, although only a single guest operating system and OSL virtualization layer as shown in  FIG. 5D , a single virtualized host system can run multiple different guest operating systems within multiple virtual machines, each of which supports one or more containers. 
     A virtual machine or virtual application, described below, is encapsulated within a data package for transmission, distribution, and loading into a virtual-execution environment. One public standard for virtual-machine encapsulation is referred to as the “open virtualization format” (“OVF”). The OVF standard specifies a format for digitally encoding a virtual machine within one or more data files.  FIG. 6  illustrates an OVF package. An OVF package  602  includes an OVF descriptor  604 , an OVF manifest  606 , an OVF certificate  608 , one or more disk-image files  610 - 611 , and one or more resource files  612 - 614 . The OVF package can be encoded and stored as a single file or as a set of files. The OVF descriptor  604  is an XML document  620  that includes a hierarchical set of elements, each demarcated by a beginning tag and an ending tag. The outermost, or highest-level, element is the envelope element, demarcated by tags  622  and  623 . The next-level element includes a reference element  626  that includes references to all files that are part of the OVF package, a disk section  628  that contains meta information about all of the virtual disks included in the OVF package, a networks section  630  that includes meta information about all of the logical networks included in the OVF package, and a collection of virtual-machine configurations  632  which further includes hardware descriptions of each virtual machine  634 . There are many additional hierarchical levels and elements within a typical OVF descriptor. The OVF descriptor is thus a self-describing XML file that describes the contents of an OVF package. The OVF manifest  606  is a list of cryptographic-hash-function-generated digests  636  of the entire OVF package and of the various components of the OVF package. The OVF certificate  608  is an authentication certificate  640  that includes a digest of the manifest and that is cryptographically signed. Disk image files, such as disk image file  610 , are digital encodings of the contents of virtual disks and resource files  612  are digitally encoded content, such as operating-system images. A virtual machine or a collection of virtual machines encapsulated together within a virtual application can thus be digitally encoded as one or more files within an OVF package that can be transmitted, distributed, and loaded using well-known tools for transmitting, distributing, and loading files. A virtual appliance is a software service that is delivered as a complete software stack installed within one or more virtual machines that is encoded within an OVF package. 
     The advent of virtual machines and virtual environments has alleviated many of the difficulties and challenges associated with traditional general-purpose computing. Machine and operating-system dependencies can be significantly reduced or entirely eliminated by packaging applications and operating systems together as virtual machines and virtual appliances that execute within virtual environments provided by virtualization layers running on many different types of computer hardware. A next level of abstraction, referred to as virtual data centers which are one example of a broader virtual-infrastructure category, provide a data-center interface to virtual data centers computationally constructed within physical data centers.  FIG. 7  illustrates virtual data centers provided as an abstraction of underlying physical-data-center hardware components. In  FIG. 7 , a physical data center  702  is shown below a virtual-interface plane  704 . The physical data center consists of a virtual-infrastructure management server (“VI-management-server”)  706  and any of various different computers, such as PCs  708 , on which a virtual-data-center management interface may be displayed to system administrators and other users. The physical data center additionally includes generally large numbers of server computers, such as server computer  710 , that are coupled together by local area networks, such as local area network  712  that directly interconnects server computer  710  and  714 - 720  and a mass-storage array  722 . The physical data center shown in  FIG. 7  includes three local area networks  712 ,  724 , and  726  that each directly interconnects a bank of eight servers and a mass-storage array. The individual server computers, such as server computer  710 , each includes a virtualization layer and runs multiple virtual machines. Different physical data centers may include many different types of computers, networks, data-storage systems and devices connected according to many different types of connection topologies. The virtual-data-center abstraction layer  704 , a logical abstraction layer shown by a plane in  FIG. 7 , abstracts the physical data center to a virtual data center comprising one or more resource pools, such as resource pools  730 - 732 , one or more virtual data stores, such as virtual data stores  734 - 736 , and one or more virtual networks. In certain implementations, the resource pools abstract banks of physical servers directly interconnected by a local area network. 
     The virtual-data-center management interface allows provisioning and launching of virtual machines with respect to resource pools, virtual data stores, and virtual networks, so that virtual-data-center administrators need not be concerned with the identities of physical-data-center components used to execute particular virtual machines. Furthermore, the VI-management-server includes functionality to migrate running virtual machines from one physical server to another in order to optimally or near optimally manage resource allocation, provide fault tolerance, and high availability by migrating virtual machines to most effectively utilize underlying physical hardware resources, to replace virtual machines disabled by physical hardware problems and failures, and to ensure that multiple virtual machines supporting a high-availability virtual appliance are executing on multiple physical computer systems so that the services provided by the virtual appliance are continuously accessible, even when one of the multiple virtual appliances becomes compute bound, data-access bound, suspends execution, or fails. Thus, the virtual data center layer of abstraction provides a virtual-data-center abstraction of physical data centers to simplify provisioning, launching, and maintenance of virtual machines and virtual appliances as well as to provide high-level, distributed functionalities that involve pooling the resources of individual physical servers and migrating virtual machines among physical servers to achieve load balancing, fault tolerance, and high availability. 
       FIG. 8  illustrates virtual-machine components of a VI-management-server and physical servers of a physical data center above which a virtual-data-center interface is provided by the VI-management-server. The VI-management-server  802  and a virtual-data-center database  804  comprise the physical components of the management component of the virtual data center. The VI-management-server  802  includes a hardware layer  806  and virtualization layer  808  and runs a virtual-data-center management-server virtual machine  810  above the virtualization layer. Although shown as a single server in  FIG. 8 , the VI-management-server (“VI management server”) may include two or more physical server computers that support multiple VI-management-server virtual appliances. The virtual machine  810  includes a management-interface component  812 , distributed services  814 , core services  816 , and a host-management interface  818 . The management interface is accessed from any of various computers, such as the PC  708  shown in  FIG. 7 . The management interface allows the virtual-data-center administrator to configure a virtual data center, provision virtual machines, collect statistics and view log files for the virtual data center, and to carry out other, similar management tasks. The host-management interface  818  interfaces to virtual-data-center agents  824 ,  825 , and  826  that execute as virtual machines within each of the physical servers of the physical data center that is abstracted to a virtual data center by the VI management server. 
     The distributed services  814  include a distributed-resource scheduler that assigns virtual machines to execute within particular physical servers and that migrates virtual machines in order to most effectively make use of computational bandwidths, data-storage capacities, and network capacities of the physical data center. The distributed services further include a high-availability service that replicates and migrates virtual machines in order to ensure that virtual machines continue to execute despite problems and failures experienced by physical hardware components. The distributed services also include a live-virtual-machine migration service that temporarily halts execution of a virtual machine, encapsulates the virtual machine in an OVF package, transmits the OVF package to a different physical server, and restarts the virtual machine on the different physical server from a virtual-machine state recorded when execution of the virtual machine was halted. The distributed services also include a distributed backup service that provides centralized virtual-machine backup and restore. 
     The core services provided by the VI management server include host configuration, virtual-machine configuration, virtual-machine provisioning, generation of virtual-data-center alarms and events, ongoing event logging and statistics collection, a task scheduler, and a resource-management module. Each physical server  820 - 822  also includes a host-agent virtual machine  828 - 830  through which the virtualization layer can be accessed via a virtual-infrastructure application programming interface (“API”). This interface allows a remote administrator or user to manage an individual server through the infrastructure API. The virtual-data-center agents  824 - 826  access virtualization-layer server information through the host agents. The virtual-data-center agents are primarily responsible for offloading certain of the virtual-data-center management-server functions specific to a particular physical server to that physical server. The virtual-data-center agents relay and enforce resource allocations made by the VI management server, relay virtual-machine provisioning and configuration-change commands to host agents, monitor and collect performance statistics, alarms, and events communicated to the virtual-data-center agents by the local host agents through the interface API, and to carry out other, similar virtual-data-management tasks. 
     The virtual-data-center abstraction provides a convenient and efficient level of abstraction for exposing the computational resources of a cloud-computing facility to cloud-computing-infrastructure users. A cloud-director management server exposes virtual resources of a cloud-computing facility to cloud-computing-infrastructure users. In addition, the cloud director introduces a multi-tenancy layer of abstraction, which partitions virtual data centers (“VDCs”) into tenant-associated VDCs that can each be allocated to a particular individual tenant or tenant organization, both referred to as a “tenant.” A given tenant can be provided one or more tenant-associated VDCs by a cloud director managing the multi-tenancy layer of abstraction within a cloud-computing facility. The cloud services interface ( 308  in  FIG. 3 ) exposes a virtual-data-center management interface that abstracts the physical data center. 
       FIG. 9  illustrates a cloud-director level of abstraction. In  FIG. 9 , three different physical data centers  902 - 904  are shown below planes representing the cloud-director layer of abstraction  906 - 908 . Above the planes representing the cloud-director level of abstraction, multi-tenant virtual data centers  910 - 912  are shown. The resources of these multi-tenant virtual data centers are securely partitioned in order to provide secure virtual data centers to multiple tenants, or cloud-services-accessing organizations. For example, a cloud-services-provider virtual data center  910  is partitioned into four different tenant-associated virtual-data centers within a multi-tenant virtual data center for four different tenants  916 - 919 . Each multi-tenant virtual data center is managed by a cloud director comprising one or more cloud-director servers  920 - 922  and associated cloud-director databases  924 - 926 . Each cloud-director server or servers runs a cloud-director virtual appliance  930  that includes a cloud-director management interface  932 , a set of cloud-director services  934 , and a virtual-data-center management-server interface  936 . The cloud-director services include an interface and tools for provisioning multi-tenant virtual data center virtual data centers on behalf of tenants, tools and interfaces for configuring and managing tenant organizations, tools and services for organization of virtual data centers and tenant-associated virtual data centers within the multi-tenant virtual data center, services associated with template and media catalogs, and provisioning of virtualization networks from a network pool. Templates are virtual machines that each contains an OS and/or one or more virtual machines containing applications. A template may include much of the detailed contents of virtual machines and virtual appliances that are encoded within OVF packages, so that the task of configuring a virtual machine or virtual appliance is significantly simplified, requiring only deployment of one OVF package. These templates are stored in catalogs within a tenant&#39;s virtual-data center. These catalogs are used for developing and staging new virtual appliances and published catalogs are used for sharing templates in virtual appliances across organizations. Catalogs may include OS images and other information relevant to construction, distribution, and provisioning of virtual appliances. 
     Considering  FIGS. 7 and 9 , the VI management server and cloud-director layers of abstraction can be seen, as discussed above, to facilitate employment of the virtual-data-center concept within private and public clouds. However, this level of abstraction does not fully facilitate aggregation of single-tenant and multi-tenant virtual data centers into heterogeneous or homogeneous aggregations of cloud-computing facilities. 
       FIG. 10  illustrates virtual-cloud-connector nodes (“VCC nodes”) and a VCC server, components of a distributed system that provides multi-cloud aggregation and that includes a cloud-connector server and cloud-connector nodes that cooperate to provide services that are distributed across multiple clouds. VMware vCloud™ VCC servers and nodes are one example of VCC server and nodes. In  FIG. 10 , seven different cloud-computing facilities are illustrated  1002 - 1008 . Cloud-computing facility  1002  is a private multi-tenant cloud with a cloud director  1010  that interfaces to a VI management server  1012  to provide a multi-tenant private cloud comprising multiple tenant-associated virtual data centers. The remaining cloud-computing facilities  1003 - 1008  may be either public or private cloud-computing facilities and may be single-tenant virtual data centers, such as virtual data centers  1003  and  1006 , multi-tenant virtual data centers, such as multi-tenant virtual data centers  1004  and  1007 - 1008 , or any of various different kinds of third-party cloud-services facilities, such as third-party cloud-services facility  1005 . An additional component, the VCC server  1014 , acting as a controller is included in the private cloud-computing facility  1002  and interfaces to a VCC node  1016  that runs as a virtual appliance within the cloud director  1010 . A VCC server may also run as a virtual appliance within a VI management server that manages a single-tenant private cloud. The VCC server  1014  additionally interfaces, through the Internet, to VCC node virtual appliances executing within remote VI management servers, remote cloud directors, or within the third-party cloud services  1018 - 1023 . The VCC server provides a VCC server interface that can be displayed on a local or remote terminal, PC, or other computer system  1026  to allow a cloud-aggregation administrator or other user to access VCC-server-provided aggregate-cloud distributed services. In general, the cloud-computing facilities that together form a multiple-cloud-computing aggregation through distributed services provided by the VCC server and VCC nodes are geographically and operationally distinct. 
     Distributed Service-Oriented Applications 
       FIG. 11  illustrates a distributed service-oriented application. In  FIG. 11 , a number of servers, such as server  1102 , are shown within a distributed computer system. The servers run various different services, such as front-end service  1104 . Services are executables that provide functionality to other computational entities through a service interface, such as a RESTful application programming interface (“API”) accessed through network communications using REST-protocol requests, although other communications protocols and application programming interfaces can be used. A distributed service-oriented application can be considered to be a collection of various different services, referred to as “service instances,” that cooperate to implement a distributed application. The services are each implemented by one or more service instantiations, or service executables, that run within servers, or within virtual machines executing within servers, of one or more distributed computer systems. Various different types of implementations are possible. The service instances of the distributed application are often registered with a registration-and-subscription service  1106  to which other services can subscribe in order to receive updates with regard to the addition, removal, and changes to the array of available service instances. In the example distributed service-oriented application illustrated in  FIG. 11 , a set of front-end-service instantiations  1104  and  1108 - 1111  communicate with remote clients and users through the Internet  1112  and communicate, via local-area networks and wide-area networks within the distributed computer system, with the many different service instantiations within the distributed computer system that together comprise the distributed service-oriented application, such as service instantiations  1116  and  1117  running within server  1118 . In a traditional distributed service-oriented application, both communications between the front-end service instances and external clients and users as well as internal communications between service instances employ the REST protocol or other similar stateless protocols. 
       FIG. 12  illustrates a sequence of service calls that implement a particular distributed-service-oriented-application API call or entry point. In a first step  1202 , a remote user or client sends a request to the distributed service-oriented application, resulting in a call to one of the front-end-service instances  1204 . The front-end-service instance, in a second step  1206 , calls a service instance  1208  in order to launch execution of the distributed-service-oriented-application request-handling machinery for the received request. In  FIG. 12A , the service instances are referred to by alphanumeric labels, such as the label “S 5 ” for service instance  1208 . In a third step  1210 , service instance S 5  calls service instance S 3   1212 . In a fourth step  1214 , service instance S 5  calls service instance S 4   1216  which, in turn, calls service instance S 6   1218  in a fifth step  1220 . Component-service instance S 6  then calls the additional service instances S 8   1222 , S 9   1224 , and S 10   1226  in steps  1228   1229  and  1230 , respectively. Each of the various service instances carry out certain tasks and functionalities that contribute to execution of the user or client request. For example, service instance S 5   1208  may receive and queue the request, call service instance S 3   1212  to authenticate and authorize the request, and then call service instance S 4   1216  to parse the request and to carry out the requested tasks. Component-service instance S 6   1218  may handle a particular type of task or set of tasks, and may call data-storage-and-retrieval service instance S 8   1222 , a data-analysis service instance S 9   1224 , and a linear-algebra-computation service instance S 10   1226 , as one example. 
     The philosophy underlying distributed service-oriented applications is, in some ways, similar to the philosophy underlying modular programming and object-oriented programming. A distributed-service-oriented-application developer can assemble a distributed application from multiple different, relatively independent, and loosely coupled services without being concerned about the implementations of any but the services specific to the service-oriented distributed application that is being developed, just as library functions and the modules can be called from a modern structured program in order to access the functionalities provided by the library functions and modules without a requirement to understand the implementations of the library functions and modules. Because stateless communication protocols are employed, many of the complexities associated with connection-oriented communications are avoided. As in object-oriented programming, services are reusable, and a useful service may be concurrently or simultaneously accessed by multiple different distributed service-oriented applications. Services can be implemented by concentrating on local considerations and constraints, without worrying about global considerations and the constraints associated with a distributed application, in its entirety. It is these and other such advantages that have propelled distributed service-oriented applications into the forefront of application development. 
     However, in addition to the many advantages, disadvantages associated with distributed service-oriented applications have become apparent. As can be seen in  FIGS. 11 and 12 , the pattern of internal service calls within a traditional service-oriented distributed application tends to be hierarchical. In many cases, particular service instances persistently store data in caches and data-storage appliances collocated with the service instance in particular computational environments. Service instances may therefore be constrained with respect to relocation to different computational environments. Additional relocation-related constraints may involve the fact that a given service often depends on external services, so that a given service instance may only be relocatable to a different computational environment that satisfies external-services dependencies. The different service instances of a service-oriented distributed application may not scale proportionately to one another, so that the number of service instances of a particular overloaded, resource-bound service may not be independently expandable, but may be expandable only on the condition that the numbers of service instances of other, dependent services are also expanded. In the case that one or more of the dependent services are third-party services, such dependencies can represent significant constraints on service-instance relocation. Additional constraints with respect to persistent-data-store collocation may add a great deal of additional complexity to the scaling operation. Additional problems are associated with communications interconnections between internal services. A given service, for example, may provide an interface accessed both by external users and clients of the distributed service-oriented application as well as by service instances of one or more distributed service-oriented applications. Changes to the interfaces of one or more services may require changes to the services that access them, and these changes may percolate further through the service-instance hierarchy. Internal to each service instance, the components of an application are coupled both to other components within the same instance and to the specifications of the compute, network and storage resources granted to that instance. While scaling is possible for currently available distributed service-oriented applications, scaling operations are generally constrained to scale large portions of the service-instance hierarchy due to rigid hierarchical constraints, including service instances that are not being used close to their capacities or bandwidths. This results in wasted computational-resource allocation. For these and additional reasons, there remain many problems and challenges with respect to development of traditional distributed service-oriented applications. 
     Currently Disclosed Methods and Systems 
       FIGS. 13A-C  provide a high-level illustration of the currently disclosed REST-actor-data-grid-based (“RAD{circumflex over ( )}2”) architecture for RAD{circumflex over ( )}2 applications. It should be understood, at the onset, that while REST is often chosen for the stateless communication protocol by which users and clients access a RAD{circumflex over ( )}2 application, other stateless communication protocols may be alternatively employed. As shown in  FIG. 13A , a RAD{circumflex over ( )}2 application consists of a set of communications-interconnected RAD{circumflex over ( )}2-application nodes  1303 - 1315 . The RAD{circumflex over ( )}2 application is implemented above the computational infrastructure of a distributed computer system  1320 , represented in  FIG. 13A  as a large collection of server computer systems, such as computer system  1322 . As discussed in preceding sections of this document, distributed computer systems tend to be large, complex systems that may be geographically distributed and that contain tens, hundreds, thousands, or more discrete computer systems, data-storage appliances, networks, power supplies, and many other, additional components. A RAD{circumflex over ( )}2 application is thus a large and complex set of physical computing components and many different control components comprising stored computer instructions that are executed by processors within the physical computing components. 
       FIG. 13B  illustrates the RAD{circumflex over ( )}2 application in greater detail. In  FIG. 13B , each RAD{circumflex over ( )}2-application node is represented by a rectangle, such as RAD{circumflex over ( )}2-application node  1326 . The RAD{circumflex over ( )}2-application nodes are interconnected with one another and to external clients and users of the RAD{circumflex over ( )}2 application through internal virtual and physical networks of the distributed computer system as well as various types of external local-area and wide-area networks, including the Internet. This set of networks is represented, in  FIG. 13B , by channels  1328 - 1330 , which represent communication links between internal virtual and physical networks of the distributed computer system and the Internet, and by continuous outer channel  1332 , which represents virtual and/or physical networks within the distributed computer system that link the RAD{circumflex over ( )}2-application nodes to one another. The portions of the RAD{circumflex over ( )}2-application nodes through which this outer channel passes, such as portion  1334  of RAD{circumflex over ( )}2-application node  1326 , are referred to as the “REST layers” of the RAD{circumflex over ( )}2-application nodes. In a prototype implementation, the VMware Spring Boot product was used to implement the REST layers of the RAD{circumflex over ( )}2-application nodes. 
     The middle portions of the RAD{circumflex over ( )}2-application nodes, such as the middle portion  1336  of RAD{circumflex over ( )}2-application node  1326 , are linked together by a message-passing channel  1338 . The middle portions of the RAD{circumflex over ( )}2-application nodes are referred to as “actor layers.” Each actor layer of a RAD{circumflex over ( )}2-application node supports execution of multiple actors. Actors, as further discussed below, carry out the bulk of the computational tasks involved in responding to requests input to the distributed service-oriented application by external clients and users. The actors running in the RAD{circumflex over ( )}2-application nodes of the RAD{circumflex over ( )}2 application communicate with one another by message passing. Message passing is, of course, also based, in part, on networking, but is a distinct logical communications layer separate from the networking layer represented by the outer channel  1332 . In a prototype implementation, the Akka Actor Model was used to implement the actor layers of the RAD{circumflex over ( )}2-application nodes. 
     A final portion of each RAD{circumflex over ( )}2-application node, such as portion  1340  of RAD{circumflex over ( )}2-application node  1326 , are linked together by a distributed data grid, represented by inner channel  1342 . These final portions of the RAD{circumflex over ( )}2-application nodes are referred to as the “data-grid layers” of the RAD{circumflex over ( )}2-application nodes. The distributed data grid is a distributed data-storage facility that can be accessed by each RAD{circumflex over ( )}2-application node and by each actor running within each RAD{circumflex over ( )}2-application node. The distributed data grid consists of a third logical communications layer that is based on virtual and physical networking as well as on physical data-storage devices and appliances and on physical memories. In a prototype implementation, the Apache Ignite® product was used to implement the data-grid layers of the RAD{circumflex over ( )}2-application nodes. 
     Each RAD{circumflex over ( )}2-application node is thus accessible by external clients and users of the distributed service-oriented application, and a client request received by any RAD{circumflex over ( )}2-application node can be either directed by the REST layer of the RAD{circumflex over ( )}2-application node to another of the RAD{circumflex over ( )}2-application nodes through the outer networking channel  1332  or processed locally within the receiving RAD{circumflex over ( )}2-application node. The actor-level implementation of the RAD{circumflex over ( )}2-application functionality is isolated from the REST-protocol-based external interface of the RAD{circumflex over ( )}2 application, which addresses the above-mentioned problem of hierarchical REST-interface dependencies among services of a distributed service-oriented application. For local processing of a client request, the REST layer of the RAD{circumflex over ( )}2-application node sends a corresponding message to one of the actors within the RAD{circumflex over ( )}2-application node. The receiving actor then processes the request or communicates, via message passing, with multiple additional actors via the message-passing channel  1338  in order to carry out the computational tasks needed to respond to the client or user request. An actor can send messages to any other actor, regardless of the RAD{circumflex over ( )}2-application node in which the other actor is currently executing. Each actor can read data from, and write data to, the distributed data grid in order to retrieve and persistently store data. Because an actor can access the data grid and communicate with all other actors regardless of the RAD{circumflex over ( )}2-application node in which the actor is currently executing, and because any actor is accessible, at least indirectly, from the REST layer of any RAD{circumflex over ( )}2-application node, an actor is not constrained with respect to the particular RAD{circumflex over ( )}2-application node that hosts the actor. An actor can therefore be freely relocated among RAD{circumflex over ( )}2-application nodes. Unconstrained distribution and redistribution of actors across the RAD{circumflex over ( )}2-application nodes addresses the execution-location dependencies and constraints inherent in traditional distributed-service-oriented-application architectures. RAD{circumflex over ( )}2-application nodes can be easily added to, or removed from, the RAD{circumflex over ( )}2 application. The actors executing within a RAD{circumflex over ( )}2-application node can be easily relocated to other of the RAD{circumflex over ( )}2-application nodes prior to removal of the RAD{circumflex over ( )}2-application node, and actors executing within current RAD{circumflex over ( )}2-application nodes can be easily relocated to a newly added RAD{circumflex over ( )}2-application node. RAD{circumflex over ( )}2 applications can therefore be more efficiently scaled, either upwards, to handle increased workloads, or downwards, to conserve computational-resource use when workloads decrease, than traditional distributed service-oriented applications. 
     As shown in  FIG. 13C , RAD{circumflex over ( )}2-application nodes, such as RAD{circumflex over ( )}2-application node  1350 , can be implemented by any of many different computational environments provided within a distributed computer system. They may be implemented by individual discrete computer systems, or servers, such as discrete computer system  1352 , they may be implemented by virtual machines, such as a virtual machine  1354  provided within computer system  1356 , and they may be implemented by finer grained execution environments, such as execution environment  1358 , provided within virtual machine  1359  of computer system  1360 . In addition, RAD{circumflex over ( )}2-application nodes can be implemented by aggregations of discrete computer systems, virtual machines, or finer-grained execution environments. 
       FIG. 14  illustrates a RAD{circumflex over ( )}2-application node in greater detail. As discussed above, an RAD{circumflex over ( )}2-application node  1402  includes a REST layer  1404 , an actors layer  1406 , and a data-grid layer  1408 . The REST layer  1404  includes a set of REST endpoints  1410  and a message-passing interface  1412  comprising one or more controllers. The REST endpoints correspond to uniform resource identifiers (“URIs”) that represent logical resources or services provided by the RAD{circumflex over ( )}2 application via the REST communications protocol. Several protocol-defined types of requests can be received by a REST endpoint. Data included in a request can further define the request types. The controllers, as further discussed below, receive REST requests directed to the REST endpoints and either forward a received request to an external RAD{circumflex over ( )}2-application node for remote processing of the request or send a corresponding message to an actor executing within the RAD{circumflex over ( )}2-application node to launch local processing of the request. The actors layer  1406  provides execution environments for multiple actors, including actor  1414  as well as an interface to the message-passing communications layer that allows the actors to send messages to one another, to receive messages from the controllers in the REST layer, and to send messages to the controllers in the REST layer. The ellipses in  FIG. 14 , such as ellipsis  1416 , indicate that execution of many additional actors may be supported within the actors layer of a RAD{circumflex over ( )}2-application node. Each actor includes a message-passing interface  1418  through which the actor receives messages  1420  and sends messages  1422 . The actor also includes a data-grid interface  1424  through which the actor reads data from the data grid  1426  and stores data persistently within the data grid  1428 . Data passes to the data grid and is received from the data grid in the form of data-transfer objects (“DTOs”). The data-grid layer  1408  of an RAD{circumflex over ( )}2-application node includes a DTO-transfer interface between the actors and the data-grid layer  1430  as well as an interface to the distributed data grid  1432 . The distributed data grid additionally interfaces  1434  to various types of data-storage appliances  1436  and may additionally include internal interfaces to memory caches and other data-storage facilities within the distributed computer system. 
       FIG. 15  illustrates basic operations of an actor. Each actor is associated with a message mailbox. In  FIG. 15 , the mailbox associated with actor  1502  is shown as an incoming-message mailbox  1504  and an outgoing mailbox  1506 , both mailboxes represented by circular queues. Each actor interfaces to an actor-creation interface  1508  that allows the actor to create one or more additional actors  1510 . Thus, a set of actors, or actor cluster, that carry out the computational tasks corresponding to a particular REST endpoint can be instantiated by an initial primary actor. Each actor processes only one message at a time. For a next message received from the incoming mailbox, as represented by arrows  1512  and  1513 , an actor selects a particular behavior  1516 , or executable logic, to process the received message. The actor then executes the selected behavior  1518 , which may involve creating one or more additional actors, generating and transmitting, to the outgoing mailbox  1506 , one or more new messages, reading data from the distributed data grid, and writing data to the distributed data grid. Actors carry out the computational tasks associated with responding to received REST requests, and do so relatively independently from the activities of other actors in the RAD{circumflex over ( )}2 application and independently from the ongoing communications traffic being handled by RAD{circumflex over ( )}2-application nodes, including the RAD{circumflex over ( )}2-application node in which the actors execute. 
       FIGS. 16A-B  illustrates logical data structures that represent the roles of message types and virtual actor addresses in a RAD{circumflex over ( )}2 application. As shown in  FIG. 16A , each actor  1602  is associated with a virtual actor address  1604  that is unique among all actors within a RAD{circumflex over ( )}2 application. In addition, each actor indicates, through a message interface, a set of message types  1606  that the actor can receive and process. Logically, a virtual actor address acts as an index into a mapping table  1608  that maps each virtual actor address to a RAD{circumflex over ( )}2-application node  1610 , a communications address  1612  which may include, as one example, an Internet protocol (“IP”) address and a port number, and additional information that constitutes a local message-passing address within the RAD{circumflex over ( )}2-application node. Thus, a message addressed to a virtual actor address and output by an actor to the outgoing mailbox is mapped to the networking address and/or additional addresses needed to transmit the message to an actor in a different RAD{circumflex over ( )}2-application node and to the actor within the local RAD{circumflex over ( )}2-application node. As shown in  FIG. 16B , another logical data structure  1620  maps each REST-endpoint/request-type pair to one or more message types. A third logical data structure  1622  maps each message type to a set of virtual actor addresses associated with currently executing actors that can receive and process messages of that type. Thus, controllers within the REST layer of a RAD{circumflex over ( )}2-application node can determine a virtual actor address for a virtual actor that can process any client or user request received by any REST endpoint. In certain cases, various types of workload-balancing logic can be applied at the controller level, including ordering the sequence of virtual actor addresses mapped to particular message types and distributing messages of each type to virtual actor addresses in proportion to sequence orders of the virtual actor addresses mapped to the type or according to other distribution policies. However, as discussed further below, redistribution of actors among RAD{circumflex over ( )}2-application nodes provides another important dimension to workload balancing. 
       FIG. 17  indicates a motivation for workload-balancing. In a first column  1702  in  FIG. 17 , a number of RAD{circumflex over ( )}2-application nodes are represented by large rectangles. Smaller rectangles inside each of the larger rectangles represent actors currently executing in the RAD{circumflex over ( )}2-application nodes represented by the larger rectangles. The distribution of actors among the RAD{circumflex over ( )}2-application nodes in column  1702  is unbalanced, since at least one of the RAD{circumflex over ( )}2-application nodes  1704  does not contain any currently executing actors. A workload-balancing operation, represented by horizontal arrow  1706 , seeks to redistribute actors among RAD{circumflex over ( )}2-application nodes to distribute, as evenly as possible, the cumulative workload represented by execution of the actors among the RAD{circumflex over ( )}2-application nodes. Following the rebalancing operation illustrated in  FIG. 17 , for example, the RAD{circumflex over ( )}2-application node represented by rectangle  1704  transitions from initially containing no actors to containing the four actors  1710 - 1713 . In addition, the RAD{circumflex over ( )}2-application node represented by rectangle  1760  transitions from containing six executing actors  1720 - 1725  to containing only four executing actors  1720 ,  1722 ,  1725 , and  1726 . Workload balancing is generally carried out periodically, during the lifetime of a RAD{circumflex over ( )}2 application, in order to maintain a fair workload distribution among the RAD{circumflex over ( )}2-application nodes despite dynamic fluctuations of workloads and dynamic fluctuations of currently executing actors. Workload balancing is also carried out during scaling operations in which the number of RAD{circumflex over ( )}2-application nodes is either increased, to respond to an increase in total workload, or decreased, to respond to a decrease in total workload. As mentioned in the preceding paragraph, a certain amount of workload-balancing can be effected at the controller level by various message-distribution policies that together promote balanced RAD{circumflex over ( )}2-application-node workloads. However, another, often more effective and simpler workload-balancing method involves redistributing actors among RAD{circumflex over ( )}2-application nodes. An actor is easily relocated from a first RAD{circumflex over ( )}2-application node to a second RAD{circumflex over ( )}2-application node, in distributed applications built according to the RAD{circumflex over ( )}2 architecture, by instantiating the actor in the second RAD{circumflex over ( )}2 architecture, since the actor equally accessible to other RAD{circumflex over ( )}2 in each RAD{circumflex over ( )}2-application node and since the data grid is equally accessible to the actor in each RAD{circumflex over ( )}2-application node. Unlike in currently available distributed service-oriented applications, there are no significant constraints associated with changing the RAD{circumflex over ( )}2-application node location of an actor. Different workload-balancing methods can be used together. 
     There are many considerations involved in determining whether a distribution of actors among RAD{circumflex over ( )}2-application nodes is balanced or unbalanced. Considerations include the number of executing actors in each of the RAD{circumflex over ( )}2-application nodes as well as the fraction of the total cumulative workload that is handled by each of the actors. In addition to the many different considerations that may be involved in workload-balancing, there are also many different types of metrics and the monitoring data that can be collected in order to facilitate workload-balancing and many different techniques and approaches to using that information to carry out effective workload-balancing. Workload balancing, however, is associated with significant computational overheads, and so the accuracy, or evenness, of workload balancing achieved by workload-balancing operations must be weighed against the computational overheads required to achieve those levels of accuracy or evenness. An optimal loan-balancing technique may, for example, produce a somewhat less even distribution of workloads than more complex techniques, but may do so at a much lower computational cost than the more complex techniques. 
       FIGS. 18A-C  illustrate a workload-balancing example. The illustration conventions introduced in  FIG. 18A  are later employed in  FIGS. 23A-C . In  FIG. 18A , a set of four REST endpoints  1802  are shown at the top of the figure. Each REST endpoint is represented by a circle, such as circle  1804 . Actors are represented by larger circles, such as circle  1806 . Message passing between REST endpoints and actors and between pairs of actors is represented by directed line segments. A double-headed arrow, such as double-headed arrow  1808 , represents passing of messages in both directions while a single-headed arrow, such as single-headed arrow  1810 , represents message passing in a single direction, only. In the example of  FIG. 18A , a request received at REST endpoint  1804  results in a corresponding message sent to either actor  1806  or to actor  1812 . Actors  1806  and  1812  are thus both primary actors for a particular type of REST request. The primary actors, in turn, may exchange messages with secondary actors  1814 - 1815  and  1816 - 1870 . These secondary actors may exchange messages with tertiary actors  1818 - 1819 ,  1820 - 1821   1822 - 1823  and  1824 - 1825 . Finally, tertiary actors  1819  and  1821  may send messages to actor  1826  and tertiary actors  1823  and  1825  may send messages to actor  1827 . The actors that indirectly receive messages from actor  1806 , including actors  1814 - 1815 ,  1818 - 1821 , and  1826 , together with actor  1806 , comprise a cluster of actors that cooperate to process REST requests received by REST endpoint  1804 . This actor cluster is encircled with dashed curve  1834 . Dashed curves  1835 - 1838  encircle four additional actor clusters. The large set of actors which receive messages resulting from REST requests arriving at REST endpoint  1830 , beginning with primary actors  1831  and  1832 , form a single cluster  1837  since message paths emanating from both of these primary actors include actor  1833 . Clusters can be determined by any of many different well-known node-cluster-determination methods that are applied to graphs, such as the graph shown in  FIG. 18A . One method assigns each graph node to its own cluster and then iteratively merges pairs of edge-connected clusters. 
     The circles representing actors each include a numeric name, or label, for the actors. These labels are used in workload-distribution data, shown in  FIG. 22  and discussed below. 
       FIG. 18B  illustrates a current, very unbalanced distribution of the actors in the example of  FIG. 18A  across five RAD{circumflex over ( )}2-application nodes. Each RAD{circumflex over ( )}2-application node is represented by an enclosed curve, such as enclosed curve  1840 , indicating the actors currently executing in the RAD{circumflex over ( )}2-application node. All but one of the actor clusters that each includes clusters that cooperate to process particular types of requests are distributed across multiple RAD{circumflex over ( )}2-application nodes. Furthermore, the numbers of actors executing in each of the five different RAD{circumflex over ( )}2-application nodes are quite unbalanced, with one RAD{circumflex over ( )}2-application node, represented by curved  1840 , currently hosting only five actors while the RAD{circumflex over ( )}2-application node represented by curve  1842  is currently hosting  15  actors. While the number of actors executing in a RAD{circumflex over ( )}2-application node may not be directly proportional to the cumulative workload handled by the RAD{circumflex over ( )}2-application node, the actors executing in the RAD{circumflex over ( )}2-application node corresponding to curve  1842  include all of the actors in the cluster that includes primary actor  1812  as well as a portion of the actors in the cluster containing primary actor  1806  and a portion of the actors in the cluster containing primary actor  1843 . If messages corresponding to user requests are being evenly distributed to primary actors  1806  and  1812  by the controller associated with REST endpoint  1804 , the RAD{circumflex over ( )}2-application node corresponding to curve  1842  is shouldering a greater portion of the cumulative workload than the RAD{circumflex over ( )}2-application node corresponding to curve  1840 . 
       FIG. 18C  illustrates the values for one type of metric that is easily collected from a RAD{circumflex over ( )}2 application. The number of messages passed in each direction between endpoints and actors and between pairs of actors can be collected from the message-passing layer with relative ease and without introducing extensive code modifications or incurring significant computational overheads. Thus, during the periods between workload-balancing operations, a RAD{circumflex over ( )}2 application can monitor application activities by collecting indications of the number of messages passed in each direction between actors and REST endpoints which exchange messages through the message-passing layer. The currently disclosed workload-balancing operation uses these collected numbers of passed messages to estimate portions of the cumulative workload handled by each of the RAD{circumflex over ( )}2-application nodes. 
       FIGS. 19A-E  illustrate a page-rank algorithm used to rank webpages based on the number of hyperlinks directed from the webpages to other webpages.  FIG. 19A  shows a small number of webpages, each represented by a circle, and the links between them, each represented by an arrow. Webpage  1902  contains hyperlinks to webpages  1904 - 1906  and webpages  1905  and  1907  contain hyperlinks to webpage  1902 . The page-rank algorithm involves initially assigning a probability to each webpage, as shown in  FIG. 19B . The initial probability is the reciprocal of the number of webpages considered. N, as indicated by expression  1910 . Thus, as indicated in  FIG. 19B  by initial probabilities inscribed in each circle, each webpage initially receives a probability of 0.1. These probabilities are then iteratively adjusted in successive transfer cycles until the probability values converge to stable values. 
     During each transfer cycle, each webpage transfers a portion of its probability either to those webpages for which it contains hyperlinks or, in the case of a sink webpage, which contains no hyperlinks to other webpages, to all of the non-sink webpages. Thus, in the first part of each cycle, as illustrated in  FIG. 19C , the probability associated with a webpage is divided by the number of outgoing hyperlinks from the webpage and this value is output by the webpage to each webpage connected to the webpage by an outgoing link. In the case of sink nodes, which lack outgoing hyperlinks, the current probabilities associated with the sink nodes can be thought of as being transferred to a virtual node  1912 , which then evenly distributes the probabilities it receives to the non-sink webpage nodes. Then, in the second portion of each cycle, as illustrated in  FIG. 19D , each node collects the probability portions transferred to it by other nodes, sums these probability portions, multiplies the sum by a factor d in the range [0.0&lt;d&lt;1.0], adds the ratio (1−d)/N to the sum, and sets its current probability to the sum. 
     The iteration of probability-transfer cycles continues until the cumulative change in the probability distribution represented by the probabilities associated with the nodes falls below a threshold value. In many cases, the cumulative change in the probability distribution is computed as the square root of the sum of the squares of the differences between the probabilities associated with each node in the two most recent cycles. The probabilities associated with each webpage following termination of the probability-transfer-cycle iterations sum to 1.0, as is the case with normalized probability distributions. The probability associated with a node, or webpage, indicates the probability that a viewer who begins surfing the webpages at a randomly selected webpage will end up viewing the page. 
     The currently disclosed RAD{circumflex over ( )}2-application architecture uses a workload-estimation method that is similar to the above-discussed page-rank method to estimate the percentages of a total workload applied to a RAD{circumflex over ( )}2 application handled by each actor during a monitoring period. These percentages can be added for each actor cluster to determine the percentages of the total workload handled by each cluster. The clusters can then be assigned to RAD{circumflex over ( )}2-application nodes based on workload-percentages in order to distribute, in a balanced fashion, the total workload among the RAD{circumflex over ( )}2-application nodes. There are a variety of different possible implementations of the modifications to the workload-estimation method based on modifications to the page-rank method. 
       FIGS. 20A-C  and  21 A-F show a simple C++ implementation of the workload-estimation method for determining the percentages of a total workload handled by each actor, as in the example discussed above with reference to  FIGS. 18A-C .  FIGS. 20A-C  provide constants and class declarations for the implementation, as generally provided in a header file.  FIGS. 20A-C  and  21 A-F provide implementations of most of the class member functions. 
     The workload-estimation method associates each actor node of a graph with a value p, as in the page-rank method. In the workload-estimation method, this value can be thought of as the fraction of a total number of messages current associated with the actor. The messages are transferred from node to node, in successive message-transfer cycles, just as probabilities are transferred in the page-rank method. Message transfer cycles continue to be carried out until the message fractions stabilize, just as in the page-rank method. In fact, certain implementations of the workload-estimation method are obtained by making only small adjustments to the page-rank method. However, in the described implementation, additional modifications are made to provide more adjustable parameters to tailor the method to particular RAD{circumflex over ( )}2 applications. One modification is to associate each actor node with a workload fraction work in addition to the message fraction p. The workload fraction is based on the amount of message passing carried out by the node during the transfer cycles, but different types of message passing are differently weighted in order to better estimate workload rather than message passing, alone. This also allows messages to be passed in only one direction, from REST endpoints to primary actors and then down through actor clusters to sink-node actors and then back to REST endpoints, while crediting actors for processing return messages that flow in the opposite direction. It also allows the workload fractions to be associated only with actors, rather than also being associated with REST endpoints. When the message fractions stabilize, the workload fractions associated with the actor nodes provide an estimate of the percentage of the total workload handled by the actor nodes. 
     At the top of  FIG. 20A , six constants are declared. The constant MAX_TO_NODES  2002  is the maximum number of actors, which are nodes in the actor graph, to which a particular actor can send messages. The constants damp and oneMinusDamp  2003  are multiplies of the message fraction p that determine the portion of p transferred in the current cycle and the portion of p retained by an actor node. The constant SINK COEFFICIENT  2004  is a parameter used to increase the workload fraction for sink nodes of the graph. The constant THRESHOLD 2005 is the value below which the cumulative change in message percentages must fall in order to consider the current percentages as converged. 
     The alias NodePtr  2006  is a type for a pointer to a node and the alias EdgePtr  2007  is a type for a pointer to an edge. Instances of the class edge  2008  represent graph edges, which indicate message passing between two nodes, as discussed above with reference to  FIG. 18A . An edge instance includes: (1) member data indicating the two nodes connected by the edge  2010 ; (2) member data indicating the number of messages passed in each direction through the edge  2011 ; and (3) Boolean data members  2012  that indicate whether a direction of an edge corresponds to response messages that respond to received incoming messages or whether the direction corresponds to messages representing requests for processing. Response messages propagate from lower-level nodes in a cluster upward to the primary node and request messages propagate from REST endpoints downward to the primary node and lower-level nodes in a cluster. A set of public function members  2013  include a constructor, an initiator function for initiating the values of the various data members of the edge class, and various in-line functions for returning the value of the data members the edge class. 
     The structure internalNode  2014  represents a node to which messages are passed by another node. The structure member node  2015  is a pointer to the node to which messages are passed. The structure member f  2016  is the fraction of the current probability associated with a transferring node that is transferred to the node represented by an instance of the internalNode structure when request messages are distributed during each cycle. The member fComb  2017  is the fraction of the work transferred to the node represented by an instance of the internalNode structure during each cycle. The structure member returnMsg  2018  indicates whether the node represented by an instance of the internalNode structure receives response messages or request messages. 
       FIG. 20B  shows a class declaration for the class node  2020 . Instances of the class node represent graph nodes, or actors. Each instance of the class node includes numerous private data members  2021 . The data members p and work  2022  store the percentage of messages and the percentage of the cumulative work, respectively, associated with the node. The array toNodes  2023  includes an internalNode structure for each node to which the node represented by an instance of the class node sends messages. The number of such nodes is stored in the data member numToNodes  2024 . The data member numOut  2025  indicates the number of nodes to which the node represented by an instance of the class node sends both portions of the percentage of messages and portions of percentage of work during each transfer cycle while the data member numOnly indicates the number of nodes to which the node represented by an instance of the class node sends only work portions. A set of data members  2027  stores the outgoing and incoming message portions, incoming work portions, and remaining message portions for the node during each cycle. The data members  2028  are used to maintain a running average of the work portions received by the node. The member functions for the class node include a constructor  2029  and a number of member functions  2034  for setting and retrieving values of data members. 
     The function member addToNode  2031  of the class node is used to add a target node to the array toNodes  2023 . The member function distributions  2032  is used to determine the fractions of message and work percentages to be distributed to the nodes represented by the internalNode structure instances in the array toNodes. The member function next  2033  is called to carry out the first part of each transfer cycle and the member function nextFinish  2034  is called to carry out the final portion of each transfer cycle. A node receives message portions and work portions via the member functions receiveMsgs and receiveWork  2035 . 
     Finally,  FIG. 20C  provides a declaration for the class graph  2040  which represents an entire graph, such as that shown in  FIG. 18A . An instance of the class graph includes a pointer to the set of nodes in the graph  2041 , and an indication of the number of nodes in the graph  2042 , a pointer to the edges in the graph  2043 , an indication of the number of edges in the graph  2044 , a pointer to the REST endpoints in the graph  2045 , an indication of the number of REST endpoint s in the graph  2046 , a pointer to the fractions of the total number of messages emitted by each REST endpoint  2047  during each transfer cycle, and a print function  2048  that displays the current work percentages for each node, or actor, in the graph. The class graph includes a constructor  2049  and destructor  2050 . In addition, the class graph includes a member function add Edge  2051  which adds an edge to the graph, a member function initiate  2052  which initiates the nodes of the graph in preparation for iterating through the transfer cycles in order to determine the percentages of the total cumulative work carried out by each node, and a member function run  2053  which carries out iterative transfer cycles until the message percentages converge. 
       FIGS. 21A-F  provide the implementations of the various member functions of the class node in the class graph. The constructor for the class node  2102  initializes the various data members. The member function addToNode  2103  adds a target node to the array toNodes. Initially, the internalNode data members f and fComb are set to actual numbers of messages collected during a monitoring interval, as discussed above with reference to  FIG. 18C . These numbers are then divided by the total number of messages in order to generate fractional percentages for transfer to the node during each cycle. The member function distributions  2104 , shown in  FIG. 21B , converts the number of messages to fractional distributions, as discussed above.  FIG. 21C  provides an implementation of the member function next of the class node. If an instance of the class node represents a REST endpoint, as indicated by the member ePoint, the node distributes its entire percentage of messages during the transfer cycle in statements  2106 . The percentage of messages currently associated with the REST entry point is computed as a fraction stored in member p times the number of messages in the virtual sink node passed to the function in the argument left. Otherwise, when the node represents a non-sink-node actor, the node retains a portion of its message percentage, indicated by the data member remaining 2108, and transfers the remaining portion of its message percentage, indicated by data member outgoing  2109 . However, when the node is a sink node, the node does not retain a message percentage  2110  and forwards all of its message percentage to the virtual sink node  2111 . The outgoing message percentages are transmitted in the for-loop  2112  in the case of nodes that pass messages to other nodes while the percentage of messages is transmitted to the virtual sink node on lines  2113 . Finally, on lines  2114 , work portions are transmitted to all nodes that either receive request messages or that receive response messages. The member function nextFinish  2116 , shown at the top of  FIG. 21D , is called to complete a transfer cycle. The squared difference of the message percentages for the current cycle and preceding cycle are computed and returned in order to detect when the message percentages have converged, on lines  2117 . The new message percentage, newP, is computed as the sum of the remaining message percentage and the incoming message percentage. The running average of the workload percentage associated with the node is updated on lines  2118 . The various data members used to accumulate messages and work during the first portion of the cycle are reset to 0, on lines  2119 . On line  2120 , the message percentage p is updated. The node class member functions next and nextFinish represent the bulk of the workload-estimation method in the described implementation. 
     The constructor for the class graph  2122  allocates data structures for nodes, edges, REST endpoints, and REST-endpoint distribution fractions. The destructor for the class graph  2123  deallocates the data structures for the nodes, edges, REST endpoints, and REST-endpoint distribution fractions. The member function addEdge for the class graph  2124  is called to add an edge to the graph. When one of the nodes connected by the edge is a REST endpoint, the number of messages sent by the REST endpoint are stored in the element of the array endPointMsgs to compute distribution fractions for the REST endpoint on lines  2125  and  2126 . For node-to-node edges, internalNode elements are added to the toNode arrays of the connected nodes  2128 . The member function initiate  2130  for the class graph initiates all of the nodes in the graph in the for-loop  2132 , computes the distribution fractions for the REST endpoints in for-loop  2134 , and initiates the REST-endpoint nodes in the for-loop  2136 . Finally, the member function run for the class graph  2140  carries out a series of transfer cycles until the cumulative difference in message percentages between successive transfer cycles falls below a threshold value. The first part of each transfer cycle is carried out in for-loops  2142  and the second part of each transfer cycle is carried out in for-loops  2144 . The do-while-loop  2146  terminates when the cumulative differences fall below the threshold value  2147 . An example main routine  2148  illustrates an example computation of the workload distributions for the actors in a graph, such as the graph shown in  FIG. 18A . An instance of the class graph is instantiated, on line  2149 . A series of calls to the member function addEdge  2150  add the edges of the graph to the graph instance. Ellipsis  2151  indicates that additional edges are added for the example graph shown in  FIG. 18C . A call to the member function initiate preparers the graph for workload-distribution calculation on line  2152 . Finally, the member function run is called, on line  2153 , to carry out a series of transfer cycles until the message percentages converge. 
       FIG. 22  illustrates the workload-distribution results generated by applying the workload-distribution calculation shown in the C++ implementation discussed above with reference to  FIGS. 20A-21F  to the graph shown in  FIG. 18C . The percentages of the total workload handled by the actors is shown in data block  2204  and the percentages of the total work carried out by the five actor clusters are shown on line  2206 . Actor cluster  1  is the cluster associated with primary actor  1806 . This cluster is responsible her handling close to 19%  2208  of the total workload during the monitoring interval. The large cluster that includes primary actors  1831  and  1832  accounts for over 40% of workload  2210 . A comparison of the workload percentages shown in  FIG. 22  to the size of the actor clusters reveals that the workload percentages attributed to the clusters are somewhat related to the size of the clusters, but not necessarily fully proportional to the cluster sizes. Instead, the workload percentages are function of the total number of messages passed to and from between the actors of the cluster and between the REST endpoint that feeds messages to the cluster and the primary nodes of the cluster, parameter values, and the distribution fractions assigned to the REST endpoints. Many additional parameters and associated considerations may be incorporated into the workload-distribution method to better estimate the workload distribution. 
       FIGS. 23A-C  illustrate, using the example of  FIGS. 18A-C , how the workload distribution computed by the above-discussed workload-distribution-estimation method can be used to distribute the workload more evenly across RAD{circumflex over ( )}2-application nodes.  FIG. 23A  shows the graph of  FIG. 18A  with edges leading from REST endpoints to primary nodes annotated with the computed workload percentages for the clusters. The cluster  2302  containing primary nodes  2303  and  2304  processes more than twice the workload processed by any of the remaining clusters. To facilitate a more even distribution of clusters among the RAD{circumflex over ( )}2-application nodes, this cluster is broken into two clusters, as shown in  FIG. 23B , by adding a new node, or actor  2306 , to receive messages from actor  2308  and removing the edge between actor  2308  and actor  2310 . As discussed above, it is quite simple to add actors to a RAD{circumflex over ( )}2 application, and this feature of RAD{circumflex over ( )}2 applications is exploited, in this example, to break up a large cluster into two smaller clusters. The work percentages for the two clusters  2312  and  2314  are estimated to be half of the computed workload distribution for the large cluster  2316  in  FIG. 23A . Then, as shown in  FIG. 23C , the enclosed curves can be redrawn to include the clusters in RAD{circumflex over ( )}2-application nodes such a way that the workload percentages assigned to each RAD{circumflex over ( )}2-application node are reasonably equivalent. The workload percentages for each RAD{circumflex over ( )}2-application node are shown in squares  2320 - 2324 . By distributing actor clusters among RAD{circumflex over ( )}2-application nodes rather than individual actors, the actors that cooperate closely to process particular types of REST requests are collocated within a single RAD{circumflex over ( )}2-application node, which minimizes inter-RAD{circumflex over ( )}2-application-node message traffic and which may also minimize network transfers of data-transfer objects. 
       FIGS. 24A-B  provide control-flow diagrams that illustrate fine-grained workload-balancing within a RAD{circumflex over ( )}2 application.  FIG. 24A  provides a control-flow diagram for the routine “distributed app,” which represents monitoring of a RAD{circumflex over ( )}2 application over the lifetime of the RAD{circumflex over ( )}2 application. In step  2402 , the RAD{circumflex over ( )}2 application is configured, resources are allocated for the RAD{circumflex over ( )}2-application nodes, and a rebalance timer is set. In step  2404 , the RAD{circumflex over ( )}2 application is launched. Steps  2406 - 2414  represent an event loop that continues until the RAD{circumflex over ( )}2 application terminates. In step  2406 , the monitor waits for a next event to occur. When the next event is an expiration of the rebalance timer, as determined in step  2407 , a routine “rebalance” is called, in step  2408 , to rebalance the workloads handled by the RAD{circumflex over ( )}2-application nodes. In step  2409 , the rebalance timer is reset. Ellipses  2416 - 2417  indicate that many other types of events are handled during the lifecycle of a RAD{circumflex over ( )}2 application. When a termination event is detected, as determined in step  2410 , the RAD{circumflex over ( )}2-application nodes are terminated and allocated resources are deallocated, in step  2411 , prior to termination of the RAD{circumflex over ( )}2 application. A default handler  2412  handles any rare or unexpected events. When there are more events that have been queued during processing of the most recently handled event, as determined in step  2413 , a next event is dequeued, in step  2414 , and control returns to step  2407  to process the next event. Otherwise, control returns to step  2406 , where the monitor waits for a next event to occur. The rebalance timer is used to partition of the lifecycle into monitoring periods, at the end of each of which workload-balancing is carried out. 
       FIG. 24  B provides a control-flow diagram for the routine “rebalance,” called in step  2408  of  FIG. 24A . In Step  2420 , collection of message-passing data is redirected to another storage location for continued metric-data collection while the collected messages in the current storage location are processed for determining the workload distribution for the actors. In step  2422 , the collected messages are processed to generate a weighted call graph, such as the example weighted call graph shown in  FIG. 18C . In Step  2424 , any of many different possible clustering methods can be applied to the nodes of the graph in order to determine the actor clusters within the graph. In step  2426 , the workload-distribution-computation method is applied to the graph in order to generate workload percentages for the actors and actor clusters, as discussed above with reference to  FIG. 22 . In step  2428 , unbalanced clusters may be split into smaller clusters to generate a set of clusters with relatively uniform workload percentages, as discussed above with reference to  FIG. 23C . Other types of workload-balancing steps may also be taken, including readjusting controller-level workload-balancing, mentioned above. In step  2430 , the set of actor clusters is sorted by estimated workload percentages and then, in step  2432 , the actor clusters are redistributed among the RAD{circumflex over ( )}2-application nodes, as discussed above with reference to  FIG. 23C . In general, there may be many thousands, hundreds of thousands, or millions of actors and large numbers of clusters. Thus, each RAD{circumflex over ( )}2-application node may receive many different clusters during the redistribution. Various different redistribution schemes can be used. In one scheme, a round-robin approach is used to distribute the clusters in descending sorted order. In another approach, the workload percentages associated with clusters are matched to the available computational resources in the RAD{circumflex over ( )}2-application nodes, so that each cluster is placed into a RAD{circumflex over ( )}2-application node with available computational resources that can best handle that clusters workload percentage. 
       FIGS. 24A-B  show only actor workload-balancing operations periodically carried out for fine-grained balancing of actors across the available RAD{circumflex over ( )}2-application nodes. Rebalancing may also occur following adding or deleting RAD{circumflex over ( )}2-application nodes from the RAD{circumflex over ( )}2 application during scaling operations periodically carried out to match the total workload to the total volume of computational resources currently available. 
     The present invention has been described in terms of particular embodiments, it is not intended that the invention be limited to these embodiments. Modifications within the spirit of the invention will be apparent to those skilled in the art. For example, any of many different implementations of the RAD{circumflex over ( )}2-application rebalancing operation and workload-distribution-computation system can be obtained by varying various design and implementation parameters, including modular organization, control structures, data structures, hardware, operating system, and virtualization layers, and other such design and implementation parameters. In alternative implementations, various different workload-distribution-computation methods based on the page-ranked method can be obtained by including additional parameters that can be adjusted to produce estimated workload distributions that faithfully mirror workload distributions that might be obtained by far more complex and invasive monitoring methods. Actor redistribution can be carried out periodically or, in alternative implementations, when monitoring routines detect workload imbalances using additional types of metrics.