Abstract:
An interface software layer is interposed between at least one application and a plurality of coprocessors. A data and command stream issued by the application(s) to an API of an intended one of the coprocessors is intercepted by the layer, which also acquires and stores the execution state information for the intended coprocessor at a coprocessor synchronization boundary. At least a portion of the intercepted data and command stream data is stored in a replay log associated with the intended coprocessor. The replay log associated with the intended coprocessor is then read out, along with the stored execution state information, and is submitted to and serviced by at least one different one of the coprocessors other than the intended coprocessor.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims priority of U.S. Provisional Patent Application No. 62/319,867, filed 8 Apr. 2016. 
     
    
     TECHNICAL FIELD 
       [0002]    This invention relates to computing systems that include co-processors. 
       BACKGROUND 
       [0003]    As both the extent and complexity of computer processing has grown even in everyday modern life, there is a well-known, ever-increasing need for greater processing power. In many cases, even the increases in processing capability predicted by Moore&#39;s Law is insufficient. 
         [0004]    One way to increase processing capacity is to distribute the load. “Cloud computing” is one known distribution scheme, in which local systems access shared processing resources such as servers remotely via a network, usually on demand. Although making essentially unlimited resources available, network delays alone preclude use of such an arrangement for many computationally intensive, time-critical or synchronized tasks. 
         [0005]    One approach to handling some such tasks is “parallel computing”, in which a task is decomposed into discrete sub-tasks that can be performed simultaneously by different processing systems. This definition indicates one of the limitations of this technique: For parallel computing to be efficient, the task must be amenable to such decomposition into independently, partially computable sub-tasks. 
         [0006]    Certain processing tasks involve operations that can be performed by a system&#39;s main processor, but that are so specialized that an auxiliary processor—a coprocessor—may instead be more efficient, thereby leaving the more general operations to the main processor. Coprocessors are thus frequently included in systems to perform such operations as floating point arithmetic, encryption, string processing, I/O interfacing, and signal and graphics processing. Such coprocessors may be locally and/or remotely attached. 
         [0007]    The specialization of coprocessors offers many obvious advantages—they are, after all, designed to perform certain tasks especially well—but they also create challenges, especially when a main hardware platform is to be able to access more than one coprocessor, which may have different API protocols, may be distributed, that is, with some or all of them remote, may have unbalanced loading, etc. Even when only a single coprocessor is being accessed, there is the issue of how to deal with a failure of that coprocessor. What is needed is therefore a system and operational method that makes the use of one or more coprocessors more efficient in terms of reliability, availability, serviceability, etc. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]      FIG. 1  illustrates where in a hardware/software stack embodiments of the invention operate in the context of use of coprocessors. 
           [0009]      FIG. 2  illustrates how embodiments intercept, at an API level, operation calls to coprocessors. 
           [0010]      FIG. 3  illustrates re-direction of coprocessor requests in the event of a failure. 
       
    
    
     DETAILED DESCRIPTION 
       [0011]    In broad terms, embodiments provide a software middleware layer that, transparent to applications, allows those applications to use compute coprocessors such as graphics processing units (GPUs). These embodiments make possible capabilities such as scale-out, resource pooling, high availability, and memory virtualization that do not now exist for coprocessors and many applications that make use of coprocessors. 
         [0012]    See  FIG. 1 , which illustrates one example of a hardware/software stack in which applications  100  run on a host hardware platform  200 , typically under the control of a system software layer such as an operating system. In addition to other standard components, the hardware platform will include one or more processors  210 , volatile (such as RAM  215 ) and usually non-volatile (such as SSD, disk, etc.,  220 ) storage components, as well as components for communicating with other systems over a network and other peripherals as needed, for example, network interface components  230 . 
         [0013]    Depending on the configuration, the hardware platform  200  may also include one or more co-processors  250 , such as GPUs; these may, however, also be located within other systems, accessible via any standard buses or networks, such that the concept of “hardware platform” may be broadened to include such “peripheral” or even remote coprocessors, such as coprocessors in cloud computing environments. Embodiments may also be used in other forms of distributed systems, such as a software-defined infrastructure (SDI). Although not shown, coprocessors may also have dedicated components of their own, such as memory. As  FIG. 1  indicates by the “duplicated” or “stacked” privileged layers, the applications may also be running in a distributed processing environment, with more than one server handling processing tasks. 
         [0014]    System software, such as a host operating system  300 , is generally included to perform well-known functions, and will typically include drivers  310  that control communication with respective peripheral devices. The software—including the applications, the system software itself (and the interception layer  1000 , described below)—that runs on the hardware platform is generally processor-executable code that is embodied in the storage components. 
         [0015]    In many modern computing environments, the application layer  100  includes, and sometimes is even totally comprised of, one or more virtual machines (VMs)  120 , in which case a system-level virtualization component such as a hypervisor  400  will typically also be included to act as the virtual-to-physical interface. In some systems, the hypervisor is co-resident with the host OS  300 , with both operating in a privileged, kernel mode; in other systems, the host OS intermediates some or all hypervisor commands to the hardware; in still other systems the hypervisor replaces the host OS altogether; and in systems without virtualized components such as VMs, no hypervisor may be needed and included at all. Embodiments of this invention do not require VMs, although they may operate with VMs as with any other applications that call coprocessors such as GPUs; moreover, system programmers will be able to adapt embodiments to different host OS/hypervisor configurations, including those with no separate host OS at all. 
         [0016]    Applications generally include various core functions  110 , such as the routines needed to communicate processing requests to the operating system, to access the file system, allocate memory, and access common functional libraries for operations such as data compression, image manipulation, accelerated math functions, etc. In the context of embodiments of this invention, one function of some applications is to properly format and issue calls to application program interfaces (APIs). As is well known, an API comprises routines and protocols that specify how software components should interact, how a software component such as an application may interact with a hardware component, etc. Of particular relevance here is that an API is usually included to form an interface between applications  100  and coprocessors. By way of example only, graphical user interface (GUI) components are referred to below as being the type of coprocessor the applications are to call into via corresponding APIs. 
         [0017]    As  FIG. 1  indicates, system software and certain other components generally run at a “privileged” or “kernel” level, meaning that they are allowed to issue, and have executed, instructions that affect such things as storage protection settings, interrupt handling, scheduling, I/O coordination, special processor status- and register settings, etc. Applications, on the other hand, typically operate in a non-privileged user space, that is, in user mode. 
         [0018]    In many systems, there is an intermediate software layer between applications and the system software. This layer, labeled  1000 * in  FIG. 1 , includes such components as application programming interfaces (APIs) libraries, both open and custom, and various system libraries. In systems that employ graphics processing units (GPUs), either for standard graphics tasks or in a GPGPU (General Purpose computing on GPUs) context, this layer may also include an API model such as the CUDA (Compute Unified Device Architecture) parallel computing platform provided by Nvidia, Inc., along with the associated libCUDA driver library. 
         [0019]    This invention provides a software layer  1000 , referred to here as the “interception layer”, which may run at the non-privileged level, and which either mirrors the intermediate layer  1000 *, or is installed to act as such a layer. As the name implies, the interception layer  1000  intercepts API calls made by applications and changes them in a way that may introduce new functionality. Depending on the chosen implementation, the layer  1000  may include both custom APIs and generally available, open APIs  1060 , as well as one or more libraries  1070 , which list application features, enumerate available devices/platforms, etc., and which may be provided by vendors, or compiled heuristically, or both. Compute APIs, specifically, deal with the management of coprocessors, execution flow, and data movement, to make full and efficient use of the coprocessors. This includes dispatching data and compiled compute routines, returning status, and synchronizing streams of execution between the coprocessors and other coprocessors, and the coprocessors and the host system. 
         [0020]    The interception layer  1000  is preferably configured so as not to require any changes to the applications running above, or modifications to the system software on which the layer itself runs. In other words, embodiments may run on commodity systems. Although this configuration leads to advantages such as ease of installation and use, portability, universality, and convenience, other configurations are possible. For example, the interception layer could be installed at the privileged level, and could even be incorporated into system software, in the OS or hypervisor. 
         [0021]    The code that comprises the interception layer may be installed in the system and configured to intercept application calls using any known method, including downloading it from a network, reading in the code from a tangible, non-volatile storage medium, etc. This is in fact one advantage of the invention: It may be installed like other user-level applications, including applications that interact with other applications, with no need to modify the system software or include dedicated or modified hardware. 
         [0022]    See  FIG. 2 , which shows an example embodiment in which applications issue calls that are intercepted by the layer  1000 , which directs the calls according to rules (see below) to one or more coprocessors. Embodiments of this invention may be used to advantage with substantially any known type of co-processor, many examples of which are mentioned above. Merely by way of a common example, embodiments are explained below and illustrated for the case in which the coprocessors are GPUs GPU 1 , GPU 2 , . . . , GPUn. If any modifications are needed at all to adapt a particular embodiment for use with another type of coprocessor, these adaptations will be within the skill of system architects. 
         [0023]    Coprocessors typically have a “master-slave” relationship relative to the normal host CPU(s)  210  that runs the applications—an application is typically run on the host CPU(s) while certain computational tasks are offloaded to coprocessors for increased performance. Compute APIs, that is, APIs that are used by applications to leverage coprocessors, have explicit functions to offload computation and move data between host memory and coprocessor device memory. 
         [0024]    The API for each coprocessor, such as a GPU, accessible by the overall system, is known and exposed to the applications that may wish to use them. Some of these APIs will be open, that is, public, whereas others may be customized for certain coprocessors. Even in the case of customized APIs, however, these will be known to all applications that may need to use the corresponding coprocessor. Since the coprocessors available at any given time are known to the system, their APIs are also known. As is known, applications  100  are programmed to properly format API calls to the coprocessors they wish to use and in many cases, the APIs that each application will call into may also be known in advance. 
         [0025]    The interception layer  1000  includes the set of APIs, both custom and open, in respective components  1050 ,  1060 , so as to be able to intercept the calls from applications and correctly interpret them and pass on calls with the proper formatting to the respective GPU. As mentioned, the set of APIs used by many applications is known in advance. Applications issuing calls via the OS  200  are identifiable using known methods even to the interception layer, which may therefore include libraries  1070  of the APIs needed by each application  100  in the system. As mentioned above, examples of such libraries include system libraries, and those offered by the OS, libraries that provide logging, image manipulation, compression, networking access, interprocess communication, etc. Alternatively, or in addition, the interception layer may build up the libraries  1070  by registering in them the APIs that the applications  100  actually call. 
         [0026]    The interception layer  1000  includes, for each GPU, a corresponding replay log RPL 0 , RPL 1 , . . . , RPLn, which may be implemented either as portions of the space of the system memory  215  or any other memory component, either local to the host hardware or remote. To access coprocessors such as the GPUs, applications issue calls that comprise respective command and data streams. Using known methods, the interception layer intercepts these streams and collects them in replay logs RPL 0 , RPL 1 , . . . , RPLn provided on a per-GPU basis. Using known methods, the interception layer also captures—“checkpoints”—the execution state of the respective GPU. The interception layer may store this execution state in the form of state and context data structures that are typically maintained by the respective compute API (for example, CUDA, OpenCL, etc.). 
         [0027]    As each GPU application runs, the layer  1000  collects the respective command and data stream in the replay log associated with that GPU. Each replay log RPL 0 , RPL 1 , . . . , RPLn is preferably sized so as, at a minimum, to be able to store all of the transactions directed to its corresponding GPU since a most recent synchronization point Ft. Note that, when it comes to coprocessors, such synchronizations points are generally well-defined, since the instruction and data stream directed to a coprocessor such as a GPU is typically “linear”, without conditional branching or time-dependent processing paths, and corresponds to discrete processing tasks having known beginning and end points. As such, if a GPU fails after a synchronization point, it is possible to restart the failed processing segment from the most recent synchronization point as long as all of the instruction and data stream from point Ft onward to the point of failure is available. A window wt, defined in terms of time or number of instructions, between consecutive synchronization points is therefore knowable in advance for each GPU, such that the replay logs may be configured so as never to have a potential “gap”. 
         [0028]    Synchronization points for GPUs are similar to other forms of “checkpointing” in that the state of the GPU memory is also known at each synchronization point. In embodiments of the invention, the GPU memory is preferably shadowed, using known mechanisms, in host memory  215 , on one or more other GPUs, on other host platforms, or even on the file system. Shadowing is preferably two-way, such that if the host makes changes to memory in the shadow region, these changes will be communicated to the GPU, and vice versa. This memory synchronization may be done using any known mechanism, such as by using a conventional unified virtual memory driver for GPUs. This means that, at each synchronization point and forward to the next, the entire state of each GPU can be replicated based on the shadowed memory state and the logged instruction stream from that point. 
         [0029]    It is not necessary for the command and data stream for a given GPU to enter and exit the corresponding replay log to be executed by the target GPU device; rather, as indicated by the active, direct stream  130   n  (for GPUn, with similar paths for other GPUs), the replay log can collect the stream in parallel to the active stream. 
         [0030]    Now see  FIG. 3  and assume that one of the coprocessors, for example, GPUn has failed or that the connection with it has failed, such that it cannot process the instructions and data from one or more of the applications  100 . There are several types of GPU failure, which include, without limitation, any failure that can be detected at runtime such as: memory, power, execution, thermal, data corruption, interconnect failures resulting in incorrect operation. The physical and logical state information of the GPU indicating such failures may generally be detected at the application programming interface (API) level, but could be detected by the interception layer by other conventional means including, but not limited to, sensing a signal from the associated driver, by sensing interrupts, polling of the GPU device, etc. 
         [0031]    Upon failure of a GPU in use, a redirection module  1080  within the interception layer  1000  first signals either the corresponding application  100  directly, or via the operating system, to pause execution of the application and then selects another, currently operational GPU according a redirection policy. This policy may be chosen in any manner, examples of which include random selection, a policy based on a utility function such as the current computational load of all GPUs in the system (for load balancing, for example), or of the least loaded GPU, based on required GPU speed, to reduce power consumption or heat, or of a degree of GPU dissimilarity (to reduce the likelihood of correlated failures), etc. This information may be programmed into the interception layer (such as in the module  1080 ) by an administrator, or sensed via either the corresponding API, via the system software, or any other known mechanisms. 
         [0032]    Assume that GPUm is selected to take over from the failed GPUn. The data and instruction stream for failed GPUn since the most recent synchronization point will be stored in RPLn, and other state information for GPUn will also be available as described above. The interception layer thus assigns the replay log RPLn of the failed GPUs to the new target GPUm, and directs the stream of instructions and data in RPLn to GPUm for execution. The replay log RPLn is thus drained into GPUm, at which point the application that was paused may be signaled to continue its execution without interruption. 
         [0033]    In some cases, an application will specify which GPU it wishes to access simply by virtue of the API call it makes; in other cases, the API call may be to a specific type of GPU, but the OS may nominally be set to make an allocation of the GPU resource. Assuming that more than one GPU is able to handle the request, however, it would not be necessary to wait for a GPU failure to switch the application-GPU assignment; rather, based on some metric or other criterion, such the GPU load imbalance being greater than some threshold, the interception layer could use the replay log of an overloaded GPU to enable forward computing progress until the application can be directed to call to a different GPU starting at the next synchronization point (if the window wt is wide enough to accommodate the entire stream from one synchronization point to the next), or starting at the point at which the RPL has been drained; the applications API stream can then also be changed to be logged in the replay log of the new GPU. 
         [0034]    Use of the replay logs to effect a change in the application-GPU assignment may be advantageous even in other situations. In particular, coprocessor redirection could be initiated for reasons other than coprocessor failure or unavailability. For example, assume that one or more additional GPUs is made available in runtime, for example through remote network attachment. For each such additional GPUj, the interception layer may create a corresponding replay log GPUj and redirect calls to a previously available GPU to one of the new ones. This might also be for the purpose of load balancing. In another embodiment, the interception layer may be used to improve the performance of the GPU-calling application  100  by splitting the application&#39;s command stream to two separate GPUs. Even this embodiment may be extended by using the replay logging of the interception layer to implement GPU RAID capabilities such as mirroring, striping, and error correction in multiple configurations. As other examples, coprocessor redirection could be initiated by the interception layer, or by system software, to improve measured performance, power consumption, GPU uptime, etc. Since the command and data stream from an application is intercepted it can be manipulated, split and mirrored in virtually any combination that allows for improved RAS (reliability, availability, serviceability), performance, and cost. 
         [0035]    Logging of the instruction and data stream for a particular GPU may also be prospective: Note that the stream for a coprocessor such as a GPU is substantially “deterministic” in the sense that there are typically no conditional branches or jumps—in a compute API, all data accesses are explicit, so there is no need to guess what the next block will be. This means that all or part of an application&#39;s API call stream, through to the next synchronization point, can be pre-fetched, assuming it is in a memory space accessible and known to the interception layer, for example, through address pointers in the API call. Many provisioning systems now assume some shared memory. For example, coprocessors now typically have two memory spaces (host vs coprocessor, coprocessor 1 vs coprocessor, etc.) that must be kept coherent, but which may also be accessed by the interception layer. Note that, in a compute API, commands can be dispatched that don&#39;t have to execute immediately to make progress. 
         [0036]    Embodiments of the invention have several features not found in the prior art, and offer corresponding benefits. For example, the interception layer may run in user space, that is, not necessarily in system software such as an OS or hypervisor, or in an application  100 . Thanks to this, the invention can be portable, installed in a wide range of environments. 
         [0037]    Moreover, because the interception layer may run in a non-privileged execution mode, security and data confidentiality may be enforced without compromising the system with additional privileged code. Furthermore, unlike other approaches, additional features can be introduced into the interception layer without having to change the underlying operating system, drivers, or virtual machine hypervisors. 
         [0038]    One main reason that systems include coprocessors such as GPUs is to increase processing speed. Assume that a GPU application  100  is running slowly. According to prior art methods, one must analyze code to understand how much time is spent on the GPU and the cost for computation and data transfer. To save time, such known systems typically change the application code such that, while computing a current batch, the next batch is being fetched. Embodiments of this invention, in contrast, may implement automatic pipelining, using the replay logs, and do this automatically, for practically any application, with no need to change code. This may therefore provide a performance advantage even for badly written code.