Patent Publication Number: US-11042413-B1

Title: Dynamic allocation of FPGA resources

Description:
BACKGROUND OF THE INVENTION 
     Data centers are facilities used to store computer systems and components, such as data storage systems and their associated components. Modern data centers are typically large facilities whose computer systems are used to store large amounts of data and perform compute tasks on such data. The tremendous increase in the generation and consumption of data in recent times has been a catalyst for the utilization of more and more data centers to store and process these vast amounts of data. One challenge associated with performing compute tasks efficiently in data center environments is configuring hardware resources for a variety of different compute tasks. A mix of compute tasks being performed in a data center requires techniques to determine how to configure the hardware resources to efficiently utilize resources and ensure the various compute tasks are completely in a timely manner. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings. 
         FIG. 1  is a block diagram illustrating an embodiment of an FPGA-based server system that dynamically allocates resources to perform a variety of compute tasks. 
         FIG. 2  illustrates an example network architecture in which servers (on which FPGAs reside) are located in different server clusters. 
         FIG. 3  is a flowchart illustrating an embodiment of a process for dynamically reallocating an FPGA resource in a server system. 
         FIG. 4  is a flowchart illustrating an embodiment of a process for performing a compute task using an FPGA. 
         FIG. 5  is a flowchart illustrating an embodiment of a process for identifying a service to perform at least a portion of a compute task. 
         FIG. 6  is a flowchart illustrating an embodiment of a process for invoking a service to perform at least a portion of a compute task. 
         FIG. 7  is a flowchart illustrating an embodiment of a process for reporting results of a compute task. 
     
    
    
     DETAILED DESCRIPTION 
     The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions. 
     A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured. 
     An FPGA-based server system that dynamically allocates resources to perform a variety of compute tasks is disclosed. The disclosed server system includes field-programmable gate arrays and a processor configured to: determine utilization metrics associated with a plurality of eligible compute tasks executable on the field-programmable gate arrays, evaluate the utilization metrics to dynamically identify a reallocation of an identified field-programmable gate array resource of the field-programmable gate arrays to handle a selected eligible compute task of the eligible compute tasks; and automatically reprogram the identified field-programmable gate array resource to handle the selected eligible compute task. 
     A practical and technological benefit of this FPGA-based server system is the capability to flexibly manage hardware resources to meet changing compute task demands. This flexibility is due to the programmable and reprogrammable nature of FPGAs, allowing them to handle specialized compute tasks and to be reconfigured to handle different specialized compute tasks as needed. Thus, the disclosed system is able to dynamically respond to computing demands. 
     FPGAs have advantages over other specialized hardware and general CPUs. Certain classes of highly-parallelizable data center compute tasks, e.g., signal processing, image processing and machine learning, can be efficiently implemented on specially designed hardware, such as ASICs. But this approach is challenging due to the substantial non-recurring engineering costs and development (e.g., design, validation, deployment, etc.) time associated with specially designed hardware such as ASICs. High-performance and high-efficiency FPGAs, which perform better and more efficiently than CPUs, present a compute alternative to accelerate algorithms which are not used frequently enough to justify the cost of a custom ASIC solution. While FPGAs may be less power efficient than ASICs, they can be significantly more power efficient than CPUs (in addition to being more computationally efficient than CPUs). 
     Furthermore, ASICs cannot be reconfigured, but FPGAs are flexible enough to allow future changes and optimization. FPGAs can also outperform graphics processing units (GPUs). Although GPUs can take advantage of data linearization to outperform CPUs, FPGAs are oftentimes more efficient still (e.g., in terms of speed and power efficiency). FPGA resources are dynamically reprogrammable and therefore are uniquely able to respond to changes in the demand for specific kinds of applications. When a CPU is performing large volumes of integer arithmetic, the remaining on-chip functional units (e.g., floating-point or vector units) go unused and simply power down. The same is true on a system level with other special-purpose compute units such as GPUs. FPGAs are capable of physically transforming into different kinds of resources on the fly. 
     In some embodiments, the system includes a plurality of application programming interface (APIs), a scheduler/router, and a plurality of FPGAs. The APIs expose the FPGAs to users (e.g., various software applications) that would like to use the FPGAs for compute tasks. In various embodiments, the APIs receive requests to perform various compute tasks (e.g., execute various algorithms, wherein different algorithms are programmed on different FPGAs and exposed to users via different APIs). The scheduler/router is responsible for receiving inbound requests, queuing them as necessary, and delivering them to appropriate interface software (interfacing with the FPGAs). In various embodiments, the interface software is capable of, when given a suitably structured request, executing an accelerated function on an FPGA and returning resulting data. In various embodiments, when an FPGA is allocated to a function, a service that is mapped one-to-one with the FPGA is initialized. In some embodiments, the service is part of the interface software. Due to the FPGA being initially unprogrammed, in various embodiments, the service reads an FPGA bitstream from persistent storage and transfers it to the FPGA, after which the service is ready to accept requests for compute tasks to be performed by the FPGA. 
     In various embodiments, the initialization process for the system is as follows: power is applied and the host operating system loads, the scheduler/router is loaded, available FPGA resources are enumerated, and available service descriptions are loaded. At this point, FPGAs are powered down and unprogrammed. In various embodiments, the lifecycle of an FPGA includes being powered down, being powered up and programmed (with a service mapped to the FPGA), potentially being dynamically reconfigured, and being powered down again. Programming an FPGA requires a non-trivial amount of time. Thus, initial allocation of FPGAs to functions/algorithms performed by the FPGAs influences overall system performance. Examples of initial configuration strategies include: keeping all FPGAs powered down and unprogrammed (not allocated) until a first compute request is received, allocating a specified number of FPGAs before the first compute request (e.g., one FPGA per API), and allocating all FPGAs before the first compute request (e.g., allocated across APIs uniformly or according to a distribution based on use statistics). 
     From a high-level perspective, the system exposes API endpoints to customers/users defined in the same way as existing remote procedure calls (RPCs) for services (e.g., REST, Thrift, gRPC, and similar models). Data dependencies are encoded within the request and all data is returned in the response, thereby yielding a stateless system. The system is therefore agnostic to the language, operating system, and conditions of the API customer, and independent of the physical location where systems are deployed. Scheduling/routing is performed by mapping incoming API requests to available FPGAs that implement the relevant functions (e.g., algorithms). In various embodiments, metrics are collected to inform subsequent scheduling/routing efforts. For example, a cost function can include such metrics as how much energy was expended executing the request, how well the work was performed, and latency and average execution durations. This configuration enables disaggregation of nodes across racks, pods, and data centers, and service requests can be intelligently mapped to nodes configured to accept the incoming function signatures. 
     FPGAs are especially suitable for compute tasks that are time-consuming and would benefit from specialized hardware but do not occur frequently enough to justify the investment costs to develop an ASIC. An example of this type of compute task is graphics interchange format (GIF) resizing (e.g., resizing a small thumbnail GIF). GIF resizing can take a long time for processing on a traditional CPU (e.g., several tens of seconds). Other example compute tasks include video encoding, natural language processing (NLP), artificial intelligence (AI) inference, complex arithmetic, model fitting, data compression, encryption, hashing, Fast Fourier Transform (FFT), and fuzzy logic hardware accelerator (FLHA) compute tasks, other image/video/audio related compute tasks, and any other compute tasks, particularly computationally intensive ones that would benefit from specialized hardware tailored to those tasks. Another advantage of FPGAs is that performance can be monitored for a period of time (e.g., a week, a month, etc.), adjustments can be made, and results can be monitored again (e.g., a learning loop). For example, if it is discovered that a specific compute task occurs very frequently, it can be decided that an investment in an ASIC implementation is warranted. 
     Reallocation of FPGAs resources can be made based on usage statistics collected over a specified period of time. For example, suppose FPGAs are configured to expose two different APIs (API A and API B) associated with two different compute tasks. The scheduler/router can collect statistics on the proportion of calls to API A versus API B over a specified period of time (or collect other utilization metrics). If API A is called only occasionally while API B is called frequently, the scheduler/router can update the allocation such that more FPGAs are programmed to implement the function associated with API B than API A. This may involve reprogramming devices previously allocated. The scheduler/router may also leave FPGAs unallocated for either power management (due to low utilization) or as backups to swap in. In some embodiments, a function repository is available to nodes for dynamic provisioning in which objects contain bitstreams and characteristic metadata. Based on geographical load and resources the repository of functions can be uniquely tailored. 
     As described above, an overall, top-level architecture includes an API, a schedules/router, interface software between the scheduler/router and hardware, and hardware (e.g., a plurality of FPGAs). The top-level architecture described above may also be extended to allow for heterogeneous hardware resources (e.g., a mix of FPGAs, ASICs, CPUs, and GPUs) to be used within the system architecture/paradigm. In this heterogeneous compute environment, certain compute tasks may only be programmed on certain hardware resources, which can be taken into consideration when assigning compute tasks to hardware resources. The scheduler/router has added responsibility to consider such additional factors. 
       FIG. 1  is a block diagram illustrating an embodiment of an FPGA-based server system that dynamically allocates resources to perform a variety of compute tasks. In the example shown, server system  100  includes API A  102 , scheduler/router  104 , hardware interface software units  106 ,  122 , and  132 , Type A FPGA  114 , Type B FPGAs  130  and  140 , and Unallocated FPGA  142 . In the example shown, each hardware interface software unit includes a service, bitstream, and driver. Each hardware interface software unit is mapped to an FPGA type and an API type. For example, hardware interface software  106  is mapped to API A  102  and Type A FPGA  114 . If API A  102  receives a certain type of function call (e.g., resizing GIFs), hardware interface software  106  is configured to handle that type of function call and Type A FPGA  114  is configured to implement that type of function call (e.g., resizing GIFs). This means that service  108 , bitstream  110 , and driver  112  are configured to handle that type of function call (e.g., resizing GIFs). 
     In the example shown, API B  120  would receive a different type of function call. In the example shown, Type B FPGAs  130  and  140  implement the function associated with API B  120  and hardware interface software units  122  and  132  are configured for the function call associated with API B  120  and Type B FPGAs  130  and  140 . In the example shown, Unallocated FPGA  142  does not implement a function associated with API A  102  or API B  120  (e.g., Unallocated FPGA  142  may be unprogrammed). In various embodiments, when Unallocated FPGA  142  is allocated for a specific type of compute task, hardware interface software is then mapped to Unallocated FPGA  142 . For example, if Unallocated FPGA  142  is programmed to implement the same function as Type A FPGA  114 , Unallocated FPGA  142  would be mapped to a hardware interface software unit identical to hardware interface software  106 . 
     In the example illustrated in  FIG. 1 , different APIs are exposed to users for different compute requests. It is also possible to instead use a single API that is configured to receive a variety of compute requests. A plurality of scheduler/routers may be used with this single API approach. For example, the single API (e.g., an API gateway) may forward compute requests to schedulers/routers mapped to FPGA resources based on what functions the FPGAs implement. Stated alternatively, each scheduler/router may be configured to handle queuing, data transfer to/from FPGAs, and communication with the API gateway for one type of compute task. It is also possible to use a single scheduler/router with a single API. In this scenario, the single scheduler/router determines where compute requests are sent based on its knowledge of which FPGAs implement which functions. This information about the FPGAs may be communicated to the scheduler/router as bitstream information from services mapped to the FPGAs. It is also possible for different APIs to be exposed for different compute tasks and use a dedicated scheduler/router for each API. In all of these scenarios, there is a defined API endpoint for receiving multiple types of compute requests and a mechanism for directing these compute requests to the appropriate FPGA resources. 
     In the example shown, API A  102 , API B  120 , scheduler/router  104 , and the hardware interface software units comprise server system  100 &#39;s software layer, and the FPGAs comprise server system  100 &#39;s hardware layer. Examples of software components in  FIG. 1  (e.g., API A  102 , scheduler/router  104 , service  108 , bitstream  110 , and driver  112 ) include any software system, component, process, and/or application. 
     In the example illustrated in  FIG. 1 , API A  102  and API B  120  interface with scheduler/router  104 . API A  102  and API B  120  also interface with users/customers of server system  100 . API A  102  and API B  120  are where a compute request from a user is initially received and they are the user&#39;s interface for receiving a response/output data associated with the compute request. By using an API approach, FPGA capabilities can be presented to users via high-level programming languages. In some embodiments, APIs are defined in the same way as traditional RPC models. Examples of RPC models include REST, Thrift, and gRPC. A customer program (customer in the sense of requesting services) of API A  102  or API B  120  may send a request message to API A  102  or API B  120  with appropriate arguments and data and then wait for a message containing the results of an executed compute on the data. 
     For example, with respect to GIF resizing, the customer program may send a desired resulting GIF size argument and as well as a GIF image. The result returned would be a GIF image that is the desired resulting GIF size. Other examples of compute task requests include video encoding, natural language processing (NLP), artificial intelligence (AI) inference, complex arithmetic, model fitting, data compression, encryption, hashing, Fast Fourier Transform (FFT), and fuzzy logic hardware accelerator (FLHA) compute tasks, other image/video/audio related compute tasks, and any other compute tasks, particularly computationally intensive ones that would benefit from specialized hardware tailored to those tasks. In various embodiments, scheduler/router  104  determines how the requests are handled. 
     In various embodiments, a function repository (e.g., implementing the above functions) is available to nodes for dynamic provisioning in which objects contain bitstreams and characteristic metadata. Based on geographical load and resources the repository of functions may be uniquely tailored. In various embodiments, when a service is initialized, it is bound to one of the FPGA devices attached to the system, the FPGA is initially unprogrammed, and the service reads an FPGA bitstream from persistent storage and transfers it to the corresponding FPGA to make the service ready to accept requests. 
     In the example illustrated in  FIG. 1 , scheduler/router  104  interfaces with API A  102  and API B  120 . Scheduler/router  104  receives user compute requests passed on from an API. Scheduler/router  104  also passes the results of compute task processing back to the requesting API. Scheduler/router  104  is configured to determine which FPGA resources are available to perform a compute task and which FPGA resources have been configured to handle which types of compute tasks. For example, if a GIF resizing request is received, scheduler/router  104  would be able to determine which FPGAs implement GIF resizing and the statuses of those FPGAs (e.g., available, occupied, etc.). In some embodiments, scheduler/router  104  assigns the compute task to a first available FPGA implementing the relevant function. If there are no currently available FPGAs implementing the relevant function, scheduler/router  104  may queue the compute task with an occupied FPGA, choosing from among a plurality of FPGAs according to a metric (e.g., a load balancing metric). 
     In addition to load balancing considerations, scheduler/router  104  may also consider where FPGAs are located when assigning compute tasks. In some embodiments, FPGAs in server system  100  are located across different servers. In these embodiments, routing functionality of scheduler/router  104  may be used to route traffic to FPGAs on multiple servers. If all FPGAs in server system  100  are located on a single server, then no routing functionality is needed and scheduler/router  104  only requires scheduling functionality. If servers on which FPGAs reside are located in different server clusters, a higher-level routing architecture may be used. 
       FIG. 2  illustrates an example network architecture in which servers (on which FPGAs reside) are located in different server clusters. In the example shown, servers (with FPGAs implementing various functions) are located in four server clusters. A master scheduler/router that interfaces with an API (e.g., API A  102  or API B  120  of  FIG. 1 ) may be located in Node 1. In order to efficiently route traffic to the four server clusters, an additional routing layer (Node 2 and Node 3) may be added. In the example shown, this additional routing layer is fully interconnected with the server clusters (e.g., Node 2 and Node 3 have connections to all four of the server clusters). Node 2 and Node 3, as well as each server in each cluster, may include a scheduler/router that coordinates with the master scheduler/router and/or other schedulers/routers in the network architecture. 
     Returning to the example illustrated in  FIG. 1 , hardware interface software  106  serves as a communication conduit between scheduler/router  104  and Type A FPGA  114 . In the example shown, other hardware interface software/FPGA stacks in server system  100  also communicate with scheduler/router  104 . As described above, FPGAs (and thus their associated hardware interface software) may be located on a single server, on different servers in a single server cluster, on different servers in different server clusters, etc. The various FPGAs implement various functions that a user could request, with the specific distribution of implemented functions across FPGAs based at least in part on demand for those functions. In various embodiments, when scheduler/router  104  assigns a compute task to an FPGA (e.g., Type A FPGA  114 ), a request that is still in a software format (e.g., RPC format) is forwarded to hardware interface software (e.g., hardware interface software  106 ). The hardware interface software (e.g., hardware interface software  106 ) then converts the request into a format compatible with an FPGA (e.g., Type A FPGA  114 ). 
     In the example illustrated in  FIG. 1 , hardware interface software units include a service, bitstream, and driver. For example, hardware interface software  106  includes service  108 , bitstream  110 , and driver  112 . Service  108  is software that decodes a request message (e.g., an RPC message) and sends it to Type A FPGA  114  after the request message is converted by driver  112  into a format compatible with Type A FPGA  114 . Bitstream  110  is software that determines the function that Type A FPGA  114  implements. This information is needed by scheduler/router  104  in order for scheduler/router  104  to match FPGAs with compute requests. In various embodiments, bitstream  110  and other bitstreams send information on what compute tasks FPGAs are configured to handle to scheduler/router  104 . Driver  112  is software that structures the request message received by service  108  into a format that Type A FPGA  114  can use to perform the requested compute task. The components of hardware interface software  122  and  132  perform the above for Type B FPGAs  130  and  140 , respectively. 
     In various embodiments, after an FPGA (e.g., Type A FPGA  114  and Type B FPGAs  130  and  140 ) performs the requested compute task, the results of the compute are transferred back to the corresponding service, which then constructs an API response object to be sent back to scheduler/router  104 , which then sends the response object back to API  102  and to the user. At this point, the service is ready to accept the next request for use of the FPGA. Services are mapped one-to-one to FPGAs (e.g., service  108  mapped to Type A FPGA  114 ). Stated alternatively, when a service is initialized, it is bound to an FPGA device. The FPGA is initially unprogrammed when it is powered on. In various embodiments, the service bound to the FPGA device reads an FPGA bitstream from persistent storage and transfers it to the FPGA, thus initializing the FPGA and making it ready to accept compute requests.  FIGS. 4-7  provide additional details associated with performing a compute task using an FPGA, from identifying a service to perform a compute task to invoking the service to reporting results of the compute task. 
     In the example illustrated in  FIG. 1 , FPGAs are the hardware resources that perform compute tasks. In various embodiments, the FPGAs are implemented as integrated circuits that include arrays of logic blocks that are programmable using a hardware description language (HDL). Examples of HDLs include VHDL and Verilog. The reprogrammable nature of FPGAs allows FPGAs the flexibility to be reconfigured to handle different specialized compute tasks as needed, which is not possible with other hardware technologies, such as ASICs. In addition, FPGAs can be much more efficient than CPUs because FPGAs can structure computation in a more parallel manner. For example, an FPGA that implements GIF resizing may be able to try 128-256 resize combinations at once because it can implement 128-256 resizing hardware cores instead of perhaps merely 4 resize combinations from a 4-core CPU. 
     In various embodiments, the initialization process for the system is as follows: power is applied and a host operating system loads, scheduler/router  104  is loaded, available FPGA resources are enumerated, and available service descriptions are loaded. At this point, FPGAs are powered down and unprogrammed. In various embodiments, the lifecycle of an FPGA includes being powered down, being powered up and programmed (with a service mapped to the FPGA), potentially being dynamically reconfigured, and being powered down again. Programming an FPGA requires a non-trivial amount of time. Thus, initial allocation of FPGAs to functions/algorithms performed by the FPGAs influences overall system performance. Examples of initial configuration strategies include: keeping all FPGAs powered down and unprogrammed (not allocated) until a first compute request is received, allocating a specified number of FPGAs before the first compute request (e.g., one FPGA per API), and allocating all FPGAs before the first compute request (e.g., allocated across APIs uniformly or according to a distribution based on use statistics). 
     Scheduler/router  104  may also allocate an unallocated FPGA to perform a specified compute task that is in high demand. For example, Unallocated FPGA  142  may be allocated, in which case hardware interface software (e.g., a service, bitstream, and driver) would be instantiated and mapped to Unallocated FPGA  142 . The function/algorithm Unallocated FPGA  142  is assigned to perform can depend on a variety of metrics. Examples of these metrics include queue depth and execution time. With respect to queue depth, Unallocated FPGA  142  may be assigned to perform a function/algorithm with longer incoming queues (indicating higher demand) on already allocated FPGAs. With respect to execution time, Unallocated FPGA  142  may be assigned to perform a function/algorithm whose outstanding compute requests are taking a longer time to execute than compute requests for other functions/algorithms. 
     When all FPGA resources have already been allocated, some FPGA resources may be reallocated (e.g., redirected to handle compute tasks that are in high demand). As with allocation of unallocated FPGA resources, reallocation of FPGA resources can depend on metrics such as queue depth and execution time. For example, if one group of FPGAs that implements a particular function has a significantly higher queue depth than other groups of FPGAs, FPGAs from the other groups may be reallocated to implement the function of the group of FPGAs with the higher queue depth in order to balance queue depths across FPGA groups. 
     In the example illustrated in  FIG. 1 , portions of the communication path between the components are shown. Other communication paths may exist, and the example of  FIG. 1  has been simplified to illustrate the example clearly. Although single instances of components have been shown to simplify the diagram, additional instances of any of the components shown in  FIG. 1  may exist. The number of components and the connections shown in  FIG. 1  are merely illustrative. For example, more APIs, hardware interface software units, and FPGAs than what is shown may exist. Components not shown in  FIG. 1  may also exist. As described above, implementations with a single API gateway and/or multiple scheduler/router units are also possible. In addition, different hardware interface software unit/FPGA stacks may be located in different geographic locations, and the scheduler/router may implement multiple layers of scheduling/routing. 
       FIG. 2  illustrates an example network architecture in which servers (on which FPGAs reside) are located in different server clusters.  FIG. 2  is discussed above with respect to the server system example of  FIG. 1 . 
       FIG. 3  is a flowchart illustrating an embodiment of a process for dynamically reallocating an FPGA resource in a server system. In some embodiments, this process is performed by server system  100  of  FIG. 1 . 
     At  302 , utilization metrics associated with a plurality of eligible compute tasks executable on FPGAs are determined. Utilization metrics can be associated with queues of incoming requests for FPGAs. Queue depth can be used as a metric for reallocating resources. For example, if one group of FPGAs that implements one particular function has a higher queue depth, then FPGAs with lower queue depths can be reallocated toward the group of FPGAs with the higher queue depth. Stated alternatively, FPGA resources can be reallocated to balance queue depths. Execution time can also be used as a metric for reallocating resources. For example, if one group of FPGAs that implements one particular function is experiencing longer execution times (e.g., taking longer to complete compute task requests) than other groups of FPGAs, then FPGA resources can be reallocated to the FPGAs that implement the function experiencing longer execution times. 
     Cost functions that take into account multiple metrics, such as queue depth, execution time, and other metrics, can be constructed and used (e.g., by giving each of the metrics weights or using them in formulas). For example, execution time may be multiplied with queue depth. Thus, compute tasks associated with a queue that is twice as long but execute in half the time would be as costly as compute tasks associated with a queue that is four times as long but execute in a quarter of the time. Reprogramming time for FPGAs can also be taken into account (e.g., reallocation may not be favored if the reprogramming time cost outweighs benefits of reallocation. Reprogramming time may vary for different FPGA bitstreams (e.g., some functions/algorithms may be associated with larger bitstreams and thus longer reprogramming time). 
     Time of day, calendar date, and time zone may be used as factors that affect allocation of resources. For example, more FPGA resources may be allocated during peak use periods. FPGA resources may be reallocated based on current events. For example, more FPGA resources may be reallocated to implement GIF resizing during active social media periods, e.g., during a Super Bowl or other major sporting event. Service-level requirements (e.g., maximum response time) can also affect whether resources are reallocated. For example, reallocation of FPGA resources may be disallowed if it would cause a service requirement to be violated. In addition, if a service is predicted to not meet a requirement, reallocation of additional resources to help meet demand may be triggered. 
     At  304 , the utilization metrics are evaluated to dynamically identify a reallocation of an identified FPGA resource of the FPGAs to handle a selected eligible compute task of the eligible compute tasks. In various embodiments, utilization metrics are tracked. For example, the number of times a function/service is called can be tracked. This may be used to determine initial allocation of FPGA resources (e.g., preloading FPGAs) as well as for dynamic reallocation of FPGA resources. Queue depth and execution time can also be tracked. This may be used to calculate and evaluate cost function associated with different groups of FPGAs associated with different eligible compute tasks. For example, a first eligible compute task may be associated with a cost function value that is twice that of a second eligible compute task. In this scenario, FPGA resources may be identified to dynamically reallocate from a group of FPGAs performing the second eligible compute task to a group of FPGAs performing the first eligible compute task. 
     With respect to initial allocation of FPGA resources (e.g., preloading FPGAs), a scheduler router (e.g., scheduler/router  104  of  FIG. 1 ) will have a mapping of API requests to bitstreams (represented as services). The scheduler/router can keep track of statistics on the types of requests that have been serviced, predict demand based on the statistics, and preload some FPGAs before requests are received. For functions/algorithms without a preloaded FPGA, when a service associated with a function/algorithm is launched to handle a request for that function/algorithm, the service can connect to the FPGA and upload a bitstream for that FPGA. For this reason, the first use of an FPGA may be associated with a relatively long setup time. 
     At  306 , the identified FPGA resource is automatically reprogrammed to handle the selected eligible compute task. In various embodiments, reprogramming of the FPGA includes mapping the FPGA to a new service associated with the selected eligible compute task. For example, in server system  100  of  FIG. 1 , this may include assigning the FPGA a new hardware interface software unit. In this scenario, if the FPGA is being reprogrammed to implement the compute task of Type A FPGA  114  of  FIG. 1 , the FPGA would need hardware interface software identical to hardware interface software  106  of  FIG. 1 . In various embodiments, reprogramming the FPGA includes a service reading an FPGA bitstream from persistent storage and transferring it to the FPGA, thereby overwriting a prior bitstream, after which the service is ready to accept requests for the selected eligible compute task to be performed by the FPGA. 
       FIG. 4  is a flowchart illustrating an embodiment of a process for performing a compute task using an FPGA. In some embodiments, this process is performed by server system  100  of  FIG. 1 . 
     At  402 , a request to perform a compute task is received. In some embodiments, the request originates from a customer/user that is exposed to an API endpoint (e.g., API A  102  or API B  120  of  FIG. 1 ). The customer/user may be a software program that invokes other ancillary software programs (e.g., via APIs). The customer/user software may request some type of compute task to return a compute result by making a function call to the API (e.g., making a library function call). In various embodiments, the customer/user is not aware of the implementation details associated with returning the compute result. For example, if the customer/user invokes a function, e.g., GIF output=resizeGIF(GIF_input, result_size), the customer/user expects a new GIF image (GIF output) that is resized from an initial GIF image (GIF_input) and resized to a new size (result_size). The customer/user may not have visibility into the compute resource that performs the processing (e.g., a CPU, an FPGA, etc.). 
     At  404 , a service to perform at least a portion of the compute task is identified, wherein the identified service is associated with an FPGA that has been programmed to perform at least the portion of the compute task. In some embodiments, the identified service is a software interface associated with the FPGA (e.g., a hardware interface software unit in server system  100  of  FIG. 1 ). The identified service is associated with a specified compute task that is to be executed on the FPGA. In some embodiments, a scheduler/router identifies the service from among many services (e.g., by comparing bitstreams associated with the services to determine which FPGAs implement which compute tasks). In some embodiments, identifying the service includes receiving the request to perform the compute task (e.g., an RPC message), identifying FPGAs that implement the compute task, considering resource management factors (e.g., load balancing) to determine a service to assign to execute the compute task, and sending a request to perform the compute to the identified service. 
     At  406 , the identified service is invoked to perform at least the portion of the compute task. In some embodiments, hardware interface software (e.g., hardware interface software of server system  100  of  FIG. 1 ) receives a request to perform a compute task from a scheduling unit (e.g., scheduler/router  104  of  FIG. 1 ). In some embodiments, the hardware interface software invokes an FPGA to perform at least the portion of the compute task. In some embodiments, invoking the identified service includes receiving a request message, decoding the request message, converting the request message into data in a hardware format, and sending the converted data to the hardware. 
     At  408 , at least the portion of the compute task is executed using the FPGA. In some embodiments, one of the FPGAs in server system  100  of  FIG. 1  executes the compute task. For example, if the compute task is to resize a GIF image, an FPGA in server system  100  of FPGA that has already been programmed to perform GIF resizing will receive data in the form of the original GIF image and a desired new GIF size (as extracted from an original request message received by a software component associated with the FPGA). The output of the FPGA is the output of the compute task (e.g., a resized GIF image), which is ultimately reported back to the requestor of the compute task. 
     At  410 , results of the compute task are reported. In various embodiments, reporting results of the compute task to the requestor of the compute task is the reverse process of forwarding a request for the compute task from the requestor to the FPGA executing the compute task. In some embodiments, reporting the results of the compute task includes transferring result data from the FPGA to the service associated with the FPGA, constructing an API response object, and forwarding the API response object up to the requestor of the compute task. 
       FIG. 5  is a flowchart illustrating an embodiment of a process for identifying a service to perform at least a portion of a compute task. In various embodiments, the service is identified from among a plurality of services, and each service is associated one-to-one with an FPGA that has been programmed to perform at least the portion of the compute task. In some embodiments, the process of  FIG. 5  is performed by scheduler/router  104  of  FIG. 1 . In some embodiments, at least a portion of the process of  FIG. 5  is performed in  404  of  FIG. 4 . 
     At  502 , a message associated with a requested compute task is received. For example, the message may be an RPC message associated with a compute task function call sent by a customer/user of an API (e.g., API A  102  or API B  120  of  FIG. 1 ). A function with parameters and data (e.g., GIF output=resizeGIF(GIF_input, result_size)) may be called. In various embodiments, the function call is encoded into a message format (e.g., RPC message) that is received by a scheduler/router (e.g., scheduler/router  104  of  FIG. 1 ). For example, an RPC message with input data (e.g., original GIF image) and input parameters (e.g., desired resulting GIF size) encoded into the RPC message can be created. 
     At  504 , FPGAs that implement the requested compute task are identified. For example, in server system  100  of  FIG. 1 , the various FPGAs in the server system implement various compute tasks (e.g., some may implement GIF resizing, others may implement video encoding tasks, audio encoding tasks, natural language processing tasks, encryption tasks, and so forth). In some embodiments, bitstream information indicating the compute task each FPGA implements is sent from hardware interface software (e.g., bitstream  110 ,  126 ,  136 , etc. of  FIG. 1 ) to scheduler/router  104  of  FIG. 1 . From the bitstream information, the FPGAs that implement the requested compute task can be identified. 
     At  506 , a service to assign to perform the requested compute task is determined based at least in part on resource management factors. For example, if a first service and a second service implement GIF resizing and the first service is currently occupied but the second service is idle, a resource management strategy may be to assign the requested compute task to the second service. If both the first service and the second service are occupied, load balancing may be considered. For example, both the first service and the second service may have queues of already assigned tasks, in which case the service with the shortest queue may be assigned to perform the requested compute task (e.g., the requested compute task is added to the shorter queue). 
     At  508 , a request to the service is sent. If the service is already processing another request, the request that is sent to the service may be queued with the service (e.g., queued to be sent to hardware resources as they become available). Alternatively, the request may be queued before being sent to a service (e.g., queued with scheduler/router  104  of  FIG. 1 ) and released from the queue to the service when the service and the hardware resource associated with the service become available. 
       FIG. 6  is a flowchart illustrating an embodiment of a process for invoking a service to perform at least a portion of a compute task. In some embodiments, the process of  FIG. 6  is performed by hardware interface software (e.g., hardware interface software  106 ,  122 ,  132 , etc. of  FIG. 1 ). In some embodiments, at least a portion of the process of  FIG. 6  is performed in  406  of  FIG. 4 . 
     At  602 , a request message is received from a scheduler. In some embodiments, the scheduler is scheduler/router  104  of  FIG. 1 . In some embodiments, the request message is a forwarded RPC message associated with a compute task function call originally sent by a customer/user of an API (e.g., API A  102  or API B  120  of  FIG. 1 ). The request message includes data and parameters associated with the compute task (e.g., in an encoded format). For example, in the case of GIF resizing, an RPC message includes input data (e.g., original GIF image) and input parameters (e.g., desired resulting GIF size). In some embodiments, the request message is received by a service (e.g., service  108 ,  124 ,  134 , etc. of  FIG. 1 ). In some embodiments, the service queues the request until an FPGA to perform the compute task is ready to begin the compute task. 
     At  604 , the request message is decoded. In some embodiments, the request message is decoded by a service (e.g., service  108 ,  124 ,  134 , etc. of  FIG. 1 ). For example, an RPC message associated with a GIF resizing compute task may be decoded by extracting the GIF data (original GIF image) and desired resulting GIF size parameter. The decoding may be performed by keeping only the portions of the message containing the desired data (e.g., ignoring message headers and other metadata). This may be accomplished by the service determining the structure of the request message from a header portion or the service already knowing the structure when it receives the request message. 
     At  606 , data from the decoded request message is converted into a format compatible with FPGAs. In some embodiments, a driver (e.g., driver  112 ,  128 ,  138 , etc. of  FIG. 1 ) converts the data from the decoded request message into a format compatible with FPGAs. The format of the data (e.g., image data, numeric parameter values, etc.) to be sent to an FPGA depends on the format the FPGA is expecting for a specified compute task. For example, an FPGA that implements GIF resizing may expect GIF image data in a certain row and column format. In addition, the FPGA may expect numeric data (e.g., GIF pixel values and desired resulting image dimensions) in a specific numeric format (e.g., binary, decimal, hexadecimal, etc.). 
     At  608 , the data is sent to an FPGA. In some embodiments, the data is sent on a bus (e.g., wire, optical, fiber, etc.). The data may be sent using a serial connection, parallel connection, etc. 
       FIG. 7  is a flowchart illustrating an embodiment of a process for reporting results of a compute task. In some embodiments, the process of  FIG. 7  is performed by server system  100  of  FIG. 1 . In some embodiments, at least a portion of the process of  FIG. 7  is performed in  410  of  FIG. 4 . 
     At  702 , computation results are sent to a service. For example, computation results from Type A FPGA  114  of  FIG. 1  may be sent to service  108  of  FIG. 1 . In some embodiments, this is the reverse of step  608  in  FIG. 6  in which input data is sent from the service to the FPGA. 
     At  704 , an API response object is constructed. In some embodiments, this is performed by the service that receives the computation results. For example, service  108  in  FIG. 1  may encode the computation results (e.g., a resized GIF image) into an API object that can serve as a response to an RPC message. In some embodiments, this is the reverse of steps  604  and  606  in  FIG. 6  in which a request message (e.g., an RPC message) is decoded and converted into a format compatible with FPGAs. 
     At  706 , the API response object is forwarded to a scheduler. In some embodiments, the scheduler is scheduler/router  104  of  FIG. 1 . In various embodiments, the API response object is forwarded to the scheduler because the scheduler is the link to where the request to perform the compute task originated. In some embodiments, the request to perform the compute task originated from a user making a call to an API (e.g., API A  102  or API B  120  of  FIG. 1 ). 
     At  708 , the API response object is forwarded from the scheduler to an API. The API (e.g., API A  102  or API B  120  of  FIG. 1 ) that receives the API response object is the API that initially received the call from a user to perform a compute task and converted the user&#39;s call to a request message (e.g., RPC message) sent to a service associated with an FPGA. 
     At  710 , a user of the API is given a response. In various embodiments, the API gives the response to the user in the format the user is expecting. For example, if the user made a GIF resizing call, such as GIF output=resizeGIF(GIF_input, result_size), to the API, the API would need to return to the user GIF output (e.g., a new GIF image). The API can extract the data that needs to be returned from the API response object. In various embodiments, a result is returned to the API caller as though the computation had occurred in place locally (without the user needing to know of any RPC message sent to an FPGA-based server system). 
     Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.