Encapsulation of an application for virtualization

Embodiments relate to a computer system comprising a service layer controller. The computer system comprises a ring interface unit configured to provide access to a host system that enables access to a plurality of virtual machines (VMs). The computer system comprises a hardware application configured to be encapsulated by the service layer controller such that the hardware application communicates to the host system via interfaces controlled by the ring interface unit and service layer controller.

BACKGROUND

The present invention relates generally to computing technology, and more specifically to the encapsulation of an application.

Applications for a computing platform may be developed to provide for one or more functions. For example, in connection with a virtual environment, a hardware application typically includes special or dedicated elements controlling the virtual nature. While many vendors offer “system on a chip” solutions, none presently offer a virtualization solution “out of the box.”

BRIEF SUMMARY

An embodiment is directed to a computer system comprising a service layer controller. The computer system comprises a ring interface unit configured to provide access to a host system that enables access to a plurality of virtual machines (VMs). The computer system comprises a hardware application configured to be encapsulated by the service layer controller such that the hardware application communicates to the host system via interfaces controlled by the ring interface unit and service layer controller.

DETAILED DESCRIPTION

Embodiments described herein are directed to methods, apparatuses, and systems for providing a set of reusable design elements that enable an application to be connected to a plurality of interconnect fabrics. These design elements may control one or more of: (1) a resetting the application, (2) a delivery of software work elements that tell the application what to do, (3) a monitoring of the application for bus or bandwidth usage and duration, (4) a handling of errors occurring within the application so that the error is contained or isolated to the virtual manager running that job, (5) a managing of the communication of the application to external endpoints so that it appears virtual to a host, (6) a stopping of the application in case of error or runaway bus usage in such a way that it does not interfere with other traffic on an interconnect fabric and that the interconnect fabric can continue to run (bus quiescing) (7) a putting or placing of the application into a known state (reset) at the beginning of each virtual job, (8) a fencing of the application into an electrical state to enable in-situ partial reconfiguration of that application for technologies that support partial reconfiguration.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

In some embodiments, a hardware application may be virtualized. The virtualization may occur in accordance with the virtualization layer62described above with respect toFIG. 3. In some embodiments, a hardware encapsulation mechanism may be used within, e.g., a field-programmable gate array (FPGA) and/or an application specific integrated circuit (ASIC) to virtualize the hardware application via one or more interfaces. The interfaces may be controlled by a trusted hardware supervisor layer. This hardware layer may be controlled by a privileged code while the application may be controlled by a device driver running within a virtual machine (VM).

Referring toFIG. 4, a system400in accordance with one or more embodiments is shown. The system may include one or more “endpoints”, such as a network402, a memory404, and a host406. The network402may be used to access an Ethernet, Fiber channel or media controller (MAC). The memory404may include an external memory bank (e.g., DRAM, Flash, etc.). The host406may include a bridge interface to a host processor, typically via a PCIexpress network or embedded via PLB or Powerbus in the case of, e.g., a system-on-a-chip (SOC) ASIC design.

The endpoints may connect to a (one or more) topologies, such as ring fabric (switch)408athrough408c. The ring fabric408a-408cmay interconnect a plurality of ring interface units (RIUs)410athrough410d. The RIUs410a-410dmay be used to provide a consistent user interface back to the application and may include one or more of an address, data, and a set of control lines to control flow, data transfer size and direction, and error indicators.

A service layer controller (SLC)412may be used to connect into the ring fabric408. Such a connection may be used to provide access to the host406.

In some embodiments, a direct memory attachment (DMA) controller, denoted by reference characters414a,414b, and414cinFIG. 4, may be used as an interface between a user interface (UI) of the RIUs410a-410cand the application.

The application itself may be designed within an encapsulation layer416. Within the application or the encapsulation layer416there may exist defined interfaces to the DMA (or RIU), a defined service queue bus (SQB) which may deliver an architected set of command to the application, and a memory mapped input output (MMIO) bus that may be used for non-virtual diagnostic purposes. In the diagnostic mode, the application may become an extension of a service layer's memory space.

The RIUs410may provide an access medium to the fabric408.

The RIUs410may provide a set of commands, such as read request, write request, read response, and write response.

The RIUs410may provide a flow control mechanism to prevent deadlocks and use ring bandwidth efficiently.

The RIUs410may provide an address bus large enough to cover the memory space for the application's resources.

The RIUs410may provide a data bus large enough to satisfy bandwidth requirements. The data bus may interleave with the address bus.

The RIUs410may provide a tag bus. A tag may be used to couple requests with corresponding responses. The RIUs410may correlate requests and responses and maintain a state indicating whether there are outstanding requests pending responses.

The RIUs410may provide a virtual function identifier (VFID). The VFID is described further below.

As described above, in some embodiments a hardware application may be virtualized. In some embodiments, the application may be reset and/or restarted without disturbing the infrastructure (e.g., the fabric408) around the application which may be performing tasks not related to the function of the application. If the SLC412could be sure the application was in a state where there are no pending requests (effectively absent from the fabric), the SLC412may change the state of the application without affecting the neighboring infrastructure.

An RIU410may keep track of the number of requests and responses made and received from the application. This may be done using a “tag” design where each request may be tagged with a unique identifier (e.g., a unique number) and when the response is returned from an endpoint, the identifier may be returned. The RIU410may watch the tags to maintain knowledge of when there are no requests out to the endpoint pending.

When a signal (sl_unit_quiesce inFIG. 4) from the SLC412is asserted indicating that the presently working job has ended is sent to the RIU410, the RIU410may stop making new requests on the fabric408. The RIU410may then wait for all responses pending to complete, corresponding to a “request neutral state.” When this occurs, the RIU410may assert a signal (unit_sl_quiesce inFIG. 4) back to the SLC412(or looped into the next RIU, which will trigger this action there). When all RIUs410connected to the application have indicated they are in the request neutral state, the SLC412may be enabled to reset the application without adversely impacting the fabric408. Otherwise, such adverse impacts could include one or more of:

(1) beaconing, wherein a runaway (unreliable) application constantly imposes requests on the bus.

(2) unsolicited responses. An unsolicited response may be a response that is returned based on a request that is no longer valid. If the application is reset and a new job is started, there cannot be a request from a previous job still outstanding on the fabric or the new job might not function properly.

(3) a complete timeout to a response. After a timeout on the return of the unit_sl_quiesce (as determined by the SLC412), the SLC412can escalate the problem to a system level error (e.g., the host is not responding).

(4) loss of tags (performance degradation). By insuring that all tags are returned after each job, performance degradation between jobs may be avoided.

(5) illegal change of state, e.g., breaking the protocol of the ring fabric408. Without this feature, a sudden reset of the application could cause a system-wide crash.

In some embodiments, the SLC412may isolate (e.g., logically isolate) the application from the fabric408. In some embodiments, the isolation may extend to electrical isolation. Electrical isolation may be used in connection with (partial) reconfiguration operations, such as a dynamic reprogramming of an FPGA device. The recfg_fence signal inFIG. 4may be asserted by the SLC412to those RIUs410that are connected to the application. The RIUs410may set their user interface outputs to the electrical state required to enable the partial reconfiguration state required by the manufacturer for that device while logically maintaining the idle state with respect to the ring fabric408. The service layer may isolate its own buses (e.g., SQB, MMIO) accordingly.

In some embodiments, an RIU410may indicate to the SLC412how much data has moved through the RIU410to the fabric408during the course of the job. Such a feature may be exploited in instances where revenue is generated or received based on available or utilized bandwidth. The data movement may be reported back to the host application (VM), without being manipulated by software running on the host machine or the hardware application itself.

In some embodiments, an RIU410and/or a DMA414may report errors back to the SLC412. An example would include bad DMA responses due to incorrectly calculated read pointers. Although the application may have the capability to report such errors, additional checking performed by the RIU410or the DMA414may be used to increase reliability since the RIU410and the DMA414may function independent of the application. Isolation may be provided between errors generated by the application and errors generated by the fabric408. Errors generated within the application and known to be caused by the application can be confined to the present job running on the application and the SLC412may merely end it, perform the quiesce operation, and reset the application and start a new job; the error might only be reported back to the VM. The application might not interfere with any job other than the one that is currently running.

As shown inFIG. 4, the SLC412may provide to one or more RIUs410a VFID signal. The VFID signal may be set on behalf of the application. The VFID may be derived by the SLC412based on a queue a job originated in. The VFID may be a hashed unique identifier that represents the queue (VM or “LPAR”) associated with a work request the application is executing. For example, in a PCIexpress system environment, a BAR address that may be used to access the queue may serve to determine a virtual function number associated with the request. This function number may become the VFID. The SLC412may assert this identifier to one or more RIUs410performing data transfers for the application. The Rills410may associated every request (and subsequent response) moving through the fabric408with that VFID so that any error seen outside of the application can be reported to the appropriate host or VM.

When the VFID reaches a designated endpoint, a respective virtualization technique may be performed by the endpoint. For example, in PCIexpress, the VFID may be replaced with the “requesterID” field in the PCIe transaction layer protocol. The requesterID may be associated by the host during configuration to correspond to the work queue the request is associated with (opci:PCISIG:Single Root IO virtualization via PCI express). In the case of a memory interface, a memory controller may perform a previously defined address translation corresponding to this VFID when the queue was initialized. For example, the memory controller may use the VFID as an extension to upper address bits to ensure that a section of memory is dedicated to that virtual function and only accessible to that virtual function. Similarly, in a networking context, an IP address may be associated with the VFID.

The application might not know of the VFID, much less adjust or manipulate the VFID. As such, the application might not influence the virtualization process of the transaction. The application might not impact or influence the operation of any other VM except potentially the VM the job is running on.

Turning now toFIG. 5, an exemplary embodiment of the SLC412is shown. The SLC412may interface to an external host processor502. The host processor502may include a number ‘n’ of VMs: VM1, VM2, VM3, . . . VMn. The host processor502may connect to the device via a bus medium504, such as PCIe. The host processor502may include a physical function (PF)506. The PF506may have supervisory access to the device, potentially using a hypervisor or supervised kernel process. Access may be obtained to the bus medium504by the SLC412via a RIU410connected to a host endpoint508.

MMIO accesses made by VM processes on the host502to the SLC412via the bus medium504may be decoded by the device as a VFID, which may correspond one-to-one back to the VM. In some embodiments, upper address bits may be used as the VFID and/or hashing techniques may be used to generate the VFID. The host502may control what address each VM can use. The VMs may be confined to accessing addresses that the SLC412may uniquely decode into a VFID and likewise target a specific queue dedicated to that VM. On these queues, the VMs may create work requests to the application. In some embodiments, the application might not be exposed to this process, the VFID, or the workings of the queue.

In some embodiments, the queue entries may contain an application specific invariant (ASIV) section532that may include data intended to be read by the application, an application specific variant (ASV) section534that may include data sent back from the application, and a service management section536maintained by the SLC412.

The SLC412may maintain control over the application via one or more (e.g., two) internal buses. A first of the buses, bus management (mgmt)552, may be connected to an RIU410around the application that controls isolating the application from the ring fabric408. The bus mgmt552may be used in conjunction with starting and completing jobs. The signaling for the bus mgmt552was described above.

A second of the buses, the service queue bus (sqb)554may interface directly to the application. The sqb554may include one or more of the following:

(1) a job availability/job busy handshake facility. A job availability signal (jobavail inFIG. 4) may be asserted to alert the application that there is a job to be executed. A job busy signal (jobbusy inFIG. 4) may be returned from the application to acknowledge the request and may be deasserted or removed when the execution is complete.

(2) a plurality of address, data, and/or control signals. These signals may be used to read and write the ASIV532/ASV534data associated with a work request. A position the data is placed in the ASIV532/ASV534may be determined a-priori by the design of the application and published in its specification (e.g., programming API). The application might only see this copy of the data for the specific queue element the SLC412has selected. The application might not access the queue element, thereby preventing the application from interfering with or corrupting the queues. In some embodiments, parity may be used on the sqb554to provide further protection against corruption of the queue.

(3) a set of control signals to indicate whether the data being sent to the SLC412is intended for the ASIV532or special control operations. The following control operations may be defined: (a) progress: this is a message that the SLC412will transmit back up to the host502on behalf of the application that goes in the service management section536of the queue element, (b) attncode: this is a message that the SLC412will transmit back up to the host502on behalf of the application and includes a pre-established protocol for generating an interrupt to the VM controlling this queue, (c) return code: this is a designation indicating how the execution of this job concluded.

In some embodiments, the service layer may overwrite one or more of the aforementioned fields if it detects an error in the operation of the application with diagnostic information intended for the device driver of the VM.

In some embodiments, the sqb554signals might only be used when the SLC412has asserted jobavail; the SLC412may ignore the sqb554signals otherwise.

Once one or more work requests are placed in the queues, operations may proceed as follows, in reference toFIG. 4,FIG. 5and the method600ofFIG. 6.

In block601, a scheduler may pick the next entry from one of the queues and retrieve from the host502the content of the queue entry (service management536and ASIV532section). Retrieval may be done via the RIU410using the VFID associated with the selected queue so the request is forwarded at the host502to the VM owning the queue. The SLC412may maintain pointers for these queues.

In block602, the retrieved contents of the queue may be placed in local memory for access by the sqb bus554. The ASIV532may be read by the application, and the ASV534may be written, although the application might not have visibility into such features yet. The service layer may perform reliability checks on the contents of these structures before proceeding.

In block603, the SLC412may select a VFID signal to match the VFID of the selected queue entry.

In block604, the SLC412may assert the sqb554signal jobavail to the application (seeFIG. 7, circle 1) and a run time job-execution timer may be set. The amount of time allowed to this queue entries execution may be set by the PF506on behalf of this processor502during an initialization of an adapter. The application or VM might not be able to manipulate this. The SLC412may enable the remaining sqb554bus signals to the control of the application.

In block605, the application may assert the sqb554signal jobbusy (seeFIG. 7, circle 2) back to the SLC412indicating it has acknowledged the work request.

In block606, using the sqb bus554the application may directly read the ASIV532from the queue entry the SLC412previously set up.

In block607, and specific to the function contained with this queue entry, the application may issue necessary transactions to the endpoints via the DMA414/RIU410connections as these are now live-connected to the application. Every request may be automatically tagged with the correct VFID and the application may be virtualized.

In block608, the application may write to the ASV memory534set up by the SLC412and may issue progress and attentions. The service layer may forward these to a previously established location (e.g., per information in the service management section536of the queue, or the configuration of the queue) using, e.g., the VFID for this queue entry to target the appropriate VM. Any interrupts generated by attention commands may be associated with the VFID and directed to a processor servicing this VM.

The method600(or a block or portion thereof) may continue or repeat until either the application ends the job (normal termination) or the SLC412ends the job (abnormal termination). While the job is running, the SLC412may accumulate the data count signals from the RIUs410keeping track of the amount of bandwidth used by the application.

The application may continue to work provided the job-execution time (block604ofFIG. 6) has not expired and an error has not be asserted by the RIU410/DMA414or service layer itself. The application may indicate completion by writing a return code instruction into the sqb bus554and then deasserting the sqb554signal jobbusy.FIG. 7illustrates a timing diagram of event when the SLC412detects the deassertion of the jobbusy signal.

In circle 3 ofFIG. 7, the SLC412may deassert the sqb bus554signal jobavail and disconnect access to the sqb bus554.

In circle 4 ofFIG. 7, the SLC412may assert the sl_unit_quiesce signal to the RIUs410. The RIUs410might not transmit any requests on behalf of the application.

The SLC412may wait for acknowledgement from the RIUs410regarding the quiesce. The job-execution timer may be running. If the RIUs410do not quiesce before this timer expires, a checkstop of the adapter may occur.

When the SLC412samples unit_sl_quiesce asserted from the RIUs410(or daisy-chained form the last one in the chain), the SLC412may assert the app_reset signal (circle 5 ofFIG. 7). The assertion of the app_reset signal may reset the application and DMA414engines. The RIU410and the fabric408may remain undisturbed.

The SLC412may deassert the sl_unit_quiesce signal, wait for the handshake unit_sl_quiesce to deassert and then release the app_reset on the application.

The SLC412may transmit the ASV534up to the host (e.g., host502) as per the queue entry on behalf of the VFID of this queue.

The SLC412may set a duration field to indicate how long this job took to execute and may place that as well as the final data counts into the job management section or service management section536of the queue entry. The SLC412may transmit the job management section of the queue entry back to the host on half of the VFID of this queue. The host may see the return code updated in its local copy and process this queue entry. The SLC412may trigger an interrupt to the respective VM to indicate the completion, potentially based on a configuration setting for the queue or information passed down in the queue entry.

The SLC412may manipulate the queue pointers for this queue so that the next entry can be retrieved.

The SLC412schedule may then pick another job from this or another queue and being the process or algorithm again. While the previous job was running, a new job may be prefetched on the adapter.

If the job-execution timer expires, the same or similar action as a normal termination condition may be taken except the service layer might not wait for the sqb bus554jobbusy signal to deassert. The service layer may merely lower the jobavail signal (circle 6 ofFIG. 7) and proceed to the quiesce operation. Despite the abnormal termination, the next selected job may be issued to the application after the app_reset sequence. This may be acceptable or safe, as there might not be any outstanding host requests, the application may be in its reset state, and the fabric may be unaffected by the error.

In some embodiments, a special VM may be designated the PF506and may control the management of the device. The PF506may typically run under the control of a kernel or supervisory process running on a host (e.g., host502). The SLC412may detect access by the PF506in a manner similar to how the SLC412decodes the VFID of the VM. This VFID (which may be referred to as VFID=0 in some embodiments) might not access the device via queues but through direct MMIO, potentially using a windowing technique (e.g., using a static register to set which VFID is targeted by subsequent MMIOs). The PF506may access memory on the device that would be associated with the VM queues for purposes of diagnostics and initialization. A register space through this VFID may map to a simple register access bus (MMIO) to extend diagnostic operation into the application.

While some of the examples described herein relate to the use of one application, in some embodiments multiple applications may each run off a different queue element from the same or different VMs all interconnected into the same fabric. This may be done by parallelizing the runtime execution and bus management elements of the SLC.

Technical effects and benefits include a definition for designing and implementing a hardware application (e.g., hardware accelerator, specialized processing core) such that the hardware application can be used in any virtual environment without the hardware application having any special design elements controlling the virtual nature. In some embodiments, the application may be physically isolated from any hardware responsible for controlling the virtual functionality of the chip or device. Reuse may be encouraged by being able to fit a plurality of different kinds of applications within a reusable virtual framework. Targeted applications or functions include compression, decompression, encryption, sorting, database queries, numerical analytics (DSP), scientific computing, etc.

Embodiments may be used to enable functional verification activities to concentrate on a core of an application, potentially without having to consider the complexities of a virtual nature of a platform. A robust design environment may be provided to develop reliable hardware accelerators for use in complex virtual memory environments. Physical encapsulation may be provided to ensure that any programming errors associated with the execution of the application can only impact the VM running the application and not impact other VMs.