Distributed bus arbiter for one-cycle channel selection using inter-channel ordering constraints in a disaggregated memory system

Embodiments using a distributed bus arbiter for one cycle channel selection with inter-channel ordering constraints. A distributed bus arbiter that orders one or more memory bus transactions originating from a plurality of master bus components to a plurality of shared remote slaves over shared serial channels attached to differing interconnect instances may be implemented.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates in general to computing systems, and more particularly to, various embodiments for serializing memory bus transactions.

Description of the Related Art

In today's interconnected and complex society, computers and computer-driven equipment are more commonplace. Processing devices, with the advent and further miniaturization of integrated circuits, have made it possible to be integrated into a wide variety of devices. Many computing systems include computer memory which may be accessed using memory bus protocols. Several memory bus standards have been developed to support different processor architectures, for example, QuickPath Interconnect (Intel®), Advanced Microcontroller Bus Architecture (ARM®), Coherent Accelerator Processor Interface (IBM®), and HyperTransport® (AMD®), etc.

SUMMARY OF THE INVENTION

Various embodiments for using a distributed bus arbiter for one cycle channel selection with inter-channel ordering constraints are provided. A distributed bus arbiter that orders one or more memory bus transactions originating from a plurality of master bus components to a plurality of shared remote slaves over shared serial channels attached to differing interconnect instances may be implemented.

DETAILED DESCRIPTION OF THE DRAWINGS

Modern day computing systems often use multiple processors for obtaining greater processing speeds. The processors often share a common command bus, and arbitrate for control of the bus using a variety of arbitration techniques. Processors requesting actions to be performed by other devices are typically referred to as masters, and memory devices, input/output (“I/O”) devices, and other processors function as slaves which communicate with the masters via the common command bus. A master/slave relationship functions such that a master initiates a request to a slave and the slave replies. Slaves may be designed to carry out a request silently without a reply. A typical bus transaction comprises a microprocessor as master, requesting data from a memory device as slave.

More specifically, non-disaggregated computing systems, such as, for example, a computer system used in cloud datacenters, the memory and processors (CPUs) are electrically interconnected via an high performance parallel bus. This allows to achieve the optimal and best performance. In non-disaggregated systems, memory bus transactions are initiated by bus masters. The non-disaggregated computing system's memory address space is statically partitioned across masters (e.g., each master is responsible for the transactions towards a disjoint partition of the address space). Memory bus transactions are received by bus slaves. Like masters, each slave is responsible of handling transactions towards a disjoint partition of the address space. Given the architecture of non-disaggregated computing, one slave always serves only one single master. One master can interact with multiple slaves, as illustrated in non-disaggregated computing system410ofFIG.4.

Disaggregated memory architectures break the electrically interconnected coupling and separate the CPUs from memory by bundling the CPUs and memory into physically different components, usually interconnected via high-speed serial links. This arrangement trades off performance for increased flexibility, better resource utilization, and ultimately for a reduced total cost of ownership (TCO) over the computing infrastructure. Requests are multiplexed over a switched network and associations between masters and the slaves are no longer statically partitioned, but may be dynamically reconfigured. For this reason, in disaggregated memory systems, one slave can serve concurrently more than one master, as illustrated in420ofFIG.4.

Moreover, in Parallel bus architectures (e.g., a shared channel that transmits data over several lines simultaneously), the parallel signal lines are grouped in logical channels. Each logical channel contains a group of signal lines that are not meaningful to handle independently. For example, all 64 signal lines that deliver a 64-bit memory address needs to be read as a unity. In addition, due to internal processor architecture, the memory address data is delivered from a different hardware unit than the one that delivers the actual data for that address. Thus, memory address data and actual data for that address may be delivered in different clock cycles. Parallel bus memory protocols have strict ordering constraints in the way input from different channels is delivered from masters to slaves. For example, in advanced extensible interface (“AXI”) protocols, 1) the order of a sequence of inputs (or protocol units “Pus”) on a “Write Address (WA)” channel (“{PU1(WA), PUn(WA)}) must match the order of the sequence of PUs on the “Write Data (WD)” channel: (“{PU1(WD), . . . , Pun(WD)}”). In this way, the slave knows that the nth input on the “Write Data” channel corresponds to the nth input of the “Write Address” channel. In addition to these ordering constraints, all channels may deliver data in parallel. In non-disaggregated memory bus architectures, given that there is no concurrency in accessing slaves, ordering constraints are guaranteed simply by ensuring that each master issues requests on the channels according to the expected order (see non-disaggregated computing system410ofFIG.4). In disaggregated systems, however, multiple masters can issue requests on the same slave concurrently (see420ofFIG.4). These requests are serialized over the network, and can reach the slave in an arbitrarily interleaved fashion if no component is in place to guarantee that the protocol ordering constraints are met.

Accordingly, the present invention provides for serializing memory bus transactions. That is, the present invention provides for a one-cycle bus arbiter logic that allows to serialize the channels of a parallel bus while satisfying inter-channel ordering constraints in a disaggregated memory system. In one aspect, the present invention provides, in a disaggregated memory system, a distributed bus arbiter for one cycle channel selection with inter-channel ordering constraints. The distributed bus arbiter may be implemented to order one or more memory bus transactions originating from a plurality of master bus components to a plurality of shared remote slaves over shared serial channels attached to differing interconnect instances. That is, the distributed bus arbiter is enabled to ensure required serialization of memory bus transactions (e.g., guaranteeing correct serialization) starting from a plurality of master bus components to a plurality of shared remote slaves over shared serial channels attached to differing interconnect instances. The distributed bus arbiter also guarantees that, at each time (“t”), the operations needed for selecting the appropriate input to deliver to a slave take at most one clock cycle.

In an additional aspect, a distributed bus arbiter guarantees the correct serialization of memory bus transactions started by many master bus components towards many shared remote slaves, over shared serial channels, that may be also attached to different interconnect instances. The distributed bus arbiter guarantees the required orderings by filtering the input mask of existing selection logic circuits (e.g., round-robin arbiters) according to inter-channel dependency constraints. The distributed bus arbiter uses bit-masks to express inter-channel dependency constraints. The distributed bus arbiter makes the correct selections to transmit over the shared serialized channel in one clock cycle. A first-in-first-out (FIFO) queue may be used to hold the dependency bitmasks in the correct order and to select the current input vector filtering bitmask. More specifically, the bitmask is a bit array where each bit is assigned to enable (value 1) or disable (value 0) that enables or disables the selectability of a specific input.

As used herein, an input channel may be an electrical line connected to the distributed bus arbiter (e.g., arbiter logic). The distributed bus arbiter handles two or more input channels. The electrical line can be either 1-bit wide or n-bit wide, depending on the bus protocol. With the goal of facilitating the present invention, 1-bit wide serial lines may be used herein; however the present invention applies to n-bit wide channels.

The input channel produces discrete input units which consist of a sequence of bits called protocol Units (PUs). The format and semantics of PUs can be different for each of the channels and are dependent on the bus protocol. In one aspect, the PUs are not dependent on the specific format.

An input request vector502(e.g., vector502,FIG.5) may be a bit mask whose length is equal to the number of input channels connected the distributed bus arbiter. Each bit corresponds to one of the input channels. Each bit of the input request vector502may be connected to a signal that indicates that the respective PU channel buffer has PUs available. Therefore, the ith bit of the vector indicates that the specific input channel buffer has a PU available that satisfies the distributed bus arbiter ordering constraints and thus is a candidate for selection.

A selection vector506may be a one-hot bit mask encoding (this encoding guarantees that in every clock cycle only one bit may have the value of one whereas the rest will have the value of zero) whose length is equal to the number of input channels connected to the arbiter. Each bit corresponds to one of the input channels. The selection vector506may be the output of the distributed bus arbiter. A single high bit in the vector corresponds to the input channel from which the next PU is to be taken to be serialized.

Selection components (e.g., selection logic) may be a set of hardware components (e.g., set of hardware logic) that takes as input an input request vector502and outputs a one-hot selection vector506. Existing arbiter designs (e.g., a round robin arbiter) can be used to implement the selection logic.

Every channel Cican have associated zero or one channel constraints. A channel constraint determines whether channel data can be forwarded or not regardless if channel data are available in a channel input buffer. For any given channel, the channel constraint may be applied or not applied based on what data have been previously forwarded from other channels, which impose the constraint to the current channel. The channel constraints ensure that memory transactions (which are assembled by PUs forwarded from different channels) have bytes of the memory transactions issued in a correct/right serial order (i.e. properly serialized). Depending on the channel data type, some channels have channel constraints whereas other channels do not have channel constraints. A channel constraint X(Ci) associates channel Ciwith another input channel Cj(e.g., Cj=X(Ci), the constrained channel).

A constraint producing channel may be any channel that is associated with a channel constraint. A constrained channel may be any channel that is subject to a constraint. (e.g., a channel Cjis constrained if there is at least one Cjso that X(Cj)=X(Ci).

A freely selectable channel may be any channel that is not a constrained channel. In one aspect, the present invention uses channel constraints to express the ordering dependencies among PUs produced by input channels of the same distributed bus arbiter.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

In the context of the present invention, and as one of skill in the art will appreciate, various components depicted inFIG.1may be located in a moving vehicle. For example, some of the processing and data storage capabilities associated with mechanisms of the illustrated embodiments may take place locally via local processing components, while the same components are connected via a network to remotely located, distributed computing data processing and storage components to accomplish various purposes of the present invention. Again, as will be appreciated by one of ordinary skill in the art, the present illustration is intended to convey only a subset of what may be an entire connected network of distributed computing components that accomplish various inventive aspects collectively.

Turning now to diagram400ofFIG.4, a non-disaggregated system410is illustrated with memory bus transactions (e.g., Read A, Write A, Write D, Read D, Write ACK “acknowledgement”) are initiated by bus masters (e.g., master1and/or master n) and are received by bus slaves (e.g., slave1, slave2, slave n-1, and/or slave n). One slave always serves only one single master. For example, slave1only servers master1. Slave2only servers master1. Slave n-1only servers master n, and slave n only servers master n. However, each master can interact with multiple slaves, as illustrated in410ofFIG.4.

Alternatively, a disaggregated system420is depicted and breaks the electrically interconnected coupling and separate the CPUs from memory by bundling the CPUs and memory into physically different components, usually interconnected via high-speed serial links. More specifically, the disaggregated system420includes a distributed bus arbiter430positioned on the master side (e.g., master1and master n) and also positioned on the slave side (e.g., slaves1,2, and n) separated only by network450.

In the disaggregated system420, the distributed bus arbiter430operates both on the master side (see distributed bus arbiter430A), before the network450, and on the disaggregated slaves side (see distributed bus arbiter430B), after the network450. On the master side (e.g., the master1and master2), the distributed bus arbiter430receives PUs from multiple masters (e.g., the master1and master2) and serializes the PUs over the shared network link(s) (e.g., network450) in a correct order. On the “disaggregated” slaves side, the distributed bus arbiter430receives serialized PUs from incoming network link(s) (e.g., network450) and delivers the serialized PUs to a parallel bus channel restoring the intended interconnect protocol ordering. Request are multiplexed over a switched network and associations between masters and the slaves are no longer statically partitioned, but may be reconfigured dynamically. For this reason, in disaggregated memory systems, one slave can serve concurrently more than one master such as, for example, slave1may serve both master1and master n, as illustrated in420ofFIG.4.

Turning now toFIG.5, diagram depicting a schematic of a distributed bus arbiter500and an example of a functional relationship between components of the distributed bus arbiter. In one aspect, each of the devices, components, modules, and/or functions described inFIGS.1-4may also apply to the devices, components, modules, and functions ofFIG.5. Also, one or more of the operations and steps ofFIGS.1-4may also be included in one or more operations or actions ofFIG.5.

In one aspect, the distributed bus arbiter500may include a grant register516(“grant reg”), which holds a grant mask (e.g., a one-hot bitmask where each bits represents one constrained channel). The only ‘1’ bit in the mask means that PUs produced by the corresponding channel are allowed to be selected by the selection logic504(e.g., round robin arbiter). The input request vector502(e.g., vector502,FIG.5) may be a bit mask whose length is equal to the number of input channels connected the distributed bus arbiter. Each bit corresponds to one of the input channels. Each bit of the input request vector502may be connected to a signal that indicates that the respective PU channel buffer has PUs available. Therefore, the ith bit of the vector indicates that the specific input channel buffer has a PU available that satisfies the distributed bus arbiter ordering constraints and thus is a candidate for selection. The selection vector506may be a one-hot bit mask encoding (this encoding guarantees that in every clock cycle only one bit may have the value of one whereas the rest will have the value of zero) whose length is equal to the number of input channels connected to the arbiter. Each bit corresponds to one of the input channels. The selection vector506may be the output of the distributed bus arbiter (e.g., distributed bus arbiter430ofFIG.4). A single high bit in the vector corresponds to the input channel from which the next PU is to be taken to be serialized. Selection logic504(e.g., selection logic) may be a set of hardware components (e.g., set of hardware logic) that takes as input the input request vector502and outputs the one-hot selection vector506.

The distributed bus arbiter500may include a grant FIFO512, which holds a queue of grant masks to be set in the grant register. The distributed bus arbiter500may include one or more channel constraint registers (e.g., channel constraint registers508A and508B), which holds, for each constraint producing channel, a one-hot bitmask. Each bit in the mask represents one constrained channel. A ‘1’ in the mask means that PUs produced by the corresponding channel are allowed to be selected by the selection logic504according to the constraint.

The distributed bus arbiter500may include a transaction status state machine510, which updates the grant register516(as illustrated in “update grant register” block514) and grant register FIFO512based on the PUs received over the arbitrated channels and based on knowledge of the target memory bus protocol.

To illustrate the functional operation of the distributed bus arbiter500, it may be assumed there are an n number of masters, each with two channels MiC1and MiC2. The 2*n input channels (e.g., MiC1and MiC2) may be arbitrated using the distributed bus arbiter500.

In the example, a master Miinitiates a transaction by producing a PU on MiC1. The transaction is completed when a PU is received on MiC2. In the example, the operations of distributed bus arbiter500may perform the following.

Given any two transactions produced by two Miand Mj(e.g., {PU(MiC1), PU(MiC2)} and {PU(MjC1), PU(MjC2)}). In any serialization produced by the distributed bus arbiter500, if PU(MiC1) appears before PU(MjC1), then PU(MiC2) will appear before PU(MjC2). Moreover, transaction order is preserved for any transaction (i.e., for any master Miand any serialization, PU(MiC1) will appear before PU(MiC2). For example, for the AXI bus protocol, assume C1is mapped on the “write address” channel and C2is mapped on the “write data” channel. The distributed bus arbiter500guarantees that the serialized order of “write data” PUs matches the serialized order of the “write address” PUs.

In one aspect, one or more constraints may be associated to all MiC1channels, so that X(MiC1) equals (“=”) MiC2for all “ith” masters. All MiC1channels may be constraint producing and freely selectable channels. All MiC2may be constrained channels. In one embodiment, one channel constraint register (e.g., channel constraint register508A and/or508b) may be used for each constraint producing channel. The (e.g., channel constraint register508A and/or508b) may hold a n-bits masks (one bit for each constrained channel). The bitmask of channel MiC1may be a one-hot mask where the only ‘1’ bit corresponds to the channel MiC2. When this mask is active, the only selectable constrained channel will be MiC2.

At time zero (“t=0”), the grant register516may be set to 0 (i.e., only freely selectable channels can be selected). These freely selectable channels may produce PUs that will start bus transactions. Whenever any channel is selected by the selection logic504, the data coming from the channel is examined by the transaction state machine510. The transaction state machine510may be aware of the specific memory bus protocol. The role of the transaction state machine510is to recognize the boundaries of PUs and trigger updates to the grant register516and grant register FIFO512, accordingly.

When a PU from a freely selectable channel is completely serialized, the transaction state machine510may push the content of the corresponding channel constraint register508and/or508B in the grant register FIFO512. If the grant register FIFO512is empty, the grant register FIFO512may update the content of the grant register516directly with the same value.

In the example, when a PU from MiC1is fully serialized, the corresponding grant mask that blocks all MjC1(i≠j) is put in the grant FIFO512. When this mask reaches the head of the grant FIFO512(this happens immediately if the FIFO is empty), all MjC2may be blocked from being selected. It should be noted that this mask does not block any other MjC1from being concurrently selected. This means that, while the channel constraint from MiC1is active, other transactions can be started from different masters (MjC1) and their PUs serialized. However, no other PU(MiC2) can be serialized, thus ensuring that the required serialization order is preserved. When a PU from the active constrained channel MiC2is eventually selected, then the grant register516is updated with next grant mask from the grant FIFO512. If the grant FIFO512is empty, the grant register516is reset to 0.

Turning now toFIG.6, a transaction status state machine diagram600for the transaction state machine510ofFIG.5is depicted. At steady state610, the transaction status state machine510keeps monitoring the data coming from input channels as it is serialized and, based on the bus protocol, the transaction status state machine510attempts to detect the boundaries of PUs being serialized. A detection of PU ends triggers state transition to “Process PU end” which indicates that the current Protocol transaction is complete and all PUs have been properly exchanged state614. The “Process PU end” state614is a pseudo-state and it transitions back to steady state610once the actions corresponding to the transition event are completed. The state transitions to an error state612upon the grant FIFO512becoming full and resets back to steady state610.

FIG.7is a flowchart diagram depicting an exemplary method700for process protocol unit (PU) end back to a steady state upon completion of actions corresponding to transition events unit, in which various aspects of the illustrated embodiments may be implemented. The functionality700may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable storage medium or one non-transitory machine-readable storage medium.

The functionality700may start in block702. A determination operation may be performed to determine a constraint producing channel, as in decision step704. If yes, the method700moves to decision step706to determine if a grant register is clear. If no, method700moves to decision step708to determine if a grant FIFO is full. If yes at decision step708, an error state occurs, as in block710. If no at decision step708, a channel constraint is pushed to the grant FIFO, as in block712.

Returning to decision step706, if yes at decision step706, a channel constraint is pushed to the grant register, as in block714. The method700moves to decision step716from block714. Returning to block704, if no at block704(and also from block714), a determination operation may be performed to determine if a channel is constrained, as in decision step716. If no, a steady state occurs, as in block718. If yes at decision step716, method700moves to decision step720to determine if a grant FIFO is empty. If yes at decision step720, the grant register is cleared, as in block724. The method700may move to block726and a steady state occurs. If no at decision step720, a bit mask from the grant FIFO may be pushed to the grant register, as in block722. The method700may move from block722to block726and a steady state occurs.

FIG.8is a flowchart diagram depicting an exemplary method800for using a distributed bus arbiter for one cycle channel selection with inter-channel ordering constraints, in which various aspects of the illustrated embodiments may be implemented. The functionality800may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable storage medium or one non-transitory machine-readable storage medium.

The functionality800may start in block802. One or more memory bus transactions from a plurality of master bus components by a distributed bus arbiter, as in block804. The distributed bus arbiter that orders the one or more memory bus transactions originating from the plurality of master bus components to a plurality of shared remote slaves over shared serial channels attached to differing interconnect instances may be implemented, as in block806. The one or more memory bus transactions may be transmitted to the plurality of shared remote slaves over the shared serial channels in a single clock cycle, as in block808. The functionality800may end in block810.

In one aspect, in conjunction with and/or as part of at least one block ofFIG.8, the operations of method800may include one or more of each of the following. The operations of method800may receive, by the distributed bus arbiter, the one or more memory bus transactions from one or more of plurality of master bus components and serializing the one or more memory bus transactions over the shared serial channels in a required order, and/or receive, by the distributed bus arbiter, serialized memory bus transactions from incoming networks.

An input mask of one or more existing arbiter components may be filtered for ensuring the required serialization of memory bus transactions. One or more channel constraints may be used to express ordering dependencies among each protocol unit, associated with the memory bus transactions, produced by one or more input channels of the distributed bus arbiter. One or more bit-masks may be used to express the inter-channel ordering constraints.

The operations of method800may use a FIFI queue to hold the one or more bit-masks in a required order and to select a current input vector filtering policy. The operations of method800may also transmit each of the memory bus transactions over the shared serial channels in one clock cycle.

Thus, as described herein, the present invention provides a memory access broker system that guarantees data integrity and deterministic program execution, while significantly improving application performance for memory write accesses. The memory access broker system may include a hardware memory write access request broker that facilitates fast EWACK and notification of failed write requests to issuing applications. The memory access broker system can be selectively enabled at runtime to serve applications that have the required support to exploit it safely, without affecting the execution integrity of concurrently active legacy applications. The memory access broker system comprises an operating system and application programming interface support for application level control, write integrity checks, and barriers for application-level handling of failed writes.

In this way, the present invention provides added feature and advantage over the current state of the art by effectively enabling, from a performance perspective, the use of remote memory in a variety of computing systems and architectures such as, for example, cloud and data-centric systems. More specifically, the memory access broker with application-controlled early write acknowledgment support may be implemented in and used with disaggregated memory (e.g., “extended memory”). Thus, without the memory access broker system, memory writes will continue to be of prohibitively high latency, thus diminishing the return and value of remote or extended memory.