Abstract:
In one embodiment, a system includes a station circuit. The station circuit includes a data layer and a transport layer. The station circuit is capable of a source mode and a destination mode. The data layer of the station circuit in source mode disassembles a source packet into one or more source parcels and sends the one or more source parcels to the transport layer. The station circuit in destination mode receives the one or more destination parcels over a ring at its transport layer, reassembles the one or more destination parcels into a destination packet, and delivers the destination packet from the transport layer to the data layer. The transport layer of the station circuit in source mode transmits the one or more source parcels over the ring. The transport layer of the station circuit in destination mode receives the one or more destination parcels over the ring.

Description:
BACKGROUND 
       [0001]    Processors include buses to connect various on- and off-chip devices. These devices can include processors, input/output interfaces, or memories. Using the bus to connect all of the devices together, the processor can utilize all of the devices together. 
       SUMMARY 
       [0002]    In one embodiment, a system includes a station circuit. The station circuit includes a data layer and a transport layer. The station circuit is capable of a source mode and a destination mode. The data layer of the station circuit in source mode disassembles a source packet into one or more source parcels and sends the one or more source parcels to the transport layer. The station circuit in destination mode receives the one or more destination parcels over a ring at its transport layer, reassembles the one or more destination parcels into a destination packet, and delivers the destination packet from the transport layer to the data layer. The transport layer of the station circuit in source mode transmits the one or more source parcels over the ring. The transport layer of the station circuit in destination mode receives the one or more destination parcels over the ring. 
         [0003]    In another embodiment, the station circuit includes at least one of a data consumption block and a data generation block. The data consumption block may include at least one of a data compression engine and an encryption engine. The data generation block may include at least a data access block. 
         [0004]    In another embodiment, the source packet and destination packet may include at least one of a command, an address, a tag, a length, and a payload. The at least one source parcel and at least one destination parcel may include at least one of a source field, a destination field, a parcel payload, a thread identification number, and a virtual machine identification number. A particular source parcel may be part of the disassembled packet and the one or more source parcels may correspond with the one or more destination parcel, which may be reassembled into the destination packet. A source parcel and a destination parcel may or may not be identical. 
         [0005]    In one embodiment, the data layer may be synchronized to a first clock. The transport layer may be synchronized to a second clock. The first and second clock are not required to have either the same clock rate or any phase relationship. 
         [0006]    In another embodiment, the station circuit is one of a plurality of station circuits interconnected via the ring. Each station circuit may include a command credit count with respect to each of the other station circuits on the ring and a parcel credit count with respect to each of the other station circuits on the ring. A particular station circuit in source mode may send the source packet to a particular station circuit in destination mode if the command credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode is non-zero, and the parcel credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode is non-zero. The particular station circuit in source mode reduces its command credit count with respect to the particular station circuit in destination mode by one and reduces its parcel credit count with respect to the particular station in destination mode by the number of parcels sent upon sending the source packet to the particular station circuit in destination mode. 
         [0007]    In another embodiment, the ring includes a credit interconnect ring and a data interconnect ring. The data interconnect ring delivers the one or more source parcels from the particular station circuit in source mode to the particular station circuit in destination mode as one or more destination parcels. The credit interconnect ring returns a command credit to the particular station circuit in source mode. The returned command credit is generated by the particular station circuit in destination mode upon the particular station circuit in destination mode consuming all of the one or more destination parcels such that the particular station circuit in source mode adds a command credit to the command credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode. 
         [0008]    In another embodiment, the credit interconnect ring further returns a parcel credit to the particular station circuit in source mode. The returned parcel credit is generated by the particular station circuit in destination mode upon consuming a particular set of the one or more destination parcels such that the returned parcel credit represents one or more parcels being consumed. The particular station circuit in source mode adds a parcel credit to the parcel credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode. The returned parcel credits are sent for the particular set of the one or more destination parcels when a slot of the credit return ring becomes available. 
         [0009]    In another embodiment, the command credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode is set by a credit-source station circuit issuing command credits over the credit interconnect ring. The parcel credit count of the particular station circuit in source mode with respect to the particular station circuit in destination mode is set by the credit-source station circuit issuing parcel credits over the credit interconnect ring. The credit-source station circuit is one of the plurality of station circuits. The credit-source station circuit may issue command credits and parcel credits at startup. 
         [0010]    In another embodiment, an active station circuit can receive at least one of a command credit and a parcel credit from an inactive station circuit. The active station circuit and inactive station circuit may be one of the plurality of station circuits. 
         [0011]    In another embodiment, parcel credits and command credits are sent over the credit interconnect ring using the same format. Command credits may be at least one of a read command credit and a write command credit. The station circuit in destination mode may include a receiving buffer reserved for a receiving destination parcels from the plurality of station circuits on the ring. 
         [0012]    In another embodiment, the source and destination packet may be tagged with an identification number of a group. The group indicates a process identification number, a thread identification number, or a virtual machine identification number, such that a particular station circuit in destination mode can organize a set of destination packets tagged with different groups identification numbers to execute work within the destination packets upon receiving the destination packets. The particular station circuit may be issued credits based on an amount of processing power allocated to a virtual machine. The virtual machine may be indicated as a virtual machine assigned to work within the destination packets that the particular station circuit accepts. The particular station circuit may be configured to accept destination packets with work from a subset of identification numbers of groups. 
         [0013]    In one embodiment, a method in a station circuit including a data layer and a transport layer, wherein the station circuit is capable of a source mode and a destination mode, may include disassembling a source packet into one or more source parcels in the data layer of the station circuit in source mode. The method may further include sending the one or more source parcels to the transport layer of the station circuit in source mode. The method may further include transmitting the one or more parcels over a ring from the transport layer of the station circuit. The method may also include receiving one or more destination parcels over the ring at the transport layer of the station circuit in destination mode. The method may further include reassembling the one or more destination parcels into a destination packet at the station circuit in destination mode. The method may additionally include delivering the destination packet from the transport layer to the data layer at the station circuit in destination mode. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]    The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention. 
           [0015]      FIG. 1  is a block diagram illustrating a processor employing a ring bus of the present invention. 
           [0016]      FIG. 2  is a block diagram illustrating an example embodiment of the ring bus at the transport layer level. 
           [0017]      FIG. 3  is a block diagram illustrating an example embodiment of data media access layer packet to transport layer parcel conversion over the ring bus. 
           [0018]      FIG. 4  is a block diagram illustrating the credit interconnect ring. 
           [0019]      FIG. 5  is a block diagram illustrating an example embodiment of a source ring station converting packets from the data media access layer to the transport layer. 
           [0020]      FIG. 6  is a block diagram illustrating an example embodiment of the credit interconnect ring and the data interconnect ring interacting with a source station and the destination station. 
           [0021]      FIGS. 7A-7D  are block diagrams illustrating an example embodiment of the ring bus configured to employ credit-based dynamic bandwidth shaping along a data interconnect ring and a resource interconnect ring. 
       
    
    
     DETAILED DESCRIPTION 
       [0022]    A description of example embodiments of the invention follows. 
         [0023]    A variable bandwidth ring-based system bus (“bus”) allows a device, such as a system-on-a-chip (SOC) block, to exchange data in arbitrarily-sized payloads. Examples of the device includes, but is not limited to, a compression engine and encryption/decryption engines. Multiple devices may be connected to the bus, and a device may choose to send write packets to or read packets from any device, including itself. Any two communicating devices may be referred to as a “doublet.” Each doublet has a command exchange debit and a data exchange debit, using credit-based mechanisms. 
         [0024]      FIG. 1  is a block diagram illustrating a processor  102  employing a ring bus of the present invention. The processor  102  includes a plurality of ring stations  104 A- 104 H. 
         [0025]    Each of the ring stations  104 A- 104 H is operatively coupled with a device of the processor  102 . In addition, each ring station  104 A- 104 H is operatively coupled with another ring station  104 A- 104 H. In one embodiment, each ring station  104 A- 104 H is coupled with two other ring stations  104 A- 104 H to form the ring bus. For example, ring station  104 A is operatively coupled with ring station  104 B and ring station  104 H. As another example, ring station  104 B is coupled with ring station  104 A and ring station  104 C. In this manner, all of the ring stations  104 A- 104 H can communicate with any of the ring stations  104 A- 104 H through the ring bus. 
         [0026]    In one embodiment of the processor  102 , ring station  104 A is coupled with 64 compression control/arbitration units  110 . Ring station  104 A transmits data to be compressed or decompressed to the compression control/arbitration units  110 . After processing, the compression control/arbitration units  110  return the compressed or decompressed data. 
         [0027]    Ring station  104 B is coupled with an instruction queue management module  108 . Ring station  104 B sends instructions to the instruction queue management module  108 , which assigns instructions to processor cores. After the instruction has executed, the instruction queue management module  108  returns the result of the instruction. 
         [0028]    Ring stations  104 C- 104 F are each coupled with cryptography unit  106 A- 106 D respectively. Each cryptography unit  106 A-D is configured to encrypt or decrypt data received from ring stations  106 A-D. 
         [0029]    Ring station  104 G is coupled with a bus interface unit  118 . Ring station  104 G sends data to the bus interface unit  118  to communicate off the processor  102  via a Serial Electrically Erasable Programmable Read-Only Memory (EEPROM), a Two-Wire Serial Interface (TWSI), or a Phase-Locked Loop (PLL)/Reset interface. The ring station  104 G can also request a key from a key memory, or request a random number be generated. The ring station  104 G receives data from off the processor  102  via the Serial Electrically Erasable Programmable Read-Only Memory (EEPROM), the Two-Wire Serial Interface (TWSI), or the Phase-Locked Loop (PLL)/Reset interface. The ring station  104 G also receives a key from the key memory or a random number from the random number generator. 
         [0030]    Lastly, ring station  104 H is coupled with a PCIe unit  112 . The PCIe unit  112  is coupled with the bus interface unit  118 . The ring station  104 H sends data to the PCIe unit  112  to be transmitted off chip on a PCIe interface. Likewise, the ring station  104 H receives data from the PCIe unit  112  which receives the data at the processor  102 . 
         [0031]    A person of ordinary skill in the art can appreciate that the devices coupled to the ring stations described above can be adjusted and that any amount of ring stations and devices can be added to or removed from the processor  102 . Likewise, other components may be added to or removed from the processor  102 . A person of ordinary skill in the art can understand that the ring stations  104 A- 104 H are described to illustrate the ring bus. 
         [0032]    Ring stations  104 A- 104 H communicate with devices connected to them over a data media access layer (DMAL). Ring stations  104 A- 104 H communicate with other ring stations over a transport layer (TL). In this manner, ring stations  104 A- 104 H receive data from devices in the form of packets on the DMAL. The ring stations  104 A- 104 H converts the packets to parcels for the transport layer. 
         [0033]      FIG. 2  is a block diagram illustrating an example embodiment of the ring bus at the TL level. Ring station  0   206 , ring station  1   208 , and ring station N−1  210  are employed in the processor  102 . The ring bus connects ring station  0   206 , ring station  1   208 , and ring station N−1  210  with a data interconnect ring (DIR)  202  and a credit interconnect ring  204  (CIR). The DIR  202  carries data between ring stations  0   206 , ring station  1   208 , and station N−1  210 . Likewise the CIR  204  carries credits between the ring station  0   206 , ring station  1   208 , and ring station N−1  210 . 
         [0034]      FIG. 3  is a block diagram  300  illustrating an example embodiment of DMAL packet  310  to TL parcel  312  conversion over the ring bus. A device connected to source station  302  generates a DMAL packet  310 , at its DMAL  306 , to be sent to destination station  304 . Source station  302  and destination station  304  are a doublet (i.e., source station  302  is sending data to destination station  304 ). The DMAL packet  310  includes a command field  310 A, an address field  310 B, a length field  310 C and a packet payload field  310 D, and a tag field  310 E. The command field  310 A indicates whether the packet is a read packet or a write packet. The source station  302 , at the TL  308 , converts the DMAL packet  310  by performing a DMAL-to-TL conversion  314 . The DMAL-to-TL-conversion  314  converts the DMAL packet  310 , on the TL  308 , into multiple TL parcels  312  by dividing the packet payload  310 D of the DMAL packet  310  into multiple parcel payloads  312 C of the multiple TL parcels  312 . The multiple TL parcels  312  include a destination field  312 A, a source field  312 B, and the parcel payload field  312 C. The combined parcel payload fields  312 C of the multiple TL parcels  312  is the same as the packet payload field  310 D of the DMAL packet  310 . 
         [0035]    The source station  302 , at the TL  308 , transfers the TL parcels  312  over the DIR  202  along the ring bus until the TL parcels  312  reach the destination station  304 . The destination station  304 , at the TL  308 , performs a TL-to-DMAL conversion  316 . The TL-to-DMAL conversion  316  converts the TL parcels  312  to the DMAL packet  310  by combining the parcel payloads  312 C of all of the TL parcels  312  into the packet payload  310 D of the DMAL packet  310 . The destination station  304 , at the DMAL  306 , receives the DMAL packet  310 . Then, the device coupled with the destination station  304  receives the DMAL packet  310 . 
         [0036]    The device coupled to the source station  302  resides in the DMAL  306 . The DMAL  306  is oriented around transactions. The DMAL  306  views a transaction at the data packet level and pays for transactions with command credits. The source station  302  at the DMAL  306  exchanges command credits with the destination station  304  to send a data packet. The DMAL  306  does not exchange parcel credits. 
         [0037]    The DMAL  306  at each station is an interface between a device on the processor and the TL  308 . DMAL packets  310  have a size (sometimes referred to as “beats”), which is determined at implementation size, which allows the device on the processor to communicate with the TL  308 . A person of ordinary skill in the art can consider the size of the DMAL packets  310  to be a data bus width of the station. 
         [0038]    DMAL packets  310  transferred between a doublet, such as between source station  302  and a destination station  304 , can be “interleaved.” At the DMAL  306 , a doublet may have a particular number of transactions pending. For each doublet, a DMAL packet  310  pertaining to any active transaction may be transmitted at any time. In other words, DMAL packets  310  regarding different transactions within a doublet can be interleaved. Data in the DMAL packets  310  are bound to their particular transactions using the tag field  310 E. Tag-based data interleaving maximizes bandwidth utilization of the DMAL  306  and the TL  308 , where each particular transaction thread may be bursty, by spreading available bandwidth between transaction threads and ring stations. The bandwidth of a doublet is defined by the maximum number of simultaneous transactions that doublet can perform. This is initially set by a parameter that issues command credits to each doublet, optionally at system start-up. Optionally, tag field  310 E reutilization can ensure a destination DMAL  306  is ready to receive a write data packet because, for example, a previous command to the destination DMAL  306  has completed. For example, should a write packet be accepted by the destination DMAL  306 , a second write packet with the value in its tag field  310 E should also be accepted by the destination DMAL  306  because it ensures that the destination DMAL  306  is not receiving a command. 
         [0039]    The DMAL  306  is agnostic of the characteristic parameters of the TL  308 , and the TL  308  is agnostic of the characteristic parameters of the DMAL  306 . For example, the TL  308  is unaware of command credits, packet lengths and memory addresses of packets in the DMAL  306  and the DMAL  306  is unaware of destination or source station identifiers and credits of parcels in the TL  308 . Therefore, device details are isolated from the physical implementation of the ring bus. In addition, the physical segregation of the TL  308  and the DMAL  306  allows for a general purpose implementation of the DIR  202  and CIR  204 , independent of the types of devices (e.g., bus widths, clock frequencies) attached to ring stations via the DMAL  306  or the types of their requests (e.g., packet sizes, reads or writes, endianness). The ring stations translate two-way between the DMAL  306  and TL  308 . Data endianness is specified for each device on the DMAL  306  and converted to a device-independent format on the TL  308  (and therefore the DIR  202 ). 
         [0040]    The TL  308  includes the CIR  204  and the DIR  202 . The CIR  204  is separate from the DIR  202 , and allows ring stations to exchange credits. Parcel credits guarantee availability of data storage in the TL  308 . Command credits guarantee the availability of command (read/write) buffers in the DMAL. Parcel and command credit distribution shapes traffic for each doublet in the TL  308  and DMAL  306 . A destination ring station&#39;s TL  308  returns a parcel credit to a source ring station when a parcel from that destination ring station&#39;s TL  308  is consumed by its DMAL  306 . A destination ring station&#39;s DMAL  306  returns a command credit to a source ring station&#39;s DMAL  306  after the destination ring station processes the last data parcel in a request. 
         [0041]    The CIR  204  payloads are formed of a header containing the addresses of the source and destination stations and a credit payload. The credit payload contains a field for each type of credit (e.g., parcel, write command, read command). A destination ring station releases a credit payload to the CIR  204  if the credit payload contains at least one non-zero field and the CIR  204  has an empty slot to transport the credit parcel. When no empty slot is available on the CIR  204 , the ring station accumulates credit return counts. Accumulated credit counts can be aggregated into one credit parcel. The CIR  204  is also used after reset or at start-up to initialize a system&#39;s credit count matrix (the initial number of parcel, read, and write credits for every doublet). 
         [0042]      FIG. 4  is a block diagram  400  illustrating the CIR  204 . The CIR  204  exists in the TL. The source station  302  and destination station  304  are a doublet. The destination station  304  generates a credit packet  402 . The credit packet  402  includes a destination field  404 , a source field  406 , a parcel credit field  410 , a write credit field  412  and a read credit  414  field. The parcel credit field  410  stores parcel credits that indicate that the DMAL of the destination station  304  consumed a parcel from the TL. The write credit field  412  and read credit field store command credits that indicate the device coupled to the destination station  304  consumed a packet from the DMAL. One instance of the parcel credit field  410  can store one individual credit, multiple credits of the same type, or multiple credits of different types. For example, the parcel credit field  410  can indicate that the credit parcel is delivering one read credit, but it can also indicate that the credit parcel is delivering three read credits, two write credits, and seven parcel credits. 
         [0043]    The destination station transmits the credit packet  402  over the CIR  204  to the source station  302 . The credit packet  402  indicates that the destination ring station  304  has freed up room on its TL  308  if the credit packet  402  includes a parcel credit, or the destination ring station  304  freed up room on its DMAL  306  if the credit packet includes a command credit (e.g., a read or write credit). The source station  302  increments its credit count by the amount of credits indicated in the credit parcel corresponding to the doublet of the source station  302  and the destination station  304 . 
         [0044]    The source station  302  may send a parcel to the destination station  304  only if it has a non-zero credit count for the destination station  304 . Each ring station has a separate buffer to store parcels for each destination station. The separate buffers prevent blocking between doublets due to zero credit counts or unresponsiveness of the destination station  304 . Further, it maximizes bandwidth utilization by providing a straightforward backpressure mechanism from the destination station  304  to the source station  302 . When the source station  302  sends the parcels  312  to the destination station  304 , it decrements its credit count to that station by an amount corresponding to the number of the parcels  312  sent, and places the TL parcels  312  on the DIR  202 . When the TL parcels  312  reach destination station  304 , the destination station  304  stores the TL parcels  312  in a queue. When the destination station  304  consumes the TL parcels  312  from the queue, therefore incrementing its parcel storage capability on the DIR  202 , it returns credits, the amount of which correspond to the number of TL parcels  312  consumed through the CIR  204  to the source station  302 , which increments its credit count with respect to the doublet corresponding to the destination station  304 . 
         [0045]    In one embodiment of the ring bus, each doublet is initialized to an initial number of credits stored in the ring stations and then modified by the parcel and credit exchange mechanisms described above. Upon start-up, a ring station can issue credits along the CIR to assign the initial number of credits to each doublet. In reference to  FIG. 1 , a first example doublet can be ring station  104 C (as a source) and ring station  104 B (as a destination), and a second example doublet can be ring station  104 A (as a source) and ring station  104 H (as a destination). Ring station  104 G can issue credits along the CIR to assign the initial number of credits to each doublet. For example, at start-up, ring station  104 G can issue 10 read credits, 10 write credits, and 50 parcel credits to the first doublet. The ring station  104 G can issue 20 read credits, 20 write credits, and 100 parcel credits to the second doublet. By doing so, the second doublet has double the bandwidth of the first doublet. A person of ordinary skill in the art can recognize that doublets can be issued any combination of read credits, write credits, and parcel credits, and the numbers described above are for illustration. 
         [0046]    In another embodiment, ring station  104 G can receive credit counts to issue to each doublet from a device off the processor  102  via bus interface unit  118 . For example, the device off the processor  102  can instruct ring station  104 G to give more credits, and therefore bandwidth, to a particular doublet. 
         [0047]      FIG. 5  is a block diagram  500  illustrating an example embodiment of a source ring station converting packets from the DMAL  306  to the TL  308 . A command interface  504  and a data interface  506  are operatively coupled to transmit to a decoder and destination selector unit  502 . In one embodiment, the command interface  504  and data interface  506  interface with the device coupled with the source stations. The decoder and destination selector unit  502  is operatively coupled to transmit commands and data to thread managers  508 A- 508 C. Each thread manager  508 A- 508 C is configured to generate packets to a particular destination ring station, e.g. thread manager  508 A is configured to generate packets to a destination station “ 0 ,” thread manager  508 B is configured to generate packets to a destination station “i,” and thread manager  508 C is configured to generate packets to a destination station “j.” A person of ordinary skill in the art can recognize that the thread managers  508 A- 508 C can be any number of thread managers. In one embodiment, the thread managers  508 A- 508 C are of the same quantity as the number of ring stations on the ring. Next, the thread managers  508 A- 508 C transmit threads to context buffers  510 A- 510 C, respectively. Then, each context buffer  510 A- 510 C transmits context to arbiters  512 A to  512 C. The arbiters  512 A- 512 C select a particular packet to send to source-to-destination FIFO queues  514 A- 514 C, respectively. A person of ordinary skill in the art can also recognize that the context buffers  510 A- 510 C, arbiters  512 A- 512 C, and source-to-destination FIFO queues  514 A- 514 C can be of any quantity. In one embodiment, the context buffers  510 A- 510 C, arbiters  512 A- 512 C, and source-to-destination FIFO queues  514 A- 514 C are of the same quantity of the thread managers  508 A- 508 C or the number of ring stations on the ring. 
         [0048]    The source-to-destination FIFO queues  514 A- 514 C are connected to a multiplexer  516  within the transport layer  308 . Each of the source-to-destination FIFO queues  514 A- 514 C store packet data until the multiplexer  516  selects the source-to-destination FIFO queue  514 A- 514 C to pop one of the packets from the queue into the source TL  518 . The source to destination FIFO queues  514 A- 514 C pop the packet by converting it to multiple TL parcels, which are sent to a source TL  518 . The multiple parcels received at the source TL  518  are then transmitted over the data interconnect ring  202 . The parcels are forwarded on the DIR  202 , described further in  FIG. 6  at reference number  612 . Further, the source TL  518  receives parcels from the DIR  202 , described further in  FIG. 6  at reference number  612 . Further the source TL  518  transmits to the CIR (not shown), described further in  FIG. 6  reference number  604 A- 604 C. 
         [0049]    The DMAL  306  runs on a device clock  520 . The TL  308  runs on a ring clock  522 . The device clock  520  and the ring clock  522  can be asynchronous with respect to each other. In one embodiment, the device clock  520  and ring clock  522  are asynchronous by having unrelated phases and frequencies. One-clock ring stations and two-clock ring stations using the same bus clock frequency may be interconnected. Two clock ring stations traverse a clock boundary in both directions at the interface between TL  308  and DMAL  306  using the source-to-destination FIFO queues  514 A- 514 C, which act as synchronizing FIFOs. 
         [0050]      FIG. 6  is a block diagram  600  illustrating an example embodiment of the CIR  204  and the DIR  202  interacting with a source station  302  and the destination station  304 . As described in  FIG. 5 , source-to-destination FIFO queues  514 A- 514 C are coupled with a multiplexer  516 , which selects packets from the source-to-destination FIFO queues  514 A- 514 C to send to the source TL  518  as multiple parcels. Upon receiving the parcels from the multiplexer  516 , the source TL  518  decreases a source-to-destination credit count  604 A- 604 C corresponding with the source-to-destination FIFO queue  514 A- 514 C where the packet originated. For example, if the packet is from source-to-destination FIFO  514 B, the source TL  518  decreases the source-to-destination credit count  604 B. The credit count  604 B is reduced by one command credit, and by as many parcel credits as there are parcels popped from the source-to-destination FIFO queue  514 A- 514 C. The source-to-destination “ 0 ” FIFO  514 A corresponds with the source-to-destination “ 0 ” credit count  604 A. Likewise the source-to-destination “I” FIFO queue  514 B corresponds with source to destination “I” credit count  604 B. In addition the source-to-destination “J” FIFO queue  514 C corresponds with the source-to-destination “J” credit count  604 C. A person of ordinary skill in the art can recognize that each destination that the source-to-destination FIFO queue  514 A- 514 C and the source-to-destination credit count  604 A to  604 C correspond with a specific destination on the ring bus. In this particular example, the doublet between the source station  302  and destination station “i”  304  is illustrated. 
         [0051]    The source TL  518 , after decrementing the credit count appropriately, transfers the parcels to the destination ring station  304 . The destination station  304  receives the parcels at a destination transport layer  612 . Upon receipt of the parcels, destination transport layer  612  pushes the parcels into a destination receipt FIFO queue  614 . When the destination DMAL  616  is ready to receive the parcels from the TL, it signals the destination receipt FIFO queue  614  with a pop signal  620 . Upon receiving the pop signal  620 , the destination receipt FIFO  614  transmits data  618 , collected from the multiple parcels in the form of a packet, to the destination DMAL  616 . Data  618  can include at least one parcel. Upon receiving data  618 , the destination DMAL  616  transmits a credit return  621  to the destination credit unit  610 . The destination credit unit  610  transmits a return credit  607  over the CIR  204  to the source credit unit  608 . The source credit unit  608  then sends a credit add  606  to the appropriate source to destination credit count  604 A- 604 C. In this manner, upon receiving the data at the destination DMAL  616 , the source to destination credit count  604 B is restored. 
         [0052]      FIGS. 7A-7D  are block diagrams illustrating an example embodiment of the ring bus configured to employ credit-based dynamic bandwidth shaping along a DIR  202  and a resource interconnect ring (RIR)  702 .  FIGS. 7A-7D  each illustrate a stage of the example embodiment.  FIG. 7A  illustrates a first stage of the embodiment,  FIG. 7B  illustrates a second stage of the embodiment,  FIG. 7C  illustrates a third stage of the embodiment, and  FIG. 7D  illustrates a fourth stage of the embodiment. 
         [0053]      FIG. 7A  is a block diagram  700  illustrating the first stage of the example embodiment of the ring bus configured to employ credit-based dynamic bandwidth shaping along a DIR  202  and a resource interconnect ring (RIR)  702  by requesting credits to send parcels. Dynamic bandwidth shaping is a manner of sharing resources among active doublets to increase bandwidth of the active doublets. In one embodiment, a default ring bus credit mechanism is static. In other words, in this embodiment, each ring station has a maximum bandwidth based on the number of credits granted, regardless of whether the ring station uses the credits. The engines commonly used (e.g., compression, cryptography, etc.) are bursty. In other words, most of the time the engines create little-to-no traffic while computing, but for short time periods they read and/or write (potentially large) chunks of data in contiguous packets. Such bursty engines can benefit from access to high bandwidth over these short time periods. 
         [0054]    When an inactive doublet becomes active, it requests a portion of bandwidth from the other active doublets. An active doublet releases its bandwidth to the other active doublet(s) upon becoming inactive. The bandwidth requests and releases are made over the RIR  702 . 
         [0055]    An inactive doublet may share credits with any active doublet (i.e., a requesting doublet) via the CIR  204  to increase the active doublet&#39;s transient bandwidth. When an inactive doublet becomes active, it requests its shared credits be returned to it by the doublets with which it shared credits. When an active doublet ceases activity and becomes inactive, it automatically releases all shared credits it received from all other inactive doublets. 
         [0056]    A person of ordinary skill in the art can also appreciate that basing dynamic credit distribution on packet activity extends management of credits from the TL  308 , where credit-related decisions are made at the parcel level, to the DMAL  306 , which is packet-aware. 
         [0057]    Each ring station has a first counter indicating how many entries remain in the destination station&#39;s receive FIFO queue. The first counter is initialized to the size of the destination FIFO queue. Each ring station has a second counter indicating the number of credits it has to send to the destination station, which is initialized to the size of the destination FIFO. Each ring station has a first and second counter for every other ring station on the ring bus. A bit mask (not shown) indicates which stations are actively sending to the destination station. Each ring station may have a back-off counter (BOC) which indicates how many empty parcel slots each station should allow to pass before allocating its own parcel on the ring since the previous parcel it allocated on the ring. The b BOC is initialized to 0 for all stations. 
         [0058]    In an example embodiment, should a station Z  704  start sending parcels to station Y  710 , but see no empty slots on the DIR  202  (as shown in  712 A), station Z  704  uses the RIR  702  to “get on” the DIR  202  by setting other stations&#39; bit masks and BOCs (not shown). Station Z  704  signals its intention to send data to station Y  710  by sending notification to all stations along the RIR  702 . Station X  706 , as an example, receives the notification and updates its bit mask (not shown) with respect to station Y and increments its BOC. Station Y+1  708 , the station that receives the notification directly before Station Y  710 , terminates the notification while updating its bit mask (not shown) and BOC. 
         [0059]    A representation  712 A of station Z&#39;s  704  view of the DIR  202  with respect to sending parcels to station Y  710  shows no empty slots, where each slot is shown as full. On the other hand, representation  712 B of station X&#39;s  706  view of the DIR  202  shows several empty slots to send a parcel. All the empty slots visible to station X  706  (see representation  712 B) are used by station X  706  causing station Z  704  to have no empty slots available to it. 
         [0060]      FIG. 7B  is a block diagram  720  illustrating the second stage of the example of operation described above where, upon receiving station Z&#39;s  704  request to send data to station Y  710 , station X  706  knows (from its BOC&#39;s value) to back off from using all empty slots, thereby leaving empty slots for station Z  704 . This is illustrated by representations  722 A and  722 B, which are the same representations as  712 A and  712 B, respectively, but represented at a different point in time. Should station Z  704  send parcels to station Y  710  while no other station is transmitting, station Z  704  determines the first counter and second counter are both non-zero for each parcel it sends to station Y  710 . Station Z  704  decrements its first counter (e.g., N_Fifo_Y in  FIG. 7B ) with respect to station Y  710 , decrements its second counter (e.g., N_Cr_Y in  FIG. 7B ) with respect to station Y  710 , puts each parcel on the DIR  202  and puts a debit request on the RIR (not shown). The debit request on the RIR (not shown) decrements the first counter (e.g., N_Fifo_Y in  FIG. 7B ) in each station it encounters and sets station Z&#39;s  704  bit in each station&#39;s bit mask (not shown) and updates its BOC. Each station&#39;s bit mask (not shown) and BOC are set to leave empty slots on the DIR  202  such that station Z  704  can send to station Y  710 . 
         [0061]    The empty slots represent empty entries on DIR  202 , shown in representation  722 A, from the perspective of station Z  704  and shown in representation  722 B, from the perspective of station X  706 . 
         [0062]      FIG. 7C  is a block diagram  730  illustrating the third stage of the example embodiment described above. In this step, should station Y  710  pop a parcel received from station Z  704 , off its receive FIFO to be consumed by the DMAL  306 , station Y  710  returns a credit to station Z  704 . The credit packet increments all of the first counters (e.g., N_Fifo_Y as in  FIG. 7C ) on the way to station Z  704 . Station Z  704  increments its first counter (e.g., N_Fifo_Y as in  FIG. 7C ) and second counter (e.g., N_Cr_Y as in  FIG. 7C ) and station Z  704  forwards the credit back to station Y  710  to accomplish the above on the other side of the ring until Y terminates it. All stations maintain an exact or a conservative count of the number of entries left in the destination receive FIFO. The entry count is decremented early (i.e., as soon as the entry is created and before the parcel enters the FIFO) and incremented late (i.e., when the return credit is received back on the CIR after the corresponding FIFO entry is popped) to keep an exact or conservative count. 
         [0063]    Ring stations do not have to send a credit request and wait for another ring station to respond, which avoids unnecessary delays. Stations are aware of available resources because bandwidth shaping is operated at the packet level in the DMAL, removing unnecessary parcel-level traffic for the maintenance of “activity” information in each station by keeping that information at packet level. 
         [0064]      FIG. 7D  is a block diagram  740  illustrating the fourth stage of the example embodiment described above. In this step, station Z  704  indicates its intention to get off the RIR  702 . Station Z  704  signals, over the RIR  702 , it is done sending parcels to station Y  710 . Station X  706  receives station Z&#39;s notification, and updates its bit mask (not shown) and its BOC. All stations until Station Y+1  708  do the same, where station Y+1  708  additionally terminates Station Z&#39;s  704  notification. 
         [0065]    If station Z  704  wants to send to station Y  710 , it performs the above steps, however, knowing that station X  706  already communicates with station Y  710  (from the bit mask), it has to pay (e.g., subtract) two credits to send a parcel to station Y  710 , which returns two when it consumes the parcel. When its debit request goes through station X  706 , station X  706  updates its mask of active stations and BOC and also starts requiring/subtracting two credits to send to station Y  710 . 
         [0066]    This automatically splits the bandwidth between station X  706  and station Z  704  within a few clock cycles. If three stations are active, each parcel costs three credits. A person of ordinary skill in the art can recognize that the cost of borrowing credits is proportional to the number of active stations. The first counter decrements or increments. A ring station sends a termination request on the CIR to indicate it is momentarily done using the ring bus. The termination request resets the requesting station&#39;s active bit in all station&#39;s resource mask, causing the stations to require one less credit to send. Further, a 1-bit ring (i.e., RIR  702 ) used only for “getting on” and “off” the ring bus sets and resets activity bit masks and BOCs to avoid saturation of the ring and blocking of a particular station from “getting on.” 
         [0067]    In yet another embodiment, a centralized bandwidth distribution slotting controller is configured to dynamically assign bandwidth to doublets based on observed bandwidth utilization. In other words, instead of requesting doublets and sharing doublets requesting credits, the centralized bandwidth distribution slotting controller is configured to observe the bandwidth needs of all the doublets. Based on this observation, the centralized bandwidth distribution slotting controller is configured to dynamically assign bandwidth to each doublet. 
         [0068]    In one embodiment, the ring bus supports virtualization and the TL and DMAL are Virtual Machine (VM) aware. VM support allows per-VM doublet bandwidth control, command credit allocation, resource allocation, traffic segregation and programmable active VM count. Traffic on the ring can be classified as belonging to a number of VMs. Each VM enables hardware resources it accesses by setting one of multiple mask registers. The mask registers dynamically map a hardware resource to a set of VMs. 
         [0069]    Each DMAL transaction on the ring can be associated with a Virtual Machine/Function Tag identifier (VFID tag). The VFID tag is included in all phases of a DMAL transaction and restricts the visibility of the transaction to hardware resources belonging to that particular VM. The VFID tag isolates VMs by protecting a particular VM from accessing another VM&#39;s resources. Source-to-destination credits associated with a VFID tag are not generic hardware resources, but are bound to a specific VM. VM specific credits are consumed when the VFID tag of the transaction matches the VFID tag stored with the credit value. 
         [0070]    Ring credits can be divided among all the VMs to give more credits to a particular VM, therefore allotting the particular VM more bandwidth on the ring bus. Similarly, assigning fewer credits to a particular VM allots a lower share of the bandwidth to the particular VM. VM Quality of Service (“QoS”) controls the ratio of command/parcel credits among the different VMs that actively send transactions on the ring bus. 
         [0071]    While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.