Abstract:
A method and apparatus is provided wherein a central Credit Controller Entity (CCE) is connected to a PCIE fabric environment by means of several buses. Flow Control information sent to the CCE over two of the buses indicates the buffer storage capacity that is available at respective Receiver components in the PCIE fabric. The CCE processes the Flow Control information, to generate updates that are sent by a third bus to Transmitter components corresponding to the Receivers. In one useful embodiment, directed to a method of Flow Control management, the CCE provides a repository adapted to store credit count information that represents the available storage capacity of respective Receivers. The method further comprises routing further credit count information from a given Receiver to the CCE, for storage in the repository, following each of successive events that affect the storage capacity of the given Receiver. The CCE is operated to selectively process the credit count information stored in the repository, in order to generate an update credit count. The update credit count is then selectively sent to a given Transmitter, to enable the given Transmitter to send a transmission to the given Receiver.

Description:
BACKGROUND OF THE INVENTION 
   1. Technical Field 
   The invention disclosed and claimed herein generally pertains to a method for managing flow control updates in a PCI Express (PCIE) environment. More particularly, the invention pertains to a method of the above type wherein a Credit Control Entity (CCE) receives credit count information that represents available storage capacity of receivers in the PCIE environment. Even more particularly, the invention pertains to a method of the above type wherein updates, derived by the CCE from the credit count information, are used to enable transmissions to the receivers. 
   2. Description of the Related Art 
   In a PCIE fabric environment, packet traffic is directed to virtual channels (VCs) by mapping packets with traffic class labels to corresponding VCs. Moreover, PCIE provides the capability of mapping multiple traffic classes onto a single VC. This is achieved by arranging for traffic flowing through a VC to be multiplexed onto a common physical Link, from Transmitters on the transmit side of the Link. Subsequently, the traffic is de-multiplexed into separate VC paths and directed to corresponding Receivers, on the receive side of the Link. 
   Within a PCIE switch, each of the VCs requires dedicated physical resources, such as RAMS, buffers or queues, in order to provide buffering or storage capacity. This is necessary to support independent traffic flows inside the switch. Accordingly, a PCIE environment is provided with a Flow Control (FC) mechanism, in order to prevent overflow of Receiver storage buffers and also to enable compliance with ordering rules. The Flow Control mechanism is used by a Requestor, that is, a device originating a transaction in the PCIE domain, to track the buffer space available in a Receiver that is on the opposite side of a Link. Such tracking is carried out by means of a credit-based Flow Control procedure, designed to ensure that a packet is transmitted only when a buffer is known to be available to receive the packet at the other end. This eliminates any packet retries, as well as associated waste of bandwidth due to resource constraints. Each virtual channel maintains an independent Flow Control credit pool. Flow Control information is conveyed between two sides of a Link, by means of Data Layer Link packets (DLLP). 
   Flow Control is generally handled by the Transaction Layer, in cooperation with the Data Link Layer, with the Transaction Layer performing Flow Control accounting for received Transaction Layer packets (TLPs). The Transaction Layer gates a Transmitter, based on available credits for transmission, in order to allow the Transmitter to send a TLP to a specified Receiver. In support of this Transmitter gating function, an initialization procedure is required, wherein Receivers must initially advertise VC credit values that are equal to or greater than certain pre-specified values. The number of credits allocated to a Transmitter is initially set according to the buffer size and allocation policies of the Receiver. As a succession of TLP transmissions occur, a count is kept of the credits being consumed. Before transmitting a given TLP, the Transmitter gating function must determine if sufficient credits are available to permit transmission of the given TLP. If the intended Receiver does not have enough credits to receive the TLP, the Transmitter must block the transmission of the TLP, possibly stalling other TLPs that are using the same virtual channel. The Transmitter must follow prescribed ordering and deadlock avoidance rules, which require that certain types of TLPs must bypass other specific types of TLPs when the latter are blocked. 
   Additionally, the credit accounting procedure tracks the count of the total number of credits granted to a Transmitter since initialization. This count may be incremented, as the Receiver side Transaction Layer makes additional received buffer space available by processing received TLPs. It would be beneficial to provide a central control that continually receives all the credit count information pertaining to each Receiver in a PCIE fabric. The central control could process such information, to provide flow control management throughout the PCIE fabric. 
   SUMMARY OF THE INVENTION 
   The invention generally pertains to a method and apparatus wherein a central Credit Controller Entity (CCE) is connected to a PCIE fabric environment by means of several buses. Flow Control information sent to the CCE over two of the buses indicates the buffer storage capacity that is available at respective Receiver components in the PCIE fabric. The CCE processes the Flow Control information, to generate updates that are sent by a third bus to Transmitter components corresponding to the Receivers. In one useful embodiment, directed to a method of Flow Control management, the CCE provides a repository adapted to store credit count information that represents the available storage capacity of respective Receivers. Embodiments of the invention thus provide a centralized entity, to significantly enhance flexibility in managing Flow Control updates generated by a PCIE root complex or end point. It is anticipated that these embodiments will enable a user to selectively maximize throughput or RAM buffering output, or to minimize latency. The method further comprises routing further credit count information from a given Receiver to the CCE, for storage in the repository, following each of successive events that affect the storage capacity of the given Receiver. The CCE is operated to selectively process the credit count information stored in the repository, in order to generate an update credit count. The update credit count is then selectively sent to a given Transmitter, to enable the given Transmitter to send a transmission to the given Receiver. Embodiments of the invention thus provide a centralized entity, to significantly enhance flexibility in managing Flow Control updates generated by a PCIE root complex or end point. It is anticipated that these embodiments will tend to maximize throughput and RAM buffering output, and to minimize latency. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself however, as well as a preferred mode of use, further objects and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein: 
       FIG. 1  is a schematic diagram showing Transmitter and Receiver components in a PCI-Express fabric that are joined together by a PCIE Link for transferring packets in accordance with an embodiment of the invention. 
       FIG. 2  is a block diagram showing an embodiment of the invention wherein a Credit Controller Entity (CCE) is connected to a PCIE fabric environment by means of buses. 
       FIG. 3  is a block diagram showing a data processing system of a type that may be used in implementing the Transmitter of  FIG. 1 , the Receiver of  FIG. 1 , and/or the CCE of  FIG. 2 . 
       FIG. 4  is a flow chart showing respective steps for implementing the embodiment of  FIG. 2 . 
   

   DETAILED DESCRIPTION 
   A PCI-Express (PCIE) fabric is composed of point-to-point links that interconnect a set of components. As an illustration,  FIG. 1  shows a PCIE Link  102  between fabric components  104  and  106 . In its most general form, a PCIE Link represents a dual-simplex communication channel between two components, and consists of two low voltage differential signal pairs, a transmit pair and a receive pair. However, for purposes of illustration,  FIG. 1  shows component  104  designated to be a Transmitter component, and component  106  designated to be a Receiver component. It is to be understood that in some applications the roles of components  104  and  106  would be reversed.  FIG. 1  further shows Receiver  106  provided with a storage buffer  108 . While not shown, component  104  could have a similar storage buffer. 
   PCIE uses packets to communicate information between components. Packets are formed in the Transaction and Data Link Layers, to carry information from the transmitting component to the receiving component. In transmitting data from Transmitter  104  to Receiver  106  of  FIG. 1 , it will be readily appreciated that storage buffer  108  of Receiver  106  has an essential role. Thus, as discussed above, PCIE requires a Flow Control mechanism, in order to prevent overflow of buffer  108 . The Flow Control mechanism also allows information packets to be transferred in accordance with prescribed PCIE ordering rules. Flow Control is handled by the Transaction Layer, in cooperation with the Data Link Layer. 
   In accordance with PCIE Flow Control procedures, Receiver  106  must initially advertise a virtual channel (VC) credit count value. The credit count value is a measure of the storage capacity of buffer  108 , at any point in time, and the advertised credit count value can be no less than a specified minimum value, for a particular intended data transfer. More particularly, the advertised value cannot be less than a minimum value required by a PCIE standard or specification. One such standard is the PCI Express Base Specification, REV. 1.0a, hereinafter referred to as “PCIE Base Specification”. 
   Following initialization, as successive Transaction Layer packets (TLPs) are received at Receiver  106 , the storage capacity of buffer  108  is correspondingly reduced. Accordingly, for each received TLP, a Received Credit Count value  110  is provided by Receiver  106 . Usefully, this value is in the form of a packet that includes an 8-bit header credit count and a 12-bit data credit count. 
   As the storage space of Receiver  106  is being diminished by received TLP transmissions, it is simultaneously also being increased, as the Transaction Layer of Receiver  106  makes additional received buffer space available by processing previously received TLPs. These increases, comprising Freed Credit Count values, offset the loss of buffer storage capacity that is caused by the receiving of TLPs. Accordingly, each time a previously received TLP is processed to add further space to buffer  108 , Receiver  106  generates a Freed Credit Count value  112 . In like manner with Received Credit Counts  110 , each Freed Credit Count value  112  is usefully in the form of a packet that includes an 8-bit header credit count and a 12-bit data credit count. 
   In accordance with the invention, the available storage capacity in buffer  108 , at a given time, will be indicated by both Received Credit Count values  110  and Freed Credit Count values  112  at the given time. Thus, both Received and Freed Credit Count values are continually routed to a Credit Controller Entity (CCE), as described hereafter in connection with  FIG. 2 . As also described, the CCE serves as a Transmitter gating mechanism by generating Update Credit Counts  114 , that are successively routed to Transmitter  104 . Each Update  114  causes Transmitter  104  to send a TLP to Receiver  106 , through Link  102 . 
   Referring to  FIG. 2 , there is shown a Credit Controller Entity (CCE)  202  located proximate to PCIE Environment  204 . Environment  204  comprises a PCIE fabric that contains a large number of Links  102 , as well as components such as Transmitter  104  and Receiver  106 .  FIG. 2  shows CCE  202  coupled to the PCIE Environment  204  by means of several buses, including Credit Received Bus (CRB)  206 , Credit Freed Bus (CFB)  208  and Credit Update Bus (CUB)  210 . CCE  202  includes a Data Processing System  212 , a Credit Update Policy (CUP)  214  and Credit State Registers (CSR)  216 . Data Processing System  212  is connected to interact with other components of CCE  202 , and could comprise, for example, the data processing system described hereinafter in connection with  FIG. 3 . 
   The bus CRB  206  is configured to route Received Credit Counts  110  from Receiver  106 , as well as from other Receiver components contained in Environment  204 , to CCE  202 . Accordingly, CRB  206  comprises buses  206   a  and  206   b , for carrying the 8-bit header credit count (7:0) and the 12-bit data credit count (11:0), respectively, of successive Received Credit Count values  110 . Received Credit Count values are thus received from both Receiver  106  and other Receiver components of PCIE Environment  204 . In addition, CRB  206  further comprises buses  206   c - 206   e , for respectively routing three mutually exclusive Credit Received Event signals. These respectively comprise Posted_Credit_Received_Event, Non-Posted_Credit_Received_Event, and Completion_Credit_Received_Event signals. 
   When any of the Received Event signals is driven to a logic 1 value, the CC  202  interprets this to mean that the number of header credits appearing on the Header_Credit_Count bus  206   a , and the number of data credits appearing on the Data_Credit_Count bus  206   b , have been received. The CCE  202  will record this information in its Credit State Registers  216 . However, if no Received Event signal is driven to a logic 1 value, the values on the Header_and Data_Credit_Count buses  206   a  and  206   b  are ignored by CCE  202 . 
   Similar to CRB  206 , bus CFB  208  is configured to route the Freed Credit Count values  112  from Receiver  106  and other Receiver components of Environment  204  to CCE  202 . CFB  208  comprises buses  208   a  and  208   b , for carrying the 8-bit header count (7:0) and the 12-bit data credit count (11:0), respectively, of successive Freed Credit Count values  112 . Such Freed Credit Count values are received from both Receiver  106  and other Receiver components of PCIE Environment  204 . In addition, CFB  208  further comprises buses  208   c - 208   e , for respectively routing three mutually exclusive Credit Freed Event signals. These respectively comprise Posted_Credit_Freed_Event, Non-Posted_Credit_Freed_Event, and Completion_Credit_Freed_Event signals. 
   When any of the Freed Event signals is driven to a logic 1 value, CCE  202  interprets this to mean that the number of header credits appearing on the Header_Credit_Count bus  208   a , and the number of data credits appearing on the Data_Credit_Count bus  208   b , have been freed. The CCE  202  will record this information in its Credit State Registers  216 . If no Freed Event signal is driven to a logic 1 value, the values on the Header_ and Data_Credit_Count buses  208   a  and  208   b  are ignored by the CCE  202 . 
   CUB  210  likewise comprises an 8-bit Header Credit Count bus (7:0)  210   a  and a 12-bit Data Credit Count (11:0) bus  210   b . CUB  210  further comprises buses  210   c - 210   e , for respectively routing three mutually exclusive Credit Update Event signals. These are respectively Posted_Credit_Update_Event, Non-Posted_Credit_Update_Event, and Completion_Credit_Update_Event signals. 
   When CCE  202  drives any of the Update Event signals to a logic 1 value, this is to be interpreted by external components of PCIE Environment  204  as a directive to generate and transmit a PCIE Flow Control update DLLP. The CUB  210  is thus used to supply Updated Credit Counts  114 , described above, to Transmitter  104 . If no Update Event signal is driven to a logic 1 value, the values on the Header_ and the Data_Credit_Count buses  210   a  and  210   b  are to be ignored by Transmitter  104  and other external components of Environment  204 . 
   Referring further to  FIG. 2 , CSR  216  is shown to comprise an array of registers, flip-flops or other storage elements R 1 -R n . These storage elements collectively store credit count information for each of the three events, as described above, that are associated with each of the buses  206 ,  208  and  210 . For each such event, the last, or most recent, credit count is stored. Also for each event, the cumulative or running total of credit counts is stored. Thus, two items of information are stored for each event. Collectively, the registers of CSR  216  store the following items of information:
         Header/Data Credit Count for last CUB Completion_Credit_Update_Event   Header/Data Credit Count for last CUB Posted_Credit_Update_Event   Header/Data Credit Count for last CUB Non-Posted_Credit_Update_Event   Header/Data Credit Count for last CRB Completion_Credit_Received_Event   Header/Data Credit Count for last CRB Posted_Credit_Received_Event   Header/Data Credit Count for last CRB Non-Posted_Credit_Received_Event   Header/Data Credit Count for last CFB Completion_Credit_Freed_Event   Header/Data Credit Count for last CFB Posted_Credit_Freed_Event   Header/Data Credit Count for last CFB Posted_Credit_Freed_Event   Running total of Header/Data Credit Counts for all CUB Completion_Credit_Update_Event   Running total of Header/Data Credit Counts for all CUB Posted_Credit_Update_Event   Running total of Header/Data Credit Counts for all CUB Non-Posted_Credit_Update_Event   Running total of Header/Data Credit Counts for all CRB Completion_Credit_Received_Events   Running total of Header/Data Credit Counts for all CRB Posted_Credit_Received_Event   Running total of Header/Data Credit Counts for all CRB Non-Posted_Credit_Update_Event   Running total of Header/Data Credit Counts for all CFB Completion_Credit_Freed_Event   Running total of Header/Data Credit Counts for all CFB Posted_Credit_Freed_Event   Running total of Header/Data Credit Counts for all CFB Posted_Credit_Freed_Event       

   The CUP  214  is a component wherein an algorithm is implemented by means of a finite state machine, a microcontroller or the like, in order to control CUB  210 . CUP  214  is disposed to receive credit count information from CSR  216 , as well as from the buses CRB  206  and CFB  208 . Moreover, the algorithm is configured to ensure that CCE  202  operates in accordance with requirements of PCIE standards, such as the PCIE Base Specification. Thus, CUP  214  and other elements of CCE  202  interact to manage Flow Control, among the external components of Environment  204 , so that Flow Control is in compliance with the PCIE Base Specification. As an example, CUP  214  will receive Received Credit Count values  110  and Freed Credit Count values  112  from Receiver  106 . By selectively processing these values, CUP  214  can determine whether or not buffer  108  has enough storage space to allow transmission of a TLP from Transmitter  104 . If buffer  108  has sufficient storage capacity, an Update Credit Count  114  is sent to Transmitter  104  from CCE  202 , enabling the transmission to take place. 
   Referring to  FIG. 3 , there is shown a block diagram of a generalized data processing system  300  which may be used in implementing embodiments of the present invention. Data processing system  300  exemplifies a computer, in which code or instructions for implementing the processes of the present invention may be located. Data processing system  300  usefully employs a peripheral component interconnect (PCI) local bus architecture, although other bus architectures may alternatively be used.  FIG. 3  shows a processor  302  and main memory  304  connected to a PCI local bus  306  through a Host/PCI bridge  308 . PCI bridge  308  also may include an integrated memory controller and cache memory for processor  302 . 
   Referring further to  FIG. 3 , there is shown a local area network (LAN) adapter  312 , a small computer system interface (SCSI) host bus adapter  310 , and an expansion bus interface  314  respectively connected to PCI local bus  306  by direct component connection. Audio adapter  316 , a graphics adapter  318 , and audio/video adapter  322  are connected to PCI local bus  306  by means of add-in boards inserted into expansion slots. SCSI host bus adapter  310  provides a connection for hard disk drive  320 , and also for CD-ROM drive  324 . 
   An operating system runs on processor  302  and is used to coordinate and provide control of various components within data processing system  300  shown in  FIG. 3 . The operating system may be a commercially available operating system such as a WINDOWS XP operating system, which is available from MICROSOFT Corporation. Instructions for the operating system and for applications or programs are located on storage devices, such as hard disk drive  320 , and may be loaded into main memory  304  for execution by processor  302 . Main memory  304  and hard disk drive  320  each comprises a computer readable medium for such instructions. 
   Referring to  FIG. 4 , there is shown a flow chart depicting steps in the general operation of CCE  202 . Function block  402  indicates that PCIE Environment  204  signals via CRB  206  that credits for TLPs have been received. Similarly, function block  404  indicates that Environment  204  signals via CFB  208  that credits for received TLPs have been freed. Alternatively, Environment  204  signals that initial credit values have been advertised by a Receiver component. Function block  406  shows that when either of these events occurs, CSR  216  is updated. Thereupon, CUP  214  must decide whether or not to send an update by means of CUB  210 , as described above. This is indicated by decision block  408 . If the CUP decides not to send an update, CCE  202  goes to a mode of waiting for the next CSR update, as indicated by function block  410 . If the CUP decides to send an update, CCE  202  operates CUB  210  to send the update credits, as shown by function  212 . Thereafter, CCE  202  goes to a mode of waiting for the next CSR update. 
   The description of the preferred embodiment of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention the practical application to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.