Abstract:
A system and method for transmitting data packets from a memory hub to a memory controller is disclosed. The system includes an upstream reception port coupled to an upstream link. The upstream reception port receives the data packets from downstream memory hubs. The system further includes a bypass bus coupled to the upstream reception port. The bypass bus transports the data packets from the upstream reception port. The system further includes a temporary storage coupled to the upstream reception port and configured to receive the data packets from the upstream reception port. The system further includes a bypass multiplexer for selectively coupling an upstream transmission port to either one of a core logic circuit, the temporary storage, or the bypass bus. The system further includes a breakpoint logic circuit coupled to the bypass multiplexer and configured to switch the bypass multiplexer to selectively connect the upstream transmission port to either one of the core logic circuit, the bypass bus, or the temporary storage. The system further includes a local memory coupled to the core logic circuit and operable to receive and send the data packets to the core logic circuit.

Description:
TECHNICAL FIELD  
       [0001]     This invention relates to computer systems, and, more particularly, to a system and method for transmitting data packets in a computer system having a memory hub architecture.  
       BACKGROUND OF THE INVENTION  
       [0002]     Computer systems use memory devices, such as dynamic random access memory (“DRAM”) devices, to store data that are accessed by a processor. These memory devices are normally used as system memory in a computer system. In a typical computer system, the processor communicates with the system memory through a processor bus and a memory controller. The processor issues a memory request, which includes a memory command, such as a read command, and an address designating the location from which data or instructions are to be read. The memory controller uses the command and address to generate appropriate command signals as well as row and column addresses, which are applied to the system memory. In response to the commands and addresses, data are transferred between the system memory and the processor.  
         [0003]     Although the operating speed of memory devices has continuously increased, this increase in operating speed has not kept pace with increases in the operating speed of processors. Even slower has been the increase in operating speed of memory controllers coupling processors to memory devices. The relatively slow speed of memory controllers and memory devices limits the data bandwidth between the processor and the memory devices.  
         [0004]     In addition to the limited bandwidth between processors and memory devices, the performance of computer systems is also limited by latency problems that increase the time required to read data from system memory devices. More specifically, when a memory device read command is coupled to a system memory device, such as a synchronous DRAM (“SDRAM”) device, the read data are output from the SDRAM device only after a delay of several clock periods. Therefore, although SDRAM devices can synchronously output burst data at a high data rate, the delay in initially providing the data can significantly slow the operating speed of a computer system using such SDRAM devices.  
         [0005]     One approach to alleviating the memory latency problem is to use multiple memory devices coupled to the processor through a memory hub. In a memory hub architecture, a system controller or memory controller is coupled over a high speed link to several memory modules. Typically, the memory modules are coupled in a point-to-point or daisy chain architecture such that the memory modules are connected one to another in series. Thus, the memory controller is coupled to a first memory module over a first high speed link, with the first memory module connected to a second memory module through a second high speed link, and the second memory module coupled to a third memory module through a third high speed link, and so on in a daisy chain fashion.  
         [0006]     Each memory module includes a memory hub that is coupled to the corresponding high speed links and a number of memory devices on the module, with the memory hubs efficiently routing memory requests and memory responses between the controller and the memory devices over the high speed links. Computer systems employing this architecture can have a higher bandwidth because a processor can access one memory device while another memory device is responding to a prior memory access. For example, the processor can output write data to one of the memory devices in the system while another memory device in the system is preparing to provide read data to the processor. Moreover, this architecture also provides for easy expansion of the system memory without concern for degradation in signal quality as more memory modules are added, such as occurs in conventional multi drop bus architectures.  
         [0007]      FIG. 1  is a block diagram of a system memory  102  that includes memory modules  104   a  and  104   b . The memory module  104   a  is coupled to a system controller  108  through a downstream link  128  and an upstream link  136 . Each of the memory modules  104   a ,  104   b  includes a memory hub  112 , which includes a link interface  116 . In the memory module  104   a , the link interface  116  is connected to the system controller  108  by the links  128 ,  136 . The link interface  116  includes a downstream reception port  124  that receives downstream memory requests from the system controller  108  over the downstream link  128 , and includes an upstream transmission port  132  that provides upstream memory responses to the system controller over the upstream link  136   
         [0008]     The system controller  108  includes a downstream transmission port  140  coupled to the downstream link  128  to provide memory requests to the memory module  104   a , and also includes an upstream reception port  144  coupled to the upstream link  136  to receive memory responses from the memory module  104   a . The ports  124 ,  132 ,  140 ,  144  and other ports to be discussed below are designated “physical” interfaces or ports since these ports are in what is commonly termed the “physical layer” of a communications system. In this case, the physical layer corresponds to components providing the actual physical connection and communications between the system controller  108  and system memory  102  as will be understood by those skilled in the art.  
         [0009]     The nature of the reception ports  124 ,  144  and transmission ports  132 ,  140  will depend upon the characteristics of the links  128 ,  136 . For example, in the event the links  128 ,  136  are implemented using optical communications paths, the reception ports  124 ,  144  will convert optical signals received through the optical communications path into electrical signals and the transmission ports  140 ,  132  will convert electrical signals into optical signals that are then transmitted over the corresponding optical communications path.  
         [0010]     In operation, the reception port  124  captures the downstream memory requests and provides the captured memory request to local hub circuitry  148 , which includes control logic for processing the request and accessing the memory devices  156  over a bus system  152  to provide the corresponding data when the request packet is directed to the memory module  104   a . The reception port  124  also provides the captured downstream memory request to a downstream transmission port  160  on a bypass bus  180 . The downstream transmission port  160 , in turn, provides the memory request over the corresponding downstream link  128  to a downstream reception port  124  in the adjacent downstream memory module  104   b . The port  124  in module  104   b  operates in the same way as the corresponding port in the module  104   a , namely to capture the memory request and provide the request to the local hub circuitry  148  for processing and to provide the request to a downstream transmission port  160 . The port  160  in the module  104   b  then operates in the same way as the corresponding port in module  104   a  to provide the memory request over the corresponding downstream link  128  to the next downstream memory module (not shown in  FIG. 1 ).  
         [0011]     The memory hub  112  in the module  104   a  further includes an upstream reception port  164  that receives memory responses over the corresponding upstream link  136  from an upstream transmission port  132  in the adjacent module  104   b . An upstream transmission port  132 , in turn, provides the response over the upstream link  136  to the upstream physical reception port  144  in the system controller  108 . Each of the memory modules  112  includes a corresponding downstream reception port  124 , upstream transmission port  132 , downstream transmission port  160 , and upstream reception port  164 . Moreover, these ports  124 ,  132 ,  160 ,  164  in each module  104   b  operate in the same way as just described for the corresponding ports in the module  104   a.    
         [0012]     In addition to the memory responses from the downstream hubs, the local hub circuitry  148  also receives memory responses from a local memory  156 . The local memory  156  may be a DRAM type memory device or other suitable memory devices as will be appreciated by those skilled in the art. The local hub circuitry  148  provides the memory responses from the local memory  156  to the upstream transmission port  132  for transmission over the upstream link  136  to the upstream reception port  144  of the controller  108 . Thus, the local hub circuitry  148  must monitor and control transmission of memory responses to the system controller  108  from the downstream memory module  104   b  and from the local memory  156 . Since the hub circuitry  148  must monitor and control transmission of memory responses to the system controller  108  from the downstream memory module  104   b  and the local memory  156 , the hub circuitry  148  must determine the priority of transmission of the memory responses. The hub circuitry  148  also must efficiently switch the transmission of memory responses from one source to another source. The hub circuitry  148  also must switch transmission of memory responses from one source to another source at an appropriate time.  
         [0013]     The system controller  108  can control the timing of the memory responses inside the memory hubs  112 . However, if there are a large number of memory hubs  112  coupled to the system controller  108 , it becomes complicated for the system controller  108  to efficiently determine the priority of transmission of memory responses and to do the scheduling in all the memory hubs  112 . Also when the system controller  108  controls the scheduling of memory responses inside the memory hubs  112 , the bandwidth available for data transmission is reduced.  
         [0014]     Accordingly, there is a need for a system and method for efficiently determining the priority of transmission of the memory responses inside the memory hub  112 . There is a need for a system and method for efficiently switching transmission of the memory responses from one source to another source inside the memory hub  112 . There is a need for a system and method for efficiently switching transmission of the memory responses from one source to another source at an appropriate point.  
       SUMMARY OF THE INVENTION  
       [0015]     The present invention is directed to a system and method for transmitting data packets from a memory hub to a memory controller. In one embodiment, the system includes an upstream reception port coupled to an upstream link. The upstream reception port receives the data packets from downstream memory hubs. The system further includes a bypass bus coupled to the upstream reception port. The bypass bus transports the data packets from the upstream reception port. The system further includes a temporary storage coupled to the upstream reception port and configured to receive the data packets from the upstream reception port. The system further includes a bypass multiplexer for selectively coupling an upstream transmission port to either one of a core logic circuit, the temporary storage, or the bypass bus. The system further includes a breakpoint logic circuit coupled to the bypass multiplexer and configured to switch the bypass multiplexer to selectively connect the upstream transmission port to either one of the core logic circuit, the bypass bus, or the temporary storage. The system further includes a local memory coupled to the core logic circuit and operable to receive and send the data packets to the core logic circuit. The bypass bus transports data packets from the downstream hubs to the upstream link when the bypass multiplexer is switched to the bypass bus. The upstream temporary storage stores the data packets from the downstream hubs when the bypass multiplexer is switched to the core logic circuit. The core logic circuit transmits the data packets from the local memory when the bypass bus is switched to the core logic circuit. The data packets from the temporary storage are transported to the upstream link when the bypass multiplexer is switched to the temporary storage.  
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0016]      FIG. 1  is a block diagram illustrating an existing memory hubs system.  
         [0017]      FIG. 2  is a block diagram of a memory hub in accordance with one embodiment of the invention.  
         [0018]      FIG. 3  shows a clock signal and upstream data packets in accordance with one embodiment of the invention.  
         [0019]      FIG. 4  shows breakpoints in upstream data packets.  
         [0020]      FIG. 5  shows a memory hub in accordance with another embodiment of the invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0021]      FIG. 2  is a block diagram of a memory hub  200  in accordance with one embodiment of the invention. The memory hub  200  includes a core logic circuit  204  coupled to the local memory  156 . The core logic circuit  204  is also coupled to the downstream reception port  124  and the downstream transmission port  160 . The upstream reception port  124  is coupled to the system controller  108  (not shown in  FIG. 2 ) via the downstream link  128 . The downstream transmission port  160  is coupled to adjacent memory hubs (not shown in  FIG. 2 ) via the downstream link  128 .  
         [0022]     The downstream reception port  124  receives read and write requests from the system controller  108  (not shown in  FIG. 2 ) over the downstream link  128 . The core logic circuit  204  receives the read and write requests from the downstream reception port  124 . The core logic circuit  204  sends to the local memory  156  those read and write requests that are destined for the local memory  156 . Read and write requests that are destined for downstream hubs (not shown in  FIG. 2 ) are moved from the reception port  124  to the transmission port  160  on the downstream bypass bus.  
         [0023]     The memory hub  200  further includes the upstream transmission port  132  that is linked to the system controller  108  by the upstream link  136 . As will be discussed further, read and write responses from the core logic circuit  204  and the downstream hubs (not shown in  FIG. 2 ) are transmitted by the upstream transmission port  132  to the system controller  108  over the upstream link  136 . A read response includes read data from the local memory  156  and a write response indicates one or more write requests have been completed.  
         [0024]     The memory hub  200  further includes a bypass multiplexer  212  coupled to the core logic  204  and a temporary storage  216 . The bypass multiplexer  212  is also connected to the upstream reception port  164  via a bypass bus  220 . The bypass multiplexer  212  selectively couples either the core logic  204 , the bypass bus  220  or the temporary storage  216  to the upstream transmission port  132 .  
         [0025]     In operation, read and write responses from the downstream hubs are received by the upstream reception port  164  over the upstream link  136  and are passed on to the upstream transmission port  132  over the bypass bus  220  and through bypass multiplexer  212 . Read responses are received by the core logic  204  from the local memory  156  and are passed on to the upstream transmission port  132  through the bypass multiplexer  212 . Write responses are generated in the core logic  204  and are also passed on to the upstream transmission port  132  through the bypass multiplexer  212 . As will be discussed further, when the bypass multiplexer  212  couples the core logic  204  to the upstream transmission port  132 , the temporary storage  216  is used to temporarily store read and write responses from the downstream hubs. In the following description, write and read responses from the core logic  204 , the downstream hubs and the temporary storage  216  will be referred to simply as “data.” 
         [0026]     As described above, the upstream transmission port  132  transmits data, over the upstream link  136 , originating from one of several sources: (1) the local memory  156 ; (2) downstream hubs; and the temporary storage  216 . The multiplexer  212  selectively couples the upstream link  136 , through the transmission port  132 , to either the core logic  204 , the bypass bus  220  or the temporary storage  216 . The multiplexer  212  is switched so that data originating from either the core logic  204 , the bypass bus  220  or the temporary storage  216  are transmitted over the upstream link  136  to the system controller  108 . A breakpoint logic  208  coupled to the bypass multiplexer  212  provides the switching algorithm to the bypass multiplexer  212 . The switching algorithm locates switch points (also referred to as breakpoints) when a switch may occur. If the switching algorithm locates a breakpoint and it is determined that a switch should be made to another data source that has data available, the bypass multiplexer is switched so that the new data source is coupled to the upstream link  136  through the upstream transmission port  132 .  
         [0027]     In general, data is transferred among the memory hub  200 , the system controller  108  and downstream hubs in a fixed data packet format. A data packet includes a beginning and an end. The breakpoint logic  208  determines the beginning or end of a data packet, and a switch is made at the beginning or end of a data packet.  
         [0028]     In one embodiment, the core logic  204  operates at 400 MHz. The reception ports  124 ,  164 , and the transmission ports  132 ,  160  operate at 1.6 GHz. The upstream link  136  and the downstream link  128  operate at 6.4 GHz.  
         [0029]     The operating speed of these devices are selected due to design requirements. The upstream and downstream links are operated at very high speed (6.4 GHz) in order to provide a large bandwidth. However, the transmission ports  136 ,  160 , the reception ports  124 ,  164 , and the core logic  204  cannot be operated at such high speed using current technology. Thus, as data is transferred from the downstream link to the reception port, the transfer speed is reduced. As data is moved to the core logic, the speed is reduced further.  
         [0030]      FIG. 3  shows a clock signal, indicated as a 4X clock, where X=400 MHz, and data packets in accordance with one embodiment of the invention. The length of the data packets depends on the type of data being transferred. A write response data packet transfers limited amount of information, primarily containing an ID number and control bits indicating that it is a write response. A read response data packet includes the same information as the write response data packet, but in addition the read response data packet includes the read data being returned. Thus the response data packet is longer than the write response data packet.  
         [0031]     In  FIG. 3 , the clock being used is a 4X clock which transfers 64 bits (8 bytes) in each clock cycle. In the example of  FIG. 3 , the read response data packet includes 64 bytes of data. These 64 bytes take 8 clock cycles to transfer. The read response data packet also includes 4 header bytes and 4 Cycle Redundancy Code (CRC) bytes, which require 1 clock cycles to transfer. Thus, the read response data packet requires a total of 9 clock cycles to transfer. The write response includes 32 bytes of data (multiple write completes), 4 bytes of header and 4 bytes of CRC. As understood by those skilled in the art, the header bytes are control bytes, and the CRC bytes are used as standard error checking mechanism.  
         [0032]      FIG. 3  also shows an idle packet, which is four clock cycles long. The idle packet contains 4 header bytes and 28 no operation (NOP) bytes. The idle packet is sent on the upstream bus by the downstream hubs when the hubs do not have any data to send. The idle packet allows the breakpoint logic to switch when no data is being sent by the downstream hubs.  
         [0033]     In one embodiment, a data packet moves from the upstream reception port  164  to the upstream transmission port  132  in one 1.6 GHz clock period. However, the breakpoint logic  208 , which switches the bypass multiplexer  212 , requires three clock periods to complete the switch because of the time required to process a decode and drive logic to switch the bypass multiplexer  212 . Thus, the beginning of the data packet is located as it enters the memory hub  200 , and then switching is initiated three clock cycles prior to the breakpoint so that the bypass multiplexer  212  is switched in time as the data packet arrives.  
         [0034]      FIG. 4  shows valid breakpoints in data packets. The bypass multiplexer  212  is switched at valid breakpoints. A valid breakpoint exists between two read responses, between a read response and a write response, and between a write response and a read response.  
         [0035]     As described before, the determination that the bypass multiplexer  212  will be switched is made three clock cycles before the arrival of a data packet. By looking ahead three clock cycles before the data arrives, the switching process of the bypass multiplexer  212  can begin so that the switch coincides with the data arrival. The write response data packet in  FIG. 4  shows that a determination that the bypass multiplexer  212  will be switched is made three clock cycles before a breakpoint.  
         [0036]      FIG. 5  shows a memory hub  500  in accordance with another embodiment of the invention. The memory hub  500  includes the elements shown in  FIG. 2  and described before. In addition, the memory hub  500  includes two temporary storages: an upstream buffer  512 , and a bypass FIFO  516  coupled to the bypass multiplexer  212  and the bypass bus  220 . The bypass FIFO is a high speed buffer operating at 4X clock speed, where X=400 MHz. The upstream buffer is a normal speed buffer operating at 1X clock speed.  
         [0037]     When the bypass multiplexer  212  is switched to the core logic  204 , incoming data packets from the downstream hubs are first stored in the bypass FIFO  516 . Since the bypass FIFO  516  operates at high speed (4X clock speed), the bypass FIFO  516  can transfer data packets from its input to its output very quickly. Thus, if the core logic  204  completes sending data packet and the bypass multiplexer switches to the temporary storages, the data from the bypass FIFO  516  is available immediately.  
         [0038]     However, if the bypass multiplexer  212  remains switched to the core logic  204 , incoming data packets from the downstream hubs fill up the bypass FIFO  516 . When the bypass FIFO  516  is filled up, the upstream buffer  512  is used to store data packets. As will be understood by those skilled in the art, the bypass FIFO  516  is fast, but is expensive to implement. Thus a small bypass FIFO  516  is typically used. The upstream buffer  512  is slower, but is less expensive to implement. Thus, a large upstream buffer  516  is used.  
         [0039]     The memory hub  500  includes clock domain change circuits  520 ,  524 ,  508 . As noted before, since the downstream ports  124 ,  160  operate at different clock frequency than the core logic  204 , the downstream ports  124 ,  160  are not synchronous with the core logic  204 . Thus, data packets cannot be directly transferred between the core logic and the downstream ports  124 ,  160 . The clock domain change circuit  520  allows transfer of data packets from the downstream port  124  to the core logic  204 , and the clock domain change circuit  524  allow the transfer of data packets from the core logic  204  to the downstream port  160 . The core logic  204  is synchronous with the bypass multiplexer  212 , and the clock domain change circuit  508  allows the transfer of data packets from the core logic  204  to the bypass multiplexer  212 .  
         [0040]     In one embodiment, after power up, the breakpoint control logic  208  initially switches the bypass multiplexer  212  to the bypass bus  220 , thus connecting the bypass bus  220  to the upstream link  136 . The bypass bus  220  remains connected to the upstream link  136  until the core logic  204  has data to be sent and a breakpoint is available on the bypass bus  220 . If the core logic  204  has data available and a breakpoint is available, the bypass multiplexer  212  is switched to the core logic  212 .  
         [0041]     When the bypass multiplexer  212  is switched to the bypass bus  220 , data on the bypass bus  220  is sent to upstream link  136 . When the bypass multiplexer  212  is switched to the core logic  204 , data from the core logic  204  is sent to the upstream link  136 . While the bypass multiplexer  212  remains switched to the core logic  204 , incoming data on the bypass bus  220  is sent first to the bypass FIFO  516 . When the bypass FIFO  316  is filled up, data is next to the upstream buffer  512 .  
         [0042]     In one embodiment, the bypass multiplexer  212  remains switched to the core logic  204  until the core logic  204  is empty or if a higher priority requires a switch. A higher priority is determined if the temporary storages, i.e., the bypass FIFO  516  or the upstream buffer  512 , have available data. When the bypass multiplexer  212  is switched away from the core logic  204 , the multiplexer  212  is first switched to the bypass FIFO  516 . The data in the bypass FIFO  516  is sent upstream over the upstream link  136  until the bypass FIFO is exhausted. In general, after the bypass FIFO  516  is exhausted, the bypass multiplexer  212  is next switched to the upstream buffer  512 , which is then emptied.  
         [0043]     If the core logic  204  has data available, a switch can be made from the bypass FIFO  516  to the core logic  204  even though the bypass FIFO has not been exhausted. If a switch is made from the bypass FIFO  516  to the core logic  204 , the next switch is made back to the bypass FIFO  516  in order to send the upstream data in the order it was received. When the bypass FIFO  516  empties, data is next taken from the upstream buffer  512 . A switch to the core logic  204  can be made from the upstream buffer  512  even though the upstream buffer has not been exhausted. However, the next switch is made back to the upstream buffer  512  in order to send the upstream data in the order it was received.  
         [0044]     After the bypass FIFO  516  and the upstream buffer  512  are cleared, the multiplexer  212  is normally switched to the bypass buss  220 . If, however, the core logic  204  has available data, the multiplexer  212  is switched to the core logic  204 . As discussed before, while the bypass multiplexer  212  is switched to the core logic  204 , upstream data is first loaded into the bypass FIFO  516  and then into the upstream buffer  512 . When the bypass multiplexer  212  is switched to the temporary storages, the bypass FIFO  516  is emptied first and then the upstream buffer  512  is emptied next. After the bypass FIFO  516  is emptied, it is not loaded again until the upstream buffer  512  has been emptied.  
         [0045]     In the preceding description, certain details were set forth to provide a sufficient understanding of the present invention. One skilled in the art will appreciate, however, that the invention may be practiced without these particular details. Furthermore, one skilled in the art will appreciate that the example embodiments described above do not limit the scope of the present invention, and will also understand that various equivalent embodiments or combinations of the disclosed example embodiments are within the scope of the present invention. Illustrative examples set forth above are intended only to further illustrate certain details of the various embodiments, and should not be interpreted as limiting the scope of the present invention. Also, in the description above the operation of well known components has not been shown or described in detail to avoid unnecessarily obscuring the present invention. Finally, the invention is to be limited only by the appended claims, and is not limited to the described examples or embodiments of the invention.