System and method for transmitting data packets in a computer system having a memory hub architecture

A system and method for transmitting data packets from a memory hub to a memory controller is disclosed. The system includes an upstream reception port coupled to an upstream link. The upstream reception port receives the data packets from downstream memory hubs. The system further includes a bypass bus coupled to the upstream reception port. The bypass bus transports the data packets from the upstream reception port. The system further includes a temporary storage coupled to the upstream reception port and configured to receive the data packets from the upstream reception port. The system further includes a bypass multiplexer for selectively coupling an upstream transmission port to either one of a core logic circuit, the temporary storage, or the bypass bus. The system further includes a breakpoint logic circuit coupled to the bypass multiplexer and configured to switch the bypass multiplexer to selectively connect the upstream transmission port to either one of the core logic circuit, the bypass bus, or the temporary storage. The system further includes a local memory coupled to the core logic circuit and operable to receive and send the data packets to the core logic circuit.

TECHNICAL FIELD

This invention relates to computer systems, and, more particularly, to a system and method for transmitting data packets in a computer system having a memory hub architecture.

BACKGROUND OF THE INVENTION

Computer systems use memory devices, such as dynamic random access memory (“DRAM”) devices, to store data that are accessed by a processor. These memory devices are normally used as system memory in a computer system. In a typical computer system, the processor communicates with the system memory through a processor bus and a memory controller. The processor issues a memory request, which includes a memory command, such as a read command, and an address designating the location from which data or instructions are to be read. The memory controller uses the command and address to generate appropriate command signals as well as row and column addresses, which are applied to the system memory. In response to the commands and addresses, data are transferred between the system memory and the processor.

Although the operating speed of memory devices has continuously increased, this increase in operating speed has not kept pace with increases in the operating speed of processors. Even slower has been the increase in operating speed of memory controllers coupling processors to memory devices. The relatively slow speed of memory controllers and memory devices limits the data bandwidth between the processor and the memory devices.

In addition to the limited bandwidth between processors and memory devices, the performance of computer systems is also limited by latency problems that increase the time required to read data from system memory devices. More specifically, when a memory device read command is coupled to a system memory device, such as a synchronous DRAM (“SDRAM”) device, the read data are output from the SDRAM device only after a delay of several clock periods. Therefore, although SDRAM devices can synchronously output burst data at a high data rate, the delay in initially providing the data can significantly slow the operating speed of a computer system using such SDRAM devices.

One approach to alleviating the memory latency problem is to use multiple memory devices coupled to the processor through a memory hub. In a memory hub architecture, a system controller or memory controller is coupled over a high speed link to several memory modules. Typically, the memory modules are coupled in a point-to-point or daisy chain architecture such that the memory modules are connected one to another in series. Thus, the memory controller is coupled to a first memory module over a first high speed link, with the first memory module connected to a second memory module through a second high speed link, and the second memory module coupled to a third memory module through a third high speed link, and so on in a daisy chain fashion.

Each memory module includes a memory hub that is coupled to the corresponding high speed links and a number of memory devices on the module, with the memory hubs efficiently routing memory requests and memory responses between the controller and the memory devices over the high speed links. Computer systems employing this architecture can have a higher bandwidth because a processor can access one memory device while another memory device is responding to a prior memory access. For example, the processor can output write data to one of the memory devices in the system while another memory device in the system is preparing to provide read data to the processor. Moreover, this architecture also provides for easy expansion of the system memory without concern for degradation in signal quality as more memory modules are added, such as occurs in conventional multi drop bus architectures.

FIG. 1is a block diagram of a system memory102that includes memory modules104aand104b. The memory module104ais coupled to a system controller108through a downstream link128and an upstream link136. Each of the memory modules104a,104bincludes a memory hub112, which includes a link interface116. In the memory module104a, the link interface116is connected to the system controller108by the links128,136. The link interface116includes a downstream reception port124that receives downstream memory requests from the system controller108over the downstream link128, and includes an upstream transmission port132that provides upstream memory responses to the system controller over the upstream link136

The system controller108includes a downstream transmission port140coupled to the downstream link128to provide memory requests to the memory module104a, and also includes an upstream reception port144coupled to the upstream link136to receive memory responses from the memory module104a. The ports124,132,140,144and other ports to be discussed below are designated “physical” interfaces or ports since these ports are in what is commonly termed the “physical layer” of a communications system. In this case, the physical layer corresponds to components providing the actual physical connection and communications between the system controller108and system memory102as will be understood by those skilled in the art.

The nature of the reception ports124,144and transmission ports132,140will depend upon the characteristics of the links128,136. For example, in the event the links128,136are implemented using optical communications paths, the reception ports124,144will convert optical signals received through the optical communications path into electrical signals and the transmission ports140,132will convert electrical signals into optical signals that are then transmitted over the corresponding optical communications path.

In operation, the reception port124captures the downstream memory requests and provides the captured memory request to local hub circuitry148, which includes control logic for processing the request and accessing the memory devices156over a bus system152to provide the corresponding data when the request packet is directed to the memory module104a. The reception port124also provides the captured downstream memory request to a downstream transmission port160on a bypass bus180. The downstream transmission port160, in turn, provides the memory request over the corresponding downstream link128to a downstream reception port124in the adjacent downstream memory module104b. The port124in module104boperates in the same way as the corresponding port in the module104a, namely to capture the memory request and provide the request to the local hub circuitry148for processing and to provide the request to a downstream transmission port160. The port160in the module104bthen operates in the same way as the corresponding port in module104ato provide the memory request over the corresponding downstream link128to the next downstream memory module (not shown inFIG. 1).

The memory hub112in the module104afurther includes an upstream reception port164that receives memory responses over the corresponding upstream link136from an upstream transmission port132in the adjacent module104b. An upstream transmission port132, in turn, provides the response over the upstream link136to the upstream physical reception port144in the system controller108. Each of the memory modules112includes a corresponding downstream reception port124, upstream transmission port132, downstream transmission port160, and upstream reception port164. Moreover, these ports124,132,160,164in each module104boperate in the same way as just described for the corresponding ports in the module104a.

In addition to the memory responses from the downstream hubs, the local hub circuitry148also receives memory responses from a local memory156. The local memory156may be a DRAM type memory device or other suitable memory devices as will be appreciated by those skilled in the art. The local hub circuitry148provides the memory responses from the local memory156to the upstream transmission port132for transmission over the upstream link136to the upstream reception port144of the controller108. Thus, the local hub circuitry148must monitor and control transmission of memory responses to the system controller108from the downstream memory module104band from the local memory156. Since the hub circuitry148must monitor and control transmission of memory responses to the system controller108from the downstream memory module104band the local memory156, the hub circuitry148must determine the priority of transmission of the memory responses. The hub circuitry148also must efficiently switch the transmission of memory responses from one source to another source. The hub circuitry148also must switch transmission of memory responses from one source to another source at an appropriate time.

The system controller108can control the timing of the memory responses inside the memory hubs112. However, if there are a large number of memory hubs112coupled to the system controller108, it becomes complicated for the system controller108to efficiently determine the priority of transmission of memory responses and to do the scheduling in all the memory hubs112. Also when the system controller108controls the scheduling of memory responses inside the memory hubs112, the bandwidth available for data transmission is reduced.

Accordingly, there is a need for a system and method for efficiently determining the priority of transmission of the memory responses inside the memory hub112. There is a need for a system and method for efficiently switching transmission of the memory responses from one source to another source inside the memory hub112. There is a need for a system and method for efficiently switching transmission of the memory responses from one source to another source at an appropriate point.

SUMMARY OF THE INVENTION

The present invention is directed to a system and method for transmitting data packets from a memory hub to a memory controller. In one embodiment, the system includes an upstream reception port coupled to an upstream link. The upstream reception port receives the data packets from downstream memory hubs. The system further includes a bypass bus coupled to the upstream reception port. The bypass bus transports the data packets from the upstream reception port. The system further includes a temporary storage coupled to the upstream reception port and configured to receive the data packets from the upstream reception port. The system further includes a bypass multiplexer for selectively coupling an upstream transmission port to either one of a core logic circuit, the temporary storage, or the bypass bus. The system further includes a breakpoint logic circuit coupled to the bypass multiplexer and configured to switch the bypass multiplexer to selectively connect the upstream transmission port to either one of the core logic circuit, the bypass bus, or the temporary storage. The system further includes a local memory coupled to the core logic circuit and operable to receive and send the data packets to the core logic circuit. The bypass bus transports data packets from the downstream hubs to the upstream link when the bypass multiplexer is switched to the bypass bus. The upstream temporary storage stores the data packets from the downstream hubs when the bypass multiplexer is switched to the core logic circuit. The core logic circuit transmits the data packets from the local memory when the bypass bus is switched to the core logic circuit. The data packets from the temporary storage are transported to the upstream link when the bypass multiplexer is switched to the temporary storage.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 2is a block diagram of a memory hub200in accordance with one embodiment of the invention. The memory hub200includes a core logic circuit204coupled to the local memory156. The core logic circuit204is also coupled to the downstream reception port124and the downstream transmission port160. The downstream reception port124is coupled to the system controller108(not shown inFIG. 2) via the downstream link128.

The downstream reception port124receives read and write requests from the system controller108(not shown inFIG. 2) over the downstream link128. The core logic circuit204receives the read and write requests from the downstream reception port124. The core logic circuit204sends to the local memory156those read and write requests that are destined for the local memory156. Read and write requests that are destined for downstream hubs (not shown inFIG. 2) are moved from the reception port124to the transmission port160on the downstream bypass bus.

The memory hub200further includes the upstream transmission port132that is linked to the system controller108by the upstream link136. As will be discussed further, read and write responses from the core logic circuit204and the downstream hubs (not shown inFIG. 2) are transmitted by the upstream transmission port132to the system controller108over the upstream link136. A read response includes read data from the local memory156and a write response indicates one or more write requests have been completed.

The memory hub200further includes a bypass multiplexer212coupled to the core logic204and a temporary storage216. The bypass multiplexer212is also connected to the upstream reception port164via a bypass bus220. The bypass multiplexer212selectively couples either the core logic204, the bypass bus220or the temporary storage216to the upstream transmission port132.

In operation, read and write responses from the downstream hubs are received by the upstream reception port164over the upstream link136and are passed on to the upstream transmission port132over the bypass bus220and through bypass multiplexer212. Read responses are received by the core logic204from the local memory156and are passed on to the upstream transmission port132through the bypass multiplexer212. Write responses are generated in the core logic204and are also passed on to the upstream transmission port132through the bypass multiplexer212. As will be discussed further, when the bypass multiplexer212couples the core logic204to the upstream transmission port132, the temporary storage216is used to temporarily store read and write responses from the downstream hubs. In the following description, write and read responses from the core logic204, the downstream hubs and the temporary storage216will be referred to simply as “data.”

As described above, the upstream transmission port132transmits data, over the upstream link136, originating from one of several sources: (1) the local memory156; (2) downstream hubs; and the temporary storage216. The multiplexer212selectively couples the upstream link136, through the transmission port132, to either the core logic204, the bypass bus220or the temporary storage216. The multiplexer212is switched so that data originating from either the core logic204, the bypass bus220or the temporary storage216are transmitted over the upstream link136to the system controller108. A breakpoint logic208coupled to the bypass multiplexer212provides the switching algorithm to the bypass multiplexer212. The switching algorithm locates switch points (also referred to as breakpoints) when a switch may occur. If the switching algorithm locates a breakpoint and it is determined that a switch should be made to another data source that has data available, the bypass multiplexer is switched so that the new data source is coupled to the upstream link136through the upstream transmission port132.

In general, data is transferred among the memory hub200, the system controller108and downstream hubs in a fixed data packet format. A data packet includes a beginning and an end. The breakpoint logic208determines the beginning or end of a data packet, and a switch is made at the beginning or end of a data packet.

In one embodiment, the core logic204operates at 400 MHz. The reception ports124,164, and the transmission ports132,160operate at 1.6 GHz. The upstream link136and the downstream link128operate at 6.4 GHz.

The operating speed of these devices are selected due to design requirements. The upstream and downstream links are operated at very high speed (6.4 GHz) in order to provide a large bandwidth. However, the transmission ports136,160, the reception ports124,164, and the core logic204cannot be operated at such high speed using current technology. Thus, as data is transferred from the downstream link to the reception port, the transfer speed is reduced. As data is moved to the core logic, the speed is reduced further.

FIG. 3shows a clock signal, indicated as a 4X clock, where X=400 MHz, and data packets in accordance with one embodiment of the invention. The length of the data packets depends on the type of data being transferred. A write response data packet transfers limited amount of information, primarily containing an ID number and control bits indicating that it is a write response. A read response data packet includes the same information as the write response data packet, but in addition the read response data packet includes the read data being returned. Thus the response data packet is longer than the write response data packet.

InFIG. 3, the clock being used is a 4X clock which transfers 64 bits (8 bytes) in each clock cycle. In the example ofFIG. 3, the read response data packet includes 64 bytes of data. These 64 bytes take 8 clock cycles to transfer. The read response data packet also includes 4 header bytes and 4 Cycle Redundancy Code (CRC) bytes, which require 1 clock cycles to transfer. Thus, the read response data packet requires a total of 9 clock cycles to transfer. The write response includes 32 bytes of data (multiple write completes), 4 bytes of header and 4 bytes of CRC. As understood by those skilled in the art, the header bytes are control bytes, and the CRC bytes are used as standard error checking mechanism.

FIG. 3also shows an idle packet, which is four clock cycles long. The idle packet contains 4 header bytes and 28 no operation (NOP) bytes. The idle packet is sent on the upstream bus by the downstream hubs when the hubs do not have any data to send. The idle packet allows the breakpoint logic to switch when no data is being sent by the downstream hubs.

In one embodiment, a data packet moves from the upstream reception port164to the upstream transmission port132in one 1.6 GHz clock period. However, the breakpoint logic208, which switches the bypass multiplexer212, requires three clock periods to complete the switch because of the time required to process a decode and drive logic to switch the bypass multiplexer212. Thus, the beginning of the data packet is located as it enters the memory hub200, and then switching is initiated three clock cycles prior to the breakpoint so that the bypass multiplexer212is switched in time as the data packet arrives.

FIG. 4shows valid breakpoints in data packets. The bypass multiplexer212is switched at valid breakpoints. A valid breakpoint exists between two read responses, between a read response and a write response, and between a write response and a read response.

As described before, the determination that the bypass multiplexer212will be switched is made three clock cycles before the arrival of a data packet. By looking ahead three clock cycles before the data arrives, the switching process of the bypass multiplexer212can begin so that the switch coincides with the data arrival. The write response data packet inFIG. 4shows that a determination that the bypass multiplexer212will be switched is made three clock cycles before a breakpoint.

FIG. 5shows a memory hub500in accordance with another embodiment of the invention. The memory hub500includes the elements shown inFIG. 2and described before. In addition, the memory hub500includes two temporary storages: an upstream buffer512, and a bypass FIFO516coupled to the bypass multiplexer212and the bypass bus220. The bypass FIFO is a high speed buffer operating at 4X clock speed, where X=400 MHz. The upstream buffer is a normal speed buffer operating at 1X clock speed.

When the bypass multiplexer212is switched to the core logic204, incoming data packets from the downstream hubs are first stored in the bypass FIFO516. Since the bypass FIFO516operates at high speed (4X clock speed), the bypass FIFO516can transfer data packets from its input to its output very quickly. Thus, if the core logic204completes sending data packet and the bypass multiplexer switches to the temporary storages, the data from the bypass FIFO516is available immediately.

However, if the bypass multiplexer212remains switched to the core logic204, incoming data packets from the downstream hubs fill up the bypass FIFO516. When the bypass FIFO516is filled up, the upstream buffer512is used to store data packets. As will be understood by those skilled in the art, the bypass FIFO516is fast, but is expensive to implement. Thus a small bypass FIFO516is typically used. The upstream buffer512is slower, but is less expensive to implement. Thus, a large upstream buffer516is used.

The memory hub500includes clock domain change circuits520,524,508. As noted before, since the downstream ports124,160operate at different clock frequency than the core logic204, the downstream ports124,160are not synchronous with the core logic204. Thus, data packets cannot be directly transferred between the core logic and the downstream ports124,160. The clock domain change circuit520allows transfer of data packets from the downstream port124to the core logic204, and the clock domain change circuit524allow the transfer of data packets from the core logic204to the downstream port160. The core logic204is synchronous with the bypass multiplexer212, and the clock domain change circuit508allows the transfer of data packets from the core logic204to the bypass multiplexer212through a core upstream FIFO504.

In one embodiment, after power up, the breakpoint control logic208initially switches the bypass multiplexer212to the bypass bus220, thus connecting the bypass bus220to the upstream link136. The bypass bus220remains connected to the upstream link136until the core logic204has data to be sent and a breakpoint is available on the bypass bus220. If the core logic204has data available and a breakpoint is available, the bypass multiplexer212is switched to the core logic212.

When the bypass multiplexer212is switched to the bypass bus220, data on the bypass bus220is sent to upstream link136. When the bypass multiplexer212is switched to the core logic204, data from the core logic204is sent to the upstream link136. While the bypass multiplexer212remains switched to the core logic204, incoming data on the bypass bus220is sent first to the bypass FIFO516. When the bypass FIFO516is filled up, data is next to the upstream buffer512.

In one embodiment, the bypass multiplexer212remains switched to the core logic204until the core logic204is empty or if a higher priority requires a switch. A higher priority is determined if the temporary storages, i.e., the bypass FIFO516or the upstream buffer512, have available data. When the bypass multiplexer212is switched away from the core logic204, the multiplexer212is first switched to the bypass FIFO516. The data in the bypass FIFO516is sent upstream over the upstream link136until the bypass FIFO is exhausted. In general, after the bypass FIFO516is exhausted, the bypass multiplexer212is next switched to the upstream buffer512, which is then emptied.

If the core logic204has data available, a switch can be made from the bypass FIFO516to the core logic204even though the bypass FIFO has not been exhausted. If a switch is made from the bypass FIFO516to the core logic204, the next switch is made back to the bypass FIFO516in order to send the upstream data in the order it was received. When the bypass FIFO516empties, data is next taken from the upstream buffer512. A switch to the core logic204can be made from the upstream buffer512even though the upstream buffer has not been exhausted. However, the next switch is made back to the upstream buffer512in order to send the upstream data in the order it was received.

After the bypass FIFO516and the upstream buffer512are cleared, the multiplexer212is normally switched to the bypass buss220. If, however, the core logic204has available data, the multiplexer212is switched to the core logic204. As discussed before, while the bypass multiplexer212is switched to the core logic204, upstream data is first loaded into the bypass FIFO516and then into the upstream buffer512. When the bypass multiplexer212is switched to the temporary storages, the bypass FIFO516is emptied first and then the upstream buffer512is emptied next. After the bypass FIFO516is emptied, it is not loaded again until the upstream buffer512has been emptied.

In the preceding description, certain details were set forth to provide a sufficient understanding of the present invention. One skilled in the art will appreciate, however, that the invention may be practiced without these particular details. Furthermore, one skilled in the art will appreciate that the example embodiments described above do not limit the scope of the present invention, and will also understand that various equivalent embodiments or combinations of the disclosed example embodiments are within the scope of the present invention. Illustrative examples set forth above are intended only to further illustrate certain details of the various embodiments, and should not be interpreted as limiting the scope of the present invention. Also, in the description above the operation of well known components has not been shown or described in detail to avoid unnecessarily obscuring the present invention. Finally, the invention is to be limited only by the appended claims, and is not limited to the described examples or embodiments of the invention.