Providing an arrangement of memory devices to enable high-speed data access

A memory subsystem includes multiple memory modules coupled by point-to-point links. A memory controller is coupled by a point-to-point link to a first memory module, which is turn is coupled by another point-to-point link to another memory module. Further memory modules may be coupled by respective point-to-point links in the memory subsystem. In some arrangements, each memory module tracks commands issued to other memory modules, such as more upstream memory modules. Also, in one example implementation, a clock is embedded within a data stream transmitted over a point-to-point link, so that an external clock is not employed in this example implementation.

BACKGROUND

Memory devices are used to store information in various processing systems, such as computer systems, handheld devices, and other electronic devices or systems. The stored information is retrieved by microprocessors or microcontrollers to perform various tasks in the systems. With improved technology, the operating speeds of microprocessors and microcontrollers have increased dramatically, which has increased demand for faster and larger memory subsystems.

Memory devices are typically coupled to a memory bus. Traditionally, memory devices are coupled to the memory bus in a multi-drop arrangement, in which a group of signal lines (making up the memory bus) is connected to multiple memory devices at multiple points. Each connection point of a memory device(s) to the memory bus is referred to as a drop. Multiple connections thus correspond to multiple drops. Memory capacity can be added by adding more “drops”—that is, additional connection points of memory device(s) to the memory bus.

However, as more memory devices are added to a multi-drop memory bus, capacitive loading is increased that may slow down memory device operation. Also, the peak data rate on a multi-drop memory bus is reduced due to impedance mismatches associated with discontinuities on the multi-drop memory bus. Multi-drop memory buses also have other electrical characteristics that reduce memory bus performance.

DETAILED DESCRIPTION

FIG. 1is a block diagram of an example computer system that includes a memory subsystem108according to some embodiments. The memory subsystem108includes multiple memory modules110A,110B,110C, and110D. The memory modules110A,110B,110C, and110D are coupled by point-to-point links106B,106C and106D. The memory module110A is coupled by point-to-point link106A to a memory controller104.

A “point-to-point” link refers to any link that is coupled to two endpoints and that establishes communication of control and/or data information between the two endpoints. A point-to-point link is contrasted to a multi-drop link that can be connected to more than two points along the link. Each point-to-point link106can be a single-bit serial link (one bit in the transmit path and one in the receive path). To enhance bandwidth, the point-to-point link106can include multiple serial links arranged in parallel between each pair of endpoints.

In the context of the memory subsystem108, a “memory bus” refers to the collection of point-to-point links106. A “memory module” refers to a module that contains one or more memory devices.

The memory controller104in the example arrangement ofFIG. 1is part of a host bridge102, which includes bridge circuitry that enables communication between a central processing unit (CPU)100and a system bus112. The system bus112is coupled to bus devices114A,114B. In another example arrangement, the memory controller104can be a separate device that is outside the host bridge102.

The arrangement shown inFIG. 1is provided for purpose of example only. Other systems will have other arrangements. For example, in some systems, the CPU100may be connected directly to the system bus112, rather than through a host bridge. Also, the number of memory modules shown in the memory subsystem108is provided as an example only. In other systems, a smaller or larger number of memory modules can be used in the memory subsystem108.

Although not shown inFIG. 1, the set of memory modules110A–110D along with the corresponding set of links106A–106D can be repeated such that multiple sets of the memory modules110and links106are coupled in parallel to the memory controller104. This provides even greater memory capacity.

In operation, the memory controller104sends a command over the link106A to memory module110A. The memory module110A retransmits that command to the next memory module110B, in addition to decoding the received command to determine whether the command is intended for the memory module110A. Similarly, the memory module110B retransmits the received command on towards the memory module110C, which also decodes the received command to determine if the command is intended for memory module110B. This is repeated with each successive memory module until the very last memory module (referred to as the most upstream module). In the arrangement ofFIG. 1, the memory module110A (the one closest to the memory controller104) is referred to as the most downstream memory module, and the memory module110D (the one furthest away from the memory controller104) is referred to as the most upstream memory module.

Read data, responsive to read commands from the memory controller104, are transmitted by the memory modules110back to the memory controller104in the reverse direction, with the read data passing through successive memory modules until the read data reaches the memory controller102. Commands and data are sent in packets, with each packet referring to a group of characters, where each character includes data or command bits of predetermined size (e.g., 8 bits, 16 bits, etc.). Each packet is transmitted in a serial stream across a link106. In one implementation, each packet has two or more characters.

Another feature of each point-to-point link106, according to some embodiments, is that a clock is embedded within a stream of information bits communicated over the link106. This provides a self-clocking feature. Self-clocking is achieved by using transitions within the information stream (transitions of successive bits in the stream between “0” and “1”) as indications of clock edges. This is contrasted to providing a separate clock signal from the memory controller104to the memory modules110. Thus, “embedding” a clock within a data or command information stream refers to using transitions of data or command bits (information bits) to define clock edges.

To ensure there in fact are information bit transitions in each transmitted character, encoding of the information in the character is performed so that there is at least one transition (usually more) within an encoded stream of information bits for all possible values of a character. For example, if an un-encoded character contains a series of all “0s” or “1s”, then the un-encoded character will have no information bit transitions. However, in accordance with some embodiments of the invention, the character is provided through an encoding algorithm to provide an encoded character. Thus, a character of all “0s” or “1s” will be encoded to a value that is not all “0s” or “1s” so that information bit transitions occur within the encoded character.

In one implementation, an 8-bit character is encoded into a 10-bit character before transmission over a point-to-point link. At the receiving end, the encoded character is decoded back to its original 8-bit form. In other implementations, other types of encoding can be performed, so long as all possible values of an N-bit un-encoded character are encoded into respective groups of information bits that have one or more transitions between “0s” and “1s”.

A benefit of using the point-to-point links106of the memory subsystem108according to some embodiments is that high capacitive loading (due to the presence of large numbers of devices on the link) and impedance mismatches (due to multiple discontinuities on a link) can be avoided. As a result, high-speed transmission over the links106is possible. Further, to add capacity to the memory subsystem108, additional memory modules110can be added with corresponding point-to-point links. In other words, memory capacity can be expanded without adding capacitive or impedance loading to an existing memory link.

Another benefit offered by the memory subsystem108according to some embodiments is that, by embedding a clock within each information stream, rather than using a separate clock, re-synchronization of received data or command bits within the memory module110to an internal clock of the memory module is not needed. Re-synchronizing information bits to an internal clock adds extra synchronization latency on each retransmit, which reduces the speed of transmission over the links106A–106D. With the memory subsystem108according to some embodiments, the only added latency for retransmit is the propagation delay through the receive and transmit buffers of each memory module and any logic in between.

Further according to one embodiment, packets from multiple memory modules can be sent back-to-back without requiring idle clocks in between, which enhances memory bandwidth and reduces latency

FIG. 2illustrates an example arrangement of a memory module110. The memory module110includes a memory buffer circuit (according to an example implementation)200and one or more memory arrays202. In one example implementation, the memory array202is a DRAM (dynamic random access memory) array, such as a synchronous DRAM (SDRAM) array. However, in other embodiments, other types of memory devices can be used, such as static RAMs (SRAMs) and so forth.

As shown inFIG. 2, each memory buffer circuit200is coupled to a downstream point-to-point link106_DNS and to an upstream point-to-point link106_UPS. The downstream link106_DNS couples the memory buffer circuit200to a memory module or memory controller that is downstream of the memory buffer circuit200, while the upstream link106_UPS couples the memory buffer circuit200to a memory module that is upstream of the memory buffer circuit. Thus, for example, the memory buffer circuit200in the memory module110A (FIG. 1) has a downstream link106A and an upstream link106B.

The downstream link106_DNS has a receive path204and a transmit path206. The receive path204receives commands and write data from the memory controller104(either directly over the link106A or through the memory buffer circuits200of one or more other memory modules). The transmit path206sends read data back to the memory controller104, either directly over link106A or through memory buffer circuits of one or more other memory modules.

The upstream link106_UPS also includes a transmit path208and a receive path210. The transmit path208sends commands and write data to further memory modules, while the receive path210receives read data from upstream memory modules110.

Note that the receive path204is a single-bit path over which a stream of serial data is received. An N-bit (e.g., N=8) stream of information bits (control or data bits) makes up a character. The other paths206,208, and210are similarly one-bit paths over which information bits are serially transmitted.

Within each memory buffer circuit200, a receive buffer212receives information from the receive path204, and a receive buffer214receives information from the receive path210. A transmit buffer216transmits information out to the transmit path208, and the transmit buffer218transmits information out to the transmit path206.

Received information is provided to command decode and clock recovery logic220, which is responsible for decoding commands received over the receive path204. The command decode and clock recovery logic220determines if the received command is intended for the current memory module, or for another memory module. If the received command is intended for the current memory module, then appropriate command signals (and any write data) are sent from the command decode logic and clock recovery logic220to a memory interface222(e.g., DRAM interface). However, if the received commands are not intended for the current memory module, the command decode and clock recovery logic220makes a note of the command (read, write, read-modify-write, etc.). The command decode and clock recovery logic220tracks commands sent to upstream memory modules so that the memory buffer circuit200knows when to provide read data (assuming a read command was received by the command decode and clock recovery logic220) in relation to read data provided by the upstream memory modules over the memory bus.

The clock recovery portion of the logic220includes a clock generator221, such as a phase-locked loop (PLL) device. The clock generator221is synchronized to the embedded clock derived from streams of information bits received by the receive buffer212. The clock generator221generates an internal clock that is provided to the DRAM interface222.

Effectively, the DRAM interface222in each memory buffer circuit200operates on a clock derived from the command stream. This ensures that the DRAM interfaces on all memory modules use the same frequency clock and thus read data can be made available at the same time on each module relative to the command being received. Thus, a memory module that is further away (further upstream) from the memory controller104receives the command later and thus starts sending data back later than a memory module that is closer to the memory controller104. The memory module that is furthest away (the most upstream memory module) uses the clock derived from the command stream to send the read data or fill data back downstream. This prevents a situation where a memory module that is more upstream starts sending data while a downstream memory module is not finished sending its own data.

The memory buffer circuit200also includes a data buffer224to store read data retrieved from the DRAM array202through the DRAM interface222. Thus, if the command decode and clock recovery logic220receives a read command, the logic220causes the DRAM interface222to send appropriate read command and address signals to the DRAM array202, which responds with read data. DRAM interface222provides the read data into the data buffer224, with the data transmitted by a data transmit circuit226through the transmit buffer218out to the transmit path206. The timing of transmission of read data by the data transmit circuit226is determined by what other data is being transmitted from upstream memory modules. The data transmit circuit226is informed (by the command decode and clock recovery logic220) of how many packets are expected from upstream memory modules before data from data buffer224can be transmitted.

FIG. 3shows components of the data transmit circuit226, according to one example implementation, along with receive and transmit buffers214and216and the memory data buffer224. The data transmit circuit226includes a multiplexer300that selects between local data (through the local data input) or external data (through the pass-through input) of the multiplexer300. The pass-through input of the multiplexer300receives data from the receive buffer214, and passes the data through to the transmit buffer218for transmission over the transmit path206. The local data input of the multiplexer300receives data from a shift register302, which in turn receives encoded data from storage312in the data buffer224. Encoding of data can be performed in one of many places. For example, as shown inFIG. 3, an encoder314can be provided in the data buffer224to perform the encoding of data received from the DRAM array202. Alternatively, the encoder can be provided in the DRAM interface222. The shift register302is clocked by an internal clock304.

The shift register302converts data in the data buffer224into a serial stream of data to be output through the multiplexer300to the transmit path206. A select input of the multiplexer300is controlled by the packet detection and clock counting logic306. The packet detection and clock counting logic306also provides a load input308to the shift register302to load data from the data buffer224into the shift register302. The packet detection and clock counting logic306is also clocked by the internal clock304.

The internal clock304is generated by a clock generator310, such as a PLL device. The phase and frequency of the internal clock304is locked with respect to the embedded clock in the data stream received over the receive path210in the downstream direction (similar to what is performed by the clock generator221(FIG. 2) in the upstream direction). Since the data stream is self-clocking, “fill” data is transmitted over the receive path210by one or more of the memory modules when there is no actual data to be sent. In the downstream path, fill data is generated by one or more of the memory modules. In the upstream path, the memory controller104generates and transmits the fill data. Fill data has one or more predefined values so that the memory controller104and memory modules can recognize a data stream as fill data.

In accordance with some embodiments of the invention, the memory subsystem108responds to commands from the memory controller104in a first-in-first-out sequence. Thus, for example, if a first command is received by the memory module110D and a second, subsequent command is received by the memory module100B, data from the memory module110D is first transmitted back to the memory controller104before data from the memory module110B is transmitted back to the memory controller104. In other words, the memory buffer circuit200in the memory module110B has to wait for data from the memory module110D to finish transmission before the memory buffer circuit200in the memory module110B can transmit data.

Since each memory buffer circuit200decodes commands intended for upstream memory modules, the memory buffer circuit200is aware of the number of packets to be transmitted from upstream memory modules. This information is provided by the command decode and clock recovery logic220to the packet detection and clock counting logic306, which determines the number of packets that have been transmitted from upstream memory modules. In effect, the packet detection and clock counting logic306tracks packets from upstream memory modules so that it knows exactly on which clock the last packet from an upstream module will occur.

When the command decode and clock recovery logic220decodes a read command intended for an upstream memory module, the command decode and clock recovery logic220stores the read command so that the memory buffer circuit200is able to track the expected number of packets from the upstream memory module. The expected number of packets for the read command is communicated to the packet detection and clock counting logic306. This is repeated for all other read commands. As a result, before the packet detection and clock counting logic306transmits local data, the packet detection and clock counting logic306first determines how many packets have been transmitted by upstream memory modules for read commands that precede the read command for which the local data is responsive. The local data is transmitted after the upstream memory modules have completed their respective transmissions.

The packet detection and clock counting logic306can perform the determination of the number of packets that have been transmitted. For example, the logic306can actually count how many packets are passed through the memory buffer circuit200by detecting for the first packet from a memory module, and then counting the number of clocks to determine how many packets have passed through the memory buffer circuit200. Note that there may be fill data between packets the logic306only counts actual packets, not fill code characters.

At the appropriate time, the packet detection logic306switches the multiplexer300(with the select input of the multiplexer) from the pass-through input to the local data input so that data contained in the shift register302can be transmitted out to the transmit path206. The packet detection logic switches the multiplexer to the local data input if one of the following conditions is satisfied: (1) there are no pending upstream read requests; or (2) there are pending upstream read requests but each such request does not precede (in time) the read request pending in the current memory module. Since the internal clock304is aligned so that the local data input to the multiplexer300switches at the same time as the pass-through data, there is no switching transition evident on the output of the multiplexer300except for some jitter specification. As a result, data from different memory modules can appear on the transmit path206without any gaps so that no clock phase shift or loss of bandwidth occurs.

In some cases, a narrow glitch on the output of the multiplexer300may result from clock skew and the clock-to-output propagation delay being slightly different between the data and select signals of the multiplexer300. To address this glitch, the memory buffer circuit200can be designed so that both inputs to the multiplexer300are stable when the switch-over occurs between the pass-through input and the local data input.

To accomplish this, the shift register302can be loaded at least one clock before switch-over to the local data input, so that the local data input to the multiplexer300is stable during the switch-over. The pass-through input of the multiplexer300can be made stable during the switch-over from the pass-through input to the local data input by using a defined algorithm that employs several different fill code characters (transmitted by the memory modules in the downstream direction when there is no valid data to be sent). In one embodiment, the fill code characters include a normal fill code character and two special fill code characters. Normally, the normal fill code character is transmitted with the normal fill code character having the-same value for the first and last bits, which ensures that the pass-through input to the multiplexer300is stable if the switch-over is done on the boundary of two normal fill code characters.

Other scenarios employ the special fill code characters to avoid glitches. Each memory module that is transmitting read data (in response to a read command) keeps track of the next read command issued after the command the memory module is currently responding to. If the next read command was issued to a more downstream module, or if no other read command has been issued yet, then the data that the current memory module (referred to as the “first memory module”) will receive after it completes sending its own data will be fill code (since the next read data will not be from a memory module that is more upstream than the first memory module).

At the completion of transmission of data by the first memory module through the local data input of the multiplexer300, rather than switch immediately to the pass-through input of the multiplexer300, the first memory module generates a special fill code character (one of two special fill code characters) for transmission immediately after transmission of the valid read data. The special fill code character has a first bit that matches the last bit of the read data that was just sent, and the last bit of the special fill code character matches the first bit of the first (normal) fill code character. This will provide a stable pass-through input at the multiplexer300at a downstream memory module (more downstream than the first memory module) that has to respond by transmitting read data after the first memory module has completed transmitting its read data. Thus, when the downstream memory module switches from the pass-through input to the local data input of the multiplexer300after the downstream memory module has detected that read data from the first memory module has completed transmission, the pass-through input of the multiplexer300in this downstream memory module is stable. To ensure that the special fill code character is received by downstream memory modules, any memory module that does not have its own data ready to send does not switch its multiplexer to the local data input until both the upstream packet and the immediately following fill code character (the special fill code character) has been retransmitted by the memory module.

In another scenario, if the next read command has been issued by a memory module that is upstream of the current memory module, then the switch-over to the pass-through input of the multiplexer300is done on the clock after the local data has completed transmitting, as usual. No special action needs to be taken in this case because a downstream memory module cannot insert its own data between two sequential upstream reads, and thus its multiplexer will not switch and thus no glitch avoidance provisions need to be made.

To provide a stable local data input to the multiplexer300when the multiplexer300switches in the other direction, from the local data input back to pass-through data, the shift register302simply holds the last data bit of the just transmitted data during the switch-over. To provide a stable pass-through input to the multiplexer300, each memory module that is sending fill codes sends a special fill code character having a last bit that is the same as the first bit of the local data for the fill code character immediately preceding valid data.

As indicated above, according to one example, there are three fill code characters. The normal fill code character has the first and last bits with the same value, while the special fill code characters have different values for the first and last bits. In one arrangement, the fill code characters are selected to have as many alternating 1s and 0s as possible to minimize jitter accumulation.

As described above, much of the logic for performing tasks according to some embodiments of the invention is provided by hardware in the memory modules110. Alternatively, the tasks can be performed by a combination of software or firmware and hardware. If the tasks are performed by software, then the software is stored as instructions and executed by one or more processing elements. The processing element includes a microprocessor, a microcontroller, a processor module or subsystem (including one or more microprocessors or microcontrollers), or other control or computing devices.