Reestablishing synchronization in a memory system

Embodiments relate to reestablishing synchronization across multiple channels in a memory system. One aspect is a system that includes a plurality of channels, each providing communication with a memory buffer chip and a plurality of memory devices. A memory control unit is coupled to the plurality of channels. The memory control unit is configured to perform a method that includes receiving an out-of-synchronization indication associated with at least one of the channels. The memory control unit performs a first stage of reestablishing synchronization that includes selectively stopping new traffic on the plurality of channels, waiting for a first time period to expire, resuming traffic on the plurality of channels based on the first time period expiring, and verifying that synchronization is reestablished for a second time period.

BACKGROUND

The present invention relates generally to computer memory, and more specifically, to reestablishing synchronization across multiple channels in a memory system.

Contemporary high performance computing main memory systems are generally composed of one or more memory devices, which are connected to one or more memory controllers and/or processors via one or more memory interface elements such as buffers, hubs, bus-to-bus converters, etc. The memory devices are generally located on a memory subsystem such as a memory card or memory module and are often connected via a pluggable interconnection system (e.g., one or more connectors) to a system board (e.g., a PC motherboard).

Overall computer system performance is affected by each of the key elements of the computer structure, including the performance/structure of the processor(s), any memory cache(s), the input/output (I/O) subsystem(s), the efficiency of the memory control function(s), the performance of the main memory devices(s) and any associated memory interface elements, and the type and structure of the memory interconnect interface(s).

Extensive research and development efforts are invested by the industry, on an ongoing basis, to create improved and/or innovative solutions to maximizing overall system performance and density by improving the memory system/subsystem design and/or structure. High-availability systems present further challenges as related to overall system reliability due to customer expectations that new computer systems will markedly surpass existing systems in regard to mean-time-between-failure (MTBF), in addition to offering additional functions, increased performance, increased storage, lower operating costs, etc. Other frequent customer requirements further exacerbate the memory system design challenges, and include such items as ease of upgrade and reduced system environmental impact (such as space, power and cooling). In addition, customers are requiring the ability to access an increasing number of higher density memory devices (e.g., DDR3 and DDR4 SDRAMs) at faster and faster access speeds.

In a high-availability memory subsystem, a memory controller typically controls multiple memory channels, where each memory channel has one or more dual in-line memory modules (DIMMs) that include dynamic random access memory (DRAM) devices and in some instances a memory buffer chip. The memory buffer chip typically acts as a slave device to the memory controller, reacting to commands provided by the memory controller. The memory subsystem can be configured as a redundant array of independent memory (RAIM) system to support recovery from failures of either DRAM devices or an entire channel. In RAIM, data blocks are striped across the channels along with check bit symbols and redundancy information. Examples of RAIM systems may be found, for instance, in U.S. Patent Publication Number 2011/0320918 titled “RAIM System Using Decoding of Virtual ECC”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety, and in U.S. Patent Publication Number 2011/0320914 titled “Error Correction and Detection in a Redundant Memory System”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety.

In a RAIM system, data is typically returned from all memory channels of the memory subsystem at close to the same time. When one channel is significantly later than the others, overall memory latency is increased by having to wait for that channel's data to return prior to sending a data block from the channels to a cache subsystem. The memory controller typically issues commands in lock-step synchronization across all channels to maintain synchronization. In the presence of an error on a channel, the memory controller corrects fetched data and initiates a recovery sequence. A recovering channel is thrown out of synchronization with the other channels. The system may be forced to wait for the recovering channel's data for subsequent data block transfers where the recovery sequence itself is not managed by the memory controller.

SUMMARY

Embodiments include a method, system, and computer program product for reestablishing synchronization across multiple channels in a memory system. A system for reestablishing synchronization across multiple channels in a memory system includes a plurality of channels, each providing communication with a memory buffer chip and a plurality of memory devices. A memory control unit is coupled to the plurality of channels. The memory control unit is configured to perform a method that includes receiving an out-of-synchronization indication associated with at least one of the channels. The memory control unit performs a first stage of reestablishing synchronization that includes selectively stopping new traffic on the plurality of channels, waiting for a first time period to expire, resuming traffic on the plurality of channels based on the first time period expiring, and verifying that synchronization is reestablished for a second time period.

A computer implemented method for reestablishing synchronization across multiple channels in a memory system includes receiving an out-of-synchronization indication associated with at least one of a plurality of channels in the memory system. A memory control unit in communication with the channels performs a first stage of reestablishing synchronization that includes selectively stopping new traffic on the plurality of channels, waiting for a first time period to expire, resuming traffic on the plurality of channels based on the first time period expiring, and verifying that synchronization is reestablished for a second time period.

A computer program product for reestablishing synchronization across multiple channels in a memory system is provided. The computer program product includes a tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method. The method includes receiving an out-of-synchronization indication associated with at least one of a plurality of channels in the memory system. A memory control unit in communication with the channels performs a first stage of reestablishing synchronization that includes selectively stopping new traffic on the plurality of channels, waiting for a first time period to expire, resuming traffic on the plurality of channels based on the first time period expiring, and verifying that synchronization is reestablished for a second time period.

DETAILED DESCRIPTION

Exemplary embodiments provide a system, method, and computer program product for reestablishing synchronization across multiple channels in a memory system. The memory system includes a processing subsystem that communicates synchronously with a memory subsystem in a nest domain. The memory subsystem also includes a memory domain that can be run synchronously or asynchronously relative to the nest domain. The memory system includes a memory controller that interfaces with the memory subsystem having multiple memory channels. The memory controller includes an out-of-sync/out-of-order detector to detect out-of-synchronization and out-of-order conditions between the memory channels while tolerating a degree of skew within the memory system. Upon detecting misalignment beyond a threshold amount, actions can be taken to re-align and synchronize the memory channels. The memory controller attempts to reestablish synchronization without having control of the underlying replay system used for recovery.

In an exemplary embodiment, a programmable quiesce sequence is used to incrementally attempt to restore channel synchronization by stopping stores and other downstream commands over a programmable time interval. The sequence may wait for completion indications from a memory buffer chip of the recovering channel and additionally inject synchronization commands across all channels. The state of synchronization may be checked before exiting the sequence, and if a channel remains out of synchronization, the quiesce sequence can be retried under programmatic control. The memory controller can use the quiesce sequence without modifications to underlying channel recovery features.

FIG. 1depicts an example memory system100which may be part of a larger computer system structure. A control processor (CP) system102is a processing subsystem that includes at least one processor104configured to interface with a memory control unit (MCU)106. The processor104can be a multi-core processor or module that processes read, write, and configuration requests from a system controller (not depicted). The MCU106includes a memory controller synchronous (MCS)108, also referred to as a memory controller, that controls communication with a number of channels110for accessing a plurality of memory devices in a memory subsystem112. The MCU106and the MCS108may include one or more processing circuits, or processing may be performed by or in conjunction with the processor104. In the example ofFIG. 1, there are five channels110that can support parallel memory accesses as a virtual channel111. In an embodiment, the memory system100is a five-channel redundant array of independent memory (RAIM) system, where four of the channels110provide access to columns of data and check-bit memory, and a fifth channel provides access to RAIM parity bits in the memory subsystem112.

Each of the channels110is a synchronous channel which includes a downstream bus114and an upstream bus116. Each downstream bus114of a given channel110may include a different number of lanes or links than a corresponding upstream bus116. In the example ofFIG. 1, each downstream bus114includes n-unidirectional high-speed serial lanes and each upstream bus116includes m-unidirectional high-speed serial lanes. Frames of commands and/or data can be transmitted and received on each of the channels110as packets that are decomposed into individual lanes for serial communication. In an embodiment, packets are transmitted at about 9.6 gigabits per second (Gbps), and each transmitting lane transmits four-bit groups serially per channel110. The memory subsystem112receives, de-skews, and de-serializes each four-bit group per lane of the downstream bus114to reconstruct a frame per channel110from the MCU106. Likewise, the memory subsystem112can transmit to the MCU106a frame of packets as four-bit groups per lane of the upstream bus116per channel110. Each frame can include one or more packets, also referred to as transmission packets.

The CP system102may also include a cache subsystem118that interfaces with the processor104. A cache subsystem interface122of the CP system102provides a communication interface to the cache subsystem118. The cache subsystem interface122may receive data from the memory subsystem112via the MCU106to store in the cache subsystem118.

FIG. 2depicts an example of a memory subsystem112aas an instance of the memory subsystem112ofFIG. 1in a planar configuration200in accordance with an embodiment. The example ofFIG. 2only depicts one channel110of the memory subsystem112a; however, it will be understood that the memory subsystem112acan include multiple instances of the planar configuration200as depicted inFIG. 2, e.g., five instances. As illustrated inFIG. 2, the planar configuration200includes a memory buffer chip202connected to a plurality of dynamic random access memory (DRAM) devices204via connectors206. The DRAM devices204may be organized as ranks of one or more dual in-line memory modules (DIMMs)208. The each of the connectors206is coupled to a double data rate (DDR) port210, also referred to as a memory interface port210of the memory buffer chip202, where each DDR port210can be coupled to more than one connector206. In the example ofFIG. 2, the memory buffer chip202includes DDR ports210a,210b,210c, and210d. The DDR ports210aand210bare each coupled to a pair of connectors206and a shared memory buffer adaptor (MBA)212a. The DDR ports210cand210dmay each be coupled to a single connector206and a shared memory buffer adaptor (MBA)212b. The DDR ports210a-210dare JEDEC-compliant memory interfaces for issuing memory commands and reading and writing memory data to the DRAM devices204.

The MBAs212aand212binclude memory control logic for managing accesses to the DRAM devices204, as well as controlling timing, refresh, calibration, and the like. The MBAs212aand212bcan be operated in parallel, such that an operation on DDR port210aor210bcan be performed in parallel with an operation on DDR port210cor210d.

The memory buffer chip202also includes an interface214configured to communicate with a corresponding interface216of the MCU106via the channel110. Synchronous communication is established between the interfaces214and216. As such, a portion of the memory buffer chip202including a memory buffer unit (MBU)218operates in a nest domain220which is synchronous with the MCS108of the CP system102. A boundary layer222divides the nest domain220from a memory domain224. The MBAs212aand212band the DDR ports210a-210d, as well as the DRAM devices204are in the memory domain224. A timing relationship between the nest domain220and the memory domain224is configurable, such that the memory domain224can operate asynchronously relative to the nest domain220, or the memory domain224can operate synchronously relative to the nest domain220. The boundary layer222is configurable to operate in a synchronous transfer mode and an asynchronous transfer mode between the nest and memory domains220,224. The memory buffer chip202may also include one or more multiple-input shift-registers (MISRs)226, as further described herein. For example, the MBA212acan include one or more MISR226a, and the MBA212bcan include one or more MISR226b. Other instances of MISRs226can be included elsewhere within the memory system100. As a further example, one or more MISRs226can be positioned individually or in a hierarchy that spans the MBU218and MBAs212aand212band/or in the MCU106.

The boundary layer222is an asynchronous interface that permits different DIMMs208or DRAM devices204of varying frequencies to be installed into the memory domain224without the need to alter the frequency of the nest domain220. This allows the CP system102to remain intact during memory installs or upgrades, thereby permitting greater flexibility in custom configurations. In the asynchronous transfer mode, a handshake protocol can be used to pass commands and data across the boundary layer222between the nest and memory domains220,224. In the synchronous transfer mode, timing of the memory domain224is phase adjusted to align with the nest domain220such that a periodic alignment of the nest and memory domains220,224occurs at an alignment cycle in which commands and data can cross the boundary layer222.

The nest domain220is mainly responsible for reconstructing and decoding the source synchronous channel packets, applying any necessary addressing translations, performing coherency actions, such as directory look-ups and cache accesses, and dispatching memory operations to the memory domain224. The memory domain224may include queues, a scheduler, dynamic power management controls, hardware engines for calibrating the DDR ports210a-210d, and maintenance, diagnostic, and test engines for discovery and management of correctable and uncorrectable errors. There may be other functions in the nest or memory domain. For instance, there may be a cache of embedded DRAM (eDRAM) memory with a corresponding directory. If the cache is created for some applications and other instances do not use it, there may be power savings by connecting a special array voltage (e.g., VCS) to ground. These functions may be incorporated within the MBU218or located elsewhere within the nest domain220. The MBAs212aand212bwithin the memory domain224may also include logic to initiate autonomic memory operations for the DRAM devices204, such as refresh and periodic calibration sequences in order to maintain proper data and signal integrity.

FIG. 3depicts a memory subsystem112bas an instance of the memory subsystem112ofFIG. 1in a buffered DIMM configuration300in accordance with an embodiment. The buffered DIMM configuration300can include multiple buffered DIMMs302within the memory subsystem112b, e.g., five or more instances of the buffered DIMM302, where a single buffered DIMM302is depicted inFIG. 3for purposes of explanation. The buffered DIMM302includes the memory buffer chip202ofFIG. 2. As in the example ofFIG. 2, the MCS108of the MCU106in the CP system102communicates synchronously on channel110via the interface216. In the example ofFIG. 3, the channel110interfaces to a connector304, e.g., a socket, that is coupled to a connector306of the buffered DIMM302. A signal path308between the connector306and the interface214of the memory buffer chip202enables synchronous communication between the interfaces214and216.

As in the example ofFIG. 2, the memory buffer chip202as depicted inFIG. 3includes the nest domain220and the memory domain224. Similar toFIG. 2, the memory buffer chip202may include one or more MISRs226, such as one or more MISR226ain MBA212aand one or more MISR226bin MBA212b. In the example ofFIG. 3, the MBU218passes commands across the boundary layer222from the nest domain220to the MBA212aand/or to the MBA212bin the memory domain224. The MBA212ainterfaces with DDR ports210aand210b, and the MBA212binterfaces with DDR ports210cand210d. Rather than interfacing with DRAM devices204on one or more DIMMs208as in the planar configuration200ofFIG. 2, the DDR ports210a-210dcan interface directly with the DRAM devices204on the buffered DIMM302.

The memory subsystem112bmay also include power management logic310that provides a voltage source for a voltage rail312. The voltage rail312is a local cache voltage rail to power a memory buffer cache314. The memory buffer cache314may be part of the MBU218. A power selector316can be used to determine whether the voltage rail312is sourced by the power management logic310or tied to ground318. The voltage rail312may be tied to ground318when the memory buffer cache314is not used, thereby reducing power consumption. When the memory buffer cache314is used, the power selector316ties the voltage rail312to a voltage supply of the power management logic310. Fencing and clock gating can also be used to better isolate voltage and clock domains.

As can be seen in reference toFIGS. 2 and 3, a number of memory subsystem configurations can be supported in embodiments. Varying sizes and configurations of the DRAM devices204can have different address format requirements, as the number of ranks and the overall details of slots, rows, columns, banks, bank groups, and/or ports may vary across different DRAM devices204in embodiments. Various stacking architectures (for example, 3 die stacking, or 3DS) may also be implemented, which may include master ranks and slave ranks in the packaging architecture. Each of these different configurations of DRAM devices204may require a unique address mapping table. Therefore, generic bits may be used by the MCU106to reference particular bits in a DRAM device204without having full knowledge of the actual DRAM topology, thereby separating the physical implementation of the DRAM devices204from the MCU106. The memory buffer chip202may map the generic bits to actual locations in the particular type(s) of DRAM that is attached to the memory buffer chip202. The generic bits may be programmed to hold any appropriate address field, including but not limited to memory base address, rank (including master or slave), row, column, bank, bank group, and/or port, depending on the particular computer system.

FIG. 4depicts a memory subsystem112cas an instance of the memory subsystem112ofFIG. 1with dual asynchronous and synchronous memory operation modes in accordance with an embodiment. The memory subsystem112ccan be implemented in the planar configuration200ofFIG. 2or in the buffered DIMM configuration300ofFIG. 3. As in the examples ofFIGS. 2 and 3, the MCS108of the MCU106in the CP system102communicates synchronously on channel110via the interface216.FIG. 4depicts multiple instances of the interface216as interfaces216a-216nwhich are configured to communicate with multiple instances of the memory buffer chip202a-202n. In an embodiment, there are five memory buffer chips202a-202nper CP system102.

As in the examples ofFIGS. 2 and 3, the memory buffer chip202aas depicted inFIG. 4includes the nest domain220and the memory domain224. Also similar toFIGS. 2 and 3, the memory buffer chip202amay include one or more MISRs226, such as one or more MISR226ain MBA212aand one or more MISR226bin MBA212b. In the example ofFIG. 4, the MBU218passes commands across the boundary layer222from the nest domain220to the MBA212aand/or to the MBA212bin the memory domain224. The MBA212ainterfaces with DDR ports210aand210b, and the MBA212binterfaces with DDR ports210cand210d. The nest domain220and the memory domain224are established and maintained using phase-locked loops (PLLs)402,404, and406.

The PLL402is a memory controller PLL configured to provide a master clock408to the MCS108and the interfaces216a-216nin the MCU106of the CP system102. The PLL404is a nest domain PLL that is coupled to the MBU218and the interface214of the memory buffer chip202ato provide a plurality of nest domain clocks405. The PLL406is a memory domain PLL coupled the MBAs212aand212band to the DDR ports210a-210dto provide a plurality of memory domain clocks407. The PLL402is driven by a reference clock410to establish the master clock408. The PLL404has a reference clock408for synchronizing to the master clock405in the nest domain220. The PLL406can use a separate reference clock414or an output416of the PLL404to provide a reference clock418. The separate reference clock414operates independent of the PLL404.

A mode selector420determines the source of the reference clock418based on an operating mode422to enable the memory domain224to run either asynchronous or synchronous relative to the nest domain220. When the operating mode422is an asynchronous operating mode, the reference clock418is based on the reference clock414as a reference clock source such that the PLL406is driven by separate reference clock and414. When the operating mode422is a synchronous operating mode, the reference clock418is based on the output416of an FSYNC block492which employs PLL404as a reference clock source for synchronous clock alignment. This ensures that the PLLs404and406have related clock sources based on the reference clock408. Even though the PLLs404and406can be synchronized in the synchronous operating mode, the PLLs404and406may be configured to operate at different frequencies relative to each other. Additional frequency multiples and derivatives, such as double rate, half rate, quarter rate, etc., can be generated based on each of the multiplier and divider settings in each of the PLLs402,404, and406. For example, the nest domain clocks405can include multiples of a first frequency of the PLL404, while the memory domain clocks407can include multiples of a second frequency of the PLL406.

In an asynchronous mode of operation each memory buffer chip202a-202nis assigned to an independent channel110. All data for an individual cache line may be self-contained within the DRAM devices204ofFIGS. 2 and 3attached to a common memory buffer chip202. This type of structure lends itself to lower-end cost effective systems which can scale the number of channels110as well as the DRAM speed and capacity as needs require. Additionally, this structure may be suitable in higher-end systems that employ features such as mirroring memory on dual channels110to provide high availability in the event of a channel outage.

When implemented as a RAIM system, the memory buffer chips202a-202ncan be configured in the synchronous mode of operation. In a RAIM configuration, memory data is striped across multiple physical memory channels110, e.g., five channels110, which can act as the single virtual channel111ofFIG. 1in order to provide error-correcting code (ECC) protection for continuous operation, even when an entire channel110fails. In a RAIM configuration, all of the memory buffer chips202a-202nof the same virtual channel111are operated synchronously since each memory buffer chip202is responsible for a portion of a coherent line.

To support and maintain synchronous operation, the MCU106can detect situations where one channel110becomes temporarily or permanently incapacitated, thereby resulting in a situation wherein the channel110is operating out of sync with respect to the other channels110. In many cases the underlying situation is recoverable, such as intermittent transmission errors on one of the interfaces216a-216nand/or interface214of one of more of the memory buffer chips202a-202n. Communication on the channels110may utilize a robust cyclic redundancy code (CRC) on transmissions, where a detected CRC error triggers a recovery retransmission sequence. There are cases where the retransmission requires some intervention or delay between the detection and retransmission. A replay system including replay buffers for each of the channels can be used to support a recovery retransmission sequence for a faulty channel110. Portions of the replay system may be suspended for a programmable period of time to ensure that source data to be stored in the replay buffer has been stored prior to initiating automated recovery. The period of time while replay is suspended can also be used to make adjustments to other subsystems, such as voltage controls, clocks, tuning logic, power controls, and the like, which may assist in preventing a recurrence of an error condition that led to the fault. Suspending replay may also remove the need for the MCU106to reissue a remaining portion of a store on the failing channel110and may increase the potential of success upon the replay.

Although the recovery retransmission sequence can eventually restore a faulty channel110to fully operational status, the overall memory subsystem112remains available during a recovery period. Tolerating a temporary out of sync condition allows memory operations to continue by using the remaining good (i.e., non-faulty) channels110until the recovery sequence is complete. For instance, if data has already started to transfer back to the cache subsystem118ofFIG. 1, there may need to be a way to process failing data after it has been transmitted. While returning data with gaps is one option, another option is to delay the start of data transmission until all error status is known. Delaying may lead to reduced performance when there is a gapless requirement. After recovering a faulty channel110, the MCU106resynchronizes the recovered channel110to the remaining good channels110thereby re-establishing a fully functional interface across all channels110of the virtual channel111ofFIG. 1.

To support timing alignment issues that may otherwise be handled using deskewing logic, the MCU106and the memory buffer chip202may support the use of tags. Command completion and data destination routing information can be stored in a tag directory424which is accessed using a received tag. Mechanisms for error recovery, including retrying of read or write commands, may be implemented in the memory buffer chips202for each individual channel110. Each command that is issued by the MCU106to the memory buffer chips202may be assigned a command tag in the MCU106, and the assigned command tag sent with the command to the memory buffer chips202in the various channels110. The various channels110send back response tags that comprise data tags or done tags. Data tags corresponding to the assigned command tag are returned from the buffer chip in each channel to correlate read data that is returned from the various channels110to an original read command. Done tags corresponding to the assigned command tag are also returned from the memory buffer chip202in each channel110to indicate read or write command completion.

The tag directory424, also associated with tag tables which can include a data tag table and a done tag table, may be maintained in the MCU106to record and check the returned data and done tags. It is determined based on the tag tables when all of the currently functioning channels in communication with the MCU106return the tags corresponding to a particular command. For data tags corresponding to a read command, the read data is considered available for delivery to the cache subsystem118ofFIG. 1when a data tag corresponding to the read command is determined to have been received from each of the currently functioning channels110. For done tags corresponding to a read or write command, the read or write is indicated as complete from a memory control unit and system perspective when a done tag corresponding to the read or write command is determined to have been received from each of the currently functioning channels110. The tag checking mechanism in the MCU106may account for a permanently failed channel110by removing that channel110from a list of channels110to check in the tag tables. No read or write commands need to be retained in the MCU106for retrying commands, freeing up queuing resources within the MCU106.

Timing and signal adjustments to support high-speed synchronous communications are also managed at the interface level for the channels110.FIG. 5depicts an example of channel110and interfaces214and216in greater detail in accordance with an embodiment. As previously described in reference toFIG. 1, each channel110includes a downstream bus114and an upstream bus116. The downstream bus114includes multiple downstream lanes502, where each lane502can be a differential serial signal path to establish communication between a driver buffer504of interface216and a receiver buffer506of interface214. Similarly, the upstream bus116includes multiple upstream lanes512, where each lane512can be a differential serial signal path to establish communication between a driver buffer514of interface214and a receiver buffer516of interface216. In an exemplary embodiment, groups508of four bits are transmitted serially on each of the active transmitting lanes502per frame, and groups510of four bits are transmitted serially on each of the active transmitting lanes512per frame; however, other group sizes can be supported. The lanes502and512can be general data lanes, clock lanes, spare lanes, or other lane types, where a general data lane may send command, address, tag, frame control or data bits.

In interface216, commands and/or data are stored in a transmit first-in-first-out (FIFO) buffer518to transmit as frames520. The frames520are serialized by serializer522and transmitted by the driver buffers504as groups508of serial data on the lanes502to interface214. In interface214, serial data received at receiver buffers506are deserialized by deserializer524and captured in a receive FIFO buffer526, where received frames528can be analyzed and reconstructed. When sending data from interface214back to interface216, frames530to be transmitted are stored in a transmit FIFO buffer532of the interface214, serialized by serializer534, and transmitted by the driver buffers514as groups510of serial data on the lanes512to interface216. In interface216, serial data received at receiver buffers516are deserialized by deserializer536and captured in a receive FIFO buffer538, where received frames540can be analyzed and reconstructed.

The interfaces214and216may each include respective instances of training logic544and546to configure the interfaces214and216. The training logic544and546train both the downstream bus114and the upstream bus116to properly align a source synchronous clock to transmissions on the lanes502and512. The training logic544and546also establish a sufficient data eye to ensure successful data capture. Further details are described in reference to process600ofFIG. 6.

FIG. 6depicts a process600for providing synchronous operation in a memory subsystem in accordance with an embodiment. In order to accomplish high availability fully synchronous memory operation across all multiple channels110, an initialization and synchronization process is employed across the channels110. The process600is described in reference to elements ofFIGS. 1-5.

At block602, the lanes502and512of each channel110are initialized and calibrated. The training logic544and546can perform impedance calibration on the driver buffers504and514. The training logic544and546may also perform static offset calibration of the receiver buffers506and516and/or sampling latches (not depicted) followed by a wire test to detect permanent defects in the transmission media of channel110. Wire testing may be performed by sending a slow pattern that checks wire continuity of both sides of the clock and data lane differential pairs for the lanes502and512. The wire testing may include driving a simple repeating pattern to set a phase rotator sampling point, synchronize the serializer522with the deserializer524and the serializer534with the deserializer536, and perform lane-based deskewing. Data eye optimization may also be performed by sending a more complex training pattern that also acts as a functional data scrambling pattern.

Training logic544and546can use complex training patterns to optimize various parameters such as a final receiver offset, a final receiver gain, peaking amplitude, decision feedback equalization, final phase rotator adjustment, final offset calibration, scrambler and descrambler synchronization, and load-to-unload delay adjustments for FIFOs518,526,532, and538.

Upon detecting any non-functional lanes in the lanes502and512, a dynamic sparing process is invoked to replace the non-functional/broken lane with an available spare lane of the corresponding downstream bus114or upstream bus116. A final adjustment may be made to read data FIFO unload pointers of the receive FIFO buffers526and538to ensure sufficient timing margin.

At block604, a frame transmission protocol is established based on a calculated frame round trip latency. Once a channel110is capable of reliably transmitting frames in both directions, a reference starting point is established for decoding frames. To establish synchronization with a common reference between the nest clock405and the master clock408, a frame lock sequence is performed by the training logic546and544. The training logic546may initiate the frame lock sequence by sending a frame including a fixed pattern, such as all ones, to the training logic544on the downstream bus114. The training logic544locks on to the fixed pattern frame received on the downstream bus114. The training logic544then sends the fixed pattern frame to the training logic546on the upstream bus116. The training logic546locks on to the fixed pattern frame received on the upstream bus116. The training logic546and544continuously generate the frame beats. Upon completion of the frame lock sequence, the detected frame start reference point is used as an alignment marker for all subsequent internal clock domains.

A positive acknowledgement frame protocol may be used where the training logic544and546acknowledge receipt of every frame back to the transmitting side. This can be accomplished through the use of sequential transaction identifiers assigned to every transmitted frame. In order for the sending side to accurately predict the returning acknowledgment, another training sequence referred to as frame round trip latency (FRTL) can be performed to account for the propagation delay in the transmission medium of the channel110.

In an exemplary embodiment, the training logic546issues a null packet downstream and starts a downstream frame timer. The training logic544responds with an upstream acknowledge frame and simultaneously starts an upstream round-trip timer. The training logic546sets a downstream round-trip latency value, when the first upstream acknowledge frame is received from the training logic544. The training logic546sends a downstream acknowledge frame on the downstream bus114in response to the upstream acknowledge frame from the training logic544. The training logic544sets an upstream round-trip delay value when the downstream acknowledge frame is detected. The training logic544issues a second upstream acknowledge frame to close the loop. At this time the training logic544goes into a channel interlock state. The training logic544starts to issue idle frames until a positive acknowledgement is received for the first idle frame transmitted by the training logic544. The training logic546detects the second upstream acknowledge frame and enters into a channel interlock state. The training logic546starts to issue idle frames until a positive acknowledgement is received for the first idle frame transmitted by the training logic546. Upon receipt of the positive acknowledgement, the training logic546completes channel interlock and normal traffic is allowed to flow through the channel110.

At block606, a common synchronization reference is established for multiple memory buffer chips202a-202n. In the case of a fully synchronous multi-channel structure, a relative synchronization point is established to ensure that operations initiated from the CP system102are executed in the same manner on the memory buffer chips202a-202n, even when the memory buffer chips202a-202nare also generating their own autonomous refresh and calibration operations. Synchronization can be accomplished by locking into a fixed frequency ratio between the nest and memory domains220and224within each memory buffer chip202. In exemplary embodiments, the PLLs404and406from both the nest and memory domains220and224are interlocked such that they have a fixed repeating relationship. This ensures both domains have a same-edge aligned boundary (e.g., rising edge aligned) at repeated intervals, which is also aligned to underlying clocks used for the high speed source synchronous interface214as well as frame decode and execution logic of the MBU218. A common rising edge across all the underlying clock domains is referred to as the alignment or “golden” reference cycle.

Multi-channel operational synchronization is achieved by using the alignment reference cycle to govern all execution and arbitration decisions within the memory buffer chips202a-202n. Since all of the memory buffer chips202a-202nin the same virtual channel111have the same relative alignment reference cycle, all of their queues and arbiters (not depicted) remain logically in lock step. This results in the same order of operations across all of the channels110. Even though the channels110can have inherent physical skew, and each memory buffer chip202performs a given operation at different absolute times with respect to the other memory buffer chips202, the common alignment reference cycle provides an opportunity for channel operations to transit the boundary layer222between the nest and memory domains220and224with guaranteed timing closure and equivalent arbitration among internally generated refresh and calibration operations.

As previously described in reference toFIG. 4, each memory buffer chip202includes two discrete PLLs, PLL404and PLL406, for driving the underlying clocks405and407of the nest and memory domains220and224. When operating in asynchronous mode, each PLL404and406has disparate reference clock inputs408and414with no inherent phase relationship to one another. However, when running in synchronous mode, the memory PLL406becomes a slave to the nest PLL404with the mode selector420taking over the role of providing a reference clock418to the memory PLL406such that memory domain clocks407align to the common alignment reference point. A common external reference clock, the master clock408, may be distributed to the nest PLLs404of all memory buffer chips202a-202nin the same virtual channel111. The PLL404can be configured into an external feedback mode to ensure that all PLLs404align their output nest clocks405to a common memory sub-system reference point. This common point is used by dedicated sync logic to drive the appropriate reference clock418based on PLL404output416into the memory domain PLL406and achieve a lock onto the target alignment cycle (i.e., the “golden” cycle).

FIG. 7depicts a process700for establishing alignment between the nest and memory domains220and224in a memory subsystem112accordance with an embodiment. The process700is described in reference to elements ofFIGS. 1-6. The process700establishes an alignment or “golden” cycle first in the nest domain220followed by the memory domain224. All internal counters and timers of a memory buffer chip202are aligned to the alignment cycle by process700.

At block702, the nest domain clocks405are aligned with a frame start signal from a previous frame lock of block604. The nest domain220can use multiple clock frequencies for the nest domain clocks405, for example, to save power. A frame start may be defined using a higher speed clock, and as such, the possibility exists that the frame start could fall in a later phase of a slower-speed nest domain clock405. This would create a situation where frame decoding would not be performed on an alignment cycle. In order to avoid this, the frame start signal may be delayed by one or more cycles, if necessary, such that it always aligns with the slower-speed nest domain clock405, thereby edge aligning the frame start with the nest domain clocks405. Clock alignment for the nest domain clocks405can be managed by the PLL404and/or additional circuitry (not depicted). At block704, the memory domain clocks407are turned off and the memory domain PLL406is placed into bypass mode.

At block706, the MCS108issues a super synchronize (“SuperSync”) command using a normal frame protocol to all memory buffer chips202a-202n. The MCS108may employ a modulo counter matching an established frequency ratio such that it will only issue any type of synchronization command at a fixed period. This establishes the master reference point for the entire memory subsystem112from the MCS108perspective. Even though the SuperSync command can arrive at the memory buffer chips202a-202nat different absolute times, each memory buffer chip202can use a nest cycle upon which this command is decoded as an internal alignment cycle. Since skew among the memory buffer chips202a-202nis fixed, the alignment cycle on each of the memory buffer chips202a-202nwill have the same fixed skew. This skew translates into a fixed operational skew under error free conditions.

At block708, sync logic of the memory buffer chip202, which may be part of the mode selector420, uses the SuperSync decode as a reference to trigger realignment of the reference clock418that drives the memory domain PLL406. The SuperSync decode is translated into a one cycle pulse signal494, synchronous with the nest domain clock405that resets to zero a modulo counter496in the FSYNC block492. The period of this counter496within the FSYNC block492is set to be the least common multiple of all memory and nest clock frequencies with the rising edge marking the sync-point corresponding to the reference point previously established by the MCS108. The rising edge of FSYNC clock416becomes the reference clock of PLL406to create the memory domain clocks. By bringing the lower-frequency output of PLL406back into the external feedback port, the nest clock405and memory clock407all have a common clock edge aligned to the master reference point. Thus, the FSYNC block492provides synchronous clock alignment logic.

At block710, the memory domain PLL406is taken out of bypass mode in order to lock into the new reference clock418based on the output416of the PLL404rather than reference clock414. At block712, the memory domain clocks407are turned back on. The memory domain clocks407are now edge aligned to the same alignment reference cycle as the nest domain clocks405.

At block714, a regular subsequent sync command is sent by the MCS108on the alignment cycle. This sync command may be used to reset the various counters, timers and MISRs226that govern internal memory operation command generation, execution and arbitration. By performing a reset on the alignment cycle, all of the memory buffer chips202a-202nstart their respective internal timers and counters with the same logical reference point. If an arbiter on one memory buffer chip202identifies a request from both a processor initiated memory operation and an internally initiated command on a particular alignment cycle, the corresponding arbiter on the remaining memory buffer chips202will also see the same requests on the same relative alignment cycle. Thus, all memory buffer chips202a-202nwill make the same arbitration decisions and maintain the same order of operations.

Embodiments may provide internally generated commands at memory buffer chip202to include DRAM refresh commands, DDR calibration operations, dynamic power management, error recovery, memory diagnostics, and the like. Anytime one of these operations is needed, it must cross into the nest domain220and go through the same arbitration as synchronous operations initiated by the MCS108. Arbitration is performed on the golden cycle to ensure all the memory buffer chips202observe the same arbitration queues and generate the same result. The result is dispatched across boundary layer222on the golden cycle which ensures timing and process variations in each memory buffer chip202is nullified.

Under normal error free conditions, the order of operations will be maintained across all of the memory buffer chips202a-202n. However, there are situations where one channel110can get out of sync with the other channels110. One such occurrence is the presence of intermittent transmission errors on one or more of the interfaces214and216. Exemplary embodiments include a hardware based recovery mechanism where all frames transmitted on a channel110are kept in a replay buffer for a prescribed period of time. This time covers a window long enough to guarantee that the frame has arrived at the receiving side, has been checked for errors, and a positive acknowledgement indicating error free transmission has been returned to the sender. Once this is confirmed, the frame is retired from the replay buffer. However, in the case of an erroneous transmission, the frame is automatically retransmitted, or replayed, along with a number of subsequent frames in case the error was a one-time event. In many cases, the replay is sufficient and normal operation can resume. In certain cases, the transmission medium of the channel110has become corrupted to the point that a dynamic repair is instituted to replace a defective lane with a spare lane from lanes502or512. Upon completion of the repair procedure, the replay of the original frames is sent and again normal operation can resume.

Another less common occurrence can be an on-chip disturbance manifesting as a latch upset which results in an internal error within the memory buffer chip202. This can lead to a situation where one memory buffer chip202executes its operations differently from the remaining memory buffer chips202. Although the memory system100continues to operate correctly, there can be significant performance degradation if the channels110do not operate in step with each other. In exemplary embodiments, the MISRs226monitor for and detect such a situation. The MISRs226receive inputs derived from key timers and counters that govern the synchronous operation of the memory buffer chip202, such as refresh starts, DDR calibration timers, power throttling, and the like. The inputs to the MISRs226are received as a combination of bits that collectively form a signature. One or more of the bits of the MISRs226are continually transmitted as part of an upstream frame payload to the MCU106, which monitors the bits received from the MISRs226of the memory buffer chips202a-202n. The presence of physical skew between the channels110results in the bits from the MISRs226arriving at different absolute times across the channels110. Therefore, a learning process is incorporated to calibrate checking of the MISRs226to the wire delays in the channels110.

In exemplary embodiments, MISR detection in the MCU106incorporates two distinct aspects in order to monitor the synchronicity of the channels110. First, the MCU106monitors the MISR bits received on the upstream bus116from each of the memory buffer chips202a-202nand any difference seen in the MISR bit stream indicates an out-of-sync condition. Although this does not pose any risk of a data integrity issue, it can negatively impact performance, as the MCU106may incur additional latency waiting for an entire cache line access to complete across the channels110. Another aspect is monitoring transaction sequence identifiers (i.e., tags) associated with each memory operation and comparing associated “data” tags or “done” tags as the operations complete. Once again, skew of the channels110is taken into account in order to perform an accurate comparison. In one example, this skew can manifest in as many as 30 cycles of difference between the fastest and slowest channel110. If the tags are 7-bits wide, with five channels110, and a maximum 30-cycle difference across channels110, this would typically require 5×7×30=1050 latches to perform a simplistic compare. There may be some cases that equate to about 40 bit-times which is about 4 cycles of deskew after aligning to a frame. To further reduce the number of latches, a MISR can be incorporated within the MCU106to encode the tag into a bit stream, which is then pipelined to eliminate the skew. By comparing the output of the MISR of the MCU106across all of the channels110, a detected difference indicates an out-of-order processing condition.

In either of these situations, the afflicted channel110can at least temporarily operate out of sync or out of order with respect to the other channels110. Continuous availability of the memory subsystem112may be provided through various recovery and self-healing mechanisms. Data tags can be used such that in the event of an out-of-order or out-of-sync condition, the MCU106continues to function. Each read command may include an associated data tag that allows the MCS108to handle data transfers received from different channels110at different times or even in different order. This allows proper functioning even in situations when the channels110go out of sync.

For out-of-sync conditions, a group of hierarchical MISRs226can be used accumulate a signature for any sync-related event. Examples of sync-related events include a memory refresh start, a periodic driver (ZQ) calibration start, periodic memory calibration start, power management window start, and other events that run off a synchronized counter. One or more bits from calibration timers, refresh timers, and the like can serve as inputs to the MISRs226to provide a time varying signature which may assist in verifying cross-channel synchronization at the MCU106. Hierarchical MISRs226can be inserted wherever there is a need for speed matching of data. For example, speed matching may be needed between MBA212aand the MBU218, between the MBA212band the MBU218, between the MBU218and the upstream bus116, and between the interfaces216a-216nand the MCS108.

For out-of-order conditions, staging each of the tags received in frames from each channel110can be used to deskew the wire delays and compare them. A MISR per channel110can be used to create a signature bit stream from the tags received at the MCU106and perform tag/signature-based deskewing rather than hardware latch-based deskewing. Based on the previous example of 7-bit wide tags, with five channels110, and a maximum 30-cycle difference across channels110, the use of MISRs reduces the 1050 latches to about 7×5+30×5=185 latches, plus the additional support latches.

To minimize performance impacts, the MCS108tries to keep all channels110in lockstep, which implies that all commands are executed in the same order. When read commands are executed, an associated data tag is used to determine which data correspond to which command. This approach also allows the commands to be reordered based on resource availability or timing dependencies and to get better performance. Commands may be reordered while keeping all channels110in lockstep such that the reordering is the same across different channels110. In this case, tags can be used to match the data to the requester of the data from memory regardless of the fact that the command order changed while the data request was processed.

Marking a channel110in error may be performed when transfers have already started and to wait for recovery for cases where transfers have not yet occurred. Data blocks from the memory subsystem112can be delivered to the cache subsystem interface122ofFIG. 1as soon as data is available without waiting for complete data error detection. This design implementation is based on the assumption that channel errors are rare. Data can be sent across clock domains from the MCS108to the cache subsystem interface122asynchronously as soon as it is available from all channels110but before data error detection is complete for all frames. If a data error is detected after the data block transfer has begun, an indication is sent from the MCS108to the cache subsystem interface122, for instance, on a separate asynchronous interface, to intercept the data block transfer in progress and complete the transfer using redundant channel information. Timing requirements are enforced to ensure that the interception occurs in time to prevent propagation of corrupt data to the cache subsystem118ofFIG. 1. A programmable count-down counter may be employed to enforce the timing requirements.

If the data error is detected before the block data transfer has begun to the cache subsystem118, the transfer is stalled until all frames have been checked for any data errors. Assuming errors are infrequent, the performance impact is minimal. This reduces the use of channel redundancy and may result in avoidance of possible uncorrectable errors in the presence of previously existing errors in the DRAM devices204.

The MCU106may also include configurable delay functions on a per-command type or destination basis to delay data block transfer to upstream elements, such as caches, until data error detection is completed for the block. Command or destination information is available for making such selections as inputs to the tag directory. This can selectively increase system reliability and simplify error handling, while minimizing performance impacts.

To support other synchronization issues, the MCU106can re-establish synchronization across multiple channels110in the event of a channel failure without having control of an underlying recovery mechanism used on the failed channel. A programmable quiesce sequence incrementally attempts to restore channel synchronization by stopping stores and other downstream commands over a programmable time interval. The quiesce sequence may wait for completion indications from the memory buffer chips202a-202nand inject synchronization commands across all channels110to reset underlying counters, timers, MISRs226, and other time-sensitive circuitry to the alignment reference cycle. If a failed channel110remains out of synchronization, the quiesce sequence can be retried under programmatic control. In many circumstances, the underlying root cause of the disturbance can be self healed, thereby resulting in the previously failed channel110being reactivated and resynchronized with the remaining channels110. Under extreme error conditions the quiesce and recovery sequence fails to restore the failed channel110, and the failed channel110is permanently taken off line. In a RAIM architecture that includes five channels110, the failure of one channel110permits the remaining four channels110to operate with a reduced level of protection.

FIG. 8depicts an example timing diagram800of synchronizing a memory subsystem in accordance with an embodiment. The timing diagram800includes timing for a number of signals of the memory buffer chip202. In the example ofFIG. 8, two of the nest domain clocks405ofFIG. 4are depicted as a higher-speed nest domain clock frequency802and a lower-speed nest domain clock frequency804. Two of the memory domain clocks407ofFIG. 4are depicted inFIG. 8as a higher-speed memory domain clock frequency806and a lower-speed memory domain clock frequency808. The timing diagram800also depicts example timing for a nest domain pipeline810, a boundary layer812, a reference counter814, a memory queue816, and a DDR interface818of a DDR port210. In an embodiment, the higher-speed nest domain clock frequency802is about 2.4 GHz, the lower-speed nest domain clock frequency804is about 1.2 GHz, the higher-speed memory domain clock frequency806is about 1.6, GHz and the lower-speed memory domain clock frequency808is about 0.8 GHz.

A repeating pattern of clock cycles is depicted inFIG. 8as a sequence of cycles “B”, “C”, “A” for the lower-speed nest domain clock frequency804. Cycle A represents an alignment cycle, where other clocks and timers in the memory buffer chip202are reset to align with a rising edge of the alignment cycle A. Upon receiving a SuperSync command, the higher and lower-speed memory domain clock frequencies806and808stop and restart based on a sync point that results in alignment after a clock sync window820. Once alignment is achieved, the alignment cycle A, also referred to as a “golden” cycle, serves as a common logical reference for all memory buffer chips202a-202nin the same virtual channel111. Commands and data only cross the boundary layer222on the alignment cycle. A regular sync command can be used to reset counters and timers within each of the memory buffer chips202a-202nsuch that all counting is referenced to the alignment cycle.

InFIG. 8at clock edge822, the higher and lower-speed nest domain clock frequencies802and804, the higher and lower-speed memory domain clock frequencies806and808, and the nest domain pipeline810are all aligned. A sync command in the nest domain pipeline810is passed to the boundary layer812at clock edge824of the higher-speed memory domain clock frequency806. At clock edge826of cycle B, a read command is received in the nest domain pipeline810. At clock edge828of the higher-speed memory domain clock frequency806, the read command is passed to the boundary layer812, the reference counter814starts counting a zero, and the sync command is passed to the memory queue816. At clock edge830of the higher-speed memory domain clock frequency806, the reference counter814increments to one, the read command is passed to the memory queue816and the DDR interface818. At clock edge832of the higher-speed memory domain clock frequency806which aligns with an alignment cycle A, the reference counter814increments to two, and a refresh command is queued in the memory queue816. Alignment is achieved between clocks and signals of the nest domain220and the memory domain224for sending commands and data across the boundary layer222ofFIG. 2.

FIG. 9Aillustrates an embodiment of the MCU106ofFIG. 1in greater detail. In MCU106, tag allocation logic901receives read and write commands from the cache subsystem118ofFIG. 1and assigns a command tag to each read and write command. The available command tags may be numbered from 0 to 31 in some embodiments; in such an embodiment, 0-23 may be reserved for read commands, and 24-31 may be reserved for write commands. Frame generation logic902sends read and write commands, and corresponding command tags, to memory buffer chips202ofFIGS. 2-4corresponding to a plurality of channels110in the virtual channel111ofFIG. 1(for example, five memory buffer chips, each corresponding to a single channel110ofFIGS. 1-4) via downstream buses903(corresponding to downstream bus114ofFIG. 1), and read data and tags (including data and done tags) are returned from the memory buffer chips202on all channels110to memory control unit interfaces905via upstream buses904(corresponding to upstream bus116ofFIG. 1). Memory control unit interfaces905may comprise a separate interface216ofFIGS. 2-4for each of the channels110. The memory control unit interfaces905separate read data from tags, and sends read data on read data buses906to buffer control blocks908A-E.

The MCU106includes a respective buffer control block908A-E for each channel110. These buffer control blocks908A-E store data that are returned on read data buses906from memory control unit interfaces905until the data may be sent to the cache subsystem118ofFIG. 1via data buffer read control and dataflow logic912. Data buffer read control and dataflow logic912may be part of the cache subsystem interface122ofFIG. 1and configured to communicate with the cache subsystem118ofFIG. 1. Accordingly, the data buffer read control and dataflow logic912of the cache subsystem interface122can be part of the MCU106or be separate but interfaced to the MCU106. An embodiment may have the data buffer read control and dataflow logic912as part of a system domain916, whereas the other blocks shown as part of the MCU106may be part of a memory controller nest domain918. In this embodiment, data buffers in buffer control blocks908A-E are custom arrays that support writing store data into the buffers which is performed in the memory controller nest domain918and reading data out of the same buffers which is performed in the system domain916.

Memory control unit interfaces905send the tag information via tag buses907to both the buffer control blocks908A-E and channel control block909. Channel control block909tracks received tags using the data tag table910and done tag table911, each of which are shown in further detail inFIG. 9B. Data tag table910and done tag table911are associated with the tag directory424ofFIG. 4, where each of the tables910and911includes a plurality of rows, each row corresponds to a command tag, and each row includes a respective bit corresponding to each of the channels110in communication with the MCU106(e.g., channels 0 through 4). In some embodiments, the rows in data tag table910may be numbered from 0 to 23, corresponding to the command tags reserved for read commands, and the rows in done tag table911may be numbered from 0 to 31, corresponding to all available command tags. The channel control block909also indicates to tag allocation logic901when a command tag may be reused, and indicates to data buffer read control and dataflow logic912that a command is completed.FIGS. 9A-Bare shown for illustrative purposes only; for example, any appropriate number of command tags may be available in a tag allocation logic901, and a data tag table910and done tag table911may each include any appropriate number of rows. Further, any appropriate number of channels110, with associated buffer control blocks908A-E and respective bits in the data tag table910and done tag table911, may be in communication with the MCU106.

In the example ofFIG. 9A, there are five memory channels110, where four channels110provide data and ECC symbols, and the fifth channel110provides redundant data used for channel recovery. In an embodiment, a data block includes 256 bytes of data which is subdivided into four 64-byte sections otherwise known as quarter-lines (QL). Each QL has associated with it 8 bytes of ECC symbols. The resulting 72 bytes are distributed across four of the five channels110using an 18-byte interleave. The fifth channel110stores channel redundancy (RAIM) information. This QL sectioning allows for recovery of any QL of data in the event of DRAM failures or an entire channel110failure.

Each QL of data and check bytes (or corresponding channel redundancy information in the case of the fifth channel110) is contained in a frame that is transmitted across a channel110. This frame contains CRC information to detect errors with the data in the frame along with attributes contained in the frame and associated with the data. One of these attributes is a data tag which is used to match incoming frame data with previously sent fetch commands to memory. Another attribute is a done tag which is used to indicate command completion in a memory buffer chip202. Each fetch command is directly associated with a 256-byte data block fetch through an assigned data tag. In this example, a 256-byte fetch requires four QL's or frames per channel. These four frames are sent consecutively by each channel110assuming no errors are present.

Data arrives independently from each of the five channels110and are matched up with data from other channels110using the previously mentioned data tag. The channel control block909tracks the received tags using the data tag table910and the done tag table911. The channel control block909sets a bit in the data tag table910based on receiving a data tag, and sets a bit in the done tag table911based on receiving a done tag. Embodiments of data tag table910and done tag table911are shown in further detail inFIG. 9B. Each row in the data tag table910corresponds to a command tag associated with a single read command (numbered from 0 to 23), and each row has an entry comprising a bit for each of the five channels110. As shown in data tag table910ofFIG. 9B, the first row corresponds to data tag0for all channels110, and the last row corresponds to data tag23for all channels110. Each row in the done tag table911corresponds to a command tag associated with a single read or write command (numbered from 0 to 31), and each row has an entry comprising a bit for each of the five channels110. As shown in done tag table911ofFIG. 9B, the first row corresponds to done tag0for all channels, and the last row corresponds to done tag31for all channels110. When a data or done tag is received from an individual channel110, the bit for that channel110in the row corresponding to the tag of the data tag table910or done tag table911is set to indicate that that particular data tag or done tag has been received. The data tags are also used as write pointers to buffer locations in buffer control blocks908A-E. Each of buffer control blocks908A-E holds read data received from the buffer control block's respective channel on read data buses906. In some embodiments, the buffer locations in buffer control blocks908A-E may be numbered from 0 to 23, corresponding to the numbers of the command tags that are reserved for read commands. Read data received on a particular channel are loaded into the location in the channel's buffer control block908A-E that is indicated by the data tag that is received with the read data.

A controller914, which may be part of the channel control block909, controls the transfer of data through the buffer control blocks908A-E to the data buffer read control and dataflow logic912. The data buffer read control and dataflow logic912can read the buffer control blocks908A-E asynchronously relative to the memory control unit interfaces905populating the buffer control blocks908A-E with data. The data buffer read control and dataflow logic912operates in the system domain916, while the memory control unit interfaces905operate in the memory controller nest domain918, where the system domain916and the memory controller nest domain918are asynchronous clock domains that may have a variable frequency relationship between domains. Accordingly, the buffer control blocks908A-E form an asynchronous boundary layer between the system domain916and the memory controller nest domain918. In an exemplary embodiment, the controller914is in the memory controller nest domain918and sends signals to the data buffer read control and dataflow logic912via an asynchronous interface920.

The MCU106also includes an out-of-sync/out-of-order (OOS/OOO) detector922that may be incorporated in the memory control unit interfaces905. The OOS/OOO detector922compares values received across multiple channels110for out-of-synchronization conditions and/or out-of-order conditions. Data tags and done tags received in frames on the channels110can be used in combination with data from the MISRs226ofFIGS. 2-4and other values to detect and compare synchronization events across the channels110. Tag-based detection allows for a greater degree of flexibility to maintain synchronization and data order as compared to stricter cycle-based synchronization. A resynchronization process may be initiated upon determining that at least one of the channels110is substantially out of synchronization or out of order. The OOS/OOO detector922and other elements of the MCU106can be implemented using a processing circuit that can include application specific integrated circuitry, programmable logic, or other underlying technologies known in the art.

The controller914can control reestablishing synchronization in the MCU106. In an exemplary embodiment, the controller914interfaces with the OOS/OOO detector922, a number of counters924, timers926, and a replay system928to reestablish synchronization. The counters924and timers926can be used for process control flow to reestablish synchronization. In an exemplary embodiment, the replay system928is a separate portion of the CP102ofFIG. 1and is not directly controlled by the MCU106. An error condition on one of the channels110, such as a loss of one or more data bits within a frame, can be detected by a CRC mismatch, which triggers a channel error recovery or replay sequence by the replay system928. A similar replay system also exists on the memory buffer chips202. The replay system928can provide status information, such as a replay-active signal930to the controller914of the MCU106. The flow of both downstream store data and commands, and upstream fetch data is interrupted as a result of a replay sequence on an individual channel110. The replay sequence causes the faulty channel110to go out of synchronization with the other four channels110. The MCU106can handle a channel110being out of synchronization for a period of time, but there is an impact to system performance if this out of synchronization condition is maintained for a long period. The MCU106is configured to restore synchronization in a timely fashion after a channel error event so that the system performance impact is minimized.

FIGS. 10A, 10B, and 10Cdepict a process1000for reestablishing synchronization across multiple memory channels110in the memory system100ofFIG. 1in accordance with an embodiment. The process1000can be implemented by the MCU106as described in reference toFIGS. 1-9A.

As is depicted inFIGS. 10A-10C, three stages are implemented under programmatic control for progressively increasing the complexity of the quiese sequence to reestablish synchronization in the example memory system100ofFIG. 1. The use of multiple programmable timers926ofFIG. 9A, e.g., timers T1-T4, enable tuning the process1000for specific system characteristics such as error rate, recovery time, asynchronous memory events and higher-level system time-out mechanisms. Asynchronous events such as refreshes, DRAM interface calibrations and memory throttling for power in particular can interfere with re-establishing synchronization. More specifically, one memory buffer chip202ofFIGS. 2-4can experience interruptions in command flow due to a replay that other memory buffer chips202on other channels110will not experience. It is possible under these conditions that the different memory buffer chips202will schedule operations differently and in a different order due to interference from asynchronous events. Scheduling of asynchronous events such a refreshes can also be disturbed on the channel110with the replay, further aggravating the out-of-sync or out-of-order behavior. These asynchronous events, although relatively infrequent, are accounted for in the incremental actions taken by the process1000.

Stage 1 of the process1000employs the simplest sequence for restoring synchronization by stopping the flow of downstream commands and store data on all channels1000until a first programmable timer T1expires. In an exemplary embodiment, the timer T1is programmable between about 32 and 1024 nanoseconds. The process1000is entered, if enabled by the MCU106, whenever an out-of-sync condition is detected as at block1002. An out-of-sync condition can be detected by the OOS/OOO detector922ofFIG. 9Aas a difference in arrival times of fetch data for an individual fetch command between the five channels110that does not match a pre-defined value determined during system initialization. At block1004, upon detecting an out-of-sync condition1002, a check is performed to determine whether a memory buffer chip out-of-sync condition exists, which may be based on the MISRs226aand226bofFIGS. 2-4. For example, the OOS/OOO detector922ofFIG. 9Acan compare values or summary values of the MISRs226aand226bofFIGS. 2-4after adjusting for skew value differences between the channels110to determine whether a memory buffer chip out-of-sync condition exists. If no memory buffer chip out-of-sync condition exists at block1004, the process1000advances to block1006.

A loop count, which may be one of the counters924ofFIG. 9A, is incremented for each pass through stages 1, 2, or 3 of process1000, and a programmable register determines how many tries each stage gets in re-establishing synchronization. A decision block exists in each stage to test the loop count, determine if it has exceeded the programmed loop count, and either repeat the stage if needed or move to the next stage. At block1006, if a stage 1 loop count is not exceeded, the process1000advances to block1008. At block1008, new traffic is selectively stopped on the channels110. One of the counters924ofFIG. 9Acan be used as a skip counter, where stopping of new traffic is performed at block1008unless the skip counter is enabled and not exceeded. The skip counter is incremented with each pass through block1008and is cleared when quiesce successfully completes. At block1010, the first timer T1is decremented if no replay is in progress. The replay-active signal930from the replay system928ofFIG. 9Acan be used to check whether a replay is in progress. At block1012, if the first timer T1has not expired, the process1000loops back to block1010; otherwise, the process1000advances to block1014.

At block1014, traffic resumes on the channels110when the first timer T1expires and no replays are in progress on any of the channels110. At block1016, a check is performed to determine whether synchronization is maintained for a time period defined using a programmable second timer T2. The second timer T2may be a programmable time window or command count. If synchronization is verified as reestablished for the time period defined using the second timer T2, the process1000returns to block1002to end the quiesce sequence; otherwise, the process1000returns to block1004.

At block1004, if a memory buffer chip MISR out-of-sync condition exists, the process1000advances to block1038in stage 3 ofFIG. 10Cas depicted by connector A. At block1006, if the stage 1 loop count is exceeded, the process1000advances to block1018in stage 2 ofFIG. 10Bas depicted by connector B.

Stage 2 is similar to stage 1 except that additional checks are added prior to starting the timer T1and waiting for it to expire. At block1018, if a stage 2 loop count is exceeded, then a check is performed at block1019to determine whether stage 3 ofFIG. 10Cis configured. If stage 3 is configured, then the process1000advances to stage 3 ofFIG. 10Cas depicted by connector D; otherwise, a failure is declared at block1020. At block1018, if the stage 2 loop count is not exceeded, the process1000advances to block1022. At block1022, new traffic is stopped on the channels110. At block1024, the process1000waits for outstanding traffic to complete on the channels110. The check for outstanding traffic may be performed by reviewing a list of pending fetches and stores. Pending fetches and stores are removed from the list when done tags respectively are received on all channels110for these operations. At block1026, the process1000waits for a write reorder queue (WRQ) empty status indicator from the memory buffer chips202ofFIGS. 2-4, if enabled. The WRQ empty status indicator may be used to indicate that no further memory writes are queued at the memory buffer chips202.

At block1028, the first timer T1is decremented if no replay is in progress. At block1030, if the first timer T1has not expired, the process1000loops back to block1028; otherwise, the process1000advances to block1032. At block1032, traffic resumes on the channels110. At block1034, a check is performed to determine whether synchronization is verified as reestablished for a time period defined using the second timer T2. If synchronization is not maintained for the time period defined using the second timer T2, the process1000advances to block1036; otherwise, the process1000returns to block1002in stage 1 ofFIG. 10Aas depicted by connector C to end the quiesce sequence. At block1036, if a memory buffer chip out-of-sync condition exists, the process1000advances to block1038in stage 3 ofFIG. 10Cas depicted by connector D; otherwise, the process1000returns to block1018to repeat stage 2.

Stage 3 adds an additional step of sending a synchronization command down all channels110after executing the same initial sequence as stage 2. At block1038, if a stage 3 loop count is exceeded, then a failure is declared at block1040; otherwise, the process1000advances to block1042. At block1042, new traffic is stopped on the channels110. At block1044, the process1000waits for outstanding traffic to complete on the channels110. At block1046, the process1000wait for the WRQ empty status indicator from the memory buffer chips202ofFIGS. 2-4, if enabled. At block1048a synchronization command is sent on the channels110. The synchronization command resets various timers and counters within the memory buffer chips202such as those associated with refresh, interface calibration and memory throttling. At block1050, a programmable third timer T3is used to wait for a guard time after issuing the synchronization command at block1048. In an exemplary embodiment, the timer T3is programmable between about 32 and 128 nanoseconds. The timer T3is used to test for successful completion of the synchronization command across channels110without replay during the time period defined by timer T3.

At block1052, a check is performed to determine whether replay was active during the guard time. If replay was active during the guard time, then the process1000returns to block1048after waiting an amount of time defined using a programmable fourth timer T4at block1054. In an exemplary embodiment, the timer T4is programmable between about 16 and 128 microseconds. The timer T4waits for the effect of the synchronization command to clear all of the memory buffer chips202. If replay was not active during the guard time, then the process1000waits an amount of time defined using the fourth timer T4at block1056and continues to block1058. At block1058, traffic resumes on the channels110. At block1060, a check is performed to determine whether synchronization is verified as reestablished for a time period defined using the second timer T2. If synchronization is not maintained for the time period defined using the second timer T2, the process1000returns to block1038; otherwise, the process1000returns to block1002in stage 1 ofFIG. 10Aas depicted by connector E to end the quiesce sequence.

Referring now toFIG. 11, in one example, a computer program product1100includes, for instance, one or more storage media1102, wherein the media may be tangible and/or non-transitory, to store computer readable program code means or logic1104thereon to provide and facilitate one or more aspects of embodiments described herein.

Program code, when created and stored on a tangible medium (including but not limited to electronic memory modules (RAM), flash memory, Compact Discs (CDs), DVDs, Magnetic Tape and the like is often referred to as a “computer program product”. The computer program product medium is typically readable by a processing circuit preferably in a computer system for execution by the processing circuit. Such program code may be created using a compiler or assembler for example, to assemble instructions, that, when executed perform aspects of the invention.

Technical effects and benefits include reestablishing synchronization across multiple memory channels in a memory subsystem. Using a multi-stage approach to reestablishing synchronization allows a faster and simpler approach to be tried initially before advancing to slower and more complex sequences of operations for reestablishing synchronization.