Memory access during memory calibration

A multi-rank memory system in which calibration operations are performed between a memory controller and one rank of memory while data is transferred between the controller and other ranks of memory. A memory controller performs a calibration operation that calibrates parameters pertaining to transmission of data via a first data bus between the memory controller and a memory device in a first rank of memory. While the controller performs the calibration operation, the controller also transfers data with a memory device in a second rank of memory via a second data bus.

BACKGROUND

The present disclosure generally relates to memory systems, and memory controllers that control the operation of memory devices in such systems. Specific embodiments described herein refer to methods and apparatus for types of memory device access during calibration operations, as executed by a memory controller.

In multi-rank memory systems, memory devices are organized into two or more ranks of memory where each rank of memory devices is independently addressable by a memory controller. Memory controllers write data to and read data from the memory devices in a rank through a data bus. In multi-rank memory systems, data buses are a shared resource. For example, devices in separate memory ranks may be connected to and share a common data bus. The memory controller transfers data with one memory rank at a time through the data bus.

Signaling interfaces in the memory controller and memory devices are responsible for transmitting signals to and receiving signals from the data bus. Due to the high-frequency nature of modern memory signaling, these interfaces are sensitive to changes in voltage and temperature. The signaling interfaces can be periodically calibrated to compensate for such changes. In conventional multi-rank memory systems, the signaling interfaces are calibrated one rank at a time. However, calibration operations tie up an entire data bus and block data access to the memory devices of other ranks that are not being calibrated.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure include a multi-rank memory system in which a memory controller calibrates parameters pertaining to transmission of data via a first data bus between the memory controller and a memory device in a first rank of memory devices. While the memory controller performs the calibration operation, the memory controller also transfers (e.g., reads or writes) data with a memory device in a second rank of memory devices via a second data bus. The solutions provided by various embodiments may allow for calibration to occur in a multi-rank memory system without completely blocking data access to the other ranks that are not being calibrated. In example embodiments, the performance degradation associated with calibration in a multi-rank memory system is reduced.

FIG. 1illustrates high-level overview of a memory system with multiple ranks of memory devices, according to an embodiment. The memory system includes a memory controller101and multiple memory devices120-1,120-2,120-3,120-4coupled to the memory controller101via data buses140-1,140-2,145-1,145-2and CMD/ADDR bus180. The memory devices are organized in a multi-rank configuration such that memory devices120-1,120-3are part of memory rank160-1and memory devices120-2,120-4are part of memory rank160-2. The memory ranks160-1,160-2may be, for example, ranks within a memory module such as a Dual Inline Memory Module (DIMM). AlthoughFIG. 1shows only two memory ranks and two memory devices in each memory rank, this is merely exemplary and in real memory systems there may be more than two memory ranks and each memory rank may have more than two memory devices.

The data buses140,145are bidirectional data buses that convey data signals between the memory controller101and the memory devices120. Data buses140-1,140-2are coupled to both memory device120-1and memory device120-2. Data buses145-1,145-2are coupled to both memory device120-3and memory device120-4. The data buses140,145are shared such that only one memory device120coupled to a given data bus140,145can transfer signals across the data bus140,145at any given time. For example, if memory device120-1is transferring signals across data bus140-1, memory device120-2cannot also transfer signals across data bus140-1at the same time.

When considered together, the individual data buses140,145(as well as data buses for any additional memory devices that are not shown) form the full data bus of the memory system. Each data bus140,145may have multiple wires or multiple pairs of wires to transmit multiple bits in parallel. For example, data bus140-1carries bits a:0, data bus140-2carries bits b:a+1, data bus145-1carries bits c:b+1, and data bus145-2carries bits d:c+1 of the full data bus. In one embodiment, each data bus140,145corresponds to four bits of the full data bus. For example, data bus140-1corresponds to bits3:0, data bus140-2corresponds to bits7:4, data bus145-1corresponds to bits11:8, and data bus145-2corresponds to bits15:12. Alternatively, each data bus140,145may be a serial bus having a single wire or pair of wires based on whether the data signals are common mode or differential signals.

As will be explained below, the memory controller101can perform a calibration operation that tunes transmission of data via a first data bus between the memory controller101and a memory device120in a first rank of memory devices. While the calibration is performed, the memory controller101can transfer data via a second data bus between the memory controller101and the same memory device120or between the memory controller and a memory device120in a different rank of memory. For example, memory controller101may send a command via command/address (CMD/ADDR) bus180that initiates a calibration operation that tunes transmission of data via data bus140-1between the controller101and memory device120-1in memory rank160-1. The memory controller101then sends a second command via CMD/ADDR bus180for transferring data between the memory controller101and memory device120-2in memory rank160-2via a different data bus140-2. Thus, the memory controller101can perform calibration operations without blocking data access to the memory devices120-1,120-2that are coupled to the bus that the calibration operations are being performed on.

FIG. 2illustrates a more detailed view of the memory system ofFIG. 1, according to one embodiment. Shown in the figure are the memory controller101and memory devices120-1and120-2fromFIG. 1. For purposes of clarity, memory devices120-3and120-4are not shown in this figure. The configuration of memory devices120-3,120-4is similar to the configuration of memory devices120-1,120-2.

Memory controller101includes multiple read queues205-1,205-2, multiple write queues210-1,210-2, a routing circuit215, multiple input/output (I/O) data interface circuits220-1,220-2(or PHYs; physical interface), controller logic225, and a control interface230. Memory device120-1includes multiple I/O interface circuits250-1,250-2, a routing circuit255-1, multiple banks including sub-banks260-1,260-2, and memory logic265-1. Similarly, memory device120-2also includes multiple I/O interface circuits250-3,250-4, a routing circuit255-2, multiple banks including sub-banks260-3,260-4, and memory logic265-2. In one embodiment, each memory device120has multiple memory banks, each of which is divided into sub-banks260.

Controller I/O interface220-1is coupled to data bus140-1, which is used to transmit data signals to and receive data signals from memory I/O interfaces250-1,250-3. Similarly, controller I/O interface220-2is coupled to data bus140-2, which is used to transmit data signals to and receive data signals from memory I/O interfaces250-2,250-4. The I/O interfaces in both the memory controller101and the memory devices120contain circuitry that is adjusted or tuned in accordance with calibration parameters. The parameters can be stored in control registers (not shown) that are updated during a calibration operation. There are a plurality of interface parameters that can be calibrated. Example parameters include timing parameters such as receiver sample phase and transmitter drive phase, voltage parameters such as receiver offset or reference voltage, receiver current bias, receiver termination impedance, transmit supply voltage, transmit drive swing voltage, and transmit termination impedance.

In more detailed embodiments, the receiver sample phase is a parameter that affects the temporal position of a received signal relative to a timing reference or other signal; transmitter drive phase is a parameter that affects the temporal position of a transmitted signal relative to a timing reference or other signal; receiver offset is a parameter that adjusts the voltage level of a received signal; receiver reference voltage is an offset that adjusts a receiver reference voltage; receiver current bias is a parameter that adjusts the bias voltage and a current source for a receiver circuit; receiver termination impedance is a parameter that affects the impedance of a transmission line termination for a receiver circuit; transmit supply voltage is a parameter that affects the supply voltage for a driver used to transmit a signal; transmit drive swing voltage is a parameter that affects the voltage swing of a transmitted signal by a transmitter; and transmit termination impedance is a parameter that affects the impedance of a transmission line termination on the transmitter (or driver) circuit used to transmit a signal or the impedance of the transmitter itself.

The impedance of a transmission line termination on the receiver side may be controlled using on-die termination (ODT) resistors (not shown) that are included in the I/O interfaces250of the memory devices120. During calibration operations, the memory controller101may adjust the ODT resistance in accordance with the receiver termination impedance parameter. The ODT resistance of the I/O interfaces250can be independently adjusted. For example, the ODT resistance of I/O interface250-1can be set differently than the ODT resistance of I/O interface250-2. The ODT resistance can be independently controlled by connecting two sets of ODT control signals between the memory controller101and the memory devices120. Alternatively, the ODT resistance can be independently controlled by bits in a command field received from the memory controller101via CMD/ADDR bus180.

In one embodiment, the memory controller101calibrates the parameters by performing calibration operations to optimize the transfer of data signals between the I/O interfaces220of the memory controller101and the I/O interfaces250of the memory devices120. For each data bus (e.g. data bus140-1), the parameters affecting the I/O interfaces coupled to the data bus (e.g. I/O220-1,250-1,250-3), are calibrated with respect to one memory device120at a time. For example, to completely calibrate parameters relating to transmission of data on data bus140-1, parameters affecting I/O220-1and I/O250-1would be first calibrated as a pair. Then, parameters affecting I/O220-1and I/O250-3would be calibrated as a pair. In one embodiment, calibration operations are performed on a periodic basis to adjust for changes in conditions such as voltage and temperature. As used herein, “calibrating a data bus” means calibrating parameters relating to transmission (and/or reception) of data, for example, by interface circuits that transmit (and/or receive) the data via the data bus.

To calibrate a data bus, the memory controller101may send test patterns to a memory device (e.g., device120-1) and receive responses to the test patterns from the memory device (e.g., device120-1) via one of the data buses (e.g., data bus140-1). As a result, calibration may temporarily remove a data bus (e.g., data bus140-1) from service, for example, in the event that the data bus is involved in the calibration operation. The bus140that is removed from service cannot be used to transfer data between the memory controller101and the memory devices in other ranks of memory (e.g. device120-2) until the calibration is complete.

Referring to memory devices120-1,120-2, each memory device has a plurality of sub-banks260-1,260-2,260-3,260-4. In the example ofFIG. 2, memory device120-1includes sub-banks260-1,260-2, and memory device120-2includes sub-banks260-3,260-4. The sub-banks260can include Dynamic Random Access Memory (DRAM) cells, static random access memory (SRAM) cells or non-volatile memory such as flash memory cells. The memory controller101maps each sub-bank260to a physical sub-bank address and writes data into and reads data from storage locations in the sub-banks260based on the physical address of the sub-bank. In an embodiment, each sub-bank260is a half bank of memory. In another embodiment, each memory device120has a plurality of banks and each sub-bank260is a set of sub-banks that represents multiple sub-banks of memory. When considered together, all of the sub-banks within a single memory device (e.g. sub-banks260-1,260-2of memory device120-1) form the full memory core of a memory device.

In one embodiment, the sub-banks260of the memory devices120are micro-threaded. Micro-threaded sub-banks are independently addressable from other sub-banks. The memory controller101can send different commands to different sub-banks260such that each sub-bank260performs a different command. For example, sub-bank260-1may perform a data write transaction while sub-bank260-2performs a data read transaction. Alternatively, a single command may be addressed to and performed by more than one sub-bank260.

In an embodiment, the memory controller101has a plurality of read queues205-1,205-2and write queues210-1,210-2. The write queues210store data waiting for transmission to the memory devices120via data buses140. The read queues205store data that is received from the memory devices120via data buses140. In one embodiment, read queues205and write queues210store data corresponding to physical addresses that are mapped to a particular sub-bank260. For example read queue205-1and write queue210-1may store data that corresponds to physical addresses mapped to sub-banks260-1,260-3. Read queue205-2and write queue210-2may store data that corresponds to physical addresses mapped to sub-banks260-2,260-4.

The controller routing circuit215is coupled to the read/write queues205,210and the I/O interfaces220. The routing circuit215can be configured by controller logic225to route signals between any of the read/write queues205,210and any of the I/O interfaces220in the memory controller101. Similarly, the routing circuit255-1in memory device120-1can be configured by memory logic265-1to route signals between any of sub-banks260-1,260-2and I/O interfaces250-1,250-2. Routing circuit255-2can be configured by memory logic265-2to route signals between any of the sub-banks260-3,260-4and I/O interfaces250-3,250-4. The routing circuits215,255enable transfer of data between any read queue205or write queue210and any sub-bank260of memory devices120-1,120-2via either data bus140-1,140-2. By properly configuring the routing circuits, the memory controller101can still access the full memory core (i.e., all sub-banks) of the memory devices120-1,120-2even if one of the data buses140-1,140-2is being used for calibration operations. Thus, the performance degradation associated with performing calibration operations in a multi-rank configuration is reduced.

FIG. 3illustrates a detailed view of a memory device showing how a routing circuit can be implemented with multiplexers, according to one embodiment. As shown, routing circuit255-1of memory device120-1can be implemented as a set of multiplexers (MUX)305-1,305-2,305-3,305-4. The inputs of MUX305-1are cross-coupled to both the sub-bank260-1and the sub-bank260-2such that MUX305-1is configured to route read data from either sub-bank260-1or sub-bank260-2. The inputs of MUX305-2are cross-coupled to both the I/O250-1(and data bus140-1) and the I/O250-2(and data bus140-2), such that MUX305-2is configured to route write data from either I/O250-1or I/O250-2. The inputs of MUX305-3are cross-coupled to both the sub-bank260-1and the sub-bank260-2such that MUX305-3is configured to route read data from either sub-bank260-1or sub-bank260-2. The inputs of MUX305-4are cross-coupled to both the I/O250-1(and data bus140-1) and the I/O250-2(and data bus140-2), such that MUX305-4is configured to route write data from either I/O250-1or I/O250-2.

Memory logic265-1is coupled to and controls the logic state of each MUX. Thus, for example, memory logic265-1can configure MUX305-1to route read data from either sub-bank260-1or260-2into I/O250-1. Memory logic265-1can configure MUX305-2to route write data from either I/O250-1or I/O250-2into sub-bank260-1. Memory logic265-1can configure MUX305-3to route read data from either sub-bank260-1or260-2into I/O250-2. Memory logic265-1can configure MUX305-4to route write data from either I/O250-1or I/O250-2into sub-bank260-2. The routing circuit255-1thus allows data to be routed between any of the I/Os250-1,250-2(and the associated data buses140-1,140-2) and any of the sub-banks250-1,250-2, depending on settings received from the memory logic265-1. As will be described in greater detail below, memory logic265-1can determine the settings for the routing circuit based on information received from the memory controller101(not shown) via the CMD/ADDR bus180.

Referring back toFIG. 2, the routing circuit265-2in device120-2and the routing circuit215in the controller101can also be implemented using multiplexers.

Referring again toFIG. 2, controller logic225is coupled to, and generates signals for controlling the operation of, the read queues205, write queues210, routing circuit215, and I/O interfaces220. Controller logic225also generates command, address, and other control information that is transmitted to memory devices120via control interface230and CMD/ADDR bus180. For example, controller logic225may transmit commands that instruct the memory devices120to read data, write data, refresh the sub-banks260, or calibrate an I/O interface250. At the other end of the CMD/ADDR bus180, memory logic265-1receives the commands from controller logic225via the CMD/ADDR bus180, decodes the commands, and generates control signals for controlling device I/Os250-1,250-1, routing circuit255-1, and sub-banks260-1,260-2of memory device120-1. Memory logic265-2also receives commands from controller logic225via the CMD/ADDR bus180, decodes the commands, and generates control signals for controlling device I/Os250-3,250-4, routing circuit255-2, and sub-banks260-3,260-4of memory device120-2. In one embodiment, memory device120-1and memory device120-2are each coupled to a different chip select signal, and the memory devices120use the logic state of the chip select signal to determine whether to decode a received command.

In one embodiment, controller logic225may provide command signals specifying different modes of operation for setting the routing circuits255, which are transmitted as command signals to the memory logic265on the memory devices, via CMD/ADDR bus180. Memory logic265decodes the signals and provides the routing settings to the routing circuit255prior to communication of read or write data. For example, the routing settings for the routing circuit255may be specified using one or more bits in a command field. In other embodiments, instead of using CMD/ADDR bus180, the routing settings can be conveyed through sideband signals. In another embodiment, information received from controller logic225is used to set a mode register (not shown) in the memory logic265. The memory logic then configures the routing circuit in accordance with the mode register. For example, in one mode set by the mode register, the routing circuit255-1routes signals directly between I/O250-1and sub-bank260-1and also between I/O250-2and sub-bank260-2. In another mode set by the mode register, the routing circuit255-1is cross coupled and routes signals between I/O250-1and sub-bank260-2and also between I/O250-2and sub-bank260-1.

FIGS. 4,5A, and5B together illustrate how calibration operations are performed in a memory system for device interfaces that operate using one data bus while transferring data between the controller and a memory device via another data bus, according to one embodiment.FIG. 4illustrates a method performed by the memory controller, according to one embodiment. At a high level, in steps405-420, the memory controller calibrates a first data bus one device at a time. While calibrating the first data bus, the memory controller transfers data between the controller and a memory device via a second data bus. In steps425-440, the memory controller calibrates a second data bus one device at a time. While calibrating the second data bus, the memory controller transfers data between the controller and a memory device via the first data bus. By calibrating one data bus while accessing data through another data bus, the method allows data access to continue during calibration operations. As a result, the performance degradation associated with periodically calibrating a data bus in a multi-rank configuration is reduced.

More specifically, in step405, the memory controller101calibrates a first data bus between the memory controller and a memory device (the “target device”). The memory controller may initiate the calibration by sending a command to the target device via the CMD/ADDR bus180. For example, referring toFIG. 5A, illustrated is how one data bus can be calibrated while data is accessed on another data bus, according to one embodiment. As shown in the example ofFIG. 5A, memory device120-1is the target device and data bus140-1is the bus being calibrated with respect to memory device120-1. Specifically, the system calibrates the data bus140-1between I/O220-1of memory controller101and I/O250-1of target memory device120-1, which are shown with cross-hatched shading. The calibration operation takes data bus140-1offline and prevents the transfer of data between the controller101and any of the memory devices120via the same data bus140-1.

In step410, the memory controller101transfers data between the memory controller101and a memory device via a second data bus while the calibration operation of step405is ongoing. The second data bus is coupled to the target device and devices in other ranks of memory. Through the second data bus, data can be transferred between the memory controller101and any of the devices120-1,120-2coupled to the second data bus. In an embodiment, the memory controller101transfers data with a memory device that is not the target device. In other words, the memory controller101transfers data with a memory device that is in a different rank of memory than the target device. For example, referring again toFIG. 5A, data bus140-1is being calibrated with respect to memory device120-1. As the calibration is ongoing, memory controller101can transfer data from write queue210-1into sub-bank260-3of a different memory device120-2via a second data bus140-2. The memory controller101can initiate the data access by generating a data transfer command and transmitting the command through the control interface230onto the CMD/ADDR bus180. The command is received by the memory logic265-2of memory device120-2, which decodes the command and performs the requested write action.

While not shown in the example ofFIG. 5A, the memory controller101can access the full memory core (i.e. all sub-banks) of any memory device through a single data bus. For example, by adjusting the settings of the routing circuits215,255, the memory controller101can transfer data from any write queue210into any sub-bank260of any memory device120via a single data bus (e.g., data bus140-2). The memory controller101can also transfer data from any sub-bank260of any device120into any read queue205via a single data bus (e.g., data bus140-2).

As explained previously, data buses140are calibrated with respect to one memory device120at a time. If there are multiple memory devices120coupled to a single data bus (e.g., data bus140-1), completely calibrating the data bus (e.g., data bus140-1) requires that calibration operations for each data bus (e.g., data bus140-1) be performed between the memory controller101and each of the memory devices120. Referring back toFIG. 4, in step415, the memory controller101determines if there are any more memory devices120coupled to the first data bus that have not yet been be calibrated. If so, it selects another memory device120as the target device. The memory controller101repeats the calibration405and data transfer410steps with respect to the new target device. For example, referring toFIG. 5B, illustrated is the calibration of parameters related to the transmission of data between the memory controller and another memory device, according to one embodiment. As shown, the new target device is memory device120-2and data bus140-1is the data bus being calibrated with respect to memory device120-2. Specifically, the system calibrates the data bus140-1between I/O220-1of memory controller101and I/O250-3of memory device120-2, which are shown with cross-hatched shading. The calibration operation takes data bus140-1offline and prevents the transfer of data between the controller and any of the memory devices120via data bus140-1. The memory controller101then transfers data from sub-bank260-1into read queue205-1via data bus140-2that is not being calibrated, in the example shown inFIG. 5B.

Referring back toFIG. 4, if there are no more memory devices for which the first data bus (e.g., data bus140-1in this example) should be calibrated in step415, then the controller101moves on to steps425-440to calibrate the second data bus. Steps425-440are similar to steps405-420, but the calibration operations are now performed for the second data bus, which would be data bus140-2in the examples illustrated inFIGS. 5A and 5B. In step425, the memory controller101calibrates the second data bus (e.g., data bus140-2) between the memory controller101and a target memory device. In step430, the memory controller101transfers data with a memory device via the first data bus while such calibration takes place. In step435, the memory controller101determines if there are any more memory devices coupled to the second data bus with parameters that have not yet been be calibrated. If so, the memory controller101selects another memory device as the target device in step440and repeats steps425-430-435. If not, the calibration operation is complete.

FIGS. 6,7A, and7B illustrate how calibration operations, core maintenance operations, and data access occur in parallel in one embodiment. Core maintenance operations include a variety of operations that affect the sub-banks260(i.e. memory core) of a memory device120. Examples of core maintenance operations include: refresh operations for refreshing the cells of DRAM based memory devices; erase operations for erasing a block of non-volatile memory; anneal operations for alleviating device degradation in some types of memory by using heat to diffuse trapped charges, and program operations in non-volatile memory that are of long enough duration to block other core transactions. Because core maintenance operations prevent data access to the sub-banks260of a memory device120, it is convenient to perform calibration operations at the same time. If the core maintenance operations are performed on only some of the sub-banks260and only one of the data buses is being calibrated, data traffic between the memory controller and the other sub-banks can be carried out through the non-calibrating bus. For purposes of clarity,FIGS. 6,7A, and7B will be described using embodiments where the core maintenance operation is a refresh operation. However, the description of these embodiments can be applied equally to any type of core maintenance operation.

In an embodiment, the memory controller101schedules refresh operations to occur in parallel with calibration operations. Specifically,FIG. 6illustrates a method performed by the memory controller101to perform refresh operations in parallel with calibration operations and data access, according to one embodiment. In step605, the memory controller101refreshes the sub-banks260of a target memory device. The controller101can initiate a refresh operation by sending a refresh command to the target memory device. The refresh command is decoded by the target device and causes the target device to refresh its sub-banks260. For example, referring toFIG. 7A, illustrated is how calibration operations, refresh operations, and data access can occur in parallel, according to one embodiment. Memory device120-1is the target device in this example ofFIG. 7A. The controller logic225generates a refresh command and transmits the refresh command through the control interface230onto CMD/ADDR bus180. Memory logic265-1of the target memory device120-1decodes the refresh command and refreshes the data in sub-banks260-1,260-2, which are shown with horizontal line shading. In another embodiment, the sub-banks260-1,260-2are refreshed one at a time. For example, the memory controller101may issue a command that only refreshes sub-bank260-1. After sub-bank260-1is refreshed, the memory controller101may issue a second command that refreshes sub-bank260-2.

Referring back toFIG. 6, in step610, the memory controller101calibrates a first data bus between the memory controller101and the target memory device. The calibration occurs while the sub-banks of the target device are being refreshed. For example, referring again toFIG. 7A, memory device120-1is the target device and data bus140-1is the bus being calibrated. Specifically, the calibration operation calibrates data bus140-1between I/O220-1of memory controller101and I/O250-1of memory device120-1, which are shown with cross-hatched shading in this example ofFIG. 7A.

Referring back toFIG. 6, in step615, the memory controller101transfers data between the memory controller101and a memory device via a second data bus. Because the sub-banks260-1,260-2of the target device120-1are being refreshed, the memory controller cannot transfer data with the target device120-1. However, the memory controller101can still transfer data with memory devices in other ranks of memory. For example, referring again toFIG. 7A, memory controller101can transfer data from write queue210-1into sub-bank260-3of memory device120-2via data bus140-2while data bus140-1is being calibrated with respect to I/O250-1of memory device120-1.

In step620, once the calibration operations of step610are complete, the memory controller101calibrates parameters relating to transmission of data via the second data bus between the memory controller101and the same target memory device as designated in step610. For example, referring now toFIG. 7B, memory device120-1is still the target memory device and data bus140-2is now the bus being calibrated in this example. Specifically, the calibration operation calibrates data bus140-2between I/O220-2of memory controller101and I/O250-2of memory device120-1, which are shown with cross-hatched shading.

Referring back toFIG. 6, in step625, the memory controller101transfers data between the memory controller101and a memory device via the first data bus, which is not being calibrated. For example, referring again toFIG. 7B, the memory controller101can transfer data from write queue210-2into sub-bank260-4of memory device120-2via data bus140-1, which is not being calibrated. Although not shown inFIG. 7B, memory controller101can transfer data with any sub-bank260, so long as the sub-bank260is not in a memory device120-1,120-2that is being refreshed in step605.

Referring back toFIG. 6, in step630, the memory controller101determines if there are any memory devices coupled to the first and second data buses that have not been refreshed. If there is a memory device that has not been refreshed yet, the memory controller101selects635another device as the target memory device for refresh and calibration and repeats steps605-625for the new target memory device. For example, referring again toFIG. 7B, the memory controller101can select memory device120-2as the new target device for refresh and calibration and send a refresh command to memory device120-2in order to refresh sub-banks260-3,260-4. If there are no more memory devices to refresh, the process is complete.

As described,FIGS. 6,7A, and7B illustrate one embodiment in which all the sub-banks260of a single memory device120are refreshed at the same time. In one embodiment, during calibration of a data bus, any number of sub-banks260across any number of memory devices120can be refreshed while data access is carried out with the non-refreshing sub-banks260. For example, three sub-banks260-1,260-2,260-3can be refreshed, data bus140-1can be calibrated, and data can be transferred between the memory controller101and sub-bank260-4via data bus140-2. As another example, one sub-bank260-4can be refreshed, data bus140-1can be calibrated, and data can be transferred between the memory controller101and any of sub-banks260-1,260-2, or260-3.

FIG. 8is a timing diagram illustrating how the memory system performs calibration operations for one data bus while transferring data between the controller and a memory device via another data bus, according to one embodiment. In the example ofFIG. 8, the data bus140-1is the bus to be calibrated with respect to memory device120-1for illustration purposes while data bus140-2is being used for data transfer with memory device120-2. The timing diagram shows a series of commands sent from the memory controller101to the memory devices120via the CMD/ADDR bus and transactions occurring on two data buses140-1,140-2as a result of those commands. The commands and transactions are broken down into individual periods of time, each of which represents an arbitrary length of time. For example, each time period can represent one or more clock cycles of a clock signal that is coupled to the memory controller101and the memory devices120.

In time period1, the memory controller101sends a calibration command via the CMD/ADDR bus180to memory device120-1. The command initiates a calibration of data bus140-1between the memory controller101and a first memory device120-1. As a result, starting from time period2, data bus140-1is removed from service for memory access. Also in time period2, the memory controller101sends a data access command (e.g. read or write command) via the CMD/ADDR bus180to a second memory device120-2. The data access command may include information for setting the routing circuit255-2of the memory device120-2so that the proper sub-bank260-3,260-4in memory device120-2can be coupled to data bus140-2for data access. The memory device120-2sets its routing circuit255-2in accordance with the command to couple the selected sub-bank260-3,260-4in memory device120-2to data bus140-2. Then, during time periods3-5, the memory controller101transfers data with the second memory device120-2via the second data bus140-2while data bus140-1is being calibrated with respect to the I/O device(s) of memory device120-1.

As shown by the embodiments, the disclosed multi-rank memory system is configured to allow data access during calibration operations. Through the use of routing circuits255in the memory devices120and the memory controller101, the memory controller101can calibrate one data bus while accessing data through another data bus. By allowing data access to continue during calibration operations, the system mitigates the performance loss that is associated with timing calibration in conventional multi-rank memory systems.

Some portions of the detailed description, such as the descriptions ofFIG. 4andFIG. 6, refer to steps performed by the memory controller101. It should be noted that in some embodiments, steps can be performed in a different order, steps can be performed concurrently with other steps, or some steps may not be performed at all. For example, referring back toFIG. 4, step410may occur before step405so that the memory controller101first initiates a transfer410of data via a second data bus. While the data transfer is ongoing, the memory controller101can then calibrate405a first data bus with respect to a target device while the data transfer is ongoing. As another example, referring back toFIG. 6, step605may occur after step615so that the sub-banks260are not refreshed605until both calibration610and data transfer615are in progress.

Also, referring back toFIG. 1, the embodiments have been described in detail with respect to the operations of the memory controller101, data buses140-1,140-2and the memory devices120-1,120-2. The principles of the described embodiments inFIG. 2throughFIG. 8also apply to the other data buses145-1,145-2and the other memory devices120-3,120-4ofFIG. 1so that calibration operations can occur across multiple data buses at the same time. For example, the memory controller101can calibrate multiple data buses (e.g., data bus140-1and data bus145-1), while at the same time transferring data with the memory devices120through the other data buses (e.g., data bus140-2and data bus145-2).

Upon reading this disclosure, those of ordinary skill in the art will appreciate still alternative structural and functional designs for accessing memory during calibration operations through the disclosed principles of the present disclosure. Thus, while particular embodiments and applications of the present disclosure have been illustrated and described, it is to be understood that the disclosure is not limited to the precise construction and components disclosed herein. Various modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus of the present disclosure herein without departing from the spirit and scope of the disclosure as defined in the appended claims.