Patent ID: 12235783

DETAILED DESCRIPTION

Overview

A memory device may have an interconnected die architecture (e.g., a stacked or linked die architecture). This architecture uses at least one set of interconnected dice (or dies). The interconnected dice can include at least one interface die that is connected “internally” to at least one linked die within a package that houses two or more dice. The multiple dice can be coupled to a controller, such as a memory controller, via an interconnect that includes a command bus and a data bus. In some cases, the interface die and the linked die share access to the command bus, which may be realized as a command and address bus. Regarding the data bus, however, in certain aspects the interface die has “direct” access, but the linked die communicates data to and from the data bus “indirectly” via the interface die. The linked die can therefore use a second data bus to send data to and receive data from the interface die.

In certain scenarios, it can be challenging to train the multiple dice relative to using the command bus due to the interconnected die architecture. This document describes techniques to address these challenges by transmitting information that provides bus training feedback with a combination of bits detected by the linked die and the interface die. By selectively combining into the feedback information bits corresponding to each of the multiple dice, the multiple dice can be trained for a common bus even though they are in an interconnected die architecture. Further, the interconnected dice can be trained together instead of sequentially. Additionally or alternatively, the interconnected dice can be trained without instructing a die to refrain from participating in a bus training process, which is called masking a die. By avoiding masking a die, such described techniques can provide bus-training compatibility with memory systems that, for example, are unable to utilize multi-purpose commands (MPCs) during a bus training procedure. Examples of techniques and features for bus training with interconnected dice are described further below.

Generally, processors and memory work in tandem to provide features to users of computers and other electronic devices. An electronic device can provide enhanced features, such as high-resolution graphics or artificial intelligence, as a processor and memory operate more quickly together in a complementary manner. Some applications, like those for AI analysis and virtual-reality graphics, can also demand ever-greater amounts of memory. These applications use increasing amounts of memory to more accurately model and mimic human thinking and the physical world.

Processors and memories can be secured to a printed-circuit board (PCB), such as a rigid or flexible motherboard. The PCB can include sockets for accepting at least one processor and one or more memories. Wiring infrastructure that enables communication between two or more components can also be disposed on at least one layer of the PCB. The PCB, however, provides a finite area for the sockets and the wiring infrastructure. Some PCBs include multiple sockets that are each shaped as a linear slot and designed to accept a double-inline memory module (DIMM). These sockets can be fully occupied by DIMMs while a processor is still able to utilize more memory. In such situations, the system is capable of greater performance if additional memory were available to the processor.

Printed circuit boards may also include at least one peripheral component interconnect (PCI) express (PCI Express®) (PCIe or PCI-E) slot. A PCIe slot is designed to provide a common interface for various types of components that may be coupled to a PCB. Compared to some older standards, PCIe can provider higher rates of data transfer or a smaller footprint on the PCB, including both greater speed and smaller size. Accordingly, certain PCBs enable a processor to access a memory device that is connected to the PCB via a PCIe slot.

In some cases, accessing a memory solely using a PCIe protocol may not offer as much functionality, flexibility, or reliability as is desired. In such cases, another protocol may be layered on top of the PCIe protocol. An example of another, higher-level protocol is the Compute Express Link™ (CXL) protocol. The CXL protocol can be implemented over a physical layer that is governed by the PCIe protocol. The CXL protocol can provide, for instance, a memory-coherent interface that offers high-bandwidth or low-latency data transfers, including data transfers having both higher bandwidth and lower latency.

The CXL protocol addresses some of the limitations of PCIe links by providing an interface that leverages, for example, the PCIe 5.0 physical layer and electricals, while providing lower-latency paths for memory access and coherent caching between processors and memory devices. It offers high-bandwidth, low-latency connectivity between host devices (e.g., processors, CPUs, SoCs) and memory devices (e.g., accelerators, memory expanders, memory buffers, smart input/output (I/O) devices). The CXL protocol also addresses growing high-performance computational workloads by supporting heterogeneous processing and memory systems with potential applications in artificial intelligence, machine learning, communication systems, and other high-performance computing. With the potential to increase memory density by utilizing improved communication protocols, such as CXL, memory devices may be specified with additional design constraints that create new challenges for designers of memory devices.

Thus, memory devices may be implemented in different forms and deployed in various environments. For example, memory dice can be secured to a PCB of a motherboard (directly or as part of a DIMM) or can be enclosed within a CXL memory module. Consider, for instance, double data rate synchronous dynamic random-access memory (DDR SDRAM), including low-power DDR (LPDDR) SDRAM, such as LPDDR5. With the LPDDR5 standard, for instance, memory density may be so high that multiple dice are packaged together—e.g., in an integrated circuit package. In some of such dice packages, at least one die may not have direct access to one or more of the pins of the package that provide an interface to an exterior interconnect. Examples of such architectures are described next.

The many different formats of memory, such as an LPDDR5 DIMM or a CXL memory module, may include multiple dice. The multiple dice that are packaged together may form a memory device with an interconnected die architecture (e.g., a stacked-die architecture or a linked-die architecture). An interconnected-die memory device includes at least one set of interconnected dice, such as an interface die and a linked die. Although described herein primarily in terms of interconnected dice that are packaged together, interconnected dice may instead be packaged separately. The interface die can “directly” send data to or receive data from a memory controller or other component over a data bus of an interconnect. In contrast, the linked die “indirectly” sends data to or receives data from the memory controller or other component through the interface die using a second bus, which may be a data bus that is internal to a package including the dice. Explained another way, the interface die can act as an interface with respect to data being passed between the linked die and the memory controller on a data bus that is coupled between the interface die and the memory controller. The interface and linked dice, however, may share joint access to a command bus, an address bus, or a combination thereof (e.g., a command address bus or a command and address bus). Alternatively, the interface die may act as an interface for the linked die with respect to a command/address bus that propagates command or address information. In some of such cases, the two dice may share joint access to a data bus.

When two entities, such as a memory controller and a memory device, communicate across a bus or interconnect, a first entity signals to a second entity using a voltage and/or a current driven on an electrical conductor. Some period of time elapses for the voltage or current value of the signal to propagate along the bus from the first entity to the second entity. Another time period elapses for the signal to affect circuitry at the second entity to an extent that the circuitry can detect the voltage or current value as information. Such information can be realized as one or more bits representing, for example, a command, an address, or data (e.g., a datum or data item). Typically, the voltage or current value is valid for a finite period of time. If the second entity fails to latch, secure, or otherwise detect the voltage or current during a suitable timeframe window in which the information is valid, the second entity can obtain incorrect information. Accordingly, there is a timing aspect associated with correctly receiving a signal over a bus.

Establishing a timing parameter for correctly receiving signaling over a bus is called bus training (BT). This document addresses, at least in part, bus training with interconnected dice. Consider bus training in which a memory controller is transmitting a signal, and a memory device is receiving the signal. Further, assume that a command bus, for instance, is being trained. To perform command bus training (CBT) in this scenario, the memory controller transmits a signal having multiple bits as a test pattern on the command bus to the memory device. The memory device detects the multiple-bit signal according to a first timing parameter.

In response to the test pattern, as part of the CBT procedure, the memory device transmits the detected values of the multi-bit signal as a feedback pattern to the memory controller over a data bus to provide feedback for the CBT analysis. If the detected values as included in the feedback pattern match the bit values of the test pattern as transmitted, the memory controller can instruct the memory device to lock in the first timing parameter for receiving on the command bus. On the other hand, if the two sets of values or patterns do not match, the memory controller can repeat the testing and feedback process with the memory device using a different, second timing parameter and one or more test patterns. The CBT procedure can continue until a suitable timing parameter is ascertained. It should be understood that the process can be more complex. For example, to fully determine a suitable timing parameter, the memory controller may transmit various bit patterns to ensure that the current timing parameter works with a variety of multi-bit signals.

In one approach to training a bus, including command bus training, each die of multiple interconnected dice can be individually trained with respect to the bus by excluding one or more other dice of the multiple interconnected dice from the training process. The other die (or dice) can be excluded by causing the other die to be masked. Thus, the other die can refrain from receiving an incoming training communication or at least decline to respond to the training communication based on a masking instruction. The masking instruction can be implemented using, e.g., a multi-purpose command (MPC).

The memory controller sends an MPC to the memory device instructing at least one die to be masked. The MPC may correspond, for instance, to a select-die access (SDA) command. The masked die refrains from responding to a bus-training test pattern, but the non-masked die returns feedback information responsive to the bus-training test pattern for the bus training process. In some situations, however, MPCs may not be available. For example, some memory systems or standards may not support an MPC. Alternatively, even if the relevant memory standard does support MPCs generally, MPCs may not be available in a relevant operational mode or scenario. For instance, during initialization, a physical (PHY) layer or PHY chip may not support the issuing of MPCs. Accordingly, a memory controller or memory system may not be able to rely, with certainty, on the availability of multi-purpose commands for bus training. Without a reliable technique for bus training, memory dice or an entire memory system may malfunction and/or produce data errors.

In another approach to training a bus, which approach may omit use of an MPC, multiple dice can be trained jointly or at least partially simultaneously. The multiple dice can include multiple interconnected dice, such as at least one interface die and at least one linked die. In example implementations, the memory controller transmits a signal test pattern on a bus, such as a command bus. In the examples described here, the command bus is common to the interface die and the linked die, so the interface die and the linked dice can each “directly” access the signal test pattern without obtaining the test pattern through another die. Each of the interface and linked dice latch a detected version of the signal test pattern, with each detected version including multiple bits. At least a portion of the detected bits are to be sent to the memory controller as feedback information for the bus training procedure.

In contrast with the command bus, for the primary examples described in this subsection, each of the interface die and the linked die do not have direct access to the data bus that is coupled to the memory controller. This data bus may be an “external” data bus for the multiple interconnected dice relative to a package containing the multiple dice. The interface die and the linked die can be coupled together, however, using a second or an “internal” data bus. Thus, the interface die can directly access the data bus, and the linked die can indirectly access the data bus via the interface die using the internal data bus. To provide joint feedback for the bus training procedure, at least a portion of the command bus bits that are detected by the linked die and at least a portion of the command bus bits that are detected by the interface die are combined. For example, for a seven-bit command bus, four bits from the detected command bus bits of the linked die can be combined with three bits from the detected command bus bits of the interface die to produce combined bits that are “jointly” detected on the command bus. The combined bits can thus provide a joint feedback pattern for multiple dice in the bus training procedure. In some cases, the bits can be combined using a pseudo-random technique or mechanism. Logic that implements the pseudo-random bit-combining may be part of the linked die, the interface die, another die, or separate from an interconnected die (e.g., disposed on a PCB).

Thus, the combined detected bits provide some feedback information regarding suitable timing for the linked die and some feedback information regarding suitable timing for the interface die. The interface die transmits the combined detected bits to the memory controller over the external data bus. Consequently, the PHY layer or chip, which may be coupled between the memory controller and the interface die, can receive a common bus training margin between, or relative to, the linked die and the interface die “automatically” in this manner. The memory controller can, for instance, interpret the combined detected bits as if they are detected bits from a single die. As such, the memory controller can test various signal patterns and timing parameters and change them based on whether the combined detected feedback bits match the transmitted test bits.

Accordingly, in some aspects, a memory controller can train a bus with respect to multiple memory dice, including multiple interconnected memory dice, using a same procedure that may be effective for training a bus with respect to a single memory die. Additionally, the bus training may be achieved without masking dice. Further, the bus training procedure can be accomplished without relying on an MPC, such as an SDA command. Described techniques can therefore be used to perform bus training even if MPCs are not available, or merely may not be available depending on an operating environment or a current operational mode (e.g., during initialization) of the memory system. In some implementations, memory-controller and/or PHY components that can achieve or comport with an LP5(x) compatibility mode can train a bus with respect to multiple interconnected dice without specialized circuitry.

Although some implementations are described above in terms of a memory controller and a memory device performing certain bus training techniques with regard to a command bus, other (e.g., non-memory) device or die types may alternatively perform the techniques with other bus types. Examples of non-memory device and die implementations are described further herein.

Example Operating Environments

FIG.1illustrates, at100generally, example apparatuses102that can implement bus training with interconnected dice. The apparatus102can be realized as, for example, at least one electronic device. Example electronic-device implementations include an internet-of-things (IoTs) device102-1, a tablet device102-2, a smartphone102-3, a notebook computer102-4(or a desktop computer), a passenger vehicle102-5(or other vehicle), a server computer102-6, a server cluster102-7that may be part of cloud computing infrastructure or a data center, and any portion thereof (e.g., a printed circuit board (PCB) or module component of a device).

Other examples of the apparatus102include a wearable device, such as a smartwatch or intelligent glasses; an entertainment device, such as a set-top box or streaming dongle, a smart television, a gaming device, or virtual reality (VR) goggles; a motherboard or blade of a server; a consumer appliance; a vehicle or drone, or the electronic components thereof; industrial equipment; a security or other sensor device; and so forth. Each type of electronic device or other apparatus can include one or more components to provide some computing functionality or feature that is enabled or enhanced by the hardware or techniques that are described herein.

In example implementations, the apparatus102can include at least one host device104, at least one interconnect106, and at least one memory device108. The host device104can include at least one processor114, at least one cache memory116, and at least one controller118. The memory device108may include at least one die110, such as a first die110-1and a second die110-2. Each die110may include at least one memory (not explicitly shown inFIG.1). The memory device108or the memory thereof may be realized with one or more memory types.

The memory of the memory device108may be realized, for example, with a dynamic random-access memory (DRAM) die or module, including with a three-dimensional (3D) stacked DRAM device, such as a high bandwidth memory (HBM) device or a hybrid memory cube (HMC) device. DRAM may include, for instance, synchronous DRAM (SDRAM) or double data rate (DDR) DRAM (DDR DRAM). The memory of the memory device108may also be realized using static random-access memory (SRAM). Thus, the memory device108may operate as a main memory or a cache memory, including as both. Additionally or alternatively, the memory device108may operate as storage memory. In such cases, the memory may be realized, for example, with a storage-class memory type, such as one employing 3D XPoint™ or phase-change memory (PCM), flash memory, a magnetic hard disk, or a solid-state drive (e.g., a Non-Volatile Memory Express® (NVMe®) device).

Regarding the host device104, the processor114can be coupled to the cache memory116, and the cache memory116can be coupled to the controller118. The processor114can also be coupled to the controller118directly (e.g., without going through a cache memory) or indirectly (e.g., via the cache memory116as depicted). The host device104may include other components to form, for instance, a system-on-a-chip or a system-on-chip (SoC). The processor114may include or comprise a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural network engine or accelerator, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) integrated circuit (IC), a communication processor (e.g., a modem or baseband processor), an SoC, and so forth.

In operation, the controller118(e.g., a memory controller) can provide a high-level or logical interface between the processor114and at least one memory device, such as a memory that is external to the host device104. The controller118can, for example, receive memory requests from the processor114and provide the memory requests to an external memory (e.g., a memory device108) with appropriate formatting, packaging, timing, reordering, and so forth. The controller118can forward to the processor114responses to the memory requests that the controller118receives from the external memory.

The controller118may communicate with multiple memory devices, or other types of devices—some of which may include one or more memory components. The controller118may also communicate with multiple memory or other devices over one or more interconnects, such as the interconnect106. Regarding connections that are external to the host device104, the host device104can be coupled to the memory device108via the interconnect106. The memory device108may be coupled to, or may include, a main memory or a storage memory, including both in some cases. Another device, such as a cache memory or a switch, may be coupled between the host device104and the memory device108and may be part of or separate from the interconnect106.

The depicted interconnect106, as well as other interconnects (not shown) that communicatively couple together various components, enables data to be transferred between two or more components of the various components. Interconnect examples include a bus, a switching fabric, a crossbar, one or more wires that carry voltage or current signals, and so forth. Each interconnect may be implemented as a unidirectional interconnect or a bidirectional interconnect. The interconnect106can be implemented as a parallel propagation pathway. For example, the interconnect106can include at least one command bus120(or command and address bus120) and at least one data bus122, each of which carries multiple bits of a particular item of information (e.g., a data byte) substantially simultaneously. As used herein, the multiple bits can be transmitted substantially simultaneously if, for example, the bits are communicated within a given clock period or half period, even if the individual bits are intentionally or inadvertently staggered slightly within the clock period or half period.

Alternatively, the interconnect106can be implemented as a serial propagation pathway that carries one bit of a particular item of information each clock cycle. For instance, the interconnect106can comport with a PCIe standard, such as version 4, 5, 6, or a future version. The interconnect106may include multiple serial propagation pathways, such as multiple lanes in a PCIe implementation.

The components of the apparatus102that are depicted inFIG.1represent an example computing architecture that may include a hierarchical memory system. A hierarchical memory system can include memories at different levels, with each level having a memory with a different speed, capacity, or volatile/nonvolatile characteristic. Thus, the memory device108may be described in terms of forming at least part of a main memory of the apparatus102. The memory device108may, however, form at least part of a cache memory, a storage memory, an SoC, and so forth of an apparatus102.

Although various implementations of the apparatus102are depicted inFIG.1and described herein, an apparatus102can be implemented in alternative manners. For example, the host device104may include multiple cache memories, including multiple levels of cache memory, or may omit a cache memory. A memory, such as the memory device108, may have a respective “internal” or “local” cache memory (not shown). In some cases, the host device104may omit the processor114and/or include other logic. Generally, the illustrated and described components may be implemented in alternative ways, including in distributed or shared memory systems. A given apparatus102may also include more, fewer, or different components than those depicted inFIG.1or described herein.

The host device104and any of the various memories may be realized in multiple manners. In some cases, the host device104and the memory device108may be located on separate blades or racks, such as in a server or data center computing environment. In other cases, the host device104and the memory device108can both be disposed on, or physically supported by, a same printed circuit board (PCB) (e.g., a rigid or flexible motherboard or PCB assembly). The host device104and the memory device108may also be integrated on a same IC or fabricated on separate ICs but packaged together.

A memory device108may also be coupled to multiple host devices104via one or more interconnects106and may be able to respond to memory requests from two or more of the multiple host devices104. Each host device104may include a respective controller118, or the multiple host devices104may share a common controller118. An example computing system architecture with at least one host device104that is coupled to a memory device108is described below with reference toFIG.2.

With continuing reference toFIG.1, however, the host device104and the memory device108can perform a bus training procedure. For example, the controller118can train at least one die110of the memory device108with respect to the command bus120or the data bus122. In some implementations, to support the bus training, the controller118includes bus training logic124, and the memory device108includes bus training logic112(BT logic112). Each respective die110can include a respective instance of bus training logic112. As shown, the first die110-1includes first bus training logic112-1, and the second die110-2includes second bus training logic112-2. Nonetheless, the bus training logic112of the memory device108may be distributed differently and/or may have a different quantity of instances of the logic.

Generally, the command bus120can be coupled to each die110or fewer than all the dice of the memory device108. Similarly, the data bus122can be coupled to each die110or fewer than all the dice of the memory device108. Two or more dice of the memory device108may also be coupled together via at least one “internal” bus, such as the bus126. Here, the bus126is not directly exposed to an interface (e.g., not directly coupled to pins or other contacts) of the memory device108or to the connections of the interconnect106.

In certain implementations, the dice110-1and110-2are each coupled to the command bus120without using another die (e.g., the dice may be “directly coupled” as used herein). In contrast, the first die110-1is directly coupled to the data bus122, but the second die110-2is indirectly coupled to the data bus122. More specifically, in these “indirectly-coupled” data-bus implementations, the second die110-2can communicate with the data bus122via the “internal” bus126using the first die110-1. In such cases, the bus126may be realized as a second data bus of the memory system. Examples of this architecture are described below with reference toFIGS.3,4,5,6-1,6-2, and7.

In a bus training procedure, the bus training logic124transmits a test pattern over a bus, such as the command bus120, to the memory device108. Generally, each die110can have a sufficiently different hardware structure with respect to a bus such that signal propagation delays may deviate for each die along the bus. Consequently, a suitable timing parameter for each die110may be different or limited. Additionally or alternatively, a timing parameter that is suitable for a die in one memory system may not be suitable for the corresponding die in another memory system. To determine suitable respective timing parameters across multiple dice, some approaches for bus training communicate with each die individually by masking one or more other dice that share the same bus. This can be problematic, however, if a component or a mode of operation limits the ability to mask a die. For example, during initialization with some LPDDR5-based memory systems, multi-purpose commands (MPCs), including a command to mask a die, can be unavailable.

To address such situations, in example implementations, the bus training logic112enables the bus training logic124to perform bus training “simultaneously” on multiple dice110-1and110-2without necessarily deviating from a process that trains a single die for bus utilization. To do so, an instance of the bus training logic112combines bits detected by each of multiple dice, such as the first and second dice110-1and110-2. The detected bits correspond to the test pattern transmitted by the bus training logic124of the controller118, but the detected bits may not match the test bits due to an unsuitable timing parameter. The combined bits correspond to a joint feedback pattern that represents the bits as detected by the memory device from the perspective of multiple dice.

Thus, a portion of the bits as detected by the first die110-1and a portion of the bits as detected by the second die110-2are selected by an instance of the bus training logic112. These bits may be selected based on a pseudo-random technique. The bus training logic112combines the selected bits into a multi-die feedback pattern. The bus training logic112transmits the combined feedback bits as the feedback pattern to the bus training logic124of the controller118over the interconnect106(e.g., over the data bus122). The bus training logic124can compare the transmitted test pattern to the joint feedback pattern. This analysis can result in a new timing parameter being used for a next test pattern round. Once the feedback pattern, which represents bit detections by multiple dice, matches the test pattern, the current or associated timing parameter can be established for use by the multiple dice.

In some implementations, the controller118can be realized as a memory controller that interfaces with the interconnect106using an SDRAM protocol or standard, such as a DDR Version 5 standard. In other implementations, the apparatus102operates with one or more protocols over the interconnect106. The apparatus102can operate, for example, a Compute Express Link™ (CXL) protocol across the interconnect106. In at least some of these cases, the apparatus102can overlay the CXL protocol on top of a PCIe protocol for the physical layer. Thus, the controller118can comport with a CXL standard or a PCIe standard, including comporting with both. Similarly, a controller (e.g., as shown inFIGS.2and4) at the memory device108can comport with a CXL standard or a PCIe standard, including with both. Examples of devices that comport with a CXL standard are described below with reference toFIG.4. As shown with respect toFIG.4, a CXL memory device may include a memory controller and a memory, with at least the memory including bus training logic112as described herein. Other circuitry, techniques, and mechanisms are also described below. Next, however, this document describes example computing architectures with one or more processors and a memory device.

FIG.2illustrates examples of a computing system200that can implement aspects of bus training with interconnected dice in conjunction with a memory device. In some implementations, the computing system200includes at least one memory device108, at least one interconnect106, and at least one processor202. The memory device108can include, or be associated with, at least one memory array206, at least one controller212, and at least one interface204. The at least one controller212can be communicatively coupled to the memory array206via at least one interconnect208(e.g., an “internal” interconnect). The memory array206and the controller212may be components that are integrated on a single semiconductor die or that are located on separate semiconductor dice (e.g., but still coupled to or disposed on a same PCB). Each of the memory array206or the controller212may also be distributed across multiple dices (or dies).

The memory device108can correspond, for example, to one or more of a cache memory, main memory, or storage memory of the apparatus102ofFIG.1. Thus, the memory array206can include an array of memory cells. These memory cells can include, but are not limited to, memory cells of Static Random-Access Memory (SRAM), Dynamic Random-Access Memory (DRAM), Synchronous DRAM (SDRAM), three-dimensional (3D) stacked DRAM, Double Data Rate (DDR) memory, low-power Dynamic Random-Access Memory (DRAM), Low-Power Double Data Rate (LPDDR) Synchronous Dynamic Random-Access Memory (SDRAM), phase-change memory (PCM), or flash memory.

The controller212can include any one or more of a number of components that can be used by the memory device108to perform various operations. These operations can include communicating with other devices, managing performance, modulating memory access rates, refreshing the memory array, training to use a bus, and performing memory read or write operations. For example, the controller212can include at least one register214, at least one receiver216, at least one transmitter218, and at least one instance of bus training logic112.

The register214may be implemented, for example, as one or more registers that can store information to be used by the controller212, by another part of the memory device108, or by a part of a host device104, such as a controller118as depicted inFIG.1. A register214may store, for instance, a mode value indicative of if a bus is being trained, a timing parameter that controls a latching of values from a bus, and so forth. The controller212may include more, fewer, different, and/or alternative components. Although depicted separately, the components of the controller212may be nested with respect to each other, may provide functionality or circuitry that is at least partially overlapping with another component, and so forth. In some cases, the receiver216or the transmitter218, including one or more instances of both, may be incorporated as part of the interface204.

The interface204can couple the controller212or the memory array206directly or indirectly to the interconnect106. The receiver216can receive information via the interconnect106, such as from a processor202. The transmitter218can transmit information onto the interconnect106. As shown inFIG.2, the register214, the receiver216, the transmitter218, and the bus training logic112can be part of a single component (e.g., the controller212). In other implementations, one or more of the register214, the receiver216, the transmitter218, or the bus training logic112may be implemented as separate components, which can be provided on a single semiconductor die or disposed across multiple semiconductor dice. These components of the controller212may be individually or jointly coupled to the interconnect106via the interface204.

The interconnect106may be implemented with any one or more of a variety of interconnects that communicatively couple together various components and enable commands, addresses, messages, packets, data, and/or other information to be transferred between two or more of the various components (e.g., between the memory device108and any of the one or more processors202). The information may be propagated over the interconnect106in a “raw” manner or using some form of encapsulation or packaging, such as with packets, frames, or flits. Although the interconnect106is represented with a single line or arrow inFIG.2, the interconnect106may include at least one bus, at least one switching fabric, at least one crossbar, one or more wires or traces that carry voltage or current signals, at least one switch, one or more buffers, at least one lane, and so forth. Accordingly, the interconnect106may contain two or more of any of these, such as three buses or a bus and a switching fabric.

In some aspects, the memory device108may be realized as a “separate” physical component relative to the host device104(ofFIG.1) or any of the processors202. Examples of physical components that may be separate include, but are not limited to, a printed circuit board (PCB), which can be rigid or flexible; a memory card; a memory stick; and a memory module, including a single in-line memory module (SIMM), a dual in-line memory module (DIMM), or a non-volatile memory express (NVMe) module. Thus, separate physical components may be located together within a same housing of an electronic device or a memory product, or such physical components may be distributed over a server rack, a data center, and so forth. Alternatively, the memory device108may be packaged or integrated with other physical components, including a host device104or a processor202, such as by being disposed on a common PCB, combined together in a single device package, or integrated into an SoC of an apparatus.

As shown inFIG.2, the one or more processors202may include a computer processor202-1, a baseband processor202-2, and an application processor202-3, which are coupled to the memory device108through the interconnect106. The processors202may each be, or may form a part of, a CPU, a GPU, an SoC, an ASIC, an FPGA, or the like. In some cases, a single “processor” can comprise multiple processing cores or resources, each dedicated to different functions, such as modem management, applications, graphics, central processing, neural network acceleration, or the like. In some implementations, the baseband processor202-2may include or be coupled to a modem (not shown inFIG.2) and may be referred to as a modem processor. The modem and/or the baseband processor202-2may be coupled wirelessly to a network via, for example, cellular, Wi-Fi®, Bluetooth®, ultra-wideband (UWB), near field, or another technology or protocol for wireless communication.

In various implementations, the processors202may be connected to different memories in different manners. For example, the processors202may be connected directly to the memory device108(e.g., via the interconnect106as shown). Alternatively, one or more of the processors202may be indirectly connected to the memory device108, such as over a network connection, through one or more other devices or components, and/or using at least one other additional interconnect. Each processor202may be realized similarly to the processor114ofFIG.1. Accordingly, a respective processor202can include or be associated with a respective controller, like the controller118depicted inFIG.1. Alternatively, two or more processors202may access the memory device108using a shared or system controller118. In any of such cases, the controller118may include bus training logic124(e.g., ofFIG.1).

Each processor202may also be separately connected to a respective memory. As shown, the computer processor202-1may be coupled to at least one DIMM210that is inserted into a DIMM slot of a motherboard. The DIMM210can be coupled to a memory controller (not shown), which may be part of the computer processor202-1. The DIMM210may be realized with a memory device108and/or include any of the components shown inFIG.2for a memory device108.

The apparatuses and methods that are described herein may be appropriate for memory that is designed for use with an SDRAM-compatible bus, a DDR-memory-related bus, a PCIe bus, and so forth. Thus, the described principles may be incorporated into a memory device with a PCIe interface. Further, the memory device can communicate over the interconnect106by overlaying a CXL protocol on the physical PCIe interface. An example of a memory standard that relates to CXL is promulgated by the Compute Express Link™ consortium and may include versions 1.0, 1.1, 2.0, and future versions. Thus, the host device104(e.g., ofFIG.1) or the memory device108, including both in some cases, may comport with at least one CXL standard. Accordingly, some terminology in this document may draw from one or more of these standards or versions thereof for clarity. The described principles, however, are also applicable to memories that comport with other standards, including earlier versions or future versions of such standards, and to memories that do not adhere to a public standard. Examples of systems that may include a PCIe interface and a CXL protocol overlay are described below with reference toFIG.4.

FIG.3illustrates an example memory device. An example memory module302includes multiple dice304. As illustrated, the memory module302includes a first die304-1, a second die304-2, a third die304-3, and a Dth die304-D, with “D” representing a positive integer. As a couple of examples, the memory module302can be a SIMM or a DIMM. As another example, the memory module302can interface with other components via a bus interconnect (e.g., a Peripheral Component Interconnect Express (PCIe®) bus). The memory device108illustrated inFIGS.1and2can correspond, for example, to a single die304, multiple dice (or dies)304-1through304-D, or a memory module302having one or more dice304. As shown, the memory module302can include one or more electrical contacts306(e.g., pins) to electrically interface the memory module302to other components.

The memory module302can be implemented in various manners. For example, the memory module302may include a PCB, and the multiple dice304-1through304-D may be mounted or otherwise attached to the PCB. The dice304(e.g., memory dice) may be arranged in a line or along two or more dimensions (e.g., forming a grid or array of dice). The dice304may have a similar size to each other or may have different sizes. Generally, each die304may be similar to another die304or different in terms of size, shape, data capacity, or control circuitries. The dice304may also be positioned on a single side or on multiple sides of the memory module302. In some cases, the memory module302may be part of a CXL memory system or module.

In some implementations, two or more dice of the multiple dice304-1to304-D may be interconnected as stacked or linked dice. As shown, the first die304-1and the second die304-2are coupled together via a bus126, such as an “internal” data bus126that is not exposed to the electrical contacts306without an intervening die304. Although not so depicted inFIG.3, two or more of the dice, including at least two interconnected dice, may be packaged together (e.g., encapsulated together in plastic).

FIG.4illustrates examples of a system400that can include a host device104and a memory device108that are coupled together via an interconnect106. The system400can implement aspects of bus training with interconnected dice and may form at least part of an apparatus102as shown inFIG.1. As illustrated inFIG.4, the host device104includes a processor114and a controller118, which can be realized with at least one initiator402. Thus, the initiator402can be coupled to the processor114or to the interconnect106(including to both), and the initiator402can be coupled between the processor114and the interconnect106. Examples of initiators402may include a leader, a primary, a master, a requester or requesting component, a main component, and so forth.

In the illustrated example system400, the memory device108includes a controller422, which can be realized with at least one target404. The target404can be coupled to the interconnect106. Thus, the target404and the initiator402can be coupled to each other via the interconnect106. Examples of targets404may include a follower, a secondary, a slave, a subordinate, a responder or responding component, a subsidiary component, and so forth. The memory device108also includes a memory424. The memory424can be realized with at least one memory module, chip, or die having at least one memory array206(ofFIG.2) or another component, such as a DRAM410as is described below.

In example implementations, the initiator402includes at least one link controller412, and the target404includes at least one link controller414. The link controller412or the link controller414can instigate, coordinate, cause, or otherwise participate in or control signaling across a physical or logical link realized by the interconnect106in accordance with one or more protocols. The link controller412may be coupled to the interconnect106. The link controller414may also be coupled to the interconnect106. Thus, the link controller412can be coupled to the link controller414via the interconnect106. Each link controller412or414may, for instance, control communications over the interconnect106at a link layer or at one or more other layers of a given protocol. Communication signaling may include, for example, a request416, a response418, and so forth.

The memory device108may further include at least one interconnect406and at least one memory controller408(MC408). Within the memory device108, and relative to the target404, the interconnect406, the memory controller408, and/or the DRAM410(or other component of the memory424) may be referred to as a “backend” or “downstream” component or memory component of the memory device108. In some cases, the interconnect406is internal to the memory device108and may operate the same as or differently from the interconnect106or operate like the interconnect208.

Thus, the memory device108can include at least one memory component. As shown, the memory device108may include multiple memory controllers408-1and408-2and/or multiple DRAMs410-1and410-2. Although two of each are shown, the memory device108may include one or more than two memory controllers and/or one or more than two DRAMs. For example, a memory device108may include four memory controllers and 16 DRAMs, such as four DRAMs per memory controller. The memory424or memory components of the memory device108are depicted as DRAM410as an example only, for one or more of the memory components may be implemented as another type of memory. For instance, the memory components may include nonvolatile memory like flash or PCM. Alternatively, the memory components may include other types of volatile memory like SRAM. Thus, the memory device108may include a dynamic random-access memory (DRAM) array, a static random-access memory (SRAM) array, or a nonvolatile memory array. A memory device108may also include any combination of memory types.

In some cases, the memory device108may include the target404, the interconnect406, the at least one memory controller408, and the at least one DRAM410within a single housing or other enclosure. The enclosure, however, may be omitted or may be merged with one for the host device104, the system400, or an apparatus102(ofFIG.1). In some cases, each of these components can be realized with a separate IC. In some of such cases, the interconnect406can be disposed on a PCB. Each of the target404, the memory controller408, and the DRAM410may be fabricated on at least one IC and packaged together or separately. The packaged IC(s) may be secured to or otherwise supported by the PCB (or PCB assembly) and may be directly or indirectly coupled to the interconnect406. In other cases, the target404of the controller422, the interconnect406, and/or the one or more memory controllers408may be integrated together into one IC. In some of such cases, this IC may be coupled to a PCB, and one or more modules for the components of the memory424may also be coupled to the same PCB, which can form a CXL memory device108. This memory device108may be enclosed within a housing or may include such a housing. The components of the memory device108may, however, be fabricated, packaged, combined, and/or housed in other manners.

As illustrated inFIG.4, the target404, including the link controller414thereof, can be coupled to the interconnect406. Each memory controller408of the multiple memory controllers408-1and408-2can also be coupled to the interconnect406. Accordingly, the target404and each memory controller408of the multiple memory controllers408-1and408-2can communicate with each other via the interconnect406. Each memory controller408is coupled to at least one DRAM410. As shown, each respective memory controller408of the multiple memory controllers408-1and408-2is coupled to at least one respective DRAM410of the multiple DRAMs410-1and410-2. Each memory controller408of the multiple memory controllers408-1and408-2may, however, be coupled to a respective set of multiple DRAMs or other memory components.

Each memory controller408can access at least one DRAM410by implementing one or more memory access protocols to facilitate reading or writing data based on at least one memory address. The memory controller408can increase bandwidth or reduce latency for the memory accessing based on a type of the memory or an organization of the memory components, such as the multiple DRAMs. The multiple memory controllers408-1and408-2and the multiple DRAMs410-1and410-2can be organized in many different manners. For example, each memory controller408can realize one or more memory channels for accessing the DRAMs. Further, the DRAMs can be manufactured to include one or more ranks, such as a single-rank or a dual-rank memory module. Each DRAM410(e.g., at least one DRAM IC chip) may also include multiple banks, such as 8 or 16 banks.

A forward path of the memory device108may include one or more memory request queues (not shown). A return path of the memory device108may include one or more memory response queues (not shown). These queues may be present in, for example, the controller422, a memory controller408, a memory array, such as the DRAM410, and so forth. Examples of a forward path and a return path are described next as part of an accessing operation for the memory device108.

This document now describes examples of the host device104accessing the memory device108. The examples are described in terms of a general memory access (e.g., a memory request) which may include a memory read access (e.g., a memory read request for a data retrieval operation) or a memory write access (e.g., a memory write request for a data storage operation). The processor114can provide a memory access request452to the initiator402. The memory access request452may be propagated over a bus or other interconnect that is internal to the host device104. This memory access request452may be or may include a read request or a write request. The initiator402, such as the link controller412thereof, can reformulate the memory access request452into a format that is suitable for the interconnect106. This reformulation may be performed based on a physical protocol or a logical protocol (including both) applicable to the interconnect106. Examples of such protocols are described below.

The initiator402can thus prepare a request416and transmit the request416over the interconnect106to the target404. The target404receives the request416from the initiator402via the interconnect106. The target404, including the link controller414thereof, can process the request416to determine (e.g., extract, decode, or interpret) the memory access request. Based on the determined memory access request, and as part of the forward path of the memory device108, the target404can forward a memory request454over the interconnect406to a memory controller408, which is the first memory controller408-1in this example. For other memory accesses, the targeted data may be accessed with the second DRAM410-2through the second memory controller408-2. Thus, the first memory controller408-1receives the memory request454via the internal interconnect406.

The first memory controller408-1can prepare a memory command456based on the memory request454. The first memory controller408-1can provide the memory command456to the first DRAM410-1over an interface or interconnect appropriate for the type of DRAM or other memory component. An applicable memory standard includes, by way of example only, LPDDR5 for SDRAM. The first DRAM410-1receives the memory command456from the first memory controller408-1and can perform the corresponding memory operation. Based on the results of the memory operation, the first DRAM410-1can generate a memory response462. If the memory request416is for a read operation, the memory response462can include the requested data. If the memory request416is for a write operation, the memory response462can include an acknowledgement that the write operation was performed successfully. As part of the return path of the memory device108, the first DRAM410-1can provide the memory response462to the first memory controller408-1.

Continuing the return path of the memory device108, the first memory controller408-1receives the memory response462from the first DRAM410-1. Based on the memory response462, the first memory controller408-1can prepare a memory response464and transmit the memory response464to the target404via the interconnect406. The target404receives the memory response464from the first memory controller408-1via the interconnect406. Based on this memory response464, and responsive to the corresponding memory request416, the target404can formulate a response418for the requested memory operation. The memory response418can include read data or a write acknowledgement and be formulated in accordance with one or more protocols of the interconnect106.

To respond to the memory request416from the host device104, the target404of the memory device108can transmit the memory response418to the initiator402over the interconnect106. Thus, the initiator402receives the response418from the target404via the interconnect106. The initiator402can therefore respond to the “originating” memory access request452, which is from the processor114in this example. To do so, the initiator402prepares a memory access response466using the information from the response418and provides the memory access response466to the processor114. In these manners, the host device104can obtain memory access services from the memory device108using the interconnect106. Example aspects of an interconnect106are described next.

The interconnect106can be implemented in a myriad of manners to enable memory-related communications to be exchanged between the initiator402and the target404. Generally, the interconnect106can carry memory-related information, such as data or a memory address, between the initiator402and the target404. In some cases, the initiator402or the target404(including both) can prepare memory-related information for communication across the interconnect106by encapsulating such information. The memory-related information can be encapsulated or incorporated into, for example, at least one packet (e.g., at least one flit). One or more packets may include at least one header with information indicating or describing the content of each packet.

In example implementations, the interconnect106can support, enforce, or enable memory coherency for a shared memory system, for a cache memory, for combinations thereof, and so forth. Thus, the memory device108can operate in a cache coherent memory domain in some cases. Additionally or alternatively, the interconnect106can be operated based on a credit allocation system. Thus, the initiator402and the target404can communicate using, for example, a credit-based flow control mechanism. Possession of a credit can enable an entity, such as the initiator402, to transmit another memory request416to the target404. The target404may return credits to “refill” a credit balance at the initiator402. Credit logic of the target404or credit logic of the initiator402(including both instances of credit logic working together in tandem) can implement a credit-based communication scheme across the interconnect106.

The system400, the initiator402of the host device104, or the target404of the memory device108may operate or interface with the interconnect106in accordance with one or more physical or logical protocols. For example, the interconnect106may be built in accordance with a Peripheral Component Interconnect Express® (PCIe or PCI-E) standard. Applicable versions of the PCIe standard may include 1.x, 2.x, 3.x, 4.0, 5.0, 6.0, and future or alternative versions of the standard.

In some cases, at least one other standard is layered over the physical-oriented PCIe standard. For example, the initiator402or the target404can communicate over the interconnect106in accordance with a Compute Express Link™ (CXL) standard. Applicable versions of the CXL standard may include 1.x, 2.0, and future or alternative versions of the standard. Thus, the initiator402and/or the target404may operate so as to comport with a PCIe standard or PCIe protocol and/or a CXL standard or CXL protocol. A device or component may comprise or operate in accordance with a CXL Type 1, Type 2, or Type 3 device. A CXL standard may operate based on credits, such as request credits, response credits, and data credits.

In some implementations, bus training with interconnected dice can be employed in a CXL or analogous environment. For example, one or more components of DRAM410, such as at least one chip or die thereof, can include an instance of bus training logic112. As shown, the first DRAM410-1includes the first BT logic112-1, and the second DRAM410-2includes the second BT logic112-2. A data bus126that can couple together two or more dice in an interconnected manner within a memory package is also shown. A memory controller408may include an instance of bus training logic124(e.g., as shown inFIG.1) to interact with the bus training logic112. The relevant bus can be coupled between the memory controller408and the associated DRAM410. Example aspects of bus training between a memory controller and multiple dice are described next, starting withFIGS.5,6-1, and6-2.

Example Techniques and Hardware

FIG.5illustrates example schemes500for bus training with interconnected dice, which schemes500can include a memory controller504and a memory package502that at least partially encases multiple memory dice110-1and110-2. The memory controller504is an example of a controller118(e.g., ofFIG.1), and the memory package502is an example of a memory device108. As shown, the memory package502is coupled to the memory controller504via an interconnect506. The interconnect506is an example of an interconnect106(e.g., ofFIGS.1,2, and4). The memory controller504includes bus training logic124. The memory package502includes bus training logic112and at least two dice: a first die110-1and a second die110-2. The first and second dice110-1and110-2are coupled together via a bus126, such as a data bus126. The bus training logic112can be part of one of the dice, part of each of the dice, separate from all the dice, distributed across two or more of the dice, part of one die but separate from other dice, or some combination thereof.

In example implementations, the interconnect506includes at least one bus, such as a first bus and a second bus (not shown inFIG.5). To perform a bus training procedure for the first bus, the bus training logic124of the memory controller504transmits a test pattern508over the first bus of the interconnect506. The first die110-1receives the test pattern508via the first bus, and the second die110-2also receives the test pattern508via the first bus. In such cases, the first and second dice110-1and110-2may share the first bus. Depending on the timing of a detection of the test pattern508, each die110may latch or otherwise detect a different detection pattern510based on receiving a same test pattern508from the bus training logic124of the memory controller504.

The first die110-1transmits a first detected pattern510-1to the bus training logic112. The second die110-2transmits a second detected pattern510-2to the bus training logic112. In a connected die architecture, the second die110-2may use the data bus126and the first die110-1to transmit the second detected pattern510-2to the bus training logic112. The bus training logic112produces a feedback pattern512based on the first and second detected patterns510-1and510-2. For example, the bus training logic112can select one or more bits from the first detected pattern510-1and one or more bits from the second detected pattern510-2. The bus training logic112then combines the selected bits from the two detected patterns510-1and510-2to produce the feedback pattern512.

The bus training logic112transmits the feedback pattern512to the bus training logic124of the memory controller504over the interconnect506. In some cases, the bus training logic112transmits the feedback pattern512to the memory controller504over the second bus of the interconnect506. The memory controller504can receive the feedback pattern512via the second bus. The bus training logic124may continue the bus training based on the feedback pattern512. Thus, the bus training logic112of the memory package502and the bus training logic124of the memory controller504may exchange additional bus training communications514.

These additional bus training communications514may pertain to the bus training logic124transmitting a different test pattern508and the bus training logic112returning a different feedback pattern512based on the different test pattern508. These additional bus training communications514may include the bus training logic124of the memory controller504sending commands to the memory package502to change a timing of the detection of a received test pattern508. The detection timing may be changed, for instance, by adjusting an amount by which a clock signal is delayed before the clock signal triggers a latching of the received test pattern508.

FIGS.6-1and6-2illustrate example architectures600-1and600-2, respectively, for bus training with interconnected dice. The architectures600-1and600-2can include at least two dice110-1and110-2that share a bus120and that are coupled together by another bus126. As depicted explicitly inFIG.6-1, the architectures600-1and600-2can include a memory package502that includes at least a first die110-1and a second die110-2. In example implementations, the memory package502is coupled to two “external” buses: a command bus120and a data bus122. The command bus120is directly coupled to the first die110-1and the second die110-2. The data bus122, on the other hand, is directly coupled to the first die110-1but indirectly coupled to the second die110-2. In some cases, this indirect coupling between the second die110-2and the data bus122includes the first die110-1and a second data bus126.

In certain aspects, the first die110-1includes a first receiver602-1, a multiplexer604, a first transmitter606-1, and first bus training logic112-1. The second die110-2includes a second receiver602-2and second bus training logic112-2. The first die110-1and the second die110-2are coupled together via a second bus: the second data bus126. The second data bus126enables the two dice to communicate data therebetween to support operations, such as a memory read operation or a memory write operation. In some cases, the first die110-1can comprise or function as an interface die with respect to one or more other dice, such as the second die110-2. In such cases, the second die110-2may comprise or function as a linked die with respect to the first die110-1.

As illustrated for certain examples, the second receiver602-2is coupled between the command bus120and the second data bus126. The second data bus126is coupled between the second receiver602-2and the multiplexer604. The first receiver602-1, the multiplexer604, and the first transmitter606-1are coupled together in series between the command bus120and the data bus122. The multiplexer604is coupled between the first receiver602-1and the transmitter606-1. The second bus training logic112-2is coupled to the second receiver602-2and can control, at least partially, operations of the second receiver602-2. The first bus training logic112-1is coupled to the first receiver602-1, the multiplexer604, and the first transmitter606-1. Thus, the first bus training logic112-1can control, at least partially, operations of the first receiver602-1, the multiplexer604, or the first transmitter606-1. In alternative implementations, an instance of the bus training logic112may include one or more other components, such as one that is depicted or another component. For example, the first bus training logic112-1may include the multiplexer604.

In example operations, the first bus training logic112-1controls the functioning of the first die110-1relative to bus training. Similarly, the second bus training logic112-2controls the functioning of the second die110-2relative to bus training. To train for using the command bus120, the bus training logic124of the memory controller504(e.g., ofFIG.5) transmits multiple bits608over the command bus120. The second receiver602-2receives the multiple bits608via the command bus120and detects the multiple bits608as multiple second bits610-2.

The multiple second bits610-2may differ from the multiple bits608because, for instance, a timing of the latching of the multiple bits608may deviate from a suitable timing for the second receiver602-2. In other words, the second receiver602-2may latch the multiple bits608“slightly” too late or “slightly” too early to correctly detect them. The first receiver602-1receives the multiple bits608via the command bus120and detects the multiple bits608as multiple first bits610-1as part of a receiving operation. Similarly, the multiple first bits610-1may differ from the multiple bits608because, for instance, a timing of the latching of the multiple bits608may deviate from a suitable timing for the first receiver602-1.

Responsive to bit detection, the second bus training logic112-2causes the multiple second bits610-2to be forwarded over the second data bus126to the multiplexer604. The first receiver602-1forwards the multiple first bits610-1to the multiplexer604. The first bus training logic112-1can control operation of the multiplexer604to produce a set of bits612. The first bus training logic112-1can combine the multiple first bits610-1and the multiple second bits610-2to produce the set of bits612. For example, the first bus training logic112-1can combine the multiple first bits610-1and the multiple second bits610-2using at least the multiplexer604to produce the set of bits612.

The first bus training logic112-1controls the multiplexer604to select at least a portion of the bits from the multiple first bits610-1and at least a portion of the bits from the multiple second bits610-2. The selected portions are included in the set of bits612. Thus, the set of bits612can include a combination (e.g., a mixture) of the bits as detected by the second receiver602-2of the second die110-2and the bits as detected by the first receiver602-1of the first die110-1. Examples of the process to produce the set of bits612are described further below, including with reference toFIG.7. The multiplexer604provides the set of bits612to the transmitter606-1(e.g., the first transmitter606-1).

Under the control of the first bus training logic112-1, the first transmitter606-1transmits the set of bits612over the data bus122. Thus, the bus training logic124of the memory controller504(ofFIG.5) can receive the set of bits612via the data bus122. In these manners, the bus training logic112can provide feedback on a test pattern to the bus training logic124of the memory controller504with the feedback indicative of how multiple dice are jointly detecting the test pattern on the bus being trained. This enables the bus training logic124to operate as if one die is being trained for bus communications without masking other dice, even with multiple dice being trained substantially simultaneously. To continue the bus training procedure, the bus training logic124of the memory controller504analyzes the set of bits612to determine what additional communications or actions for bus training are to be performed next. Examples of this are described below with reference toFIG.8.

With reference toFIG.6-2, additional example aspects are depicted in the architectures600-2as compared to the architectures600-1ofFIG.6-1. As shown, the second die110-2includes a second transmitter606-2, and the first die110-1includes a third receiver602-3. The second data bus126is coupled between an output of the second transmitter606-2and an input of the third receiver602-3. Further, the first bus training logic112-1includes a delay unit644and a bit sequencer646. A clock signal642is also shown coupled to the delay unit644, the first receiver602-1, and the second receiver602-2.

The first and second receivers602-1and602-2detect (e.g., latch or secure a voltage or current level of) the multiple bits608responsive to the clock signal642. The clock signal642can have an associated timing parameter that establishes a delay of at least one edge (e.g., a rising edge or a falling edge) of the clock signal642. The delay may be instituted using, for instance, a variable quantity of delay units (not shown with respect to the clock signal642). Thus, the first and second receivers602-1and602-2can detect the multiple bits608received from the command bus120based on at least one edge of the clock signal642that occurs at least partially based on the timing parameter. The command bus120may have any bit-width, such 1, 2, 7, 16, and so forth. Accordingly, the multiple bits608may have any such quantity of bits.

To communicate the multiple second bits610-2between the second die110-2and the first die110-1, the second transmitter606-2accepts the multiple second bits610-2from the second receiver602-2. At least partially under the control of the second bus training logic112-2, the second transmitter606-2of the second die110-2transmits the multiple second bits610-2over the second data bus126to the third receiver602-3of the first die110-1. The third receiver602-3receives the multiple second bits610-2and forwards them to the multiplexer604.

In example implementations, the multiplexer604can include at least two inputs (e.g., a first input and a second input), an output, and a control input. The multiplexer604receives the multiple first bits610-1at the first input and the multiple second bits610-2at the second input. The multiplexer604produces the set of bits612for the feedback pattern512(e.g., ofFIG.5) based on the multiple first bits610-1and the multiple second bits610-2and responsive to a selection indication signal648. The first bus training logic112-1can generate the selection indication signal648as described below. The multiplexer604provides the set of bits612to the first transmitter606-1for transmission on the data bus122.

The first bus training logic112-1includes at least one delay unit644and the bit sequencer646. The delay unit644delays the clock signal642to enable one or more receivers (e.g., the first and second receivers602-1and602-2) or other circuitry to process the multiple bits608. For instance, the delay unit644can delay the clock signal642by a time period sufficient to enable the multiplexer604to be receiving the multiple first bits610-1and the multiple second bits610-2while a selection indication signal648for the corresponding multiple bits608is valid at the control input of the multiplexer604.

The bit sequencer646can operate responsive to the clock signal642, including a delayed version of the clock signal642. The bit sequencer646generates the selection indication signal648. The selection indication signal648indicates to the multiplexer604which one or more bits of the multiple first bits610-1and which one or more bits of the multiple second bits610-2are to be selected for inclusion in the set of bits612. In some cases, the bit sequencer646generates at least one value as the selection indication signal648using a pseudo-random number mechanism. Example operations of the bit sequencer646are described next with reference toFIG.7.

FIG.7illustrates, generally at700, examples of bit combination logic that combines bits of a test pattern as detected by multiple dice and that produces a combined feedback pattern. The components shown inFIG.7can be part of the first die110-1ofFIGS.6-1and6-2. In example operations, the first die110-1and the second die110-2each receives test bits702from the bus training logic124(e.g., of a memory controller504) via the command bus120(e.g., ofFIGS.6-1and6-2). As is described above with reference toFIGS.6-1and6-2, the second die110-2uses the second receiver602-2to detect the received test bits702as second detected bits704-2. The second die110-2uses the second transmitter606-2to transmit the second detected bits704-2to the third receiver602-3at the first die110-1, as is depicted inFIG.7.

The third receiver602-3forwards the second detected bits704-2to a second input of the multiplexer604. The first receiver602-1of the first die110-1detects the received test bits702at an input thereof as first detected bits704-1. The first receiver602-1forwards from an output thereof the first detected bits704-1to a first input of the multiplexer604. The multiplexer604therefore accepts or receives the first detected bits704-1and the second detected bits704-2at two inputs thereof. If there are two or more linked dice, or if more than two dice are interconnected as described herein, the multiplexer604may have more than two inputs.

In example implementations, the multiplexer604selects at least one bit (e.g., a first set of bits) from the first detected bits704-1and at least one bit (e.g., a second set of bits) from the second detected bits704-2based on one or more selection bits706from the bit sequencer646. In some cases, the bit sequencer646operates to exclude at least one bit of the multiple first detected bits704-1to produce a first set of bits that become part of the feedback bits708and/or to exclude at least one bit of the multiple second detected bits704-2to produce a second set of bits that become part of the feedback bits708. The multiplexer604provides the selected bits as feedback bits708. The first transmitter606-1then transmits the feedback bits708over the data bus122as part of a bus training procedure. The bit sequencer646can operate as a pseudo-random bit sequencer (e.g., can operate pseudo-randomly) that produces the selection bits706using at least one pseudo-random value. At least one pseudo-random number (PRN) can be generated by the first bus training logic112-1, including by the bit sequencer646, or by another component of the first die110-1. The PRN can also be generated or seeded in an alternative manner.

The PRN can be used to select one or more first bits from the first detected bits704-1and one or more second bits from the second detected bits704-2. For example, the bit sequencer646can choose how many bits from each group of detected bits704are to be selected based on the PRN. For instance, if each of the detected bits704have a quantity of 12, the PRN can indicate whether the split between two dice is to be 6 and 6 bits, 4 and 8 bits, 3 and 9 bits, and so forth. Additionally or alternatively, the bit sequencer646can use the PRN to determine which bits from each group of detected bits are to be selected. For instance, if each of the detected bits704have a quantity of five, a value of the PRN for a current clock cycle may determine that bit positions 1, 2, and 4 of the five bits of the first detected bits704-1and that bit positions 3 and 5 of the five bits of the second detected bits704-2are selected for inclusion in the feedback bits708.

Thus, the selection bits706can at least partially control production of the feedback bits708. The bit sequencer646may, however, use one or more pseudo-random values differently to produce the selection bits706, which bit-values at least partially control operation of the multiplexer604to produce the feedback bits708. Examples of bit selections, bit combinations, timing parameters or delays, bit test patterns, and bit feedback patterns are described next with reference toFIG.8and Table 1.

FIG.8illustrates a diagram800that includes multiple example test patterns for bus training in which different bit-detection timing parameters are depicted in conjunction with a selection of detected bits from two memory dice. In this example, a command and address (CA) bus with seven (7) bits is being trained. The diagram800therefore includes seven rows, one for each bit: CA<0>, CA<1>, CA<2>, CA<3>, CA<4>, CA<5>, and CA<6>. Nonetheless, the described principles are applicable to other bus types and to buses with a different quantity of bits. Two example test patterns are depicted across two columns: test pattern “A” and test pattern “B.” Each of the different bit patterns is applied across at least three time offsets or delays for the clock signal: T1, T2, and T3. The described principles are, however, applicable to more or fewer test patterns, different test patterns, and more or fewer time delays.

The diagram800is described with reference to Table 1. Examples of bit patterns, such as test patterns and feedback (FB) patterns, that can be exchanged between a memory controller and a memory device are presented below in Table 1.

TABLE 1Examples of test bit patterns and combinedfeedback bit patterns for multiple dice.Time IndexPattern APattern BTest PatternLHLLHLHResultHLHHLHHResultFB Pattern: T1LXHLLHHNOTHLLHXHXNOTMatchedMatchedFB Pattern: T2LHLLHLHMatchedHLHHLHHMatchedFB Pattern: T3HHLXHLLNOTXHHXLXXNOTMatchedMatched

As shown in the diagram800and presented in Table 1, the test pattern “A” is LHLLHLH, and the test pattern “B” is HLHHLHH. The “H” represents a high voltage (or current), and the “L” represents a low voltage (or current). The test patterns are listed in descending order from left to right of CA<6> to CA<0>. InFIG.8, for each row or CA bit, there are two signals per test pattern. An upper signal corresponds to the test pattern signal as received by one die, and a lower signal corresponds to the test pattern signal as received by another die.

In the diagram800, a selected bit is indicated at the intersection of each CA bit row and test pattern column by an encircled “S.” Thus, for the CA<2> bit and test pattern “B,” the upper signal is selected for inclusion in the feedback bits instead of the lower signal. For the CA<5> bit and test pattern “A,” the lower signal is selected for inclusion in the feedback bits instead of the upper signal. As described herein, the selection of the received signal or the detected bit may be made by the multiplexer604responsive to a selection indication signal648, which may be based on a pseudo-random value. In some cases, with two dice and seven bits on the CA bus, a quantity of selected detected bits can alternate between the two dice at three bits and four bits apiece.

In Table 1 above, the three rows correspond to the three example time indices T1, T2, and T3. Each row includes a feedback pattern generated at the memory device and an associated result (e.g., “Matched” or “NOT Matched”) respectively for each of the test patterns “A” and “B.” The feedback pattern includes a combination of bits from the detected bits of the “same” signal as received differently by the two dice. In Table 1, the feedback pattern is coded as follows. A non-emphasized, standard letter (e.g., “L” or “H”) indicates that the corresponding bit of the feedback pattern matches the test pattern. An underlined letter (e.g., “L” or “H”) indicates that the corresponding bit of the feedback pattern does not match the test pattern. A bolded “X” character (e.g., “X”) indicates that the corresponding bit of the feedback pattern is undetermined, such as if the signal is “detected” during a transition point.

The following specific examples pertain to the feedback pattern for the test pattern “A” at time T1. This “FB Pattern: T1” is “LXHLLHH,” which does not match the test pattern “A.” Consider the CA<0> bit for test pattern “A” at time T1 at802in the diagram800. This bit is indicated by the “H” character on the far right of the feedback pattern in Table 1. As indicated by the encircled “S,” the upper signal is selected for the feedback pattern for test pattern “A.” With this T1 timing parameter, the upper signal correctly detects a high “H” voltage level that matches with the corresponding bit of the test pattern.

Consider the CA<1> bit for test pattern “A” at time T1 at804in the diagram800. This bit is indicated by the “H” character one space from the far right of the feedback pattern in Table 1 for “FB Pattern: T1.” As indicated by the encircled “S,” the lower signal is selected for inclusion in the feedback pattern. With this T1 timing parameter, the lower signal incorrectly detects a high “H” voltage level while the correct voltage level that is intended to be transmitted for this bit of the test pattern is a low voltage level.

The other bits of the feedback patterns can be similarly determined to decode Table 1. As indicated in Table 1 by the upper row for time offset T1 and the lower row for time offset T3, the feedback patterns for these two time offsets do not match the corresponding test patterns. In contrast, as indicated by the middle row for time offset T2, the feedback patterns for this time offset do match the corresponding test patterns “A” and “B.” Accordingly, the bus training logic124of a memory controller504can command the bus training logic112of the memory (and/or general mode register logic) to establish a timing parameter corresponding to the time offset T2. Examples of processes and signal timings for bus training (BT), such as command bus training (CBT), are described next with reference toFIGS.9,10-1,10-2,11-1, and11-2.

FIG.9illustrates a flow chart for example methods900for performing a bus training procedure by a controller, such as a controller118(ofFIG.1), a memory controller408(ofFIG.4), or a memory controller504(ofFIG.5). By way of example but not limitation, the flow chart is directed to command bus training (CBT) for an LPDDR5-compatible memory. Nonetheless, the principles are applicable to bus training generally and/or to memories that are compatible with other standards. The example buses, bits, and times that are referenced by the flow chart ofFIG.9are further described and/or depicted inFIGS.10-1,10-2,11-1, and11-2.

At902, the controller issues a command to enter CBT mode 1 (or mode 2) with MR13 OP[6]=0B(1Bfor mode 2). At904, the controller issues at least one command with MRW1, MRW2 for MR16 OP[4]=1B. This command can be issued to all dice that are coupled to the bus being trained. At906, the controller waits for the following two time periods to transpire: tCBTWCKPRE_static+tWCK2DQ7H. Examples of these time periods are depicted relative to the DQ[7] bits inFIG.10-1.

At908, the controller causes the target dice to enter CBT by driving the signal (or multiple bit signals) of the DQ[7] bits high. For a x16 bus or memory configuration, the DQ[7] bits and the DQ[15] bits are driven from low to high to enter the CBT mode. At910, the controller waits for the following two time periods to transpire: tDQ7HWCK+tCAENT. Examples of these time periods are depicted relative to the DQ[7] bits inFIG.10-1. At912, the controller performs operations for the CBT by transmitting a test pattern on the CA bus and then asynchronously reading from the DQ bus the corresponding responsive feedback pattern.

At914, the controller can ascertain if a suitable timing margin (e.g., an optimal timing margin or one within specified or targeted parameters) has been determined. For example, the controller can compare the test pattern read from the DQ bus to the feedback pattern transmitted on the CA bus to determine if the two match. If not, then at916, the controller can change the test pattern and/or the timing delay and then repeat the test pattern transmission and the feedback pattern reception operations by continuing the process at912. If, on the other hand, a suitable margin has been ascertained, then at918, the controller can cause the target dice to exit the CBT mode by driving the DQ[7] signal low.

At920, the controller waits for the following two time periods to transpire: tDQ7HWCK+tXCBT. Examples of these time periods are depicted relative to the DQ[7] bits inFIG.11-1with regard to exiting a CBT mode. At922, the controller commands the target dice to write a trained timing parameter value with MRW1, MRW2 for MR16 OP[4]=0B. This command may be issued to all dice that are coupled to the bus being trained.

Examples of signaling between a memory controller and a memory device for a CBT procedure, which may correspond to the methods900of the flow chart inFIG.9, are presented below in Table 2.

TABLE 2Examples of a Bus Training Procedure.Bus TrainingStageSignal ActivityOperationPre-Command BusN/ADie masking modeTraining (CBT)obviatedCBTMRW13 OP[6]CBT Mode (1 or 2)MRW16 OP[4] = 1BNo FSP, CBT enabledMRW16 OP[6]VRCGMRW16 OP[7]CBT-PhaseDQ7 (DQ15) to HCBT-processMemory device returnscombined bits as afeedback pattern tomemory controller.DQ7 (DQ15) to LPost-CBTMRW16 OP[4] = 0BNormal Operation Mode

FIGS.10-1and10-2jointly illustrate an example timing diagram at1000-1and1000-2for entering a bus training mode. The timing diagram includes, for example, a CA bus, a command indication row (COMMAND), the DQ[7] bits under the control of a memory controller, and the DQ[6:0] bus under the control of the memory device. The timing diagram1000-1depicts, for instance, the mode register writes 1 and 2 at1002for MRW CBT entry. The timing diagram1000-1also depicts the driving low of the DQ[7] bits by the memory controller at1004.

At1006, the memory controller transmits a pattern “A” to the memory device over the CA bus. As described herein, bus training logic112(e.g., ofFIGS.1,2,4,5,6-1,6-2, and7) of the memory device produces a feedback pattern version of the test pattern “A,” with the feedback pattern including a combination of bits detected by two or more dice. In an environment in which an interface die (IF) and a linked die (Li) are present, logic of the interface die can combine selected bits that are detected by each of the interface and linked dice to produce the combined feedback pattern. At1008, the interface die transmits the combined feedback bits for the pattern “A” on the DQ[6:0] bus to the memory controller.

FIGS.11-1and11-2jointly illustrate an example timing diagram1100-1and1100-2for exiting a bus training mode. As also shown inFIG.10-1, the communication of the test pattern “A” is indicated at1006, and the return communication of the feedback pattern “A” is indicated at1008. At1102, the memory controller transmits a test pattern “B” to the memory device over the CA bus, and the memory device receives the pattern “B.” The memory device detects the bits of the pattern “B” based on some timing parameter across multiple dice.

After combining detected bits as selected from individual ones of the multiple dice, the memory device produces combined feedback bits. At1104, the interface die transmits the combined feedback bits for the pattern “B” over the DQ[6:0] bus to the memory controller. Further, the memory controller receives the combined feedback bits via the DQ[6:0] bus. The timing diagram1100-1additionally depicts, for example, the driving low of the DQ[7] bits by the memory controller at1106. After expiration of two time periods (tDQ7LWCK and tXCBT), the memory controller issues the mode register writes 1 and 2 at1108for the MRW CBT exit. Although specific aspects of example implementations are depicted in the timing diagrams and described above, other implementations may deviate from these timing diagrams.

Particular circuit implementations and hardware environments have been illustrated in the accompanying figures and described above. Nonetheless, the principles described with reference toFIGS.5to11-2, as well as the other figures, are applicable to other types of memory devices, communication exchanges, and/or environments. Although certain concepts are described herein in the context of LPDDR5 SDRAM, the described techniques can be applied to other memory device types or standards and/or to non-memory dice. Also, although certain concepts are described herein in the context of CXL Type 3 devices (“Memory Expanders”), the described techniques can be applied to other CXL device types and/or to non-CXL devices.

Example Methods

This section describes example methods with reference to the flow chart(s) and flow diagram(s) ofFIGS.12and13for implementing aspects of bus training with interconnected dice. These descriptions may also refer to components, entities, and other aspects depicted inFIGS.1to11-2, which reference is made only by way of example.

The processes1200ofFIG.12and the processes1300ofFIG.13may be performed by, for example, a memory device108, a memory module302, a DRAM410, a memory package502, bus training logic112, some combination thereof, and so forth. In a memory environment, for instance, the communications may be accomplished across a command bus and a data bus that are coupled between a memory device and a host device or memory controller. Although the operations are described with reference to components of a memory environment, the operations may be performed by circuitry that is not necessarily directed to memory but that implements bus training.

FIG.12illustrates a flow diagram for example processes1200that implement aspects of bus training with interconnected dice. The processes1200can include blocks1202to1212. At block1202, a first die receives multiple bits via a bus. For example, a first die110-1can receive multiple bits608via a bus. For instance, a first receiver602-1of the first die110-1may receive the multiple bits608from a command bus120, with the multiple bits608corresponding to a test pattern for a bus training procedure.

At block1204, the first die detects the multiple bits as multiple first bits based on the receiving by the first die. For example, the first die110-1can detect the multiple bits608as multiple first bits610-1based on the receiving by the first die110-1. In some cases, a timing parameter may establish a delay for an edge of a clock signal642that controls a timing of when the first receiver602-1latches a current value of a signal that carries the multiple bits608to obtain the multiple first bits610-1.

At block1206, a second die receives the multiple bits via the bus. For example, the second die110-2can receive the multiple bits608via the bus. To do so, a second receiver602-1of the second die110-2may receive the multiple bits608from the “same” command bus120.

At block1208, the second die detects the multiple bits as multiple second bits based on the receiving by the second die. For example, the second die110-2can detect the multiple bits608as multiple second bits610-2based on the receiving by the second die110-2. Here, the timing parameter may establish a same delay for an edge of the clock signal642that controls a timing of when the second receiver602-2latches a current value of the signal that carries the multiple bits608to obtain the multiple second bits610-2. Further, the second die110-2may transmit the multiple second bits610-2to bus training logic112.

At block1210, the multiple first bits and the multiple second bits are combined to produce a set of bits. For example, the bus training logic112can combine the multiple first bits610-1and the multiple second bits610-2to produce a set of bits612. The bus training logic112may be present at the first die110-1. If so, the first die110-1may combine one or more selected bits from the multiple first bits610-1and one or more selected bits from the multiple second bits610-2to produce the set of bits612as a feedback pattern.

At block1212, the set of bits is transmitted over a data bus. For example, the bus training logic112can transmit the set of bits612over a data bus122. This may be performed by a transmitter606-1of the first die110-1, and the transmitter606-1may transmit the set of bits612to bus training logic124at a controller.

FIG.13illustrates a flow diagram for other example processes1300that implement aspects of bus training with interconnected dice. The processes1300can include blocks1302and1304. At block1302, a memory device package receives multiple test bits. For example, a memory device package502can receive multiple test bits702as at least part of a test pattern508. For instance, a first die110-1may receive the multiple test bits702via a first bus and obtain first detected bits704-1based on the received multiple test bits702and at least one timing parameter, which may be realized with a clock signal642having a first timing. Similarly, a second die110-2may receive the multiple test bits702via the first bus and obtain second detected bits704-2based on the received multiple test bits702and the at least one timing parameter, which may be realized with the clock signal642having a second timing.

At block1304, the memory device package transmits multiple feedback bits, with the multiple feedback bits including first bits indicative of how a first die detected the multiple test bits and second bits indicative of how a second die detected the multiple test bits. For example, the memory device package502can transmit multiple feedback bits708as at least part of a feedback pattern512. Here, the multiple feedback bits708can include first bits (e.g., the first detected bits704-1) indicative of how the first die110-1detected the multiple test bits702and second bits (e.g., the second detected bits704-2) indicative of how the second die110-2detected the multiple test bits702.

In some cases, the memory device package502may transmit, from the first die110-1, the multiple feedback bits708together substantially in parallel over another bus that is external to the memory device package502. Further, the memory device package502may perform a bus training procedure by repeating the receiving and the transmitting for different values of the multiple test bits702and for different timings of the clock signal642that triggers detection of the multiple test bits702.

For the flow chart(s) and flow diagram(s) described above, the orders in which operations are shown and/or described are not intended to be construed as a limitation. Any number or combination of the described process operations can be combined or rearranged in any order to implement a given method or an alternative method. Operations may also be omitted from or added to the described methods. Further, described operations can be implemented in fully or partially overlapping manners.

Aspects of these methods may be implemented in, for example, hardware (e.g., fixed-logic circuitry or a processor in conjunction with a memory), firmware, software, or some combination thereof. The methods may be realized using one or more of the apparatuses, components, or other aspects shown inFIGS.1to7, the components or aspects of which may be further divided, combined, rearranged, and so on. The devices and components of these figures generally represent hardware, such as electronic devices, packaged modules, IC chips, or circuits; firmware or the actions thereof; software; or a combination thereof. Thus, these figures illustrate some of the many possible systems or apparatuses capable of implementing the described methods.

Unless context dictates otherwise, use herein of the word “or” may be considered use of an “inclusive or,” or a term that permits inclusion or application of one or more items that are linked by the word “or” (e.g., a phrase “A or B” may be interpreted as permitting just “A,” as permitting just “B,” or as permitting both “A” and “B”). Also, as used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. For instance, “at least one of a, b, or c” can cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c, or any other ordering of a, b, and c). Further, items represented in the accompanying figures and terms discussed herein may be indicative of one or more items or terms, and thus reference may be made interchangeably to single or plural forms of the items and terms in this written description.

CONCLUSION

Although implementations for bus training with interconnected dice have been described in language specific to certain features and/or methods, the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations for bus training with interconnected dice.