MEMORY SUBSYSTEM WITH NARROW BANDWIDTH REPEATER CHANNEL

A system with memory includes a repeater architecture where memory connects to a host with one bandwidth, and repeats the channel with a lower bandwidth. A memory circuit includes a first group of signal lines to couple point-to-point between a first group of memory devices and a host device. The memory circuit includes a second group of signal lines to couple point-to-point between the first group of memory devices and a second group of memory devices. The second group of signal lines extends the memory channel to the second group of memory devices. The second group of signal lines includes fewer data signal lines than the first group of signal lines, to support a lower bandwidth than the first group of signal lines on the memory channel.

FIELD

The descriptions are generally related to memory channels, and more particular descriptions are related to a narrow bandwidth backend channel from a channel repeater.

Portions of the disclosure of this patent document may contain material that is subject to copyright protection. The copyright owner has no objection to the reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The copyright notice applies to all data as described below, and in the accompanying drawings hereto, as well as to any software described below: Copyright ©2016, Intel Corporation, All Rights Reserved.

BACKGROUND

With ever-improving designs and manufacturing capability, processors continue to become more capable and achieve higher performance. As their capabilities increase, the demand for more functionality from devices increases. The increased functionality in turn increases processor bandwidth demand. The processor bandwidth demand is related to higher overall data throughput. Thus, the inability to move data into and out of the processor with higher bandwidths can impede the continuation of processor performance improvements.

Modern computing systems include on die memory storage, or cache which the processor can access relatively quickly, and main memory that has higher capacity but takes longer to access. Higher processor throughput means the desire for memory continues to increase, and systems incorporate more and more main memory capacity. Traditional connection of the main memory to the processor is via native memory channels. Native memory channels rely on a direct connection from the memory devices to a controller circuit or memory manager/driver on the host processor. Thus, traditional memory connection occurs through multidrop channels, where the signal lines of a memory channel extend from devices mounted physically closest to the processor to the devices mounted physically farthest from the processor. The devices connect in turn and drive the same signal lines.

Multidrop channels limit the number of memory devices that can be connected to a processor, which limits the memory capacity of a system. When more memory devices and memory modules are connected with native connections, the loading on the memory channel can degrade the communication over the bus. Thus, there is a tradeoff between increasing the speed and increasing the capacity in a memory subsystem. The tradeoff represents a potential limitation in the ability of memory bandwidth to continue to scale to processor performance. Current systems are already hitting the data rate limit for a native multidrop memory channel with two memory modules installed. While increasing the number of channels can help with the data rate problem, it is also not scalable with processor performance increases. Increasing channel bandwidth may also be impractical because of increasing the number of signal lines, as well as increasing device and connector size to accommodate the additional signal lines.

Descriptions of certain details and implementations follow, including a description of the figures, which may depict some or all of the embodiments described below, as well as discussing other potential embodiments or implementations of the inventive concepts presented herein.

DETAILED DESCRIPTION

As described herein, a system with memory includes a repeater architecture where memory connects to a host with one bandwidth, and repeats the channel with a lower bandwidth. Instead of a multidrop memory channel, such a system can employ a point-to-point memory channel. Whereas a multidrop memory channel has all memory devices or memory modules loading the same signal lines that couple directly to the host, a point-to-point memory channel introduces a buffer that repeats the channel. The buffer can result in the host seeing the loading of only a single connection, while the channel can be extended through the repeating to additional devices.

As described herein, a memory circuit includes a first group of signal lines to couple point-to-point between a first group of memory devices and a host device. The memory circuit includes a second group of signal lines to couple point-to-point between the first group of memory devices and a second group of memory devices. The second group of signal lines extends the memory channel to the second group of memory devices. The second group of signal lines includes fewer data signal lines than the first group of signal lines, to support a lower bandwidth than the first group of signal lines on the memory channel. Thus, a repeater or a buffer architecture can convert a multidrop connection into a point-to-point connection. The point-to-point connection can enable higher data rates, due to the decrease of capacitive loading that would otherwise degrade high-speed communication on the connection.

In one embodiment, a memory subsystem includes point-to-point channels to connect the memory modules to a central processing unit (CPU). A point-to-point channel can scale to much higher data rates than a multidrop channel traditionally used in memory standards. In one embodiment, the memory subsystem supports the connection of two memory modules per channel to a CPU, with a point-to-point channel connected between the CPU and the first module, and a repeater on the first memory module to provide a second point-to-point channel to the second module. In some descriptions, the connection can be described as a first point-to-point channel and a second point-to-point channel, or a primary channel and a secondary channel, or some other description. It will be understood that such descriptions are for purposes of illustration and refer to a memory channel that is repeated. Thus, reference to a second channel or a secondary channel can be understood as referring to a second portion of the memory channel or a secondary portion of the memory channel. Thus, a single memory channel can be considered to be separated into a first channel and an extension channel, or to be considered to be treated as separate sub-channels of the memory channel.

With the repeater channel architecture, the extended or repeated point-to-point channel is narrower than the first portion of the channel. More specifically, the second portion of the channel extending from a buffer or repeater has fewer signals lines than the portion of the channel between the host and the first point-to-point connection. By sharing the bandwidth to the CPU between a pair of memory modules, the second memory module can operate on a narrower channel to deliver requested bandwidth.

Reference is made throughout to sharing bandwidth between a first memory module and a second memory module. For purposes of simplicity in description, examples and embodiments are described with respect to a secondary channel having half the bandwidth of the primary channel. Such descriptions will be understood as non-limiting examples. While certain implementations can share half the bandwidth on a repeated channel, other implementations can share other portions of the bandwidth. As one example, the secondary channel could be allocated a third of the bandwidth, such as by alternating packets to/from the secondary memory module where every third memory access packet is allocated to the secondary memory module. It will be understood that other divisions and other portions of the memory channel could be shared in accordance with what is described herein. One of skill in the art will understand how to modify the system to share a portion of the bandwidth other than one half.

Furthermore, reference is made throughout to primary and secondary memory modules coupled to the same memory channel, with bandwidth shared between the memory modules. While connecting two memory modules or DIMMs to the same memory channel provides one example, it will be understood that it is also possible to extend the memory channel to additional memory modules over repeated channels in accordance with what is described herein. Thus, descriptions to a primary memory module can refer to a memory module that couples to a repeated channel, and further repeats the channel to another memory module. In such an implementation, the secondary module can be a secondary module from the perspective of the upstream memory module coupled closer to the host and can be a primary module from the perspective of a downstream memory module coupled farther from the host. The secondary memory module could thus repeat the channel on another repeated channel segment.

In a practical implementation of a system in which there are two memory modules coupled to a memory channel, it may not be practical to repeat any more than half the bandwidth. Thus, the repeated segment or portion may be up to half the native channel. In accordance with an embodiment where more than two memory modules are coupled to the same memory channel, the repeated portion may be more than half the bandwidth. For example, consider an example where each of three connected memory modules is allocated a third of total bandwidth. The repeater on the first memory module could repeat ⅔ bandwidth on a repeated channel, and the repeater on the second memory module could repeat ⅓ bandwidth on a second repeated channel. The repeater logic may be more complex and the timing may be more complex in such implementations, but it will be understood that such implementation can fall within the scope of what is described herein. Thus, descriptions below with reference to first and second memory modules and a repeated channel with half the bandwidth will be understood as non-limiting examples, and system configurations with more memory modules and/or with different portions of the native memory channel repeated can also be applied to the descriptions herein.

Reference to memory devices can apply to different memory types. Memory devices generally refer to volatile memory technologies. Volatile memory is memory whose state (and therefore the data stored on it) is indeterminate if power is interrupted to the device. Nonvolatile memory refers to memory whose state is determinate even if power is interrupted to the device. Dynamic volatile memory requires refreshing the data stored in the device to maintain state. One example of dynamic volatile memory includes DRAM (dynamic random access memory), or some variant such as synchronous DRAM (SDRAM). A memory subsystem as described herein may be compatible with a number of memory technologies, such as DDR3 (dual data rate version 3, original release by JEDEC (Joint Electronic Device Engineering Council) on Jun. 27, 2007, currently on release 21), DDR4 (DDR version 4, initial specification published in September 2012 by JEDEC), DDR4E (DDR version 4, extended, currently in discussion by JEDEC), LPDDR3 (low power DDR version 3, JESD209-3B, August 2013 by JEDEC), LPDDR4 (LOW POWER DOUBLE DATA RATE (LPDDR) version 4, JESD209-4, originally published by JEDEC in August 2014), WIO2 (Wide I/O 2 (WideIO2), JESD229-2, originally published by JEDEC in August 2014), HBM (HIGH BANDWIDTH MEMORY DRAM, JESD235, originally published by JEDEC in October 2013), DDR5 (DDR version 5, currently in discussion by JEDEC), LPDDR5 (currently in discussion by JEDEC), HBM2 (HBM version 2), (currently in discussion byJEDEC), and/or others, and technologies based on derivatives or extensions of such specifications.

In addition to, or alternatively to, volatile memory, in one embodiment, reference to memory devices can refer to a nonvolatile memory device whose state is determinate even if power is interrupted to the device. In one embodiment, the nonvolatile memory device is a block addressable memory device, such as NAND or NOR technologies. Thus, a memory device can also include a future generation nonvolatile devices, such as a three dimensional crosspoint memory device, or other byte addressable nonvolatile memory devices, or memory devices that use a chalcogenide phase change material (e.g., chalcogenide glass). In one embodiment, the memory device can be or include multi-threshold level NAND flash memory, NOR flash memory, single or multi-level Phase Change Memory (PCM), a resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, or spin transfer torque (STT)-MRAM, or a combination of any of the above, or other memory.

Descriptions herein referring to a “DRAM” can apply to any memory device that allows random access, whether volatile or nonvolatile. The memory device or DRAM can refer to the die itself and/or to a packaged memory product.

FIG. 1is a block diagram of an embodiment of a system with a repeater channel architecture having a lower bandwidth repeater channel to extend the memory channel to another memory module. System100includes elements of a memory subsystem in a computing device. Processor110represents a processing unit of a host computing platform that executes an operating system (OS) and applications, which can collectively be referred to as a “host” for the memory. The OS and applications execute operations that result in memory accesses. Processor110can include one or more separate processors. Each separate processor can include a single and/or a multicore processing unit. The processing unit can be a primary processor such as a CPU (central processing unit) and/or a peripheral processor such as a GPU (graphics processing unit). System100can be implemented as an SOC (system on a chip), or be implemented with standalone components.

Memory controller120represents one or more memory controller circuits or devices for system100. Memory controller120represents control logic that generates memory access commands in response to the execution of operations by processor110. Memory controller120accesses one or more memory devices140. Memory devices140can be DRAMs in accordance with any referred to above. In one embodiment, memory devices140are organized and managed as different channels, where each channel couples to buses and signal lines that couple to multiple memory devices in parallel. Each channel is independently operable. Thus, each channel is independently accessed and controlled, and the timing, data transfer, command and address exchanges, and other operations are separate for each channel. As used herein, coupling can refer to an electrical coupling, communicative, and/or a physical coupling. Physical coupling can include direct contact. Electrical coupling includes an interface or interconnection that allows electrical flow and/or signaling between components. Communicative coupling includes connections, including wireless, that enable components to exchange data.

In one embodiment, settings for each channel are controlled by separate mode registers or other register settings. In one embodiment, each memory controller120manages a separate memory channel, although system100can be configured to have multiple channels managed by a single controller, or to have multiple controllers on a single channel. In one embodiment, memory controller120is part of host processor110, such as logic implemented on the same die or implemented in the same package space as the processor.

Memory controller120includes I/O interface logic122to couple to a system bus and/or a memory bus, such as a memory channel as referred to above. I/O interface logic122(as well as I/O interface logic142of memory device140) can include pins, pads, connectors, signal lines, traces, wires, and/or other hardware to connect the devices. I/O interface logic122can include a hardware interface. As illustrated, I/O interface logic122includes at least drivers/transceivers for signal lines. Commonly, wires within an integrated circuit interface couple with a pad, pin, or connector to interface signal lines or traces or other wires between devices. I/O interface logic122can include drivers, receivers, transceivers, termination, and/or other circuitry to send and/or receive signal on the signal lines between the devices. The system bus can be implemented as multiple signal lines coupling memory controller120to memory devices140. The system bus includes at least clock (CLK)132, command/address (CMD) and write data (DQ)134, read DQ136, and zero or more other signal lines138. In one embodiment, a bus or connection between memory controller120and memory can be referred to as a memory bus. The signal lines for CMD can be referred to as a “C/A bus” (or ADD/CMD bus, or some other designation indicating the transfer of commands and address information) and the signal lines for write and read DQ can be referred to as a “data bus.” In one embodiment, independent channels have different clock signals, C/A buses, data buses, and other signal lines. Thus, system100can be considered to have multiple “system buses,” in the sense that an independent interface path can be considered a separate system bus. It will be understood that in addition to the lines explicitly shown, a system bus can include strobe signaling lines, alert lines, auxiliary lines, and other signal lines.

It will be understood that the system bus includes a command and write data bus134configured to operate at a bandwidth. In one embodiment, the command and write signal lines can include unidirectional lines for write and command data from the host to memory, and read DQ136can include unidirectional lines for read data from the memory to the host. In one embodiment, the data bus can includes bidirectional lines for read data and for write/command data. Based on design and/or implementation of system100, the data bus can have more or less bandwidth per memory device140. For example, the data bus can support memory devices that have either a x32 interface, a x16 interface, a x8 interface, or other interface. The convention “xW,” where W is a binary integer refers to an interface size of memory device140, which represents a number of signal lines to exchange data with memory controller120. The interface size of the memory devices is a controlling factor on how many memory devices can be used concurrently per channel in system100or coupled in parallel to the same signal lines.

Memory devices140represent memory resources for system100. In one embodiment, each memory device140is a separate memory die. In one embodiment, each memory device140can interface with multiple (e.g., 2) channels per device or die. Each memory device140includes I/O interface logic142, which has a bandwidth determined by the implementation of the device (e.g., x16 or x8 or some other interface bandwidth). I/O interface logic142enables the memory devices to interface with memory controller120. I/O interface logic142can include a hardware interface, and can be in accordance with I/O122of memory controller, but at the memory device end. In one embodiment, multiple memory devices140are connected in parallel to the same data buses. For example, system100can be configured with multiple memory devices140coupled in parallel, with each memory device responding to a command, and accessing memory resources160internal to each. For a Write operation, an individual memory device140can write a portion of the overall data word, and for a Read operation, an individual memory device140can fetch a portion of the overall data word.

In one embodiment, memory devices140are disposed directly on a motherboard or host system platform (e.g., a PCB (printed circuit board) on which processor110is disposed) of a computing device. In one embodiment, memory devices140can be organized into memory modules130. In one embodiment, memory modules130represent dual inline memory modules (DIMMs). In one embodiment, memory modules130represent other organization of multiple memory devices to share at least a portion of access or control circuitry, which can be a separate circuit, a separate device, or a separate board from the host system platform. Memory modules130can include multiple memory devices140, and the memory modules can include support for multiple separate channels to the included memory devices disposed on them.

Memory devices140each include memory resources160. Memory resources160represent individual arrays of memory locations or storage locations for data. Typically memory resources160are managed as rows of data, accessed via wordline (rows) and bitline (individual bits within a row) control. Memory resources160can be organized as separate channels, ranks, and banks of memory. Channels are independent control paths to storage locations within memory devices140. Ranks refer to common locations across multiple memory devices (e.g., same row addresses within different devices). Banks refer to arrays of memory locations within a memory device140. In one embodiment, banks of memory are divided into sub-banks with at least a portion of shared circuitry (e.g., drivers, signal lines, control logic) for the sub-banks. It will be understood that channels, ranks, banks, and/or other organizations of the memory locations can overlap physical resources. For example, the same physical memory locations can be accessed over a specific channel as a specific bank, which can also belong to a rank. Thus, the organization of memory resources will be understood in an inclusive, rather than exclusive, manner.

In one embodiment, memory devices140include one or more registers144. Register144represents one or more storage devices or storage locations that provide configuration or settings for the operation of the memory device. In one embodiment, register144can provide a storage location for memory device140to store data for access by memory controller120as part of a control or management operation. In one embodiment, register144includes one or more Mode Registers. In one embodiment, register144includes one or more multipurpose registers. The configuration of locations within register144can configure memory device140to operate in different “mode,” where command and/or address information or signal lines can trigger different operations within memory device140depending on the mode. Settings of register144can indicate configuration for I/O settings (e.g., timing, termination or ODT (on-die termination), driver configuration, and/or other I/O settings).

In one embodiment, memory device140includes ODT146as part of the interface hardware associated with I/O142. ODT146can be configured as mentioned above, and provide settings for impedance to be applied to the interface to specified signal lines. The ODT settings can be changed based on whether a memory device is a selected target of an access operation or a non-target device. ODT146settings can affect the timing and reflections of signaling on the terminated lines. Careful control over ODT146can enable higher-speed operation with improved matching of applied impedance and loading. ODT146can be applied to specific signal lines of I/O interface142,122, and is not necessarily applied to all signal lines.

Memory device140includes controller150, which represents control logic within the memory device to control internal operations within the memory device. For example, controller150decodes commands sent by memory controller120and generates internal operations to execute or satisfy the commands. Controller150can be referred to as an internal controller. Controller150can determine what mode is selected based on register144, and configure the access and/or execution of operations for memory resources160based on the selected mode. Controller150generates control signals to control the routing of bits within memory device140to provide a proper interface for the selected mode and direct a command to the proper memory locations or addresses.

Referring again to memory controller120, memory controller120includes command (CMD) logic124, which represents logic or circuitry to generate commands to send to memory devices140. Typically, the signaling in memory subsystems includes address information within or accompanying the command to indicate or select one or more memory locations where the memory devices should execute the command. In one embodiment, controller150of memory device140includes command logic152to receive and decode command and address information received via I/O142from memory controller120. Based on the received command and address information, controller150can control the timing of operations of the logic and circuitry within memory device140to execute the commands. Controller150is responsible for compliance with standards or specifications within memory device140, such as timing and signaling requirements. Memory controller120can also ensure compliance with standards or specifications by access scheduling and control.

In one embodiment, memory controller120includes refresh (REF) logic126. Refresh logic126can be used where memory devices140are volatile and need to be refreshed to maintain a deterministic state. In one embodiment, refresh logic126indicates a location for refresh, and a type of refresh to perform. Refresh logic126can trigger self-refresh within memory device140, and/or execute external refreshes by sending refresh commands. For example, in one embodiment, system100supports all bank refreshes as well as per bank refreshes, or other all bank and per bank commands. All bank commands cause an operation of a selected bank within all memory devices140coupled in parallel. Per bank commands cause the operation of a specified bank within a specified memory device140. In one embodiment, controller150within memory device140includes refresh logic154to apply refresh within memory device140. In one embodiment, refresh logic154generates internal operations to perform refresh in accordance with an external refresh received from memory controller120. Refresh logic154can determine if a refresh is directed to memory device140, and what memory resources160to refresh in response to the command.

In one embodiment, system100includes multiple memory modules130that include memory devices140. In one embodiment, memory modules130provide a repeater architecture for a memory channel. The memory channel can include clock132, command and write DQ134, read DQ136, and zero or more other control signals138. In a repeater architecture, memory module130[1] connects to memory controller120via buffer or repeater172of memory module130[0]. Thus, memory module130[0] has a point-to-point connection to memory controller120, and memory module130[1] has a point-to-point connection to memory module130[0]. In the repeater architecture described herein, memory module130[0] connects to memory controller120with a higher bandwidth connection than memory module130[1]. In one embodiment, buffer172provides an input point for all memory channel signals from memory controller120to implement the point-to-point connection, and provides clock, command/write, read, and control interfaces to all memory devices140of memory module130[0]. It will be understood that clock132and possibly other control signal lines138can be repeated via buffer172. For purposes of illustration, the command and write signal lines as well as the read signal lines on the repeated channel have different reference numbers (174and176, respectively) to identify that the repeated channel or repeated channel portion has a lower bandwidth.

For purposes of description, the point-to-point connection from memory module130[0] to memory controller120will be referred to as a primary channel or a primary connection, and the point-to-point connection from memory module130[1] to memory module130[0] will be referred to as a secondary channel or a secondary connection. In either case, it will be understood that memory controller120can treat a memory channel as having memory module130[0] and memory module130[1] connected to it. The timing of signaling to memory module130[1] may be different than the timing from memory module130[0].

As illustrated in system100, memory module130[0] has a native connection with memory controller120with N command and write DQ signal lines, and M read DQ signal lines. In one embodiment, M is greater than N, although in one embodiment M could be equal to N. Memory module130[1] does not have a native connection with memory controller120, and in one embodiment connects with N/2 command and write DQ signal lines174and M/2 read signal lines176. It will be understood that buffer172includes hardware to interface the primary connection to the secondary connection. In one embodiment, buffer172can include software or other logic to control the connection of command and write signal lines134to write signal lines174and/or of read signal lines136to read signal lines176.

With point-to-point connections, the loading on the memory channel at memory controller120is reduced relative to a multidrop architecture. With reduced loading, system100can operate the memory channel at a higher data rate than a corresponding multidrop memory channel. In one embodiment, the higher data rate can enable the use of fewer data pins to provide the same throughput. In one embodiment, the signaling timing on the repeated channel is the same as the signaling timing (e.g., the same data rate per signal line) on the primary channel. However, with a lower bandwidth connection or fewer signal lines, the overall data throughput on the repeated channel would be less than the data throughput on the primary channel due to the reduced bandwidth of the channel. The system can compensate for the lower bandwidth by increasing the number of cycles (e.g., unit intervals or UIs) used to transfer the data on the primary channel. For example, data sent on N signal lines over X cycles might be sent over 2X cycles on N/2 signal lines.

Traditional memory systems included only bidirectional data channels, which allowed only a single operation on the data bus. In one embodiment, system100includes separate write and read data buses, including unidirectional signal lines. Such unidirectional signal lines can enable a read to be performed with one memory module130while a write is performed with the other memory module130.

In one embodiment, system100includes a first group of signal lines to couple point-to-point as a memory channel, such as the signal lines between memory controller120or other host device and memory module130[0]. Memory module130[0] can be considered a first group of memory devices140, referring to the memory devices on the memory module. System100includes a second group of signal lines to couple point-to-point between memory module130[0] and memory module130[1], which can be considered a second group of memory devices140. Thus, buffer172can extend the memory channel to memory module130[1], but with fewer data lines and a narrow or lower bandwidth. Buffer172includes a repeater that can repeat signals from the host on the first group of signal lines to the second group of signal lines, and repeat signal from the memory devices on the second group of signal lines to the host. In general, a buffer enables the point-to-point connection functionality, and provides a single instance of loading on the memory channel for multiple connected memory devices, which can include devices on a repeated channel. A repeater enables the repeating of the memory channel to another memory module and/or group of memory devices.

In one embodiment, the repeated channel from memory module130[0] to memory module130[1] includes half as many data signal lines as the primary channel from memory controller120to memory module130[0]. In one embodiment, buffer172provides for approximately one half the total bandwidth of the primary channel for use by memory module130[0], and approximately one half the total bandwidth of the primary channel to be repeated on the secondary channel for use by memory module130[1]. In one embodiment, depending on the configuration of the primary and secondary channels, buffer172can provide more than half the available bandwidth to memory module130[0], and thus use more bandwidth than memory module130[1].

FIG. 2Ais a block diagram of an embodiment of a system with a repeater channel architecture in which dual data rate (DDR) modules occupy both primary and secondary positions on the lower bandwidth repeated channel. System202provides one example of a system in accordance with system100ofFIG. 1. System202includes central processing unit (CPU)210, which represents a host or host device to which memory resources connect. CPU210includes processing resources to perform operations in system202. CPU210also includes a memory controller circuit or equivalent circuitry and logic to drive commands to the memory resources and receive data from the memory resources.

As illustrated, system202includes 2X separate channels. In one embodiment, multiple channels connect to the same memory modules, for example in an implementation where a memory module includes memory devices configured to couple to two separate channels. The number of channels in system202will be an even number, for example, 2, 4, 6, 8, or some other number of channels. It will be understood that a system can be designed to use an odd number of channels in an alternate embodiment. The placement in the drawing of the channels is merely for illustration, and is not necessarily representative of a practical implementation. In one embodiment, the channels connect memory device groups organized as memory modules, such as DIMMs.

In system202, each channel includes two memory modules, a primary memory module and a secondary memory. As illustrated, all memory modules include the same type of memory devices. In accordance with the specific example, the memory modules include DDR DRAMs. System204ofFIG. 2Bincludes an example with memory modules having different types of memory, as described in more detail below. Referring again to system202, primary DDR module220[0] couples to CPU210with a bandwidth of R for read data and a bandwidth of W for commands and write data. It will be understood that R and W are not necessarily equal, but they could be in one implementation. Secondary DDR module220[0] couples to primary DDR module220[0] with a bandwidth of R/2 for read data and a bandwidth of W/2 for commands and write data. Thus, the bandwidth for the secondary connection has half the bandwidth as the primary connection on channel[0]. The remaining channels are similar.

There are known solutions for fully buffered DIMMs that use repeaters on the DIMM. Such solutions implement a full width channel between the first DIMM and the host, as well as a full width channel between the first and second DIMMs. However, in most cases the secondary DIMM does not require a full bandwidth connection, which means the secondary connection is overprovisioned in traditional implementations. System202includes a narrow repeated channel. Reducing the repeated channel bandwidth can reduce the number of signal pins required on a package of CPU210, and on corresponding connectors and PCB trace connections to the secondary DDR module. The reduction of signal pins and signal line traces results in lower silicon costs, lower power, and lower module connector pin counts.

In one embodiment, the X channels include unidirectional command and write data links and unidirectional read data links. In one embodiment, the width of the channel between the primary module and secondary module is half as wide as the channel width between CPU210and the primary module. In one embodiment, modules220include DIMMs with DRAMs or SDRAMs compliant with a DDR5 standard. In such an implementation, the primary and secondary DIMMs can each deliver approximately half of the CPU channel bandwidth as dictated by an even address distribution between the two modules. The read data link can deliver full read bandwidth from the memory modules to CPU210. In one embodiment, CPU210can read from the primary DIMM simultaneously with writing data to the secondary DIMM. In one embodiment, system202can bias bandwidth usage towards the primary DIMM. For example, system202can provide up to full bandwidth (all of R and/or all of W) to the primary DIMM, and provide whatever bandwidth is requested for the secondary DIMM.

FIG. 2Bis a block diagram of an embodiment of a system with a repeater channel architecture in which a dual data rate (DDR) module occupies a primary position and a nonvolatile memory occupies a secondary position on the lower bandwidth repeated channel. System204provides one example of a system in accordance with system100ofFIG. 1, and is an alternative to system202ofFIG. 2A. System204includes central processing unit (CPU)230, which represents a host or host device to which memory resources connect. CPU230includes processing resources to perform operations in system204. CPU230also includes a memory controller circuit or equivalent circuitry and logic to drive commands to the memory resources and receive data from the memory resources.

As illustrated, system204includes 2Y separate channels, where 2Y can be the same or different number as 2X. In one embodiment, multiple channels connect to the same memory modules, for example in an implementation where a memory module includes memory devices configured to couple to two separate channels. The number of channels in system204will be an even number, for example, 2, 4, 6, 8, or some other number of channels, but can be implemented with an odd number of channels in an alternate embodiment. In one embodiment, the channels connect memory device groups organized as memory modules, such as DIMMs.

In system204, each channel includes two memory modules, a primary memory module and a secondary memory. Whereas in system202all memory modules include memory devices of the same type of technology (e.g., DDR5 DRAMs), system204illustrates one example of memory modules of different types connected to the same channel. For example, using channel[0] as an illustration, channel[0] includes DDR module240[0] as the primary memory module connected to CPU230, and NV (nonvolatile) module240[0] as the secondary memory module connected to the primary module. In system204, primary DDR module240[0] couples to CPU230with a bandwidth of R for read data and a bandwidth of W for commands and write data. It will be understood that R and W are not necessarily equal, but they could be in one implementation. Secondary NV module240[0] couples to primary DDR module240[0] with a bandwidth of R/2 for read data and a bandwidth of W/2 for commands and write data. Thus, the bandwidth for the secondary connection has half the bandwidth as the primary connection on channel[0]. The remaining channels are similar.

In one embodiment, the Y channels include unidirectional command and write data links and unidirectional read data links. In one embodiment, the width of the channel between the primary module and secondary module is half as wide as the channel width between CPU210and the primary module. In one embodiment, primary DDR modules240include DIMMs with DRAMs or SDRAMs compliant with a DDR5 standard, and secondary DDR modules240includes DIMMs with nonvolatile, byte-addressable, random access memory devices. In one embodiment, CPU210can read from the primary DIMM simultaneously with writing data to the secondary DIMM. In one embodiment, system204can bias bandwidth usage towards the primary DIMM. For example, system202can provide up to full bandwidth (all of R and/or all of W) to the primary DDR memory module, and provide whatever bandwidth is needed for the DIMMs with the nonvolatile memory type.

In one embodiment, NV modules240include three dimensional memory devices including memory cells with a chalcogenide glass. For example, one implementation of nonvolatile memory devices includes three dimensional crosspoint (3DXP). If DDR modules240include DDR5 memory devices and NV modules240include 3DXP memory devices, the 3DXP operates more efficiently at a narrow bandwidth channel than a traditional native memory channel connection. DDR5 memory devices operate at higher speed than 3DXP and at lower power. But 3DXP operates at speeds near DDR memory, and is nonvolatile. Thus, system204provides both high speed, low power memory with somewhat lower speed, nonvolatile memory on the same memory channel.

In one embodiment, when DDR modules240include DDR5 DIMMs and NV modules240include 3DXP modules, system204shares bandwidth between the DDR5 memory and the 3DXP memory (e.g., via a buffer or repeater) on the DDR modules. In one embodiment, the DDR5 DIMM can deliver full channel bandwidth to CPU230when there is no 3DXP access demand. Due to the bandwidth headroom enabled by the dedicated write and read links, a significant amount of 3DXP bandwidth can be delivered to CPU230without reducing DDR5 bandwidth.

FIG. 3Ais a block diagram of an embodiment of a system with a repeater channel architecture in which both primary and secondary memory modules include a buffer. System302provides one example of a system in accordance with system100, and/or systems202or204. Host310represents logic to control operations in a system in which the memory is included. Host310includes processor316, which can include one or more processing resources that generate requests for access to data stored in memory. Memory controller318represents hardware and logic control access to memory in response to the data needs of processor316. In one embodiment, memory controller318is part of processor316. Host310includes interface312, which represents hardware such as pins, balls, pads, and/or other hardware to couple host310to the memory.

Host connector314represents hardware in system302to mount host310to a PCB or other substrate to interconnect host310with the memory resources of primary module330and secondary module350. In one embodiment, host310includes discrete components coupled to mounting locations on a PCB. In one embodiment, host310is incorporated in an SOC that mounts into a socket, and interface312can provide electrical and communicative coupling to the socket connector.

Host connector314connects to primary module connector334with W command and write signal lines322(which can also be referred to as a command/write link) and with R read signal lines324(which can also be referred to as a read link). Primary module330couples to primary module connector334. Primary module330includes memory devices336, which can be considered a first group of memory devices. Memory devices336represent memory resources to be accessed by requests from host310. In one embodiment, primary module330includes buffer338, which represents logic to enable a point-to-point connection from primary module connector334to host connector314, and a point-to-point connection from primary module connector334to secondary module connector354. Buffer338includes hardware components to provide the interconnections. In one embodiment, buffer338includes firmware and/or software logic to provide an interconnection.

In one embodiment, buffer338includes primary interface332A with a bandwidth of W command/write signal lines and R read signal lines for the primary channel connection to host310. In one embodiment, buffer338includes secondary interface332B with a bandwidth of W/2 command/write signal lines and R/2 read signal lines for the secondary channel connection to secondary module350. Interface332B can represent a repeater circuit of buffer338. It will be understood that in a practical implementation of a memory subsystem, there are limitations on interconnection based on the pin counts of connectors. Thus, it would impractical with traditional systems to repeat the memory channel from primary module330to secondary module350, at least because there would not be a high enough pin count available to support the connection. Thus, buffered DIMMs can repeat the signal within a DIMM, but traditionally memory channels are not repeated as point-to-point connections from one DIMM to another at least because of physical pin count limitations.

As provided in system302, the repeated channel from primary module330to secondary module350has fewer signal lines, which can enable a point-to-point connection without having to increase the connection pin count on current industry-standard connectors. Such a repeated channel includes fewer signal lines, which reduces the bandwidth of the channel. However, the reduced bandwidth can be compensated by increasing the signaling speed on the channel.

Thus, host connector314connects to secondary module connector354via buffer338with W/2 command and write signal lines342(which can also be referred to as a command/write link) and with R/2 read signal lines344(which can also be referred to as a read link). Secondary module350includes memory devices356, which can be considered a second group of memory devices. Memory devices356represent memory resources to be accessed by requests from host310. In one embodiment, secondary module350includes buffer358, which can be the same or similar to buffer338. Thus, buffer358can include primary interface352A and secondary interface352B. In one embodiment, buffer358includes a repeater similar to buffer338, which would enable buffer358to support a repeated channel to a next DIMM (not shown).

In the configuration of system302, in one embodiment, a single DIMM can be manufactured that can be placed in either primary module connector334or in secondary module connector354. The advantage to such an implementation is that the same type of DIMM can be connected in any available DIMM slot. Such a configuration may require the DIMM to be aware of its connection type. In one embodiment, secondary module350discovers that it is connected to the memory channel with W/2 signal lines for the command/write link and R/2 signal lines for the read link. In one embodiment, secondary module350configures internal routing and connections for use of the reduced bandwidth channel, which can include buffering read data to be sent on more UIs. Based on being connected to the lower bandwidth interface, at least some of primary interface352A and/or secondary interface352B will not be used by secondary module350.

In one embodiment, a practical implementation of system302can include a circuit board or circuit that includes host connector314, primary module connector334, and secondary module connector354. The signal lines for connection between host connector314and primary module connector334, and the signal lines for connection between primary module connector334and secondary module connector354can exist in the circuit even prior to connection of host310, primary module330, and/or secondary module350. Assume for a first configuration (e.g., a 1DPC (one DIMM per channel)) that host310and primary module330are connected. In such a configuration, in one embodiment, primary module330can utilize all bandwidth to host310. Assume then that a user connects secondary module350into system302. In such a case (e.g., a 2DPC (two DIMM per channel)), in one embodiment, buffer338can share the bandwidth between primary module330and secondary module350to host310. In one embodiment, in the 2DPC configuration system302can generate a higher throughput to host310based on shared utilization of the separate command/write link and read link. For example, host310can coordinate a write to one module while reading from the other.

In one embodiment, careful coordination of data addresses between primary module330and secondary module350can reduce stress on the modules. For example, with proper address location striping, half of the data can be stored on primary module330and half of the data on secondary module350. Greater distribution of the data results in less repeated access to a single module, which can result in lower power consumption per memory module. The lower power consumption can reduce the amount of throttling that might otherwise occur at a memory module. For example, when a threshold amount of traffic (read and/or write traffic) is focused on one module, the components tend to heat up, resulting in lower efficiency. To prevent overheating failures, a controller (not specifically shown) on the memory modules will cause throttling of data access by the host. Spreading data tends to spread the access, which can result in less stress on a single module, and therefore less throttling.

Theoretical predictions suggest peak performance with half of the total bandwidth provided to each memory module. In one embodiment, in system302, primary module330utilizes approximately one half the total memory channel bandwidth, and secondary module350utilizes approximately one half the total memory channel bandwidth. It will be understood that the total memory channel bandwidth refers to the memory channel connection at host310(e.g., via host connector314). While the total memory channel bandwidth is provided at primary module330at buffer338, secondary module350is only presented with half the total bandwidth.

With the reduced secondary command/write link and read link, system302favors lower pin count for higher implementation complexity. The higher complexity comes in management of the data accesses from host310to the memory modules. In one embodiment, memory controller318manages the data access complexity, for example, by following rules for timing and access frequency to each module. In one embodiment, instead of sharing half the bandwidth for each memory module, system302can be configured for primary module330to utilize more bandwidth than secondary module350.

FIG. 3Bis a block diagram of an embodiment of a system with a repeater channel architecture in which a primary memory module includes a buffer that supports a secondary channel, and a secondary memory module includes a buffer that does not provide a repeater function. System304represents an alternative to system302. System304includes host310with processor316, memory controller318, and interface312, connected to host connector314. The descriptions of these components above can be the same as what is described above with respect to system302. System304includes primary module330with memory devices336, buffer338, primary interface332A, and secondary interface332B, connected to primary module connector334. The descriptions of these components above can be the same as what is described above with respect to system302.

In one embodiment, system304includes secondary module360with memory devices366. Secondary module360connects to host connector314via buffer338with W/2 command and write signal lines342(which can also be referred to as a command/write link) and with R/2 read signal lines344(which can also be referred to as a read link). Secondary module360includes memory devices366, which can be considered a second group of memory devices. Memory devices366represent memory resources to be accessed by requests from host310.

Secondary module360is different from primary module330. Namely, secondary module360does not include a repeater. Buffer368does not repeat the connection of interface362. Secondary module360includes interface362to couple to secondary module connector364. In one embodiment, secondary module connection364is the same as secondary module connector354of system302. In one embodiment, secondary module360is different than primary module330because memory devices366are of a different type of memory than memory devices336. For example, secondary module360can be a different type of module with nonvolatile memory devices366. In one embodiment, secondary module360includes the same type of memory, in that memory devices366and memory devices336are the same type of memory, and secondary module360is specifically designed to be mounted in a secondary position.

FIG. 4Ais a block diagram of an embodiment of a system with a repeater channel architecture illustrating primary and secondary channel connections for a single primary channel on a memory module. System402represents elements of a memory subsystem in accordance with embodiments described herein. System402includes memory module410, which represents a memory module that repeats a memory channel with reduced bandwidth, in accordance with any embodiment described herein, such as a memory module of system100, a DDR module of systems202or204, or a primary module of systems302or304.

In one embodiment, memory module410includes DRAMs412, which represent memory device resources. The number of memory devices will be understood to vary based on the desired implementation for total capacity and density. Memory module410includes buffer420, which represents a buffer and/or repeater in accordance with any embodiment described herein. Buffer420enables memory module410to repeat a memory channel to a second memory module (not specifically shown) with a lower bandwidth connection.

In one embodiment, buffer420includes separate read and command/write connections. Buffer420sends read data to the host over a primary read channel from host read interface432. In one embodiment, host read interface432includes M data (DQ) signals and a clock (CLK) signal. In one embodiment, unidirectional signal lines provide the signals. In one embodiment, system402includes bidirectional lines to provide the signals. In one embodiment, buffer420receives read data from a secondary DIMM or next memory module connected on the memory channel over a secondary read channel via repeat read interface434. In one embodiment, repeat read interface434includes ½ M data signals and the clock signal.

Buffer420receives command and write data from the host over a primary command/write channel at host command/write interface436. In one embodiment, host command/write interface436includes N data signals and a clock signal. In one embodiment, unidirectional signal lines provide the signals. In one embodiment, system402includes bidirectional lines to provide the signals. In one embodiment, buffer420sends command/write data to the secondary DIMM or next memory module connected on the memory channel via repeat command/write interface438. In one embodiment, repeat command/write interface438includes ½ N data signals and the clock signal.

In one embodiment, system402applies different data rates (DR) to different aspects of exchanging data with memory module410. For example, interfaces432,434,436, and438are illustrated as operating at a data rate of DR. DRAMs412couple to buffer420at DRAM interface414. In one embodiment, buffer420exchanges signals with DRAMs412at a data rate of ½ DR. In one embodiment, the individual DRAMs412operate at a data rate of ¼ DR. In one embodiment, buffer420includes hardware interface logic such as the illustrated interfaces to route or exchange data to its target. In one embodiment, buffer420includes controller logic422to control the flow of data from primary channel connections to secondary channel connections and vice versa. In one embodiment, control logic422includes software logic.

FIG. 4Bis a block diagram of an embodiment of a system with a repeater channel architecture illustrating primary and secondary channel connections for first and second channels of a memory module. System404represents elements of a memory subsystem in accordance with embodiments described herein. System404includes memory module450, which represents a memory module that repeats a memory channel with reduced bandwidth, in accordance with any embodiment described herein.

In one embodiment, memory module450includes two channels, channel0and channel1. Channel0includes DRAMs452and channel1includes DRAMs456, which can be understood to be independent memory channels. DRAMs452and456represent memory device resources in memory module450. The number of memory devices will be understood to vary based on the desired implementation for total capacity and density. The memory devices can be split evenly among the two channels, although such a configuration is not necessary. In one embodiment, the separate channels include separate buffers, buffer460for channel0and buffer464for channel1. In one embodiment, the same buffer can service a narrow bandwidth repeated channel connection for both channels.

In one embodiment, channel0includes buffer460, which can be a buffer or repeater in accordance with any embodiment described herein. Buffer460enables memory module450to repeat channel0from memory module450to a second memory module (not specifically shown) with a lower bandwidth connection. In one embodiment, channel1includes buffer464, which can be a buffer and/or repeater in accordance with any embodiment described herein. Buffer464enables memory module450to repeat channel1from memory module450to a second memory module (not specifically shown) with a lower bandwidth connection, which can be the same or a different second memory module connected by repeated channel0.

In one embodiment, buffer460is similar or the same as buffer420of system402. In one embodiment, buffer460includes control logic462to control the flow of data between the host, memory module450, and a “next DIMM” or additional memory module connected to the repeated channel connection. In one embodiment, buffer460includes host read interface482to send read data to the host over the primary read channel for channel0(CH0). In one embodiment, the primary read channel for channel0has M data (DQ) signals and a clock signal, and operates at a data rate of DR. In one embodiment, buffer460includes repeat read interface484to receive read data from a secondary module over the secondary read channel for channel0. In one embodiment, the secondary read channel for channel0has M/2 data signals and a clock signal, and operates at a data rate of DR.

In one embodiment, buffer460includes host command/write (CMD/WR) interface486to receive commands and write data from the host over the primary command/write channel for channel0(CH0). In one embodiment, the primary command/write channel for channel0has N data signals and a clock signal, and operates at a data rate of DR. In one embodiment, buffer460includes repeat command/write interface488to send commands and write data to a secondary module over the secondary command/write channel for channel0. In one embodiment, the secondary command/write channel for channel0has N/2 data signals and a clock signal, and operates at a data rate of DR.

In one embodiment, buffer464is similar or the same as buffer460. In one embodiment, buffer464includes control logic466to control the flow of data between the host, memory module450, and a “next DIMM” or additional memory module connected to the repeated channel connection. In one embodiment, buffer464includes host read interface492to send read data to the host over the primary read channel for channel1(CH1). In one embodiment, the primary read channel for channel1has M data (DQ) signals and a clock signal, and operates at a data rate of DR. In one embodiment, buffer464includes repeat read interface494to receive read data from a secondary module over the secondary read channel for channel1. In one embodiment, the secondary read channel for channel1has M/2 data signals and a clock signal, and operates at a data rate of DR.

In one embodiment, buffer464includes host command/write (CMD/WR) interface496to receive commands and write data from the host over the primary command/write channel for channel1(CH1). In one embodiment, the primary command/write channel for channel1has N data signals and a clock signal, and operates at a data rate of DR. In one embodiment, buffer464includes repeat command/write interface498to send commands and write data to a secondary module over the secondary command/write channel for channel1. In one embodiment, the secondary command/write channel for channel1has N/2 data signals and a clock signal, and operates at a data rate of DR.

In one embodiment, system404applies different data rates (DR) to different aspects of exchanging data with memory module450. For example, external interfaces482,484,486,488,492,494,496, and498are illustrated as operating at a data rate of DR. DRAMs452couple to buffer460at DRAM interface454at a data rate of ½ DR. In one embodiment, the individual DRAMs452operate at a data rate of ¼ DR. Similarly, DRAMs456couple to buffer464at DRAM interface458at a data rate of ½ DR. In one embodiment, the individual DRAMs456operate at a data rate of ¼ DR.

FIG. 4Cis a block diagram of an embodiment of a system with a repeater channel architecture illustrating differential signaling for the primary and secondary channel connections. System406represents elements of a memory subsystem in accordance with embodiments described herein. System406includes memory module440, which represents a memory module that repeats a memory channel with reduced bandwidth, in accordance with any embodiment described herein, such as a memory module of system100, a DDR module of systems202or204, or a primary module of systems302or304. Memory module440provides an example of one embodiment of a memory module in accordance with module410of system402, or module450of system404.

Memory module440more specifically illustrates differential signaling with a memory subsystem. In one embodiment, system406multiplexes DRAM data onto narrow high speed differential links to the host. It will be understood that differential signaling within the memory subsystem can enable further data rate increases due to the improved signal quality of differential signaling. The improvements in signal quality would be beneficial to memory access operations, but differential signaling traditionally would require too many signal lines to fit within a DIMM slot connector. However, as described herein with a point-to-point differential data link and a repeated channel having lower bandwidth, a differential implementation can be implemented with existing memory module connector pin counts.

In one embodiment, memory module440includes DRAMs442, which represent memory device resources. The number of memory devices will be understood to vary based on the desired implementation for total capacity and density. Memory module440includes buffer424, which represents a buffer and/or repeater in accordance with any embodiment described herein. Buffer424enables memory module440to repeat a memory channel to a second memory module (not specifically shown) with a lower bandwidth connection. Both the primary channel and the repeated or secondary channel are differential.

In one embodiment, buffer424includes separate read and command/write connections. Buffer424sends read data to the host over a primary read channel from host read interface472, with a data rate of DR. In one embodiment, host read interface472includes M data (DQ) signal pairs and a clock (CLK) signal pair. In systems402and404, reference is made to a number of signals, which can refer to the number of signal lines, or the number of signal pairs, depending on the implementation of those systems. In one embodiment, buffer424receives read data from a secondary DIMM or next memory module connected on the memory channel over a secondary read channel via repeat read interface474, with a data rate of DR. In one embodiment, repeat read interface474includes ½ M data signal pairs and the clock signal pair.

Buffer424receives command and write data from the host over a primary command/write channel at host command/write interface476, with a data rate of DR. In one embodiment, host command/write interface476includes N data signal pairs and a clock signal pair. In one embodiment, buffer424sends command/write data to the secondary DIMM or next memory module connected on the memory channel via repeat command/write interface478, with a data rate of DR. In one embodiment, repeat command/write interface478includes ½ N data signal pairs and the clock signal pair.

In one embodiment, the outward facing interfaces of buffer424and memory module440include differential signal pairs for data. Outward facing interfaces refer to interfaces with devices external to memory module440. Thus, connections to the host and to the next DIMM are outward facing interfaces. In one embodiment, the inward facing interfaces of buffer424to the DRAMs is single-ended instead of differential. Thus, for example, DRAM interface444includes only single-ended signal lines instead of differential signal lines to couple to DRAMs442. In one embodiment, DRAMs442operate at a speed equivalent to ¼ DR, and couple to DRAM interface444with a data rate of ½ DR. In one embodiment, buffer424includes hardware interface logic such as the illustrated interfaces to route or exchange data to its target. In one embodiment, buffer424includes controller logic426to control the flow of data from primary channel connections to secondary channel connections and vice versa. In one embodiment, control logic426includes software logic.

FIG. 5is a block diagram of an embodiment of a system in which a memory controller schedules commands for a repeater channel architecture having primary and secondary channel segments of different bandwidth. System500represents an example of an embodiment of a system in accordance with system100ofFIG. 1. System500includes processor510, which includes hardware resources to execute processes512. Processor510can include one or more processing devices, which can include multicore processors. Processes512represent agent, threads, routines, software programs, or other software executed on processor510. Processes512generate requests for data stored in memory, to read and/or to write the data.

System500includes memory controller520, which can be a standalone component or integrated as a controller circuit of processor510. Memory controller520controls access to memory in response to the requests for data from processes512. Memory controller520includes scheduler522, which enables memory controller520to schedule the timing of requests to memory. System500includes at least two groups of memory devices, as represented by memory devices502of primary module530and memory devices504of secondary module540. Scheduler522manages the ordering of commands to memory devices502and504. Management of the ordering of the commands enables memory controller520to schedule access between the two separate groups of memory devices. While illustrated as separate groups of memory devices, in one embodiment, similar techniques could be used to separately access two memory devices with point-to-point connections.

In one embodiment, primary module530includes buffer532, which represents a buffer and/or repeater in accordance with any embodiment described herein. Buffer532receives commands and write data with host command/write interface552. The link to host command/write interface552has an associated bandwidth that can be the bandwidth of a native memory channel connection. In one embodiment, secondary module540includes buffer542, which represents a buffer that does not execute a repeat function. Buffer542receives commands and write data from primary module530with host command/write interface558, with a reduced bandwidth.

In one embodiment, buffer532includes command/write engine554to manage the repeat of the memory channel that enables a point-to-point connection to secondary module540. Buffer532can repeat command and write data traffic from the host to secondary module540. Command/write engine554can transfer commands and write data via backend command/write interface556, which is an interface to repeat the commands and data to secondary module540. Backend command/write interface556can be referred to as a repeat interface. Backend command/write interface556interfaces with host command/write interface558of secondary module540. In one embodiment, host command/write interface558is part of buffer542, but buffer542of secondary module540does not necessarily include a repeater in the buffer.

For read traffic, host read interface548of secondary module540enables the secondary module to send read data to memory controller520and processor510. In one embodiment, host read interface548is part of buffer542at secondary module540. Host read interface548interfaces with backend read interface546of buffer532of primary module530. In one embodiment, buffer532includes read engine544to manage the repeat of the memory channel that enables a point-to-point connection to secondary module540. Buffer532can repeat read data traffic from secondary module540to the host. Read engine544can transfer read data via host read interface542, which is an interface to memory controller520and processor510.

In one embodiment, memory controller520via scheduler522can be configured to schedule the data access to memory devices502and to memory devices504based on the management of traffic by read engine544and command/write engine554. Thus, scheduler522can schedule reads and writes to primary module530and to secondary module540based on expected timing to the memory devices, and available bandwidth to the modules.

FIG. 6is a block diagram of an embodiment of a timing representation for a repeater channel architecture in which primary and secondary channel segments have different bandwidths. System600includes host602, primary DIMM buffer604, and secondary DIMM606. System600includes a representation of timing between the host, the buffer, and the second DIMM. System600provides one example of an embodiment of a system in accordance with system100ofFIG. 1. System600illustrates an embodiment of a repeated channel architecture as described herein. In the specific example of system600, the secondary or backend channel is repeated with half the bandwidth as the primary or frontend channel. To adjust for the reduced bandwidth, the secondary channel may double the amount of time it takes to exchange data from the secondary module to the primary module.

Host602represents a host in accordance with any embodiment described herein, and includes at least logic to manage access to the memory. For example, host602can include a host processor and a memory controller circuit. Primary DIMM buffer604represents a buffer on a primary memory module, where the buffer provides a point-to-point connection to host602, and provides a point-to-point connection to a next connected memory module. Secondary DIMM606represents the next connected memory module, which has a point-to-point connection to primary DIMM buffer604.

As illustrated, host602generates clock612to control the timing of signaling across the memory channel. Primary DIMM buffer604repeats clock612to secondary DIMM606. The memory devices (not specifically shown) time their operations based on clock612, on both the primary and the secondary memory modules.

In one embodiment, system600includes separate read and write links. As illustrated, buffer604includes primary command/write interface620and primary read interface640, both of which interface with host602. Buffer604repeats the primary command/write channel to secondary DIMM606through secondary command/write interface630, and repeats the primary read channel to secondary DIMM606through secondary read interface650.

In one embodiment, primary command/write interface620includes M signal lines, and secondary command/write interface630includes M/2 signal lines. In one embodiment, primary read interface640includes N signal lines, and secondary read interface650includes N/2 signal lines. In one embodiment, the primary channel includes M command/write data signal lines and N read data signal lines, managed as independent interfaces. Similarly, the secondary channel can include M/2 command/write data signal lines and N/2 read data signal lines, managed as separate interfaces.

As illustrated, the primary channel operates over X UI, which can be referred to as a burst length (BL) for the memory access operation. For example, X can be 8 or 16 UI to transfer a sequence of 8 or 16 bits, respectively, on each signal line. Thus, a memory access operation (either read or write) includes the transfer of a total number of bits equal to the interface size times the burst length. Thus, commands and writes transfer X*M bits from host602to primary DIMM buffer604. If primary DIMM buffer604repeats the bits to secondary DIMM606, the transfer occurs over half as many command/write data signal lines (M/2), but over a longer burst cycle (2X). Thus, the total number of bits transferred is the same: (2X)*(M/2)=X*M.

It will be understood that system600illustrates the write bits as flowing from host602towards the memory modules, while the read bits are illustrated as flowing from the memory modules to host602. Reads transfer X*N bits from primary DIMM buffer604to host602. If a read access requests data bits from secondary DIMM606, the transfer includes the transfer of bits over N/2 read data signal lines for 2X UI for a total number of bits equal to (2X)*(N/2)=X*N. Primary DIMM buffer604then repeats the X*N bits over N signal lines for X UI to transfer the X*N bits to host602.

FIG. 7is a flow diagram of an embodiment of a process for accessing data in a repeated channel architecture with primary channel and secondary channel segments having different bandwidths. Process700illustrates a flow for data access in a repeated channel architecture. Process700provides one example of a flow in accordance with any repeated channel architecture described herein.

During operation of an electronic device, a host generates requests for data stored in memory resources coupled to a memory channel. The host can include a host processor or a peripheral processor, a host operating system (OS) that controls the hardware and software platforms of the electronic device, or a process or program operating from an application or service under the host OS. The host requests the data,702. A memory controller or equivalent circuitry generates a memory access request to send to the memory resources over a memory channel,704. The memory controller schedules the memory access to the memory resources,706. In one embodiment, the memory controller schedules the access requests in light of longer access times from memory resources on a secondary memory module. In one embodiment, the scheduling can account for different repeating techniques applied by a command and write engine for sending commands and data and/or different repeating techniques applied by a read engine for receiving read data. The write engine and read engine can be applied within a buffer or repeater in the primary memory module in accordance with what is described above.

In one embodiment, the system includes separate command/write data and read data links. In one embodiment, if the access request is for command or write data,708COMMAND/WRITE branch, the memory controller sends the scheduled command or write data on a full bandwidth point-to-point connection to the first memory module,710. The first memory module or the memory module coupled physically closest to the memory controller or host on the memory channel includes a buffer or repeater to extend the memory channel to a next memory module. In one embodiment, the buffer can determine if there is a second or next memory module coupled to the memory channel,712. If there is no second module,712NO branch, the buffer only distributes the request to local memory resources on the memory devices of the first memory module,714.

If there is a second memory module connected to the repeated memory channel,712YES branch, in one embodiment the buffer repeats the command or write data on a narrow bandwidth channel to the second memory module,716. Thus, the repeated channel has a different bandwidth than the link to the first memory module. The second memory module can then process the request and perform the required operations in accordance with the command and/or write data,718.

In one embodiment, if the access request is for read data,708READ branch, the memory controller sends the read request to the first memory module,720. It will be understood that sending a read request is done by sending a command, but for purposes of description here, the sending of the read request is illustrated in a read branch. In a practical implementation with separate read and command/write buses, the memory controller will typically send a read request on the command/write bus, which the first memory module either processes or forwards to the second memory module. The selected module then returns the data on an appropriate read bus.

Thus, if the buffer of the first memory module determines that there is either not a second memory module connected to the memory channel or that a received read request is addressed to the first memory module instead of the second memory module,722NO branch, the first memory controller receives read data sent from the first memory module itself on a full bandwidth channel,724. If the buffer determines that the read request is for a connected second memory module, the buffer forwards the request to the second memory module. The buffer subsequently receives read data on a narrow bandwidth channel from the second memory module,726. The buffer then repeats the read data to the controller over the full bandwidth channel,728.

FIG. 8is a flow diagram of an embodiment of a process for setting up a memory module in a system with a repeated channel architecture with primary channel and secondary channel segments having different bandwidths. Process800illustrates a flow for setting up memory for a narrow repeated channel architecture. Process800provides one example of a flow for a memory module in a repeated channel architecture in accordance with any embodiment described herein.

In one embodiment, the memory modules perform a discovery operation to determine where they are connected in the system. The memory modules can perform such a discovery on power up or other initialization, such as on a reset from a low power or off state, when the memory module configures itself for operation. In one embodiment, the memory module configures itself as part of a periodic configuration check. Thus, in one embodiment, a memory module configures,802. In one embodiment, the memory module determines if it is the primary memory module on a memory channel,804. In one embodiment, the memory controller makes the determination and sends configuration information to the memory module. In one embodiment, the memory module itself makes the determination via a handshake procedure with the memory controller.

If the memory module determines that it is not the primary memory module on the memory channel,806NO branch, in one embodiment the memory module configures itself for use of a narrow bandwidth interface to the memory channel,808. For example, the memory module includes a controller (separate from the memory controller of the host) that controls the exchange of data between the memory module and the host. The controller can configure for use of more UIs to send read data, or to receive commands and write data for more UIs.

If the memory module determines that it is the primary memory module on the memory channel,806YES branch, in one embodiment the memory module further tries to determine if there is a secondary memory module connected to the memory channel,810. If a secondary memory module is not also connected to the memory channel,812NO branch, in one embodiment the primary memory module utilizes all bandwidth of the memory channel point-to-point connection to the memory controller. If a secondary memory module is connected to the memory channel,812YES branch, in one embodiment the primary memory module shares bandwidth of the primary link with the secondary memory module,816. The primary memory module shares bandwidth with the secondary module through the operation of a channel repeater in accordance with any embodiment described herein.

FIG. 9is a block diagram of an embodiment of a computing system in which a repeated memory channel architecture can be implemented. System900represents a computing device in accordance with any embodiment described herein, and can be a laptop computer, a desktop computer, a server, a gaming or entertainment control system, a scanner, copier, printer, routing or switching device, or other electronic device. System900includes processor920, which provides processing, operation management, and execution of instructions for system900. Processor920can include any type of microprocessor, central processing unit (CPU), processing core, or other processing hardware to provide processing for system900. Processor920controls the overall operation of system900, and can be or include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of such devices. Processor920can execute data stored in memory932and/or write or edit data stored in memory932.

Memory subsystem930represents the main memory of system900, and provides temporary storage for code to be executed by processor920, or data values to be used in executing a routine. Memory subsystem930can include one or more memory devices such as read-only memory (ROM), flash memory, one or more varieties of random access memory (RAM), or other memory devices, or a combination of such devices. Memory subsystem930stores and hosts, among other things, operating system (OS)936to provide a software platform for execution of instructions in system900. Additionally, other instructions938are stored and executed from memory subsystem930to provide the logic and the processing of system900. OS936and instructions938are executed by processor920. Memory subsystem930includes memory device932where it stores data, instructions, programs, or other items. In one embodiment, memory subsystem includes memory controller934, which is a memory controller to generate and issue commands to memory device932. It will be understood that memory controller934could be a physical part of processor920.

Processor920and memory subsystem930are coupled to bus/bus system910. Bus910is an abstraction that represents any one or more separate physical buses, communication lines/interfaces, and/or point-to-point connections, connected by appropriate bridges, adapters, and/or controllers. Therefore, bus910can include, for example, one or more of a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (commonly referred to as “Firewire”). The buses of bus910can also correspond to interfaces in network interface950.

Power source912couples to bus910to provide power to the components of system900. In one embodiment, power source912includes an AC to DC (alternating current to direct current) adapter to plug into a wall outlet. Such AC power can be renewable energy (e.g., solar power). In one embodiment, power source912includes only DC power, which can be provided by a DC power source, such as an external AC to DC converter. In one embodiment, power source912includes wireless charging hardware to charge via proximity to a charging field. In one embodiment, power source912can include an internal battery or fuel cell source.

System900also includes one or more input/output (I/O) interface(s)940, network interface950, one or more internal mass storage device(s)960, and peripheral interface970coupled to bus910. I/O interface940can include one or more interface components through which a user interacts with system900(e.g., video, audio, and/or alphanumeric interfacing). In one embodiment, I/O interface940generates a display based on data stored in memory and/or operations executed by processor920. Network interface950provides system900the ability to communicate with remote devices (e.g., servers, other computing devices) over one or more networks. Network interface950can include an Ethernet adapter, wireless interconnection components, USB (universal serial bus), or other wired or wireless standards-based or proprietary interfaces. Network interface950can exchange data with a remote device, which can include sending data stored in memory and/or receive data to be stored in memory.

Storage960can be or include any conventional medium for storing large amounts of data in a nonvolatile manner, such as one or more magnetic, solid state, or optical based disks, or a combination. Storage960holds code or instructions and data962in a persistent state (i.e., the value is retained despite interruption of power to system900). Storage960can be generically considered to be a “memory,” although memory930is the executing or operating memory to provide instructions to processor920. Whereas storage960is nonvolatile, memory930can include volatile memory (i.e., the value or state of the data is indeterminate if power is interrupted to system900).

Peripheral interface970can include any hardware interface not specifically mentioned above. Peripherals refer generally to devices that connect dependently to system900. A dependent connection is one where system900provides the software and/or hardware platform on which operation executes, and with which a user interacts.

In one embodiment, system900includes channel repeater980. Channel repeater980can be implemented as a repeater and/or in a buffer on a primary memory module in accordance with any embodiment described herein. Channel repeater980provides a point-to-point connection for a primary connection to a host from a primary memory module or a memory module physically coupled closest to the host on a memory channel. Channel repeater980provides a secondary connection from the primary memory module to a secondary memory module, where the secondary connection has a lower bandwidth than the primary connection.

FIG. 10is a block diagram of an embodiment of a mobile device in which a repeated memory channel architecture can be implemented. Device1000represents a mobile computing device, such as a computing tablet, a mobile phone or smartphone, a wireless-enabled e-reader, wearable computing device, or other mobile device. It will be understood that certain of the components are shown generally, and not all components of such a device are shown in device1000.

Device1000includes processor1010, which performs the primary processing operations of device1000. Processor1010can include one or more physical devices, such as microprocessors, application processors, microcontrollers, programmable logic devices, or other processing means. The processing operations performed by processor1010include the execution of an operating platform or operating system on which applications and/or device functions are executed. The processing operations include operations related to I/O (input/output) with a human user or with other devices, operations related to power management, and/or operations related to connecting device1000to another device. The processing operations can also include operations related to audio I/O and/or display I/O. Processor1010can execute data stored in memory and/or write or edit data stored in memory.

In one embodiment, device1000includes audio subsystem1020, which represents hardware (e.g., audio hardware and audio circuits) and software (e.g., drivers, codecs) components associated with providing audio functions to the computing device. Audio functions can include speaker and/or headphone output, as well as microphone input. Devices for such functions can be integrated into device1000, or connected to device1000. In one embodiment, a user interacts with device1000by providing audio commands that are received and processed by processor1010.

Display subsystem1030represents hardware (e.g., display devices) and software (e.g., drivers) components that provide a visual and/or tactile display for a user to interact with the computing device. Display subsystem1030includes display interface1032, which includes the particular screen or hardware device used to provide a display to a user. In one embodiment, display interface1032includes logic separate from processor1010to perform at least some processing related to the display. In one embodiment, display subsystem1030includes a touchscreen device that provides both output and input to a user. In one embodiment, display subsystem1030includes a high definition (HD) display that provides an output to a user. High definition can refer to a display having a pixel density of approximately 100 PPI (pixels per inch) or greater, and can include formats such as full HD (e.g., 1080p), retina displays, 4K (ultra high definition or UHD), or others. In one embodiment, display subsystem1030generates display information based on data stored in memory and/or operations executed by processor1010.

I/O controller1040represents hardware devices and software components related to interaction with a user. I/O controller1040can operate to manage hardware that is part of audio subsystem1020and/or display subsystem1030. Additionally, I/O controller1040illustrates a connection point for additional devices that connect to device1000through which a user might interact with the system. For example, devices that can be attached to device1000might include microphone devices, speaker or stereo systems, video systems or other display device, keyboard or keypad devices, or other I/O devices for use with specific applications such as card readers or other devices.

As mentioned above, I/O controller1040can interact with audio subsystem1020and/or display subsystem1030. For example, input through a microphone or other audio device can provide input or commands for one or more applications or functions of device1000. Additionally, audio output can be provided instead of or in addition to display output. In another example, if display subsystem includes a touchscreen, the display device also acts as an input device, which can be at least partially managed by I/O controller1040. There can also be additional buttons or switches on device1000to provide I/O functions managed by I/O controller1040.

In one embodiment, I/O controller1040manages devices such as accelerometers, cameras, light sensors or other environmental sensors, gyroscopes, global positioning system (GPS), or other hardware that can be included in device1000. The input can be part of direct user interaction, as well as providing environmental input to the system to influence its operations (such as filtering for noise, adjusting displays for brightness detection, applying a flash for a camera, or other features).

In one embodiment, device1000includes power management1050that manages battery power usage, charging of the battery, and features related to power saving operation. Power management1050manages power from power source1052, which provides power to the components of system1000. In one embodiment, power source1052includes an AC to DC (alternating current to direct current) adapter to plug into a wall outlet. Such AC power can be renewable energy (e.g., solar power). In one embodiment, power source1052includes only DC power, which can be provided by a DC power source, such as an external AC to DC converter. In one embodiment, power source1052includes wireless charging hardware to charge via proximity to a charging field. In one embodiment, power source1052can include an internal battery or fuel cell source.

Memory subsystem1060includes memory device(s)1062for storing information in device1000. Memory subsystem1060can include nonvolatile (state does not change if power to the memory device is interrupted) and/or volatile (state is indeterminate if power to the memory device is interrupted) memory devices. Memory1060can store application data, user data, music, photos, documents, or other data, as well as system data (whether long-term or temporary) related to the execution of the applications and functions of system1000. In one embodiment, memory subsystem1060includes memory controller1064(which could also be considered part of the control of system1000, and could potentially be considered part of processor1010). Memory controller1064includes a scheduler to generate and issue commands to memory device1062.

Connectivity1070includes hardware devices (e.g., wireless and/or wired connectors and communication hardware) and software components (e.g., drivers, protocol stacks) to enable device1000to communicate with external devices. The external device could be separate devices, such as other computing devices, wireless access points or base stations, as well as peripherals such as headsets, printers, or other devices. In one embodiment, system1000exchanges data with an external device for storage in memory and/or for display on a display device. The exchanged data can include data to be stored in memory and/or data already stored in memory, to read, write, or edit data.

Connectivity1070can include multiple different types of connectivity. To generalize, device1000is illustrated with cellular connectivity1072and wireless connectivity1074. Cellular connectivity1072refers generally to cellular network connectivity provided by wireless carriers, such as provided via GSM (global system for mobile communications) or variations or derivatives, CDMA (code division multiple access) or variations or derivatives, TDM (time division multiplexing) or variations or derivatives, LTE (long term evolution—also referred to as “4G”), or other cellular service standards. Wireless connectivity1074refers to wireless connectivity that is not cellular, and can include personal area networks (such as Bluetooth), local area networks (such as WiFi), and/or wide area networks (such as WiMax), or other wireless communication. Wireless communication refers to transfer of data through the use of modulated electromagnetic radiation through a non-solid medium. Wired communication occurs through a solid communication medium.

Peripheral connections1080include hardware interfaces and connectors, as well as software components (e.g., drivers, protocol stacks) to make peripheral connections. It will be understood that device1000could both be a peripheral device (“to”1082) to other computing devices, as well as have peripheral devices (“from”1084) connected to it. Device1000commonly has a “docking” connector to connect to other computing devices for purposes such as managing (e.g., downloading and/or uploading, changing, synchronizing) content on device1000. Additionally, a docking connector can allow device1000to connect to certain peripherals that allow device1000to control content output, for example, to audiovisual or other systems.

In addition to a proprietary docking connector or other proprietary connection hardware, device1000can make peripheral connections1080via common or standards-based connectors. Common types can include a Universal Serial Bus (USB) connector (which can include any of a number of different hardware interfaces), DisplayPort including MiniDisplayPort (MDP), High Definition Multimedia Interface (HDMI), Firewire, or other type.

In one embodiment, system1000includes channel repeater1090. Channel repeater1090can be implemented as a repeater and/or in a buffer on a primary memory module in accordance with any embodiment described herein. Channel repeater1090provides a point-to-point connection for a primary connection to a host from a primary memory module or a memory module physically coupled closest to the host on a memory channel. Channel repeater1090provides a secondary connection from the primary memory module to a secondary memory module, where the secondary connection has a lower bandwidth than the primary connection.

In one aspect, a memory circuit to couple multiple memory devices to a memory channel includes: a first group of signal lines to couple point-to-point as a memory channel between a first group of memory devices and a host device; a second group of signal lines to couple point-to-point between the first group of memory devices and a second group of memory devices to extend the memory channel to the second group of memory devices, wherein the second group of signal lines includes fewer data signal lines than the first group of signal lines, to support a lower bandwidth than the first group of signal lines on the memory channel; and a repeater coupled to repeat signals on the memory channel between the first group of signal lines and the second group of signal lines.

In one embodiment, the first group of signal lines and the second group of signal lines comprises single ended data signal lines. In one embodiment, the first group of signal lines and the second group of signal lines comprises differential data signal lines. In one embodiment, the second group of signal lines includes half as many data signal lines as the first group of signal lines. In one embodiment, the first group of memory devices is to utilize approximately one half total bandwidth of the memory channel and the second group of memory devices is to utilize approximately one half total bandwidth of the memory channel. In one embodiment, the first group of memory devices is to utilize more bandwidth than the second group of memory devices. In one embodiment, the first group of memory devices comprises memory devices of a first memory module, and the second group of memory devices comprises memory devices of a second memory module. In one embodiment, the first memory module comprises a dual inline memory module (DIMM) including synchronous dynamic random access memory (SDRAM) devices compliant with a dual data rate version 5 (DDR5) standard, and the second memory module comprises a DIMM including SDRAM devices compliant with a dual data rate version 5 (DDR5) standard. In one embodiment, the first memory module comprises a dual inline memory module (DIMM) including synchronous dynamic random access memory (SDRAM) devices compliant with a dual data rate version 5 (DDR5) standard, and the second memory module comprises a DIMM including byte-addressable, random access nonvolatile memory devices. In one embodiment, the byte-addressable, random access nonvolatile memory devices comprise three dimensional memory devices including memory cells with chalcogenide glass.

In one aspect, a system to couple multiple memory devices to a memory channel includes: a memory controller circuit to control access to memory; a first memory module to couple point-to-point to the memory controller circuit via a first group of signal lines of a memory channel between the memory controller circuit and the first memory module, the first memory module including a repeater; and a second memory module to couple to the memory controller circuit through the repeater via a second group of signal lines to couple point-to-point between the first memory module the second memory module, wherein the second group of signal lines includes fewer data signal lines than the first group of signal lines, to support a lower bandwidth than the first group of signal lines on the memory channel; wherein the repeater is to repeat signals on the memory channel between the first group of signal lines and the second group of signal lines.

The system can include a memory circuit in accordance with any embodiment of an aspect of the above memory circuit. In one embodiment, further comprising a processor coupled to the memory controller, at least one core of the processor to execute data stored in the memory modules. In one embodiment, further comprising a network adapter coupled to exchange data between the memory modules and a remote network location. In one embodiment, further comprising a display communicatively coupled to the processor.

In one aspect, a method for accessing multiple memory devices on a memory channel includes: receiving a memory access signal on a first group of signal lines to couple point-to-point as a memory channel between a first group of memory devices and a host device; and repeating at least a portion of the memory access signal on a second group of signal lines to couple point-to-point between the first group of signal lines and a second group of memory devices to extend the memory channel to the second group of memory devices; wherein the second group of signal lines includes fewer data signal lines than the first group of signal lines, to support a lower bandwidth than the first group of signal lines on the memory channel.

In one embodiment, the first group of signal lines and the second group of signal lines comprises single ended data signal lines. In one embodiment, the first group of signal lines and the second group of signal lines comprises differential data signal lines. In one embodiment, the second group of signal lines includes half as many data signal lines as the first group of signal lines. In one embodiment, further comprising allocating approximately one half total bandwidth of the memory channel to the first group of memory devices and allocating approximately one half total bandwidth of the memory channel to the second group of memory devices. In one embodiment, further comprising allocating more than one half total bandwidth of the memory channel to the first group of memory devices. In one embodiment, the first group of memory devices comprises memory devices of a first memory module, and the second group of memory devices comprises memory devices of a second memory module. In one embodiment, the first memory module comprises a dual inline memory module (DIMM) including synchronous dynamic random access memory (SDRAM) devices compliant with a dual data rate version 5 (DDR5) standard, and the second memory module comprises a DIMM including SDRAM devices compliant with a dual data rate version 5 (DDR5) standard. In one embodiment, the first memory module comprises a dual inline memory module (DIMM) including synchronous dynamic random access memory (SDRAM) devices compliant with a dual data rate version 5 (DDR5) standard, and the second memory module comprises a DIMM including byte-addressable, random access nonvolatile memory devices. In one embodiment, the byte-addressable, random access nonvolatile memory devices comprise three dimensional memory devices including memory cells with chalcogenide glass.

In one aspect, an article of manufacture comprising a computer readable storage medium having content stored thereon to cause execution of operations to execute a method for accessing multiple memory devices on a memory channel in accordance with any embodiment of an aspect of the above. In one aspect, an apparatus for accessing multiple memory devices on a memory channel, comprising means for performing operations to execute a method in accordance with any embodiment of an aspect of the above.

Besides what is described herein, various modifications can be made to the disclosed embodiments and implementations of the invention without departing from their scope. Therefore, the illustrations and examples herein should be construed in an illustrative, and not a restrictive sense. The scope of the invention should be measured solely by reference to the claims that follow.