Multiple channel cache memory and system memory device utilizing a pseudo-multiple port for commands and addresses and a multiple frequency band QAM serializer/deserializer for data

A high performance, low power, and cost effective multiple channel cache-system memory system is disclosed.

TECHNICAL FIELD

A high performance, low power and cost effective multiple channel cache memory/system memory is disclosed.

BACKGROUND OF THE INVENTION

The performance of both cache memory and system memory is critical in high performance computing systems containing multiple cores processors or multiple processors, particularly systems using additional hardware accelerators such as graphics processing units (GPUs). These computing systems increasingly not only perform general purpose computing but also perform deep machine learning and large data mining.

To handle the demands placed on the computing system, the memory systems need to be optimized for memory access latency and memory bandwidth at the same time. In order to optimize memory access latency and memory bandwidth on the same computing system, one must increase cache performance as well as reduce the frequency of memory collisions on the buses.

Prior art approaches have included increasing cache performance by integrating a large cache RAM on the same silicon as the processor cores it is servicing. However, this approach is limited due to the cost of large size RAM.

Other prior art approaches have used large off-chip RAM located on different silicon than the processor cores it is serving. However, this approach requires a large number of connection pins between the RAM chip and the processor core chip, and the system design cost become unfavorable.

Another prior art approach is a “brute force” approach that increases memory bandwidth by increasing the bus width and clock rate of the memory bus. However, under this approach, memory requests from different memory masters of the system can easily collide when the system has multiple hardware accelerators.

Another prior art approach is a multiple concurrent memory channel, which is the most effective solution to maintain a high effective memory bandwidth and support the high bandwidth to multiple hardware accelerators. The limitation of this approach is that once again you need a large number of interface connection pins between the processor chip and the RAM chip. Using a large number of connection pins increases the cost, size, and manufacturing complexity of the chips.

The prior art also includes multiple approaches for transmitting data from one chip to another chip. In U.S. Pat. No. 9,369,318, titled “Scalable Serial/De-serial I/O for Chip-to-Chip Connection Based on Multi Frequency QAM Scheme,” which is incorporated by reference herein and which shares an inventor with this application, an embodiment of a serializer/deserializer (“SerDes”) was disclosed.

This prior art technique is shown inFIG. 5. Serializer530and deserializer540typically are located on different chips and are connected by I/O interface510. Serializer540receives parallel digital data, in this example shown as eight bits, D0to D7. The data is converted into analog form by digital-to-analog 2-bit converters501,502,503, and504. Each analog output from the digital-to-analog 2-bit converters501,502,503, and504is coupled to QAM mixers. Output from DAC501is received at the QAM I channel at mixer505, which also receives a 90 degree out-of-phase modulation carrier F1_I. Output from DAC502is received at the QAM I channel at mixer506, which also receives a 90 degree out-of-phase modulation carrier F1_Q. Mixers505and506are both associated with QAM modulator524. Similarly, output from DAC503is coupled to mixer507which also receives 90 degree out-of-phase modulation carrier F2_I, while output from DAC504is received at mixer508which also receives 90 degree out-of-phase modulation carrier F2_Q. Mixers507and508are both associated with QAM modulator525. Outputs from the mixers of both QAM modulators524and525are summed at adder509and output over I/O interface510from the chip containing serializer530. Through this modulation process, the parallel input is thus serialized into a series output as an analog signal.

The analog signal over I/O connection510is received by deserializer540in a second chip. Deserializer540preferable includes amplifier511which receives the signal and provides a gain stage to compensate for loss in the low pass filter. The amplified signal is provided to mixers512and513in a first QAM demodulator526, which receives 90 degree out-of-phase modulation carriers F1_I, F1_Q, respectively, and to mixers514and515in a second QAM demodulator527, which receives the amplified signal as well as 90 degree out-of-phase modulation carriers F2_I, F2_Q. Four analog signal channels are output from mixers512,513,514, and515to low pass filters516,517,518, and519, respectively. The low pass filters may be of any desired configuration and order (i.e., 2nd order, 3rd order and so forth). Output from low pass filters516,517,518, and519is received by two-bit analog-to-digital converters (ADC)520,521,522, and523, respectively, which output the digital data. Through this demodulation process, the analog serial input is thus deserialized back to a parallel digital output. Typically, each chip will contain serializer530and deserializer540, such that either chip can send data and either chip can receive data.

Thus, in the prior art system ofFIG. 5, an 8 bit parallel input is serialized by two frequency bands of QAM16 to one I/O interface in a first chip, sent to a second chip, and then deserialized in the second chip by two frequency bands of QAM16 demodulation back into the original parallel data. It should be appreciated that other configurations can be used, such as using 16 bits of parallel data and four frequency bands of QAM16 or two frequency bands of QAM256. To date, the SerDes design ofFIG. 5has not been used in a memory system.

The prior art includes other techniques for maximizing the amount of information that can be transmitted over a given channel. U.S. Pat. No. 5,206,833, titled “Pipelined Dual Port RAM,” which is incorporated by reference herein and which shares an inventor with this application, introduced a technique of a pseudo multiple port memory. This prior art technique is shown inFIG. 6. Devices601and602each output data, which is received by multiplexor603. Multiplexor603is controlled by an arbitration signal. Multiplexor603combines the data from devices601and602into a single channel, as shown. In one embodiment, device601is given priority over device602, and the data from device602is output from multiplexor603only when device601is in a lull. This effectively creates a pseudo-multiple port device even though in reality only one port exists. To date, this technique has not been used in handling command and address data in a memory system.

What is needed is a new architecture for cache memory and system memory that allows multiple memory channels that can operate concurrently while also optimizing memory access latency and memory bandwidth without adding a large number of additional pins between a processor chip and a memory chip.

SUMMARY OF THE INVENTION

The embodiments described herein comprise a cache memory and system memory architecture that utilizes the SerDes technique for the data bus and the pseudo-multiple port technique for the command/address bus, resulting in multiple channel, concurrent cache-system memory. The proposed multiple channels of cache-system memory results in good cache performance as well as good system memory performance.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1depicts an embodiment of computing device100comprising a 4-channel concurrent cache-system memory system. Computing device100comprises system-on-chip processor (labeled as “SOC”)110, which contains last level cache and system memory controller (labeled as “LLC/DRAM controller”)104. SOC processor110is coupled to cache system chip120, which is separate from SOC processor110.

Cache system chip120integrates multiple channels of cache RAM and legacy PHY to connect different type of system memory. Cache system chip120comprises data router127that regulates the traffic between cache system chip120, system memory bus117, and processor cache bus129. Cache system chip120further comprises cache memory arrays126a. . .126j(where j is an integer, such as 4) and legacy physical interfaces (PHY)128a. . .128d(where d is an integer, such as 4) to connect cache system chip120to system memory130such as LPDDR, DDR, GDR, HBM, HMC etc. In this example, cache memory arrays126comprise four arrays of 4 MB each. One of ordinary skill in the art will understand that the example ofFIG. 1can be expanded to include additional memory channels and larger cache RAM size depending the target performance and complexity of the system-on-chip.

SOC processor110further comprises CPU cores101a. . .101i(where i is an integer, such as 2 or 4), each of which is coupled to an L1 cache memory system102a. . .102i, each of which in turn is coupled to L2 cache memory system103. L2 cache memory system103is coupled to LLC/DRAM controller104(specifically, to last level cache controller105) over processor cache bus129. Here, LLC or last-level cache refers to the last level of cache memory utilized by CPU cores101a. . .101i, such as L3 or L4 (the latter of which would require another level of cache memory between L2 cache memory103and last level cache controller105).

SOC processor110further comprises graphics processor unit (GPU) cores115a. . .115k(where k is an integer), each of which is coupled to shared GPU cache memory system116.

SOC processor110further comprises system bus117, to which the following devices are coupled: shared GPU cache memory system116, Serial MFI SerDes controller118, bus bridge121, display engine123, DSP video multimedia processor124, and SOC processor110. Serial MFI SerDes controller118is also coupled to Serial MFI SerDes119, which in turn connects to external devices140such as a solid state drive or hardware accelerator. Bus bridge121is coupled to PCIe (PCI Express) controller122, which in turn is coupled to Legacy Phy125. Display engine123also is coupled to Legal Phy125, which in turn connects to I/O ports150, such as USB, Ethernet, and HDMI ports.

LLC/DRAM controller104further comprises last level cache (LLC) controller105, quad channel memory arbiter106, MFI SerDes107, DRAM controller108, and MFI SerDes109. Quad channel memory arbiter106is coupled to system bus117.

As shown inFIG. 1, it is likely that a system will have multiple hardware accelerators and memory masters. A multiple memory channel can reduce the probability of memory requests colliding and can achieve a high sustained memory bandwidth. If two memory requests do collide at a particular memory channel, then memory arbiter106performs request arbitration to grant the service to a specific memory master based on the arbitration algorithm.

FIG. 2contains additional details regarding the quad-channel concurrent cache memory/system memory system of computing device100. Processor cache bus129and system bus117are depicted. System memory bus117is connected to quad channel memory arbiter106, which in turn is coupled to system memory controller108, SerDes107, and pseudo multi-port CMD/ADR205a,205b,205c, and205d. Processor cache bus129is coupled to LLC Tag206, which in turn is coupled to LLC controller105, SerDes107, and pseudo multi-port CMD/ADR204a,204b,204c, and204d. . . . Pseudo multi-port CMD/ADR204,204b,204c,204d,205a,205b,205c, and205dare coupled to cache system chip120, specifically, to system CMD/ADR interface202a,202b,202c, and202d. SerDes107is coupled to cache system chip120, specifically, to SerDes interface201a,201b,20c,201d,203a,203b,203c, and203d. Data router127is coupled to Legacy Phy128a,128b,128c, and128d, which in turn are coupled to system memory130.

InFIG. 2, the memory requests are asserted not only by memory masters on system memory bus117but also by last level cache105when there is a cache miss. That is, for a particular memory request, LLC controller105will first check the last level cache memory for the data. If the data is not present in the last level cache memory or if the data is stale, that will be considered a cache miss, and the request will be forwarded to cache system chip120. The data path for memory access will be arranged by data router127in cache memory chip120for various bus conditions. Once arbiter106decides which memory master to serve, the bus request is forwarded to the system memory controller108. The state machine inside system memory controller108will issue the correct sequence of command and address to system memory130.

When arbiter106forwards a request to system memory controller108, it also needs to forward the same memory request to the last level cache (LLC) TAG206, for snooping comparison if the memory request is on cacheable exchange memory region. If there is a snooping hit, data router127in the cache memory chip120must perform the data path re-route from the fast cache RAM126a. . .126jinstead of the relatively slow system memory130.

The SOC processor110accesses memory concurrently to other system memory masters when the processor memory request is a cache hit. But when there is a cache miss, the LLC TAG206must forward the processor memory request to the arbiter106, and the arbiter106grants system memory bus117to LLC TAG206. Once this cache miss cycle is executed, the data router127in the cache memory chip120must perform data path re-outing.

Table 1 shows how data router127within the cache memory chip120performs routing for all possible cycles:

In this embodiment, data router127in cache memory chip120must perform the memory data path routing described in the Table 1.

Utilizing the SerDes architecture can reduce the number of pins required for the data bus between the SOC processor110and cache memory chip120as shown inFIG. 2. But the SerDes design cannot be used for the command and address bus because the latency of a half-duplex SerDes degrades the memory performance. A full duplex SerDes can solve the latency problem but the cost of doubling the pin number simply defeats the advantage of SerDes as compared to a parallel bus.

When the system memory bus117transfers the data in the fixed burst length transfer, one can observe the idle cycles between consecutive commands and addresses on the bus. In a typical memory design, the burst length is a fixed burst of 4 or 8 for each memory access. The choice of burst length for memory bus is to synchronize the processor cache line size and DRAM fixed burst length.

For the case of memory burst length is 8, the fastest consecutive command and address will be no earlier than the 8th clock.FIG. 3shows timing sequence of the interface bus between processor and cache RAM chip with a pipeline latency for the memory array is 2-1-1-1-1-1-1-1.

The first command/address is asserted by memory master 0 at 1st clock. The memory array returns 8 consecutive data starting at 2nd clock and ending at 9th clock. As shown in theFIG. 3, the fastest next consecutive command address by the memory master 0 is at 9th clock after issuing the command address at 1st clock. That is to say, between 1st clock and 9th clock, the command/address bus is idle to memory master 0 and is available for other memory masters to issue the command address to the same bus. Thus, other memory masters can perform “cycle stealing” from the idle command address bus. Through this type of cycle stealing on a single command address bus, one can support multiple memory channels on a single command address bus without multiple dedicated command address bus to each memory channel.FIG. 3shows the memory burst length of 8 can support up to 8 concurrent memory channel command address cycles on a single command address bus.

Because the memory array architecture consists of a row and column decoder, further reduction of the command and address bus protocol can be achieved without performance degradation. A cache memory chip120typically consists of SRAM where one can decode a column address later than the row address without slowing down the memory access. Therefore, an SRAM command address bus protocol can strobe the row address at a rising clock edge and strobe the column address at a falling edge. In this way, one can reduce the number of address signals by half.

The command address bus protocol to cache RAM chip120consists of two sets of command address protocol, namely, one to cache RAM and the other to legacy system memory. Data router127must re-route the data between the cache RAM126a. . .126jand system memory130as described in Table 1, whenever there is a cache miss or a snoop hit. Thus, in the command to cache RAM chip, one must include the instruction to the data router127as to how to re-route the memory data.

The four separated cache RAM command address bus shown in theFIG. 2can be implemented by a single command address bus based on fixed burst length cycle stealing technique described above. That is, a single command address bus performs as a pseudo multiple port of memory command address bus. The four legacy system memory command can also reduce to a single command address bus like a cache RAM command address bus.

High-level differences between the design of the prior art and these embodiments is shown inFIGS. 4A and 4B.

InFIG. 4A, a prior art system400is shown, where a processor SOC410contains a last-level cache420, and the SOC410interfaces to system memory430(e.g., a DRAM chip) over numerous data pins440and command/address pins450. Increasing the size of the last-level cache420would improve the cache hit rate. However, the SOC semiconductor process is relatively expensive, so using a larger last-level cache420on the SOC410would be expensive.

InFIG. 4B, computing device100is shown, where processor SOC110interfaces to a cache-system memory120over a SerDes interface112and a command/address interface410. The cache-system memory module120comprises last-level cache memory126a. . .126jas well as system memory128a. . .128dand130. Because the last-level cache is on a separate chip from the SOC, the last-level cache can be made larger than inFIG. 4Abecause the cache-system memory module can be manufactured with a cheaper semiconductor process. Typically, an SOC is manufactured with around 11 metal layers, whereas a memory chip can be manufactured with around 6 metal layers.

Another benefit of the embodiment shown inFIG. 4Bis that no pins are required between the SOC and system memory. Those pins typically require substantial power and also require termination circuitry.

In summary, we describe an invention for multiple channel concurrent cache RAM and system memory based on a short latency SerDes and a pseudo multiple port command address bus. The invention not only reduces the number of interface pin between processor and memory system for multiple memory channel for cost competitive manufacturing but also maintains high memory performance of short latency and high concurrency.