Partitioned memory with locally aggregated copy pools

An aspect includes receiving a request to access data in a memory, the request from a requesting processor and including a virtual address of the data. It is determined, based on contents of a page table that a plurality of physical addresses in the memory corresponds to the virtual address. The physical addresses include a first physical address of a primary memory location in a first partition accessed via a bus that is communicatively coupled to a port of a first processor, and a second physical address of a secondary memory location in a second partition accessed via a bus that is communicatively coupled to a port of a second processor. Contents of the primary memory location in the first partition were previously copied into the secondary memory location. Based on the requesting processor, one of the physical addresses is selected and data at the selected physical address is accessed.

BACKGROUND

Embodiments of the invention relate to computer memory, and more specifically to partitioned memory systems with locally aggregated copy pools.

SUMMARY

Embodiments of the invention include methods, systems, and computer program products for implementing a partitioned memory system with locally aggregated cache pools. An example method includes receiving a request to access data in a partitioned memory, the request from a requesting processor and including a virtual address of the data. It is determined, based at least in part on contents of a page table, that a plurality of physical addresses in the memory corresponds to the virtual address. The physical addresses include a first physical address of a primary memory location in a first partition accessed via a first memory bus that is communicatively coupled to a port of a first processor, and a second physical address of a secondary memory location in a second partition accessed via a second memory bus that is communicatively coupled to a port of a second processor. Contents of the primary memory location in the first partition were previously copied into the secondary memory location in the second partition. Based at least in part on the requesting processor, one of the physical addresses is selected and data at the selected physical address is accessed.

DETAILED DESCRIPTION

One or more embodiments of the invention described herein provide a partitioned memory system that includes a unified memory physically separated into disjoint partitions connected to a processor through different ports, with a portion of each partition used to store copies of data that resides in a different partition. In accordance with one or more embodiments of the invention, the segments that are copied can be chosen by standard hardware mechanisms, such as most recently used (MRU). The segments can also be copied under the direction of software that is at the application level or within the operating system (OS). In one or more embodiments of the invention, coherence is maintained between the copied segments using any known coherence methods or protocols.

As described herein, in one or more embodiments of the invention, the partitioned memory system stores multiple copies of the same data at different physical memory locations. When a request to access the data is received from a processor, one of the physical memory locations is selected for the access. The selection can be based at least in part on attributes of the requesting processor combined with attributes of the different physical memory locations. For example, if one of the physical memory locations is in a memory partition that is communicatively coupled to the requesting processor via a memory bus, then that physical memory location can be selected for the requested access to the data. In accordance with one or more embodiments of the invention, a page table that allows storage of multiple physical addresses for each virtual address is utilized to track the multiple physical memory locations of the data. Copies of the page table can be stored on each of the requesting processors and their contents kept synchronized.

The ability to replicate and provide local access to the data (e.g., via a memory bus communicatively coupled to a port on the requesting processor) can provide performance improvements when compared to traditional methods where a single copy of the data is stored in a physical memory location that is not local to the requesting processor. Using traditional methods, when the data is not local to the requesting processor, the requesting processor requests the data from another processor via, for example, a symmetric multiprocessing (SMP) bus. This request uses bandwidth on the SMP bus, as well as processing cycles on the other processor to receive and service the request, and can result in performance penalties for the data access. By providing local access to a copy of the data, the extra processing on the SMP bus and the other processor can be eliminated during data accesses.

As used herein, the term “locally aggregated copy pools” refers to physical memory locations of copies of data that are stored locally to processors that are accessing the data. For example, a system can include a first processor with a first port that is connected to a first memory bus that is connected to a first partition of a physical memory. The system can also include a second processor with a second port that is connected to a second memory bus that is connected to a second partition of the physical memory. The first partition of the physical memory can have a copy pool (e.g., a portion of physical memory) that includes copies of data that are accessed frequently by the first processor. The copies can be made of data in other partitions, such as the second partition. By having the locally aggregated copy pool (also referred to herein as a “copy block”) in the first partition, access to the data by the first processor is local, and therefore the accesses can be performed using fewer resources and less elapsed time when compared to non-local data accesses (e.g., via another processor).

As used herein, the terms “communicatively coupled” and “connected” are used interchangeably to refer to a communication path, wired or wireless, between two entities such as, but not limited to: a memory bus and a processor; a first processor and a second processor; and a memory bus and a physical memory.

Turning now toFIG. 1, a block diagram of a system100for implementing locally aggregated copy pools in accordance with one or more embodiments of the invention is generally shown in accordance with one or more embodiments of the invention. The system100includes a processor102that executes an application and/or OS that can request access to data stored in a memory106. Requests of reads or writes to the memory106, from the processor102, are sent to a memory controller104for processing. The memory controller104can perform a number of tasks including, but not limited to, translation of virtual addresses to real addresses, and buffering of requests. In embodiments of the invention described herein, the memory controller104can also execute instructions to implement locally aggregated copy pools in accordance with one or more embodiments of the invention. Though shown as separate physical components inFIG. 1, one or more of the memory controller104, the memory106, and the processor102can also be co-located on a single physical component. In an embodiment, the memory106is main memory that is internal to the processor and implemented by a dynamic random access memory (DRAM) device.

Turning now toFIG. 2, a block diagram of a system200with locally aggregated copy pools is generally shown in accordance with one or more embodiments of the invention. The system200shown inFIG. 2includes a plurality of processors202204each connected to a plurality of memory partitions210212214216via memory busses208, and connected to each other via a SMP bus206. Two copy pools or copy blocks218are also depicted in the system200. Any known technology can be used to implement the SMP bus206and the memory busses208. The SMP bus206is not dedicated to servicing memory requests, and can also be utilized for a variety of communications between the processors202204such as, but not limited to: cache coherence or I/O operations (e.g. networking or storage accesses). In accordance with one or more embodiments of the invention, the SMP bus206can have interface bandwidths of around 40 Gigabyte/second (GB/s). This is contrasted with memory busses208which in accordance with one or more embodiments of the invention is dedicated to servicing memory requests and can have total bandwidth in excess of 200 GB/s.

In accordance with one or more embodiments of the invention, each of the processors202204also include a translation look-aside buffer (TLB) and a page table. As is known in the art, the TLB is a cache of the page table and stores recent translations between virtual addresses and physical addresses. References herein to the page table refer to both the TLB and the page table. In accordance with one or more embodiments of the invention, conventional page tables are extended to include additional entries that allow more than one physical address to correspond to each virtual address. In this manner, the copies of the data described herein are tracked by the page table. Thus, when a request is received to access data at a virtual address, the virtual address can be translated using the page table into one of two (or more) different physical addresses.

In accordance with one or more embodiments of the invention, processor202requests access to data (labeled “DATA A”) that is natively stored in memory partition214. In contemporary implementations, this is performed by processor202continually sending requests to processor204via the SMP bus206. In contrast, in accordance with one or more embodiments of the invention described herein, large blocks of memory can be copied from memory locations in memory partition214into copy block218that is contained in memory partition210. Once the blocks of memory corresponding to the data are stored in copy block218, processor202can access the data locally (e.g., via a memory bus208) from memory partition210. This avoids the requesting processor202from having to send requests to processor204. One or more embodiments of the invention can be used when threads on both processor202and processor204need to touch the data (“DATA A”) stored natively in memory partition210, and will reduce traffic on the shard SMP bus206. This example includes moving data from memory partition214to copy block218. In other scenarios in accordance with one or more embodiments of the invention, the data is moved from memory partition210or memory partition212to copy blocks218in memory partition214or memory partition216. In further scenarios the data is moved from memory partition216to copy blocks218in memory partition210or memory partition212.

In accordance with one or more embodiments of the invention, processor202and processor204are both performing processing on data stored in memory partition214which is attached to processor204. The SMP fabric on one or both of the processors202204notices this condition (e.g., by detecting more than a specified amount of coherence traffic). In accordance with one or more embodiments of the invention, if both processors202204are primarily reading the data, with very few writes, the data copying is initiated. A section or region of memory large enough to contain the data, a copy block218, is allocated in memory partition210and the read/write transactions to copy the data from memory partition214to the copy block218in memory partition210are initiated. In accordance with one or more embodiments of the invention, after allocation, but before the copying of the data to the new location in copy block218is completed, the regions of the copy block218without proper data are marked as invalid (e.g., using spare error correcting code bits or cache entries). After valid data is written, the copies are marked valid and processor202can access data in memory partition210instead of memory partition214. The page tables, including the TLB if needed, in both of the processors202204are updated to indicate that multiple physical memory locations contain valid data for the copied virtual address.

In accordance with one or more embodiments of the invention, all of the physical memory locations are marked as (local) read-only copies. When a write occurs to the addresses that have multiple physical locations for a single virtual address, a page table update can be broadcast to all processors so that all copies are marked as invalid. A write of the data can then be performed to one physical address, and the page table updated to correlate the virtual address with the one physical address. In accordance with one or more embodiments of the invention, when a write occurs to a virtual address that corresponds to multiple physical locations, the write is performed to all of the physical locations specified by the page table for the virtual address.

For ease of description, the example inFIG. 2shows two processors, however embodiments of the invention are not limited to two processors. The number of physical addresses corresponding to one logical address can increase based at least in part on the number of processors connected by SMP busses. For example, when there are three processors, each logical address can corresponding to up to three physical addresses; and when there are “N” processors, each logical address can correspond to up to “N” physical addresses.

Turning now toFIG. 3, a flow diagram300of a process for utilizing locally aggregated copy pools in a partitioned memory system is generally shown in accordance with one or more embodiments of the invention. All or a portion of the processing shown inFIG. 3can be performed by computer instructions located, for example, in memory controller104or processor202. At block302, a request to access data in a partitioned memory is received from a requesting processor. The request can include a virtual address of the data. At block304, it is determined, based at least in part on contents of a page table that a plurality of physical addresses in the memory corresponds to the virtual address. The physical addresses can include, in the case of two processors, a first physical address of a primary memory location in a first partition accessed via a first memory bus communicatively coupled to a port of a first processor. The physical addresses can also include a second physical address of a secondary memory location in a second partition accessed via a second memory bus communicatively coupled to a port of a second processor. Referring toFIG. 2, the requesting processor can be processor202, the first physical address can correspond to the data (e.g., “DATA A”) stored in memory partition214, and the second physical address can correspond to the data (e.g., “DATA A”) stored in copy block218in memory partition210. Referring back toFIG. 3, at block306, one of the physical addresses is selected based, for example, on attributes of the requesting processor and on physical locations of the data. At block308, the data is accessed by the requesting processor at the selected physical memory location.

In accordance with one or more embodiments of the invention, prior to receiving the request to access data in the partitioned memory, contents of the primary memory location are copied into the secondary memory location and the page table is updated to indicate that the virtual address corresponds to both the first physical address and the second physical address. The copying can be responsive to the second processor accessing the data more than a threshold number of times and/or responsive to a software instruction (application and/or OS). Prior to the copying, the second processor accesses the data at the primary memory location via the first processor. Subsequent to the copying the second processor accesses the data at the secondary location.

In accordance with one or more embodiments of the invention, the accesses can include reads, writes and/or deletions. When the access is a write access, the data stored in the secondary memory location can be invalidated, new data written to the primary memory location as indicated by the first physical address, and the contents of the page table updated to indicate that only the first physical address corresponds to the virtual address. When the access is a delete, the data stored in both the primary memory location and the secondary memory location can be invalidated and the contents of the page table updated to remove the corresponding virtual address.

In accordance with one or more embodiments of the invention, one of the physical addresses is selected to service the request based at least in part on attributes of the requestor and attributes of the physical locations of the data. For example, an attribute of a requesting processor can include, but is not limited to memory partitions that the processor is locally connected to by a memory bus. If one of the physical addresses is contained in a memory partition that is locally connected to the requesting processor, then the physical location in the locally connected partition can be selected. In accordance with one or more embodiments of the invention, when it is determined that the requesting processor is the second processor the second physical address is selected, and when it is determined that the requesting processor is the first processor, the first physical address is selected.

If one of the physical addresses is not contained in a memory partition that is locally connected to the requesting processor then other attributes such as, but not limited to available bandwidth, latency, queue depths, or bus speed between the requesting processor and the memory partition can be used to select the physical address.

Turning now toFIG. 4, a block diagram of a system400with locally aggregated copy pools is generally shown in accordance with one or more embodiments of the invention. The system400shown inFIG. 4includes a plurality of processors402404each connected to a plurality of memory partitions410412414416via memory busses408, and connected to connected to each other via a SMP bus406. Four copy pools, or copy blocks418are also depicted in the system400. Also as shown inFIG. 4, each of the processors402404also include a translation look-aside buffer (TLB) and a page table. When compared to the system200shown inFIG. 2, the embodiment of the system400shown inFIG. 4includes a copy block418in every partition410412414416, and a separate physical copy bus420for moving data between the memory partitions410412414416. By having the copy bus420as a secondary bus between memory subsystems, contents of memory locations can be copied without adding additional traffic to the SMP bus406. In addition, the copy bus420can be dedicated to copying data between memory subsystems and not shared with other tasks. In accordance with one or more embodiments of the invention, the copy bus420can have bandwidths on the order of 40 GB/s significantly reducing pressure on the inter-processor SMP bus. In one or more embodiments of the invention, a copy bus420is used in conjunction with off-chip memory buffer.

For ease of description, the example inFIG. 4shows two processors, however embodiments of the invention are not limited to two processors. The number of physical addresses corresponding to one logical address can increase based at least in part on the number of processors connected by SMP busses. For example, when there are three processors, each logical address can corresponding to up to three physical addresses; and when there are “N” processors, each logical address can correspond to up to “N” physical addresses.

Turning now toFIG. 5, a block diagram of a computer system500for use in implementing some or all aspects of a partitioned memory system with locally aggregated copy pools is generally shown according to some embodiments of the invention. The processing described herein may be implemented in hardware, software (e.g., firmware), or a combination thereof. In an exemplary embodiment, the methods described may be implemented, at least in part, in hardware and may be part of the microprocessor of a special or general-purpose computer system500, such as a personal computer, workstation, minicomputer, or mainframe computer.

In an exemplary embodiment, as shown inFIG. 5, the computer system500includes a processor505, memory510coupled to a memory controller515, and one or more input devices545and/or output devices540, such as peripherals, that are communicatively coupled via a local I/O controller535. These devices540and545may include, for example, a printer, a scanner, a microphone, and the like. A conventional keyboard550and mouse555may be coupled to the I/O controller535. The I/O controller535may be, for example, one or more buses or other wired or wireless connections, as are known in the art. The I/O controller535may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications.

The processor505is a hardware device for executing hardware instructions or software, particularly those stored in memory510. The processor505may be a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the computer system500, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or other device for executing instructions. The processor505can include a cache such as, but not limited to, an instruction cache to speed up executable instruction fetch, a data cache to speed up data fetch and store, and a translation look-aside buffer (TLB) used to speed up virtual-to-physical address translation for both executable instructions and data. The cache may be organized as a hierarchy of more cache levels (L1, L2, etc.).

The memory510may include one or combinations of volatile memory elements (e.g., random access memory, RAM, such as DRAM, SRAM, SDRAM, etc.) and nonvolatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like, etc.). Moreover, the memory510may incorporate electronic, magnetic, optical, or other types of storage media. Note that the memory510may have a distributed architecture, where various components are situated remote from one another but may be accessed by the processor505.

The instructions in memory510may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. In the example ofFIG. 5, the instructions in the memory510include a suitable operating system (OS)511. The operating system511essentially may control the execution of other computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.

Additional data, including, for example, instructions for the processor505or other retrievable information, may be stored in storage520, which may be a storage device such as a hard disk drive or solid state drive. The stored instructions in memory510or in storage520may include those enabling the processor to execute one or more aspects of the dispatch systems and methods of this disclosure.

The computer system500may further include a display controller525coupled to a display530. In an exemplary embodiment, the computer system500may further include a network interface560for coupling to a network565. The network565may be an IP-based network for communication between the computer system500and an external server, client and the like via a broadband connection. The network565transmits and receives data between the computer system500and external systems. In an exemplary embodiment, the network565may be a managed IP network administered by a service provider. The network565may be implemented in a wireless fashion, e.g., using wireless protocols and technologies, such as WiFi, WiMax, etc. The network565may also be a packet-switched network such as a local area network, wide area network, metropolitan area network, the Internet, or other similar type of network environment. The network565may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN) a personal area network (PAN), a virtual private network (VPN), intranet or other suitable network system and may include equipment for receiving and transmitting signals.

Systems and methods for providing a partitioned memory system with locally aggregated copy pools as described herein can be embodied, in whole or in part, in computer program products or in computer systems500, such as that illustrated inFIG. 5.

Technical effects and benefits of embodiments of the invention include the ability to reduce traffic on a SMP bus between processors that require access to the same data that is natively stored in a partition local to one of the processors. In addition, by adding a second copy of the data in a partition that is local to the requesting processor, the time to service a request to access the data can be decreased.