Patent Description:
In processing systems, main memory and/or non-volatile memory stores data in page table that includes data pages that may be accessed by an application and/or kernel via an operating system of the processing system. In order for the operating system to access the main memory and/or non-volatile memory, a page table may be used. A page table is a data structure that stores a mapping between virtual addresses used by the operating system and physical addresses of the main memory and/or non-volatile memory. In this manner, when the operating system attempts to access data from a data page in the main memory and/or non-volatile memory, the operating system sends the virtual address corresponding to the data page to hardware of the processing system. The hardware then uses the page table to determine the physical address of the data page in the main memory and/or non-volatile memory based on the virtual address and access the data for the operating system. However, accessing data from remote memory attached to a first processor socket from a second processor socket takes a large amount of time to perform. Accordingly, local memory may be used to store portions of
the data pages of the remote memory and/or non-volatile memory. The local memory is faster than the remote memory and/or non-volatile memory.

<CIT> describes techniques to identify activity levels of large pages in a computer system having memory that is partitioned and accessed as small pages and large pages. In operation, mappings to selected large pages are temporarily demoted to mappings to small pages and accesses to these small pages are then tracked. For each selected large page, an activity level is determined based on the tracked accesses to the small pages included in the large page.

The object to be solved is to profile data pages in an efficient manner. The object is achieved by the present invention in the aspects of a method, an apparatus and machine-readable storage having the features of the independent claims. Additional features for advantageous embodiments are provided in the dependent claims.

The figures are not to scale. In general, the same reference numbers will be used throughout the drawing(s) and accompanying written description to refer to the same or like parts.

Unless otherwise specified or understood based on their context of use, such descriptors are not intended to impute any meaning of priority or ordering in time but merely as labels for referring to multiple elements or components separately for ease of understanding the disclosed examples.

A processing system (e.g., a computer, a server, etc.) includes electronic circuitry that executes instructions making up a program or workload of an application and/or a kernel. Additionally, the processing system includes hardware components (e.g., main memory) that stores data that the application and/or kernel can access. The processing system includes an operating system (e.g., system software) that manages the computer hardware, software resources, and provides services for the application and/or kernels implemented on the processing system. Accordingly, when an application and/or a kernel needs to access main memory (e.g., to read and/or write data to/from the main memory), the operating system transmits a request to the hardware to access data pages from a page table stored in the memory.

In some processing systems (e.g., a non-uniform memory access (NUMA) system), local memory is the memory dual inline memory module (DIMM) that is attached to a same central processing unit (CPU) socket and remote memory is the memory DIMM that is attached to another CPU socket on the same system. The size and type of local and remove memory may be the same, similar, or different. However, accessing data from local memory (e.g., memory attached to the same CPU socket_) is faster compared to accessing the data in remote memory that is attached to a different (e.g., remove CPU socket). For example, on system with two sockets (S1 attached to memory M1 and S2 attached to memory M2), accessing M1 from S1 is local and faster while accessing M2 from S1 is remote and slower.

In other processing system (e.g., including tiered or heterogeneous memory system), the local memory may be Dynamic Random Access Memory (DRAM) and the remote memory may be non-volatile memory. Non-volatile memory is slower than to DRAM, but is larger and can store more data. The non-volatile memory may be part of the system's main memory and is attached to the CPU socks.

In some examples, a page table is stored in the main memory that maps virtual addresses used by the operating system to physical addresses in the remote memory. In this manner, when the operating system transmits a request to access a data page (e.g., using a virtual address) in the remote memory, the hardware can use the page table to determine the physical address corresponding to the location of the data page. To overcome the amount of time (e.g., latency) it takes to access data from remote memory, the operating system may obtain data pages from the remote memory and store them locally in local memory. In this manner, the operating system can use and/manipulate the data page locally in local memory with less latency by interfacing with the remote memory less often.

To most efficiently utilize the local memory, the operating system attempts to store the most frequently accessed data pages (e.g., hot data pages) in the local memory. As used herein, data pages that are frequently accessed are defined as hot data pages. For example, data pages that are accessed more than a threshold number of times within a duration of time are defined as hot data pages. Data pages that are not frequently accessed are defined as cold data pages. For example, data pages that are accessed less than the threshold number of times within the duration of time are defined as cold data pages. To determine which data pages are hot and which data pages are cold, the operating system performs a profiling protocol to profile the data pages as hot or cold. After the operating system profiles the data pages, the operating system or another component can (A) store hot data pages into local memory, (B) move cold data pages from out of local memory, etc. Examples disclosed herein profile pages as hot or cold, and/or otherwise determine the frequency of data page access, so that the OS can move the data pages between local and remote memory, between fast and slow memory, etc..

Additionally or alternatively, data page profiling may be used to promote and/or demote virtual address-to-physical address mappings to/from huge pages. Huge pages are hardware supported feature that reduces translation look-ahead buffer misses during a virtual-to-physical address translation. To improve the efficiency of huge pages, examples disclose herein quickly identify and promote data pages to huge pages, rather than promoting an arbitrary set of data pages to huge data pages. Accordingly, profiling of data pages is helpful to promote a mapping of a hot data page into the huge table and/or demote a mapping of a cold data page from the huge table, thereby ensuring that the huge table includes hot data pages that are frequently accessed.

A prior data page profiling protocol includes linearly scanning every data page in the main memory to determine which data pages are accessed within a predetermined duration of time. Accordingly, if there are <NUM>,<NUM> data pages in remote memory, prior data page profiling protocols have to access and process <NUM>,<NUM> data pages from remote memory to determine whether the data pages are hot or cold. Accordingly, linearly scanning every data page using prior techniques is time consuming and requires processor resources. Additionally, when the operating system is profiling, the operating system cannot use resources to perform other tasks. Examples disclosed herein reduce the amount of time and resources needed to profile data pages. Thus, data pages can be profiled faster and with less resources using examples disclosed herein than using prior techniques, thereby allowing the operating system more time and resources to perform other tasks.

The page table used by the operating system and memory that maps virtual addresses to physical addresses is structured in a hierarchy of levels (e.g., also referred to as layers) that map to the data pages stored in main memory. For example, the page table includes a first-highest level including a page (e.g., page global director page (PGD)) that is split into a second-highest level that includes a plurality of pages (e.g., page upper director pages (PUDs)). Each of the PUDs of the second-highest level is split into a third-highest level that includes a plurality of pages (e.g., page middle directory pages (PMDs)). Each of the PMDs of the third-highest level is split into a leaf level that includes a plurality of pages (e.g., page table entry pages (PTEs)). Each of the PTEs of the leaf level corresponds to a plurality of the data pages stored in main memory that are allocated by the operating system to the application. Whenever the hardware accesses a data page from main memory, the access is flagged for the data page and all the higher level pages that correspond to the data page (e.g., the PTE, the PMD, the PUD, and the PGD that correspond to the data page). Examples disclosed herein leverage the hierarchy of the page table corresponding to the data pages to identify large sets of data pages that are cold. For example, if during profiling the operating system determines that a highest-level page (e.g., a PUD) has not been accessed during a duration of time, examples disclosed herein determine that all of the data pages that correspond to the highest-level page are cold. In this manner, examples disclosed herein can profile a plurality of data pages based on a single page scan of the high-level data as opposed to the plurality of profiling scans for the plurality of data pages in the prior profiling protocols.

When examples disclosed herein determine that the highest-level page has been accessed during a duration of time, examples disclosed herein perform another round of profiling for the second-highest level pages that correspond to the accessed highest-level page to determine which of the second-highest level pages has been accessed, and the process continues until examples disclosed herein determine the data pages of the main memory that have been accessed. In this manner, for a <NUM>-tier page table (e.g., PUD, PMD, PTE, and data page), examples disclosed herein can profile all data pages as target pages or non-target pages (e.g., hot or cold) using <NUM> profiling rounds, regardless of the number of data pages in the page table.

<FIG> is a block diagram of an example implementation of an example server <NUM>. The example server <NUM> of <FIG> includes an example operating system (OS) <NUM>, an example page table profiler <NUM>, example hardware <NUM>, example main memory <NUM>, an example page table <NUM>, example page table pages <NUM>, and local memory <NUM>. Although <FIG> corresponds to the example server <NUM>, examples disclosed herein may be implemented in any type of processing system and/or computing system.

The example server <NUM> of <FIG> is a computer system the includes software, hardware, and/or firmware to perform tasks defined by an application and/or kernel. The example server <NUM> utilizes processor resources (e.g., the example memories <NUM>, <NUM>, register(s) and/or logic circuitry of processor core(s)) and utilizes the OS <NUM> to execute instructions to implement an application and/or kernel.

The example OS <NUM> of <FIG> is system software that executes (e.g., using processor cores) instructions and/or a workload from an application and/or kernel (e.g., by reading and/or writing data). The example OS <NUM> manages the computer hardware, software resources, etc. to be able to access (e.g., read and/or write) data to/from data pages (e.g., the data pages <NUM>) stored in the example main memory <NUM>. To access the data pages <NUM> in the main memory <NUM>, the example OS <NUM> transmits a virtual address to the hardware <NUM> and the hardware <NUM> reads and/or writes the data to the data page that corresponds to the virtual address. The example OS <NUM> includes the example page table profiler <NUM>. After the example page table profiler <NUM> profiles the data pages, the OS <NUM> causes a copy of the hot data pages to be stored in the local memory <NUM> and/or causes the cold data pages stored in the local memory <NUM> to be removed from the local memory <NUM>.

The example page table profiler <NUM> of <FIG> profiles the pages of the page table <NUM> by leveraging the hierarchy of the page table <NUM>. For example, to profile the data pages of the page table(s) <NUM> as target pages or non-target pages (e.g., hot and/or cold), the example page table profiler <NUM> profiles the highest-level pages of the page table <NUM> to determine which pages of the highest-level have been accessed (e.g., hot) and which pages of the highest-level have not been accessed (e.g., cold). The example page table profiler <NUM> tags (e.g., labels) all the data pages that correspond to the cold pages at the highest-level as cold data pages. The example page table profiler <NUM> then performs a subsequent profiling round using the second-highest level of data pages that correspond to the hot highest level pages, and the processes is repeated per level (e.g., tagging data pages as cold when a corresponding higher level is tagged as cold and repeating the process at the next level (e.g., a lower level) for hot data pages), until all of the main memory of the process/application have been identified as hot or cold. The example page table profiler <NUM> is further described below in conjunction with <FIG>.

The example hardware <NUM> of <FIG> obtains access requests (e.g., to read and/or write data to/from the main memory <NUM>) from the example OS <NUM>. As described above, the access request includes a virtual address corresponding to a data page to be accessed. The example hardware <NUM> uses the page table <NUM> to determine the physical memory address corresponding to the location of the data page based on the virtual address. In some examples, the OS <NUM> may promote and/or demote data to/from huge pages based on the profiled data. For example, the OS <NUM> may promote mappings corresponding to hot data pages to a huge page and/or may demote huge page mappings corresponding to cold data pages to base pages. The example hardware <NUM> of <FIG> includes the main memory <NUM>.

The example main memory <NUM> of <FIG> stores data pages in the page table pages <NUM> that the OS <NUM> may access (e.g., to be read from and/or to be written to). The example main memory <NUM> may be non-volatile memory and/or memory that is located at a remote CPU socket. Additionally, the example main memory <NUM> includes a section of memory to store the example page table <NUM>. As described above, the page table <NUM> is a tree-based structure that includes levels (e.g., also referred to as layers) that map a virtual address to a physical address. In this manner, the hardware <NUM> can use the page table <NUM> to access data pages from the page table pages <NUM> based on the virtual address from the OS <NUM>. The data pages <NUM> include data that an application and/or kernel may access via the OS <NUM>. An example implementation of the page table <NUM> and/or the data pages <NUM> is further described below in conjunction with <FIG>.

The example local memory <NUM> is memory that is implemented in the same (e.g., local) socket as the OS <NUM>. Accordingly, the example local memory <NUM> is faster than the example remote memory. The example local memory <NUM> stores a subsection of the data pages <NUM> stored in the example main memory <NUM>. As described above, the example OS <NUM> profiles the page table pages <NUM> to attempt to identify more frequently accessed data pages. In this manner, the OS <NUM> can store the hot data pages in the local memory <NUM> to increase efficiency and speed of executing instructions from an application and/or kernel. In some examples, the local memory <NUM> may be located in a different section of the example server <NUM> (e.g., part of the main memory <NUM>).

<FIG> is a block diagram of an example implementation of the page table profiler <NUM> of <FIG>. The example page table profiler <NUM> includes an example component interface <NUM>, an example access tracker <NUM>, an example flag controller <NUM>, and an example timer <NUM>.

The example component interface <NUM> interfaces with other components of the example server <NUM>. For example, the component interface <NUM> may obtain instructions to profile page table pages from the OS <NUM>. Additionally, the component interface <NUM> may interface with the page table <NUM> of the example main memory <NUM> (e.g., directly or via the hardware <NUM>) as part of the profiling of the page table pages <NUM>. For example, if the profiling includes setting and/or reading flags corresponding to the page table <NUM>, the component interface <NUM> may access the page table <NUM> to set and/or read the flags. In some examples, another component may be used to profile the page table pages <NUM>. For example, a memory address tracker may capture and process telemetry data on memory access patterns to identify hot and/or cold pages of the page table <NUM>. In some examples, the memory address tracker may be implemented by the access tracker <NUM>. In such examples, the component interface <NUM> may obtain the profiling data from the memory address tracker.

The example access tracker <NUM> of <FIG> tracks accesses to one or more pages in the page table <NUM> and/or data pages <NUM> using a profiling protocol. For example, in an OS-triggered fault-profiling protocol, the access tracker <NUM> may utilize the example flag controller <NUM> to flag the page table page as invalid by changing the access permission, removing an entry to trigger a page fault, etc. In this manner, the access tracker <NUM> can wait a duration of time using the example timer <NUM> and then check the flags and/or faults to determine if a page was accessed. For example, a minor fault is triggered by the hardware <NUM> when any data page mapped under a page in the page table <NUM> is accessed. In this manner, the access tracker <NUM> can track the accesses to the page table pages and label as hot or cold by checking for (a) changes in the flag and/or (b) triggered faults. Additionally or alternatively, for a memory address tracker profiling protocol, the example access tracker <NUM> can capture and/or obtain telemetry data on memory access patterns to profile a page of the page table <NUM> to identify access to the page. As described above, the access tracker <NUM> performs a first round based on the highest-level of pages and then repeats at lower levels when one or more of the higher levels is labeled as hot. In this manner, the example access tracker <NUM> is able to label data pages as hot or cold with just four profiling rounds (e.g., for the four levels in the hierarchy).

The example timer <NUM> of <FIG> tracks time. In this manner, the example access tracker <NUM> can determine how long to wait after flagging pages as invalid to check if the page has been accessed. In some examples, the timer <NUM> can include or be replaced with a counter that counts clock cycles. The amount of time may be any duration of time based on user and/or manufacturer preferences. Additionally, the duration of time may be based on the profiling. For example, the duration of time may be shorter to profile hot data pages and may be longer to profile cold data pages that have already been implemented in the local memory.

While an example manner of implementing the example OS <NUM> and/or the example page table profiler <NUM> of <FIG> is illustrated in <FIG> and/or <NUM>, one or more of the elements, processes and/or devices illustrated in <FIG> and/or <NUM> may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example component interface <NUM>, the example access tracker <NUM>, the example flag controller <NUM>, and the example timer <NUM>, and/or, more generally, the example page table profiler <NUM> of <FIG> and/or the OS <NUM> of <FIG> may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example component interface <NUM>, the example access tracker <NUM>, the example flag controller <NUM>, and the example timer <NUM>, and/or, more generally, the example page table profiler <NUM> of <FIG> and/or the OS <NUM> of <FIG> could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example component interface <NUM>, the example access tracker <NUM>, the example flag controller <NUM>, and the example timer <NUM>, and/or, more generally, the example page table profiler <NUM> of <FIG> and/or the OS <NUM> of <FIG> is/are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example page table profiler <NUM> and/or the OS <NUM> of <FIG> may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated in <FIG> and/or <NUM>, and/or may include more than one of any or all of the illustrated elements, processes, and devices. As used herein, the phrase "in communication," including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events.

Flowcharts representative of example hardware logic, machine readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the example page table profiler <NUM> and/or the OS <NUM> of <FIG> and/or <NUM> are shown in <FIG>. The machine readable instructions may be one or more executable programs or portion(s) of an executable program for execution by a computer processor such as the processor <NUM> shown in the example processor platform <NUM> discussed below in connection with <FIG>. The program(s) may be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor <NUM>, but the entirety of the program(s) and/or parts thereof could alternatively be executed by a device other than the processor <NUM> and/or embodied in firmware or dedicated hardware. Further, although the example program(s) is/are described with reference to the flowchart illustrated in <FIG>, many other methods of implementing the example page table profiler <NUM> and/or the OS <NUM> of <FIG> and/or <NUM> may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

The machine readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a fragmented format, a compiled format, an executable format, a packaged format, etc. Machine readable instructions as described herein may be stored as data (e.g., portions of instructions, code, representations of code, etc.) that may be utilized to create, manufacture, and/or produce machine executable instructions. For example, the machine readable instructions may be fragmented and stored on one or more storage devices and/or computing devices (e.g., servers). The machine readable instructions may require one or more of installation, modification, adaptation, updating, combining, supplementing, configuring, decryption, decompression, unpacking, distribution, reassignment, compilation, etc. in order to make them directly readable, interpretable, and/or executable by a computing device and/or other machine. For example, the machine readable instructions may be stored in multiple parts, which are individually compressed, encrypted, and stored on separate computing devices, wherein the parts when decrypted, decompressed, and combined form a set of executable instructions that implement a program such as that described herein.

In another example, the machine readable instructions may be stored in a state in which they may be read by a computer, but require addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc. in order to execute the instructions on a particular computing device or other device. In another example, the machine readable instructions may need to be configured (e.g., settings stored, data input, network addresses recorded, etc.) before the machine readable instructions and/or the corresponding program(s) can be executed in whole or in part. Thus, the disclosed machine readable instructions and/or corresponding program(s) are intended to encompass such machine readable instructions and/or program(s) regardless of the particular format or state of the machine readable instructions and/or program(s) when stored or otherwise at rest or in transit.

As mentioned above, the example processes of <FIG> may be implemented using executable instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a local memory, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.

<FIG> illustrate a flowchart representative of example machine readable instructions <NUM> that may be executed to implement the example page table profiler <NUM> and/or the example OS <NUM> (<FIG> and/or <NUM>) to profile data pages as hot or cold. Although the flowchart of <FIG> is described in conjunction with the example page table profiler <NUM> of the example server <NUM>, other type(s) of page table profilers(s), and/or other type(s) of computing system(s) may be utilized instead. Although the example of <FIG> correspond to an OS-triggered fault-profiling protocol, examples disclosed herein can profile pages using any profiling protocol (e.g., memory address tracker profiling).

At block <NUM>, the example flag controller <NUM> of the access tracker <NUM> marks the highest-level page(s) (e.g., the PUD page(s)) in the page table hierarchy as invalid. As described above, if the hardware <NUM> accesses a page of the page table <NUM> that is marked as invalid, a minor fault is triggered when any data page mapped under the highest-level page, which the example access tracker <NUM> can track to define the corresponding page as hot. additionally or Alternatively, the example flag controller <NUM> can reset a flag that is set by the hardware <NUM> whenever any data page mapped under the PUD page is accessed. In this manner, the example access tracker <NUM> can identify hot pages based on set flags.

At block <NUM>, the example access tracker <NUM> waits a duration of time by tracking the timer <NUM>. As described above in conjunction with <FIG>, the duration of time may be defined by a user and/or manufacturer via a user interface and/or a configuration file. At block <NUM>, the example access tracker <NUM> profiles the highest-level page(s) as part of a target group or not part of a target group because they were accessed during the duration of time (e.g., as hot or cold) based on flags and/or faults that are set by the hardware <NUM> whenever any data page mapped under the PUD page is accessed. For example, if a PUD corresponds to a triggered fault and/or flag that corresponds to access of a page mapped under the PUD, the access tracker <NUM> profiles the PUD as hot. If the PUD does not correspond to a fault and/or flag, the access tracker <NUM> profiles the PUD as cold. At block <NUM>, the example access tracker <NUM> determines if there is/are one or more highest level page(s) profiled as cold. If the example access tracker <NUM> determines that there is not one or more highest level page(s) profiled as not part of the target group (e.g., profiled as cold) (block <NUM>: NO), control continues to block <NUM>. If the example access tracker <NUM> determines that there is/are one or more highest level page(s) profiled as not part of the target group (e.g., cold) (block <NUM>: YES), the example access tracker <NUM> labels the corresponding lower level page(s) (e.g., the page(s) that (a) correspond to the cold highest level page and (b) are at lower levels than the cold highest level page) as not part of the target group (e.g., cold) (block <NUM>).

At block <NUM>, the example access tracker <NUM> determines if one or more of the highest-level page(s) (e.g., PUDs) were profiled as part of the target group (e.g., hot). If the example access tracker <NUM> determines that one or more of the highest-level page(s) (e.g., PUDs) were not profiled as part of the group (e.g., hot) (block <NUM>: NO), the process returns to block <NUM> to repeat the process until a highest-level page is profiled as hot. If the example access tracker <NUM> determines that one or more of the highest-level page(s) (e.g., PUDs) were profiled as part of the target group (e.g., hot) (block <NUM>: YES), the example flag controller <NUM> of the access tracker <NUM> marks the middle level page(s) (e.g., the PMDs) corresponding to the target highest-level page(s) (e.g., hot PUD(s)) of the previously profiling round as invalid (block <NUM>). Additionally or alternatively, the example flag controller <NUM> can reset a flag that is set by the hardware <NUM> whenever any data page mapped under the PMD page is accessed.

At block <NUM>, the example access tracker <NUM> waits a duration of time by tracking the timer <NUM>. As described above in conjunction with <FIG>, the duration of time may be defined by a user and/or manufacturer. At block <NUM>, the example access tracker <NUM> profiles the middle level page(s) as part of the target group or not part of the target group (e.g., hot or cold) based on flags and/or faults that are set by the hardware <NUM> whenever any data page mapped under the PMD page is accessed. For example, if a PMD corresponds to a triggered fault and/or flag that corresponds to access of a page mapped under the PMD, the access tracker <NUM> profiles the PMD as part of the target group or hot. If the PMD does not correspond to a fault and/or flag, the access tracker <NUM> profiles the PMD as not part of the target group (e.g., cold). At block <NUM>, the example access tracker <NUM> determines if there is/are one or more middle level page(s) profiled as not part of the target group (e.g., cold). If the example access tracker <NUM> determines that there is not one or more middle level page(s) profiled as not part of the target group (e.g., cold) (block <NUM>: NO), control continues to block <NUM>. If the example access tracker <NUM> determines that there is/are one or more middle level page(s) profiled as not part of the target group (e.g., cold) (block <NUM>: YES), the example access tracker <NUM> labels the corresponding lower level page(s) (e.g., the page(s) that (a) correspond to the non-target (e.g., cold) middle level page and (b) are at lower levels than the non-target and/or cold middle level page) as not part of the target group (e.g., cold) (block <NUM>).

At block <NUM>, the example access tracker <NUM> determines if one or more of the middle level page(s) (e.g., PMDs) were profiled as part of the target (e.g., hot). If the example access tracker <NUM> determines that one or more of the middle level page(s) (e.g., PMDs) were not profiled as part of the target group (e.g., hot) (block <NUM>: NO), the process continues to block <NUM> of <FIG>. If the example access tracker <NUM> determines that one or more of the middle level page(s) (e.g., PMDs) were profiled as part of the target group (e.g., hot) (block <NUM>: YES), the example flag controller <NUM> of the access tracker <NUM> marks the low level page(s) (e.g., the PTEs) corresponding to the target (e.g., hot) middle level page(s) (e.g., hot PMD(s)) of the previously profiling round as invalid (block <NUM>). Additionally or alternatively, the example flag controller <NUM> can reset a flag that is set by the hardware <NUM> whenever any data page mapped under the PTE page is accessed.

At block <NUM>, the example access tracker <NUM> waits a duration of time by tracking the timer <NUM>. At block <NUM>, the example access tracker <NUM> profiles the lowest level page(s) as part of the target group or not part of the target group (e.g., hot or cold) based on flags and/or faults that are set by the hardware <NUM> whenever any data page mapped under the PTE page is accessed. For example, if a PTE corresponds to a triggered fault and/or flag that corresponds to access of a page mapped under the PTE, the access tracker <NUM> profiles the PTE as part of the target group (e.g., hot). If the PTE does not correspond to a fault and/or flag, the access tracker <NUM> profiles the PTE as not part of the target group (e.g., cold). If none of the faults and/or flags are triggered, none of the pages are labelled as not part of the target group (e.g., cold). At block <NUM>, the example access tracker <NUM> determines if there is/are one or more lowest level page(s) profiled as not part of the target group (e.g., cold). If the example access tracker <NUM> determines that there is not one or more lowest level page(s) profiled as cold (block <NUM>: NO), control continues to block <NUM>. If the example access tracker <NUM> determines that there is/are one or more lowest level page(s) profiled as not part of the target group (e.g., cold) (block <NUM>: YES), the example access tracker <NUM> labels the corresponding data page(s) (e.g., the data page(s) that correspond to the cold lowest level page) as not part of the target group (e.g., cold) (block <NUM>).

At block <NUM>, the example access tracker <NUM> determines if one or more of the lowest level page(s) (e.g., PTE(s)) were profiled as part of the target group (e.g., hot). If the example access tracker <NUM> determines that one or more of the lowest level page(s) (e.g., PTE(s)) were not profiled as part of the target group (e.g., hot) (block <NUM>: NO), the process continues to block <NUM>. If the example access tracker <NUM> determines that one or more of the lowest level page(s) (e.g., PTE(s)) were profiled as part of the target group (e.g., hot) (block <NUM>: YES), the example flag controller <NUM> of the access tracker <NUM> marks the data page(s) corresponding to the target lowest level page(s) (e.g., hot PTE(s)) of the previously profiling round as invalid (block <NUM>). Additionally or alternatively, the example flag controller <NUM> can reset a flag that is set by the hardware <NUM> whenever any data page is accessed.

At block <NUM>, the example access tracker <NUM> waits a duration of time by tracking the timer <NUM>. At block <NUM>, the example access tracker <NUM> profiles the data page(s) as target (e.g., hot) based on flags and/or faults that are set by the hardware <NUM> whenever the data page is accessed. For example, if a data page corresponds to a triggered fault and/or flag that corresponds to access of the data page, the access tracker <NUM> profiles the data page as part of the target group (e.g., hot). If none of the faults and/or flags are triggered, none of the data pages are labelled as part of the target group (e.g., hot). At block <NUM>, the example access tracker <NUM> profiles one or more of the data pages as not part of the target group (e.g., cold) based on flags or faults that are not set by the hardware <NUM>. For example, if a data page does not correspond to a fault and/or flag, the access tracker <NUM> profiles the data page as not part of the target group (e.g., cold). If none of the faults and/or flags are triggered, none of the data pages are labelled as not part of the target group (e.g., cold).

At block <NUM>, the example operating system <NUM> may instruct the hardware components <NUM> to read hot data pages from remote memory to be stored in the example local memory. At block <NUM>, the example OS <NUM> determines if data pages labelled as cold that are currently stored in the local memory. If the example OS <NUM> determines that cold data page(s) is/are not stored in the example local memory <NUM> (block <NUM>: NO), control continues to block <NUM>. If the example OS <NUM> determines that cold data page(s) is/are stored in the example local memory <NUM> (block <NUM>: YES), the example OS <NUM> instructs the hardware components <NUM> to move the cold data pages in the local memory <NUM> and store in the remote memory (block <NUM>).

At block <NUM>, the example OS <NUM> determines if mapping(s) of pages(s) (e.g., virtual address to physical address mappings) should be promoted/demoted to/from the huge pages list. For example, the user and/or manufacturer settings may define when page(s) should be promoted and/or demoted. If the example OS <NUM> determines that mapping(s) of page(s) should not be promoted and/or demoted (block <NUM>: NO), the instructions end. If the example OS <NUM> determines that mapping(s) of page(s) should be promoted and/or demoted (block <NUM>: YES), the example OS <NUM> instructs the hardware <NUM> to promote mappings of hot pages that are not stored in the huge pages (block <NUM>). At block <NUM>, the example OS <NUM> instructs the hardware <NUM> to demote mappings of cold pages that are included in the huge pages (block <NUM>). After block <NUM>, the instructions end.

<FIG> illustrates an example profiling protocol corresponding to the example page table <NUM> and the example data pages <NUM> of <FIG>. The example page table <NUM> includes an example page global director (PGD) page (also referred to as a node) <NUM>, example PUD pages 404a-b, example PMD pages 406a-d, example PTEs 408a-h, and the example data pages <NUM> of <FIG>. Although the example of <FIG> illustrates a particular structure with a particular number of pages at four levels (e.g., also referred to as layers), the example page table <NUM> may include any number of levels and/or pages in any tree structure.

The example PGD <NUM> of <FIG> is the highest node and/or page of the tree structure. Under the highest node is a subsequent level of high-level pages corresponding to the example PUDs 404a-b. Under each of the PUDs 404a-b is a next highest level (e.g., a middle level) of the PMDs 406a-d. The example PMDs 406a-b correspond to (e.g., are implemented below) the example PUD 404a and the PMDs 406c-d correspond to the example PUD 404b. Under each of the PMDs 406a-d is the lowest level of PTEs 408a-h (also referred to as leaf pages). The example PTEs 408a-b correspond to (e.g., are implemented below) the example PMD 406a, the example PTEs 408c-d correspond to the example PMD 406b, the example PTEs 406e-f correspond to the example PMD 406c, and the example PTEs <NUM>-h correspond to the example PMD 406d.

During a first profiling round, the example page table profiler <NUM> profiles the PUDs 404a-b, as described above. After the first profiling round, the example page table profiler <NUM> determines that the example PUD 404a is hot and the example PUD 404b is cold. Accordingly, the example page table profiler <NUM> labels all the pages below the PUD 404b as cold (e.g., the PMDs 406c-d, PTEs 408e-h, and the data pages that correspond to the PTES 408e-h). After the PUD 404a is profiled as hot, the example page table profiler <NUM> resets the flags and performs a subsequent profiling round at the PMD level. During the subsequent profile round, the example page table profiler <NUM> determines that both PMDs 406a-b are hot. Accordingly, the example page table profiler <NUM> resets the flags and performs a subsequent profiling round at the PTE level.

During the subsequent profile round, the example page table profiler <NUM> determines that PTE 408a is cold and PTEs 408b-d are hot. Accordingly, the example page table profiler <NUM> labels all the data pages that correspond to the PTE 408a as cold and resets the flags and performs a subsequent profiling round at the data page level for the data pages that correspond to hot PTEs 408b-d. In this manner, the example page table profiler <NUM> can identify the hot data pages that correspond to the hot PTEs 408b-d to complete the profiling protocol for all data pages.

<FIG> is a block diagram of an example processor platform <NUM> structured to execute the instructions of <FIG> to implement the example page table profiler <NUM> and/or the OS <NUM> of <FIG> and/or <NUM>. The processor platform <NUM> can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), or any other type of computing device.

For example, the processor <NUM> can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor implements the example component interface <NUM>, the example access tracker <NUM>, the example flag controller <NUM>, and the example timer <NUM> of <FIG>.

The processor <NUM> of the illustrated example includes a local memory <NUM> (e.g., a local memory). In the example of <FIG>, the example main memories <NUM>, <NUM> implements the example remote memory.

In the example of <FIG>, the interface circuit <NUM> implements the interface <NUM> of <FIG>. However, the interface <NUM> may be a separate component from the interface circuit <NUM> of <FIG>.

The interface circuit <NUM> of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip, and/or a graphics driver processor.

The machine executable instructions <NUM> of 3A-3C may be stored in the mass storage device <NUM>, in the volatile memory <NUM>, in the non-volatile memory <NUM>, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.

A block diagram illustrating an example software distribution platform <NUM> to distribute software such as the example computer readable instructions <NUM>, <NUM> of <FIG> and/or <NUM> to third parties is illustrated in <FIG>. The example software distribution platform <NUM> may be implemented by any computer server, data facility, cloud service, etc., capable of storing and transmitting software to other computing devices. The third parties may be customers of the entity owning and/or operating the software distribution platform. For example, the entity that owns and/or operates the software distribution platform may be a developer, a seller, and/or a licensor of software such as the example computer readable instructions <NUM>, <NUM> of <FIG> and/or <NUM>. The third parties may be consumers, users, retailers, OEMs, etc., who purchase and/or license the software for use and/or re-sale and/or sub-licensing. In the illustrated example, the software distribution platform <NUM> includes one or more servers and one or more storage devices. The storage devices store the computer readable instructions <NUM>, which may correspond to the example computer readable instructions <NUM>, <NUM> of <FIG> and/or <NUM>, as described above. The one or more servers of the example software distribution platform <NUM> are in communication with a network <NUM>. In some examples, the one or more servers are responsive to requests to transmit the software to a requesting party as part of a commercial transaction. Payment for the delivery, sale and/or license of the software may be handled by the one or more servers of the software distribution platform and/or via a third party payment entity. The servers enable purchasers and/or licensors to download the computer readable instructions <NUM> from the software distribution platform <NUM>. For example, the software, which may correspond to the example computer readable instructions <NUM>, <NUM> of <FIG> and/or <NUM>, may be downloaded to the example processor platform <NUM>, which is to execute the computer readable instructions <NUM> to implement the OS <NUM> and/or the page table profiler <NUM>. In some example, one or more servers of the software distribution platform <NUM> periodically offer, transmit, and/or force updates to the software (e.g., the example computer readable instructions <NUM>, <NUM> of <FIG> and/or <NUM>) to ensure improvements, patches, updates, etc. are distributed and applied to the software at the end user devices.

From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed herein to profile page table pages for memory management. Disclosed methods, apparatus and articles of manufacture improve the efficiency of a computer by leveraging the hierarchy structure of a page table to profile data pages in main memory based on the status of the pages in the profile table. In this manner, the amount of data pages that are processed to profile the data pages is reduced, thereby reducing profiling time, decreasing resources, and opening up the OS to perform different tasks. Accordingly, disclosed methods, apparatus and articles of manufacture are directed to one or more improvement(s) in the functioning of a computer.

Claim 1:
A method to profile data pages that are stored in remote memory (<NUM>), the method comprising:
profiling a first page (406c) at a first level of a page table (<NUM>) as not part of a target group; and
in response to profiling the first page (406c) as not part of the target group, labelling a data page (408e), at a second lower level of the page table, that corresponds to the first page (406c), as not part of the target group.