mm: add memlayout & dnuma to track pfn->nid & transplant pages between nodes

On some systems, the hypervisor can (and will) relocate physical
addresses as seen in a VM between real NUMA nodes. For example, IBM
Power systems which are using particular revisions of PHYP (IBM's
proprietary hypervisor)

This change set introduces the infrastructure for tracking & dynamically
changing "memory layouts" (or "memlayouts"): the mapping between page
ranges & the actual backing NUMA node.

A memlayout is stored as an rbtree which maps pfns (really, ranges of
pfns) to a node. This mapping (combined with the LookupNode pageflag) is
used to "transplant" (move pages between nodes) pages when they are
freed back to the page allocator.

Additionally, when a new memlayout is commited the currently free pages
that are now in the 'wrong' zone's freelist are immidiately transplanted.

Hooks that tie it into the page alloctor to actually perform the
"transplant on free" are in later patches.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
add CLAMP_MIN(val, min)

CLAMP_MIN(val, min) ensures @val is at least @min, and does so in a way usable outside of functions.
mm: memlayout+dnuma: add debugfs interface

Add a debugfs interface to dnuma/memlayout. It keeps track of a
variable backlog of memory layouts, provides some statistics on dnuma
moved pages & cache performance, and allows the setting of a new global
memlayout.

TODO: split out statistics, backlog, & write interfaces from each other.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
drivers/base/memory.c: alphabetize headers.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
drivers/base/node,memory: rename function to match interface

Rename register_mem_sect_under_node() to register_mem_block_under_node()
and rename unregister_mem_sect_under_nodes() to
unregister_mem_block_under_nodes() to reflect that both of these
functions are given memory_blocks instead of mem_sections

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
drivers/base/node: rename unregister_mem_blk_under_nodes() to be more acurate

unregister_mem_block_under_nodes() only unregisters a single section in
the mem block under all nodes, not the entire mem block. Rename it to
unregister_mem_block_section_under_nodes(). Also rename the phys_index
param to indicate that it is a section number.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
drivers/base/node: add unregister_mem_block_under_nodes()

Provides similar functionality to
unregister_mem_block_section_under_nodes() (which was previously named
identically to the newly added funtion), but operates on all memory
sections included in the memory block, not just the specified one.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
mm: memory,memlayout: add refresh_memory_blocks() for Dynamic NUMA.

Add refresh_memory_blocks(), which re-registers memory blocks under the
appropriate nodes based on a given memlayout.

Used in the later patch adding baseline DNUMA functionality.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
mm/memlayout: refresh sysfs memory_block info when a memlayout is commited

Use refresh_memory_blocks() to reconfigure the sysfs registrations
following a (non-initial) memlayout being commited.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
dnuma: factor out ml_stat_count_moved_pages() into dnuma_post_free_to_new_zone()

This essentially exposes ml_stat_count_moved_pages() to things other
than memlayout & dnuma allowing usage (in this case) in page_alloc.c

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
mm/page_alloc: add free_one_page_to_new_zone()
page_alloc: use dnuma to transplant newly freed pages in __free_pages_ok()

__free_pages_ok() handles higher order (order != 0) pages. Transplant
hook is added here as this is where the struct zone to free to is
decided.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
page_alloc: use dnuma to transplant newly freed pages in free_hot_cold_page()

free_hot_cold_page() is used for order == 0 pages, and is where the
page's zone is decided.

In the normal case, these pages are freed to the per-cpu lists. When a
page needs transplanting (ie: the actual node it belongs to has changed,
and it needs to be moved to another zone), the pcp lists are skipped &
the page is freed via free_one_page().

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
page_alloc: transplant pages that are being flushed from the per-cpu lists

In free_pcppages_bulk(), check if a page needs to be moved to a new
node/zone & then perform the transplant (in a slightly defered manner).

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
x86: memlayout: add a arch specific inital memlayout setter.

On x86, we have numa_info specifically to track the numa layout, which
is precisely the data memlayout needs, so use it to create an initial
memlayout.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
init/main: call memlayout_global_init() in start_kernel().

memlayout_global_init() initializes the first memlayout, which is
assumed to match the initial page-flag nid settings.

This is done in start_kernel() as the initdata used to populate the
memlayout is purged from memory early in the boot process (XXX: When?).

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
dnuma: memlayout: add memory_add_physaddr_to_nid() for memory_hotplug

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
x86/mm/numa: when dnuma is enabled, use memlayout to handle memory hotplug's physaddr_to_nid.

When a memlayout is tracked (ie: CONFIG_DYNAMIC_NUMA is enabled), rather
than iterate over numa_meminfo, a lookup can be done using memlayout.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
mm/memory_hotplug: VM_BUG if nid is too large.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>
mm: add a early_param "extra_nr_node_ids" to increase nr_node_ids above the minimum by a percentage.

For dynamic numa, sometimes the hypervisor we're running under will want
to split a single NUMA node into multiple NUMA nodes. If the number of
numa nodes is limited to the number avaliable when the system booted (as
it is on x86), we may not be able to fully adopt the new memory layout
provided by the hypervisor.

This option allows reserving some extra node ids as a percentage of the
boot time node ids. While not perfect (idealy nr_node_ids would be fully
dynamic), this allows decent functionality without invasive changes to
the SL{U,A}B allocators.

Signed-off-by: Cody P Schafer <cody@linux.vnet.ibm.com>