mm: vmscan: fix endless loop in kswapd balancing

Kswapd does not in all places have the same criteria for a balanced
zone.  Zones are only being reclaimed when their high watermark is
breached, but compaction checks loop over the zonelist again when the
zone does not meet the low watermark plus two times the size of the
allocation.  This gets kswapd stuck in an endless loop over a small
zone, like the DMA zone, where the high watermark is smaller than the
compaction requirement.

Add a function, zone_balanced(), that checks the watermark, and, for
higher order allocations, if compaction has enough free memory.  Then
use it uniformly to check for balanced zones.

This makes sure that when the compaction watermark is not met, at least
reclaim happens and progress is made - or the zone is declared
unreclaimable at some point and skipped entirely.

Change-Id: I17a375f356189bde63406e2127153287208157a8
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-by: George Spelvin <linux@horizon.com>
Reported-by: Johannes Hirte <johannes.hirte@fem.tu-ilmenau.de>
Reported-by: Tomas Racek <tracek@redhat.com>
Tested-by: Johannes Hirte <johannes.hirte@fem.tu-ilmenau.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: 60cefed485a02bd99b6299dad70666fe49245da7
Git-repo: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
mm: vmscan: do not keep kswapd looping forever due to individual uncompactable zones

When a zone meets its high watermark and is compactable in case of
higher order allocations, it contributes to the percentage of the node's
memory that is considered balanced.

This requirement, that a node be only partially balanced, came about
when kswapd was desparately trying to balance tiny zones when all bigger
zones in the node had plenty of free memory.  Arguably, the same should
apply to compaction: if a significant part of the node is balanced
enough to run compaction, do not get hung up on that tiny zone that
might never get in shape.

When the compaction logic in kswapd is reached, we know that at least
25% of the node's memory is balanced properly for compaction (see
zone_balanced and pgdat_balanced).  Remove the individual zone checks
that restart the kswapd cycle.

Otherwise, we may observe more endless looping in kswapd where the
compaction code loops back to reclaim because of a single zone and
reclaim does nothing because the node is considered balanced overall.

See for example

  https://bugzilla.redhat.com/show_bug.cgi?id=866988

Change-Id: I9c8f180466fea814c3038fa019f317799ddc0bcb
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Reported-and-tested-by: Thorsten Leemhuis <fedora@leemhuis.info>
Reported-by: Jiri Slaby <jslaby@suse.cz>
Tested-by: John Ellson <john.ellson@comcast.net>
Tested-by: Zdenek Kabelac <zkabelac@redhat.com>
Tested-by: Bruno Wolff III <bruno@wolff.to>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: c702418f8a2fa6cc92e84a39880d458faf7af9cc
Git-repo: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
msm: Allow lowmem to be non contiguous and mixed.

Any image that is expected to have a lifetime of
the entire system can give the virtual address
space back for use in vmalloc.

Change-Id: I81ce848cd37e8573d706fa5d1aa52147b3c8da12
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
msm: Increase the kernel virtual area to include lowmem

Even though lowmem is accounted for in vmalloc space, allocation
comes only from the region bounded by VMALLOC_START and VMALLOC_END.
The kernel virtual area can now allocate from any unmapped region
starting from PAGE_OFFSET.

Change-Id: I291b9eb443d3f7445fd979bd7b09e9241ff22ba3
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
mm: Remove __init annotations from free_bootmem_late

free_bootmem_late is currently set up to only be used in init
functions. Some clients need to use this function past initcalls.
The functions themselves have no restrictions on being used later
minus the __init annotations so remove the annotation.

Change-Id: I7c7e15cf2780a8843ebb4610da5b633c9abb0b3d
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
mm: panic on the first bad page table entry access

Sometimes having a number of bad page table entries precipitates in
a crash much later. Because of this, we do not have any context for
the point at which the first bad pte entry was encountered. Hence,
panic on first such instance to help gather context for debug.

Change-Id: Idddf2b977214eb1463d08e16630e98264b9af487
Signed-off-by: Pushkar Joshi <pushkarj@codeaurora.org>
debug-pagealloc: Panic on pagealloc corruption

Currently, we just print the pagealloc corruption warnings and
proceed. Sometimes, we are getting multiple errors printed down
the line. It will be good to get the device state as early as
possible when we get the first pagealloc error.

Change-Id: I79155ac8a039b30a3a98d5dd1384d3923082712f
Signed-off-by: Subbaraman Narayanamurthy <subbaram@codeaurora.org>
mm: Update is_vmalloc_addr to account for vmalloc savings

is_vmalloc_addr current assumes that all vmalloc addresses
exist between VMALLOC_START and VMALLOC_END. This may not be
the case when interleaving vmalloc and lowmem. Update the
is_vmalloc_addr to properly check for this.

Change-Id: I5def3d6ae1a4de59ea36f095b8c73649a37b1f36
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
mm: methods for teaching filesystems about PG_swapcache pages

In order to teach filesystems to handle swap cache pages, three new page
functions are introduced:

  pgoff_t page_file_index(struct page *);
  loff_t page_file_offset(struct page *);
  struct address_space *page_file_mapping(struct page *);

page_file_index() - gives the offset of this page in the file in
PAGE_CACHE_SIZE blocks.  Like page->index is for mapped pages, this
function also gives the correct index for PG_swapcache pages.

page_file_offset() - uses page_file_index(), so that it will give the
expected result, even for PG_swapcache pages.

page_file_mapping() - gives the mapping backing the actual page; that is
for swap cache pages it will give swap_file->f_mapping.

Change-Id: I13d18bb25be606760eac26cc842eb7c9fc9e4766
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Eric Paris <eparis@redhat.com>
Cc: James Morris <jmorris@namei.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Neil Brown <neilb@suse.de>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Xiaotian Feng <dfeng@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: f981c5950fa85916ba49bea5d9a7a5078f47e569
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[ohaugan@codeaurora.org: Resolved merge issues]
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
mm: remove compressed copy from zram in-memory

Swap subsystem does lazy swap slot free with expecting the page would be
swapped out again so we can avoid unnecessary write.

But the problem in in-memory swap(ex, zram) is that it consumes memory
space until vm_swap_full(ie, used half of all of swap device) condition
meet.  It could be bad if we use multiple swap device, small in-memory
swap and big storage swap or in-memory swap alone.

This patch makes swap subsystem free swap slot as soon as swap-read is
completed and make the swapcache page dirty so the page should be
written out the swap device to reclaim it.  It means we never lose it.

I tested this patch with kernel compile workload.

1. before

   compile time : 9882.42
   zram max wasted space by fragmentation: 13471881 byte
   memory space consumed by zram: 174227456 byte
   the number of slot free notify: 206684

2. after

   compile time : 9653.90
   zram max wasted space by fragmentation: 11805932 byte
   memory space consumed by zram: 154001408 byte
   the number of slot free notify: 426972

Change-Id: Ida7dcf6fbd67408e2429483c28c75f59f83310d3
[akpm@linux-foundation.org: tweak comment text]
[artem.savkov@gmail.com: fix BUG due to non-swapcache pages in end_swap_bio_read()]
[akpm@linux-foundation.org: invert unlikely() test, augment comment, 80-col cleanup]
Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Artem Savkov <artem.savkov@gmail.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Seth Jennings <sjenning@linux.vnet.ibm.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Cc: Konrad Rzeszutek Wilk <konrad@darnok.org>
Cc: Shaohua Li <shli@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: b430e9d1c6d416306d44dbf3aa3148be7af78abc
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[ohaugan@codeaurora.org: Resolved merge issues]
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
swap: add per-partition lock for swapfile

swap_lock is heavily contended when I test swap to 3 fast SSD (even
slightly slower than swap to 2 such SSD).  The main contention comes
from swap_info_get().  This patch tries to fix the gap with adding a new
per-partition lock.

Global data like nr_swapfiles, total_swap_pages, least_priority and
swap_list are still protected by swap_lock.

nr_swap_pages is an atomic now, it can be changed without swap_lock.  In
theory, it's possible get_swap_page() finds no swap pages but actually
there are free swap pages.  But sounds not a big problem.

Accessing partition specific data (like scan_swap_map and so on) is only
protected by swap_info_struct.lock.

Changing swap_info_struct.flags need hold swap_lock and
swap_info_struct.lock, because scan_scan_map() will check it.  read the
flags is ok with either the locks hold.

If both swap_lock and swap_info_struct.lock must be hold, we always hold
the former first to avoid deadlock.

swap_entry_free() can change swap_list.  To delete that code, we add a
new highest_priority_index.  Whenever get_swap_page() is called, we
check it.  If it's valid, we use it.

It's a pity get_swap_page() still holds swap_lock().  But in practice,
swap_lock() isn't heavily contended in my test with this patch (or I can
say there are other much more heavier bottlenecks like TLB flush).  And
BTW, looks get_swap_page() doesn't really need the lock.  We never free
swap_info[] and we check SWAP_WRITEOK flag.  The only risk without the
lock is we could swapout to some low priority swap, but we can quickly
recover after several rounds of swap, so sounds not a big deal to me.
But I'd prefer to fix this if it's a real problem.

"swap: make each swap partition have one address_space" improved the
swapout speed from 1.7G/s to 2G/s.  This patch further improves the
speed to 2.3G/s, so around 15% improvement.  It's a multi-process test,
so TLB flush isn't the biggest bottleneck before the patches.

Change-Id: Ic8d600653479c0e6767f04462c3f057b0fe94332
[arnd@arndb.de: fix it for nommu]
[hughd@google.com: add missing unlock]
[minchan@kernel.org: get rid of lockdep whinge on sys_swapon]
Signed-off-by: Shaohua Li <shli@fusionio.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Seth Jennings <sjenning@linux.vnet.ibm.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
Cc: Dan Magenheimer <dan.magenheimer@oracle.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: ec8acf20afb8534ed511f6613dd2226b9e301010
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[lauraa@codeaurora.org: Various context fixups due to backport]
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
mm: vmscan: fix do_try_to_free_pages() livelock

This patch is based on KOSAKI's work and I add a little more description,
please refer https://lkml.org/lkml/2012/6/14/74.

Currently, I found system can enter a state that there are lots of free
pages in a zone but only order-0 and order-1 pages which means the zone is
heavily fragmented, then high order allocation could make direct reclaim
path's long stall(ex, 60 seconds) especially in no swap and no compaciton
enviroment.  This problem happened on v3.4, but it seems issue still lives
in current tree, the reason is do_try_to_free_pages enter live lock:

kswapd will go to sleep if the zones have been fully scanned and are still
not balanced.  As kswapd thinks there's little point trying all over again
to avoid infinite loop.  Instead it changes order from high-order to
0-order because kswapd think order-0 is the most important.  Look at
73ce02e9 in detail.  If watermarks are ok, kswapd will go back to sleep
and may leave zone->all_unreclaimable =3D 0.  It assume high-order users
can still perform direct reclaim if they wish.

Direct reclaim continue to reclaim for a high order which is not a
COSTLY_ORDER without oom-killer until kswapd turn on
zone->all_unreclaimble= .  This is because to avoid too early oom-kill.
So it means direct_reclaim depends on kswapd to break this loop.

In worst case, direct-reclaim may continue to page reclaim forever when
kswapd sleeps forever until someone like watchdog detect and finally kill
the process.  As described in:
http://thread.gmane.org/gmane.linux.kernel.mm/103737

We can't turn on zone->all_unreclaimable from direct reclaim path because
direct reclaim path don't take any lock and this way is racy.  Thus this
patch removes zone->all_unreclaimable field completely and recalculates
zone reclaimable state every time.

Note: we can't take the idea that direct-reclaim see zone->pages_scanned
directly and kswapd continue to use zone->all_unreclaimable.  Because, it
is racy.  commit 929bea7c71 (vmscan: all_unreclaimable() use
zone->all_unreclaimable as a name) describes the detail.

Change-Id: I49970a0fa751cf33af293fd1ee784e36422785b1
[akpm@linux-foundation.org: uninline zone_reclaimable_pages() and zone_reclaimable()]
Cc: Aaditya Kumar <aaditya.kumar.30@gmail.com>
Cc: Ying Han <yinghan@google.com>
Cc: Nick Piggin <npiggin@gmail.com>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Bob Liu <lliubbo@gmail.com>
Cc: Neil Zhang <zhangwm@marvell.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Minchan Kim <minchan@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Lisa Du <cldu@marvell.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 6e543d5780e36ff5ee56c44d7e2e30db3457a7ed
[lauraa@codeaurora.org: Some context fixups and variable name changes due
to backporting. Dropped parts that don't apply to older kernels]
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
mm: vmscan: Move pages that fail swapout to LRU active list

Move pages that fail swapout to the LRU active list to reduce
pressure on swap device when swapping out is already failing.
This helps when using a pseudo swap device such as zram which
starts failing when memory is low.

Change-Id: Ib136cd0a744378aa93d837a24b9143ee818c80b3
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
mm: swap: Rate limit swap write errors

If an error occurs in the swap subsystem when writing pages
out to a swap device the system might get overflowed with
logging errors because the mm subsystem will continously try
to swap out pages even in the event of error on the swap
device.

Reduce the amount of logging to prevent the kernel log
from being filled with these error messages.

Change-Id: Iddaf6c32ae87132817482cc4cb68b909a1c527e6
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Kconfig: Enable config to allow vmalloc savings

Select the config to reclaim the vmalloc space in lowmem
and disable the map around the hole config, where selected,
as both the configs cannot be enabled together

CRs-Fixed: 568470
Change-Id: I9dd3e0123e4fe3e33554cdbdf5a82e344c230f47
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
msm: Adjust the low memory boundary

The unused virtual address space in low memory is given to
vmalloc for use. This reduces the low memory space and
increases vmalloc space. Adjust the vmalloc_min in order
to increase the low memory.

Change-Id: I0dbef5b6e5ec3d19f5f93f06ed03a4cf1215dc4d
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
msm: Update the kernel log messages

After reclaiming vmalloc and adjusting lowmem, the two regions
are no longer contiguous. Update the messages printed during
bootup to list all regions corresponding to low memory and vmalloc
respectively

Change-Id: I2aeb25026c068e27860c3df1496fc6f5c420b818
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
msm: acpuclock-krait: Program both high- and low-performance sources

The latest LDO/BHS switching sequence employed in the 8974
krait-regulator driver involves mandatory operations that cause
the CPUs to switch to the low-performance clock source when the
CPU is powered from the LDO, and to the high-performance clock
source when powered from the BHS.

This results in potential undervolting of the CPUs since the
acpuclock driver does not program the low-performance source,
and the clock source it selects be default is running at the
same speed as DDR (which can be very fast).

Fix this by programming both the high and low-performance clock
sources to be the same at all times.

CRs-Fixed: 538629
Change-Id: Iaa39cca9165b892e52acd8ec3aac5d0784112623
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
msm: acpuclock-krait: Fix HFPLL regulator voting in regulator_init()

Simply incrementing l2_vreg_count is not enough to ensure the
regulators necessary for the L2 HFPLL are enabled. Instead,
enable_l2_regulators() must be called to place votes for the
regulators.

This problem will only result in incorrect behaviour (a missing
vote for the HFPLL regulators) if the L2 rate detected at boot is
not in the frequency table. In all other cases, the vote will be
applied as part of the hfpll_enable call when init_clock_sources()
is called for the L2.

Fix the code so that the voting works correctly even if the initial
L2 rate is not in the frequency table.

Change-Id: I8e84cd81de17a0efcc91281848180cf34ef9bb57
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
msm: acpuclock-krait: Update 'unknown rate' debug prints to warnings

While not fatal, boot performance will be suboptimal if the L2 or
CPU frequencies at boot are not in the frequency tables. This is
because acpuclock 'plays it safe' by applying the lowest possible
clock rate in such situations to ensure clocks are not set higher
than boot-up voltages can support.

Change-Id: I07fcff70d742b2590fa996ea4ca7a1f1c560eee3
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>