better io request merge changes: cfq-avoid-move-request-from-different-queue.patch and block-recursive-merge-request.patch.
to reduce the overhead of finding a swap pages cluster: swap-simple-buddy-allocation.patch
changes to enable 0-200 vm.swappiness for file page preservation: swappiness-exceed-100.patch and swappiness-fix-zone-reclaim.patch
improvement on swap readahead: hugh2.patch and swap-readahead-heuristic-improvement.patch
swap: make cluster allocator per-cpu

swap cluster allocatior is to get better request merge to improve performance.
But the cluster is shared globally, if multiple tasks are doing swap, this will
cause interleave disk access. While multiple tasks swap is quite common, for
example, each numa node has a kswapd thread doing swap.

We makes the cluster allocator per-cpu here. The interleave disk access issue
goes away. All tasks will do sequential swap.

In my test (swap to a 2-disk raid0 partition), this improves around 10%
throughput, and request size is increased significantly.

This might improve swap readahead too. Considering page reclaim always swaps
several adjancent pages, this will make swap readahead not mis-prefetch data
for sequential access pattern workload. For random access pattern workload,
there should be no overhead as we only enable the per-cpu allocator for SSD, so
no seek penality.

Signed-off-by: Shaohua Li <shli@fusionio.com>
swap: make each swap partition have one address_space

When I use several fast SSD to do swap, swapper_space.tree_lock is heavily
contended. This makes each swap partition have one address_space to reduce the
lock contention. There is an array of address_space for swap. The swap entry
type is the index to the array.

In my test with 3 SSD, this increases the swapout throughput 20%.

There are some code here which looks unnecessary, for example, moving some code
from swapops.h to swap.h and soem changes in audit_tree.c. Those are to make
the code compile.

Signed-off-by: Shaohua Li <shli@fusionio.com>
swap: add per-partition lock for swapfile

swap_lock is heavily contended when I test swap to 3 fast SSD (even slightly
slower than swap to 2 such SSD). The main contention comes from
swap_info_get(). This patch tries to fix the gap with adding a new
per-partition lock.

global data like nr_swapfiles, total_swap_pages, least_priority and swap_list are
still protected by swap_lock.

nr_swap_pages is an atomic now, it can be changed without swap_lock. In theory,
it's possible get_swap_page() finds no swap pages but actually there are free
swap pages. But sounds not a big problem.

accessing partition specific data (like scan_swap_map and so on) is only
protected by swap_info_struct.lock.

Changing swap_info_struct.flags need hold swap_lock and swap_info_struct.lock,
because scan_scan_map() will check it. read the flags is ok with either the
locks hold.

If both swap_lock and swap_info_struct.lock must be hold, we always hold the
former first to avoid deadlock.

swap_entry_free() can change swap_list. To delete that code, we add a new
highest_priority_index. Whenever get_swap_page() is called, we check it. If
it's valid, we use it.

It's a pitty get_swap_page() still holds swap_lock(). But in practice,
swap_lock() isn't heavily contended in my test with this patch (or I can say
there are other much more heavier bottlenecks like TLB flush). And BTW, looks
get_swap_page() doesn't really need the lock. We never free swap_info[] and we
check SWAP_WRITEOK flag. The only risk without the lock is we could swapout to
some low priority swap, but we can quickly recover after several rounds of
swap, so sounds not a big deal to me. But I'd prefer to fix this if it's a real
problem.

Signed-off-by: Shaohua Li <shli@fusionio.com>
mm: make madvise(MADV_WILLNEED) support swap file prefetch

Make madvise(MADV_WILLNEED) support swap file prefetch. If memory is swapout,
this syscall can do swapin prefetch. It has no impact if the memory isn't
swapout.

Signed-off-by: Shaohua Li <shli@fusionio.com>
swap: fix a typo

9a7d901aca1 has a typo.

Signed-off-by: Shaohua Li <shli@fusionio.com>
mm/rmap: Convert the struct anon_vma::mutex to an rwsem

Convert the struct anon_vma::mutex to an rwsem, which will help
in solving a page-migration scalability problem. (Addressed in
a separate patch.)

The conversion is simple and straightforward: in every case
where we mutex_lock()ed we'll now down_write().

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Turner <pjt@google.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Mel Gorman <mgorman@suse.de>
mm/rmap, migration: Make rmap_walk_anon() and try_to_unmap_anon() more scalable

rmap_walk_anon() and try_to_unmap_anon() appears to be too
careful about locking the anon vma: while it needs protection
against anon vma list modifications, it does not need exclusive
access to the list itself.

Transforming this exclusive lock to a read-locked rwsem removes
a global lock from the hot path of page-migration intense
threaded workloads which can cause pathological performance like
this:

    96.43%        process 0  [kernel.kallsyms]  [k] perf_trace_sched_switch
                  |
                  --- perf_trace_sched_switch
                      __schedule
                      schedule
                      schedule_preempt_disabled
                      __mutex_lock_common.isra.6
                      __mutex_lock_slowpath
                      mutex_lock
                     |
                     |--50.61%-- rmap_walk
                     |          move_to_new_page
                     |          migrate_pages
                     |          migrate_misplaced_page
                     |          __do_numa_page.isra.69
                     |          handle_pte_fault
                     |          handle_mm_fault
                     |          __do_page_fault
                     |          do_page_fault
                     |          page_fault
                     |          __memset_sse2
                     |          |
                     |           --100.00%-- worker_thread
                     |                     |
                     |                      --100.00%-- start_thread
                     |
                      --49.39%-- page_lock_anon_vma
                                try_to_unmap_anon
                                try_to_unmap
                                migrate_pages
                                migrate_misplaced_page
                                __do_numa_page.isra.69
                                handle_pte_fault
                                handle_mm_fault
                                __do_page_fault
                                do_page_fault
                                page_fault
                                __memset_sse2
                                |
                                 --100.00%-- worker_thread
                                           start_thread

With this change applied the profile is now nicely flat
and there's no anon-vma related scheduling/blocking.

Rename anon_vma_[un]lock() => anon_vma_[un]lock_write(),
to make it clearer that it's an exclusive write-lock in
that case - suggested by Rik van Riel.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Paul Turner <pjt@google.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Mel Gorman <mgorman@suse.de>
smp: make smp_call_function_many use the similar logic like smp_call_function_single

I'm testing swapout workload in a two-socket Xeon machine. The workload has 10
threads, each thread sequentially accesses separate memory region. TLB flush
overhead is very big in the workload. For each page, page reclaim need move it
from active lru list and then unmap it. Both need a TLB flush. And this is a
multthread workload, TLB flush happens in 10 CPUs. In X86, TLB flush uses
generic smp_call)function. So this workload stress smp_call_function_many
heavily.

Without patch, perf shows:
+  24.49%  [k] generic_smp_call_function_interrupt
-  21.72%  [k] _raw_spin_lock
   - _raw_spin_lock
      + 79.80% __page_check_address
      + 6.42% generic_smp_call_function_interrupt
      + 3.31% get_swap_page
      + 2.37% free_pcppages_bulk
      + 1.75% handle_pte_fault
      + 1.54% put_super
      + 1.41% grab_super_passive
      + 1.36% __swap_duplicate
      + 0.68% blk_flush_plug_list
      + 0.62% swap_info_get
+   6.55%  [k] flush_tlb_func
+   6.46%  [k] smp_call_function_many
+   5.09%  [k] call_function_interrupt
+   4.75%  [k] default_send_IPI_mask_sequence_phys
+   2.18%  [k] find_next_bit

swapout throughput is around 1300M/s.

With the patch, perf shows:
-  27.23%  [k] _raw_spin_lock
   - _raw_spin_lock
      + 80.53% __page_check_address
      + 8.39% generic_smp_call_function_single_interrupt
      + 2.44% get_swap_page
      + 1.76% free_pcppages_bulk
      + 1.40% handle_pte_fault
      + 1.15% __swap_duplicate
      + 1.05% put_super
      + 0.98% grab_super_passive
      + 0.86% blk_flush_plug_list
      + 0.57% swap_info_get
+   8.25%  [k] default_send_IPI_mask_sequence_phys
+   7.55%  [k] call_function_interrupt
+   7.47%  [k] smp_call_function_many
+   7.25%  [k] flush_tlb_func
+   3.81%  [k] _raw_spin_lock_irqsave
+   3.78%  [k] generic_smp_call_function_single_interrupt

swapout throughput is around 1400M/s. So there is around a 7% improvement, and
total cpu utilization doesn't change.

Without the patch, cfd_data is shared by all CPUs.
generic_smp_call_function_interrupt does read/write cfd_data several times
which will create a lot of cache ping-pong. With the patch, the data becomes
per-cpu. The ping-pong is avoided. And from the perf data, this doesn't make
call_single_queue lock contend.

Next step is to remove generic_smp_call_function_interrupt from arch code.

Signed-off-by: Shaohua Li <shli@fusionio.com>
Fix some problem in madvise WILL_NEED implementation

1. build error without CONFIG_SWAP
2. warning in i386 system
3. madvise early failure triggers warning
per-partition swapfile fix

1. fix two warning reported by lockdep
2. fix a potential deadlock