slub: Add frozen check in __slab_alloc

Verify that objects returned from __slab_alloc come from slab pages
in the correct state.

Change-Id: I873ca332ca1f8dd1aa58c6110a7d038a6e596a9e
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: Simplify control flow in __slab_alloc()

Simplify control flow a bit avoiding nesting.

Change-Id: I68ca675217885e320cd4193c81f87c70bcb47aad
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: Get rid of the node field

The node field is always page_to_nid(c->page). So its rather easy to
replace. Note that there maybe slightly more overhead in various hot paths
due to the need to shift the bits from page->flags. However, that is mostly
compensated for by a smaller footprint of the kmem_cache_cpu structure (this
patch reduces that to 3 words per cache) which allows better caching.

Change-Id: I323344ed066891315b4a3692ab33054fe7562b97
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: refactoring unfreeze_partials()

Current implementation of unfreeze_partials() is so complicated,
but benefit from it is insignificant. In addition many code in
do {} while loop have a bad influence to a fail rate of cmpxchg_double_slab.
Under current implementation which test status of cpu partial slab
and acquire list_lock in do {} while loop,
we don't need to acquire a list_lock and gain a little benefit
when front of the cpu partial slab is to be discarded, but this is a rare case.
In case that add_partial is performed and cmpxchg_double_slab is failed,
remove_partial should be called case by case.

I think that these are disadvantages of current implementation,
so I do refactoring unfreeze_partials().

Minimizing code in do {} while loop introduce a reduced fail rate
of cmpxchg_double_slab. Below is output of 'slabinfo -r kmalloc-256'
when './perf stat -r 33 hackbench 50 process 4000 > /dev/null' is done.

** before **
Cmpxchg_double Looping
------------------------
Locked Cmpxchg Double redos   182685
Unlocked Cmpxchg Double redos 0

** after **
Cmpxchg_double Looping
------------------------
Locked Cmpxchg Double redos   177995
Unlocked Cmpxchg Double redos 1

We can see cmpxchg_double_slab fail rate is improved slightly.

Bolow is output of './perf stat -r 30 hackbench 50 process 4000 > /dev/null'.

** before **
 Performance counter stats for './hackbench 50 process 4000' (30 runs):

     108517.190463 task-clock                #    7.926 CPUs utilized            ( +-  0.24% )
         2,919,550 context-switches          #    0.027 M/sec                    ( +-  3.07% )
           100,774 CPU-migrations            #    0.929 K/sec                    ( +-  4.72% )
           124,201 page-faults               #    0.001 M/sec                    ( +-  0.15% )
   401,500,234,387 cycles                    #    3.700 GHz                      ( +-  0.24% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
   250,576,913,354 instructions              #    0.62  insns per cycle          ( +-  0.13% )
    45,934,956,860 branches                  #  423.297 M/sec                    ( +-  0.14% )
       188,219,787 branch-misses             #    0.41% of all branches          ( +-  0.56% )

      13.691837307 seconds time elapsed                                          ( +-  0.24% )

** after **
 Performance counter stats for './hackbench 50 process 4000' (30 runs):

     107784.479767 task-clock                #    7.928 CPUs utilized            ( +-  0.22% )
         2,834,781 context-switches          #    0.026 M/sec                    ( +-  2.33% )
            93,083 CPU-migrations            #    0.864 K/sec                    ( +-  3.45% )
           123,967 page-faults               #    0.001 M/sec                    ( +-  0.15% )
   398,781,421,836 cycles                    #    3.700 GHz                      ( +-  0.22% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
   250,189,160,419 instructions              #    0.63  insns per cycle          ( +-  0.09% )
    45,855,370,128 branches                  #  425.436 M/sec                    ( +-  0.10% )
       169,881,248 branch-misses             #    0.37% of all branches          ( +-  0.43% )

      13.596272341 seconds time elapsed                                          ( +-  0.22% )

No regression is found, but rather we can see slightly better result.

Change-Id: I0e7fe4134beab1f844efc96834d87fc643fecfd1
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Joonsoo Kim <js1304@gmail.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
Merge "slub: Simplify control flow in __slab_alloc()" into android-4.4
Merge "slub: Get rid of the node field" into android-4.4
Merge "slub: refactoring unfreeze_partials()" into android-4.4
Linux 3.4.1

Change-Id: I817059fa823cc7bee1390c191b64a359acffd86a
net: tcp: Enable westwood and set as default

Change-Id: I44666f0d4f605b815185c97064d6eb6dfdd947af
block: Added zen, vr, sio, fiops and fifo

Change-Id: Ice9f59e0487ddfd8ae82803104d4d48bbd505206
intelli_plug: performance tune-up continued...

Change-Id: I59f6ccdbe9ce97f7247bc510d1cfff6b79565c42
Signed-off-by: Paul Reioux <reioux@gmail.com>
Linux 3.4.2

Change-Id: I4974751213bc87ccf01bc0a244919ad310cde44e
Linux 3.4.3

Change-Id: I12b03860d34fec13af354463f261aa1e4f342594
Linux 3.4.4

Change-Id: I68f7382e5517631a1442ea374dc83aa991cc58f1
Linux 3.4.5

Change-Id: Id52aa8c9e4b7d74b90658e7e3b33fe50a1e2b820
Linux 3.4.6

Change-Id: I8eb9ce1af8dab0a5e859b9e248d40ab6ac8d78ab
Linux 3.4.7

Change-Id: I69e3ca59fa38c60172b5ed102ee9975f32524699
Linux 3.4.8

Change-Id: Idb3c2a6cb01d2af2ff255d52897b5171e5fe44ad
Linux 3.4.9

Change-Id: I3396e916fa93b9ee27f7e908e6867f2b84af9662
Linux 3.4.10

Change-Id: Ic2c258b8dd6262c3b2d872f5164dd106dde341e5