staging: zram: add Crypto API support

Date	Tue, 30 Jul 2013 14:30:48 +0200

Current version of zram does not allow any substitution of a default
compression algorithm. Therefore, I decided to change the existing
implementation of page compression by adding Crypto API compability.

All direct calls to lzo1x compression/decompression methods are now
replaced by calls consistent with Crypto. Also, I removed "workmem"
field from struct zram_meta, as it was there for lzo1x purposes only
and is no longer needed. Finally, I added a set of functions required
by Crypto API to work properly.

In order to substitute the default algorithm (lzo), change the value
of zram.compressor module parameter to a proper name (e.g. lz4).

Signed-off-by: Piotr Sarna <p.sarna@partner.samsung.com>
Acked-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>

backported to Linux 3.4 by faux123

Signed-off-by: Paul Reioux <reioux@gmail.com>
staging: zram: add per-cpu support to Crypto

Date	Tue, 30 Jul 2013 14:30:49 +0200

Since original zram code did not implement any per-cpu operations,
my previous patch (staging: zram: add Crypto API support) did not
include them either.

This patch complements the first one with per-cpu support for Crypto,
allocating tfms buffer separately for each online processor.
Changes are based on zswap and zcache per-cpu code.

Basic tests (concurrent writing several 10-40MB chunks to zram) performed
on an ARM-based EXYNOS4412 Quad-Core showed that per-cpu code provides
noticeable time saving, ranging between 30-40% for LZO and LZ4 compressors.
Sample data (LZO): writing 160MB, 40MB per thread took 0.60s with per-cpu
code included and approximately 0.80s without per-cpu support.

Signed-off-by: Piotr Sarna <p.sarna@partner.samsung.com>
Acked-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Signed-off-by: Kyungmin Park <kyungmin.park@samsung.com>

backported to Linux 3.4 by faux123
Signed-off-by: Paul Reioux <reioux@gmail.com>
zram_drv: switch default compressor to new lz4

Signed-off-by: Paul Reioux <reioux@gmail.com>
zcache-main.c: use MACRO define instead of hardcoded default of lzo

Signed-off-by: Paul Reioux <reioux@gmail.com>
zcache-main: switch to lz4 as default

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: Reduce overestimating rq->avg_idle

Date	Thu, 29 Aug 2013 13:05:34 -0700

When updating avg_idle, if the delta exceeds some max value, then avg_idle
gets set to the max, regardless of what the previous avg was. This can cause
avg_idle to often be overestimated.

This patch modifies the way we update avg_idle by always updating it with the
function call to update_avg() first. Then, if avg_idle exceeds the max, we set
it to the max.

Signed-off-by: Jason Low <jason.low2@hp.com>
Reviewed-by: Rik van Riel <riel@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: Consider max cost of idle balance per sched domain

Date    Thu, 29 Aug 2013 13:05:35 -0700

In this patch, we keep track of the max cost we spend doing idle load balancing
for each sched domain. If the avg time the CPU remains idle is less then the
time we have already spent on idle balancing + the max cost of idle balancing
in the sched domain, then we don't continue to attempt the balance. We also
keep a per rq variable, max_idle_balance_cost, which keeps track of the max
time spent on newidle load balances throughout all its domains. Additionally,
we swap sched_migration_cost used in idle_balance for rq->max_idle_balance_cost.

By using the max, we avoid overrunning the average. This further reduces the chance
we attempt balancing when the CPU is not idle for longer than the cost to balance.

I also limited the max cost of each domain to 5*sysctl_sched_migration_cost as
a way to prevent the max from becoming too inflated.

Signed-off-by: Jason Low <jason.low2@hp.com>

backported for Linux 3.4

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: Periodically decay max cost of idle balance

This RFC patch builds on patch 2 and periodically decays that max value to
do idle balancing per sched domain.

Though we want to decay it fairly consistently, we may not want to lower it by
too much each time, especially since avg_idle is capped based on that value.
So I thought that decaying the value every second and lowering it by half a
percent each time appeared to be fairly reasonable.

This change would allow us to remove the limit we set on each domain's max cost
to idle balance. Also, since the max can be reduced now, we now have to
update rq->max_idle balance_cost more frequently. So after every idle balance,
we loop through the sched domain to find the max sd's newidle load balance cost
for any one domain. Then we will set rq->max_idle_balance_cost to that value.

Since we are now decaying the max cost to do idle balancing, that max cost can
also become not high enough. One possible explanation for why is that
besides the time spent on each newidle load balance, there are other costs
associated with attempting idle balancing. Idle balance also releases and
reacquires a spin lock. That cost is not counted when we keep track of each
domain's cost to do newidle load balance. Also, acquiring the rq locks can
potentially prevent other CPUs from running something useful. And after
migrating tasks, it might potentially have to pay the costs of cache misses and
refreshing tasks' cache.

Because of that, this patch also compares avg_idle with max cost to do idle
balancing + sched_migration_cost. While using the max cost helps reduce
overestimating the average idle, the sched_migration_cost can help account
for those additional costs of idle balancing.

Signed-off-by: Jason Low <jason.low2@hp.com>
[peterz: rewrote the logic, but kept the spirit]
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
backported to Linux 3.4
Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: change load balance number to h_nr_running of run queue

Date    Sun, 18 Aug 2013 16:25:15 +0800

Since rq->nr_running would include both migration and rt task, it is not
reasonable to seek to move nr_running number of task in the load_balance
function, since it only apply to cfs type.

Change it to cfs's h_nr_running, which could well present the task
number in current cfs queue.

Signed-off-by: Lei Wen <leiwen@marvell.com>
backported to Linux 3.4 by faux123
Signed-off-by: Paul Reioux <reioux@gmail.com>
kernel/sched/fair.c: fix merge derp

 from patch sched: change load balance number to h_nr_running of run queue

Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: change cpu_avg_load_per_task using h_nr_running

Date	Sun, 18 Aug 2013 16:25:16 +0800

Since cpu_avg_load_per_task is used only by cfs scheduler, its meaning
should present the average cfs type task load in the current run queue.
Thus we change it to h_nr_running for well presenting its meaning.

Signed-off-by: Lei Wen <leiwen@marvell.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: change pick_next_task_fair to h_nr_running

Date	Sun, 18 Aug 2013 16:25:18 +0800

Since pick_next_task_fair only want to ensure there is some task in the
run queue to be picked up, it should use the h_nr_running instead of
nr_running, since nr_running cannot present all tasks if group existed.

Signed-off-by: Lei Wen <leiwen@marvell.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: change find_busiest_queue to h_nr_running

Date    Sun, 18 Aug 2013 16:25:20 +0800

Since find_busiest_queue try to avoid do load balance for runqueue
which has only one cfs task and its load is above the imbalance
value calculated, we should use h_nr_running of cfs instead of
nr_running of rq.

Signed-off-by: Lei Wen <leiwen@marvell.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: change active_load_balance_cpu_stop to use h_nr_running

Date	Sun, 18 Aug 2013 16:25:21 +0800

We should only avoid do the active load balance when there is no
cfs type task. If just use rq->nr_running, it is possible for the
source cpu has multiple rt task, while zero cfs task, so that it
would confuse the active load balance function that try to move,
but find no task it could move.

Signed-off-by: Lei Wen <leiwen@marvell.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
sched: document the difference between nr_running and h_nr_running

Date    Sun, 18 Aug 2013 16:25:22 +0800

Signed-off-by: Lei Wen <leiwen@marvell.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
slub: Add frozen check in __slab_alloc

Verify that objects returned from __slab_alloc come from slab pages
in the correct state.

Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: Simplify control flow in __slab_alloc()

Simplify control flow a bit avoiding nesting.

Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: new_slab_objects() can also get objects from partial list

Moving the attempt to get a slab page from the partial lists simplifies
__slab_alloc which is rather complicated.

Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: Get rid of the node field

The node field is always page_to_nid(c->page). So its rather easy to
replace. Note that there maybe slightly more overhead in various hot paths
due to the need to shift the bits from page->flags. However, that is mostly
compensated for by a smaller footprint of the kmem_cache_cpu structure (this
patch reduces that to 3 words per cache) which allows better caching.

Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
slub: Separate out kmem_cache_cpu processing from deactivate_slab

Processing on fields of kmem_cache_cpu is cleaner if code working on fields
of this struct is taken out of deactivate_slab().

Acked-by: David Rientjes <rientjes@google.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>