Optimized ARM RWSEM algorithm

RWSEM implementation for ARM using atomic functions.
Heavily based on arch/sh/include/asm/rwsem.h

Signed-off-by: Ashwin Chaugule <ashwinc@codeaurora.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: 7583/1: decompressor: Enable unaligned memory access for v6 and above

Modern GCC can generate code which makes use of the CPU's native
unaligned memory access capabilities.  This is useful for the C
decompressor implementations used for unpacking compressed kernels.

This patch disables alignment faults and enables the v6 unaligned
access model on CPUs which support these features (i.e., v6 and
later), allowing full unaligned access support for C code in the
decompressor.

The decompressor C code must not be built to assume that unaligned
access works if support for v5 or older platforms is included in
the kernel.

For correct code generation, C decompressor code must always use
the get_unaligned and put_unaligned accessors when dealing with
unaligned pointers, regardless of this patch.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
arch:arm: enable unaligned capability for ARM

Signed-off-by: Dan Trifan <jimsth@gmail.com>
lib/lzo: Update LZO compression to current upstream version

This commit updates the kernel LZO code to the current upsteam version
which features a significant speed improvement - benchmarking the Calgary
and Silesia test corpora typically shows a doubled performance in
both compression and decompression on modern i386/x86_64/powerpc machines.

Signed-off-by: Markus F.X.J. Oberhumer <markus@oberhumer.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
lib:lzo: updated LZO libs

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b0af9cd9aab60ceb17d3ebabb9fdf4ff0a99cf50

Pull LZO compression update from Markus Oberhumer:
 "Summary:
  ========

  Update the Linux kernel LZO compression and decompression code to the
  current upstream version which features significant performance
  improvements on modern machines.

  Some *synthetic* benchmarks:
  ============================

    x86_64 (Sandy Bridge), gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:

                     compression speed   decompression speed

    LZO-2005    :         150 MB/sec          468 MB/sec
    LZO-2012    :         434 MB/sec         1210 MB/sec

    i386 (Sandy Bridge), gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:

                     compression speed   decompression speed

    LZO-2005    :         143 MB/sec          409 MB/sec
    LZO-2012    :         372 MB/sec         1121 MB/sec

    armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:

                     compression speed   decompression speed

    LZO-2005    :          27 MB/sec           84 MB/sec
    LZO-2012    :          44 MB/sec          117 MB/sec
  **LZO-2013-UA :          47 MB/sec          167 MB/sec

  Legend:

    LZO-2005    : LZO version in current 3.8 kernel (which is based on
                     the LZO 2.02 release from 2005)
    LZO-2012    : updated LZO version available in linux-next
  **LZO-2013-UA : updated LZO version available in linux-next plus experimental
                     ARM Unaligned Access patch. This needs approval
                     from some ARM maintainer ist NOT YET INCLUDED."

Andrew Morton <akpm@linux-foundation.org> acks it and says:
 "There's a new LZ4 on the block which is even faster than the sped-up
  LZO, but various filesystems and things use LZO"

* tag 'lzo-update-signature-20130226' of git://github.com/markus-oberhumer/linux:
  crypto: testmgr - update LZO compression test vectors
  lib/lzo: Update LZO compression to current upstream version
  lib/lzo: Rename lzo1x_decompress.c to lzo1x_decompress_safe.c

Signed-off-by: Dan Trifan <jimsth@gmail.com>
lib/lzo: huge LZO decompression speedup on ARM by using unaligned access

Signed-off-by: Markus F.X.J. Oberhumer <markus@oberhumer.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
decompressor: Add LZ4 decompressor module

Date	Tue, 26 Feb 2013 15:24:27 +0900

This patch adds support for LZ4 decompression in the Linux Kernel.
LZ4 Decompression APIs for kernel are based on LZ4 implementation
by Yann Collet.

LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
LZ4 source repository : http://code.google.com/p/lz4/

Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>

v2:
- Clean up code
- Enable unaligned access for ARM v6 and above with
  CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
- Add lz4_decompress() for faster decompression with
  uncompressed output size

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
lib: Add support for LZ4-compressed kernel

Date	Tue, 26 Feb 2013 15:24:28 +0900

This patch adds support for extracting LZ4-compressed kernel images,
as well as LZ4-compressed ramdisk images in the kernel boot process.

This depends on the patch below
decompressor: Add LZ4 decompressor module

Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>

v2:
- Clean up code
- Use lz4_decompress() for LZ4-compressed kernel during boot-process

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
arm: Add support for LZ4-compressed kernel

Date	Tue, 26 Feb 2013 15:24:29 +0900

This patch integrates the LZ4 decompression code to the arm pre-boot code.
And it depends on two patchs below

lib: Add support for LZ4-compressed kernel
decompressor: Add LZ4 decompressor module

Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>

v2:
- Apply CFLAGS, -Os to decompress.o to improve decompress
  performance during boot-up process

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
lib: Add lz4 compressor module

From	Chanho Min <>

This patch adds support for LZ4 compression in the Linux Kernel.
LZ4 Compression APIs for kernel are based on LZ4 implementation
by Yann Collet and changed with kernel coding style.

LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
LZ4 source repository : http://code.google.com/p/lz4/
svn revision : r90

Two APIs are added:
lz4_compress() support basic lz4 compression whereas lz4hc_compress() support
high compression or CPU performance get lower but compression ratio get higher.
Also, we require the pre-allocated working memory with the defined size and
destination buffer must be allocated with the size of lz4_compressbound.

Signed-off-by: Chanho Min <chanho.min@lge.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
crypto: Add lz4 Cryptographic API

From	Chanho Min <>

This patch adds support for lz4 and lz4hc compression algorithm
using the lib/lz4/* codebase.
Signed-off-by: Chanho Min <chanho.min@lge.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
LZ4: compression/decompression signedness mismatch (v2)

LZ4 compression and decompression functions require different
in signedness input/output parameters: unsigned char for
compression and signed char for decompression.

Change decompression API to require "(const) unsigned char *".

v2: minor coding style fix.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
logger: Add an enable toggle for entry storage; TNX AndreiLux and Dorimanx

You can disable logging by echo-ing 0 in /sys/module/logger/parameters/log_enabled
By default the logging is enabled.

Signed-off-by: Dan Trifan <jimsth@gmail.com>
sync: don't block the flusher thread waiting on IO

When sync does it's WB_SYNC_ALL writeback, it issues data Io and
then immediately waits for IO completion. This is done in the
context of the flusher thread, and hence completely ties up the
flusher thread for the backing device until all the dirty inodes
have been synced. On filesystems that are dirtying inodes constantly
and quickly, this means the flusher thread can be tied up for
minutes per sync call and hence badly affect system level write IO
performance as the page cache cannot be cleaned quickly.

We already have a wait loop for IO completion for sync(2), so cut
this out of the flusher thread and delegate it to wait_sb_inodes().
Hence we can do rapid IO submission, and then wait for it all to
complete.

Effect of sync on fsmark before the patch:

FSUse%        Count         Size    Files/sec     App Overhead
.....
     0       640000         4096      35154.6          1026984
     0       720000         4096      36740.3          1023844
     0       800000         4096      36184.6           916599
     0       880000         4096       1282.7          1054367
     0       960000         4096       3951.3           918773
     0      1040000         4096      40646.2           996448
     0      1120000         4096      43610.1           895647
     0      1200000         4096      40333.1           921048

And a single sync pass took:

  real    0m52.407s
  user    0m0.000s
  sys     0m0.090s

After the patch, there is no impact on fsmark results, and each
individual sync(2) operation run concurrently with the same fsmark
workload takes roughly 7s:

  real    0m6.930s
  user    0m0.000s
  sys     0m0.039s

IOWs, sync is 7-8x faster on a busy filesystem and does not have an
adverse impact on ongoing async data write operations.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
writeback: fix writeback cache thrashing

From: Namjae Jeon <namjae.jeon@samsung.com>

Consider Process A: huge I/O on sda
        doing heavy write operation - dirty memory becomes more
        than dirty_background_ratio
        on HDD - flusher thread flush-8:0

Consider Process B: small I/O on sdb
        doing while [1]; read 1024K + rewrite 1024K + sleep 2sec
        on Flash device - flusher thread flush-8:16

As Process A is a heavy dirtier, dirty memory becomes more
than dirty_background_thresh. Due to this, below check becomes
true(checking global_page_state in over_bground_thresh)
for all bdi devices(even for very small dirtied bdi - sdb):

In this case, even small cached data on 'sdb' is forced to flush
and writeback cache thrashing happens.

When we added debug prints inside above 'if' condition and ran
above Process A(heavy dirtier on bdi with flush-8:0) and
Process B(1024K frequent read/rewrite on bdi with flush-8:16)
we got below prints:

[Test setup: ARM dual core CPU, 512 MB RAM]

[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  56064 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  56704 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84720 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 94720 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   384 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   960 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =    64 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92160 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   256 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   768 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =    64 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   256 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   320 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =     0 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92032 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 91968 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   192 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =  1024 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =    64 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   192 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   576 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =     0 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 84352 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   192 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =   512 KB
[over_bground_thresh]: wakeup flush-8:16 : BDI_RECLAIMABLE =     0 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92608 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE = 92544 KB

As mentioned in above log, when global dirty memory > global background_thresh
small cached data is also forced to flush by flush-8:16.
If removing global background_thresh checking code, we can reduce cache
thrashing of frequently used small data.
And It will be great if we can reserve a portion of writeback cache using
min_ratio.

After applying patch:
$ echo 5 > /sys/block/sdb/bdi/min_ratio
$ cat /sys/block/sdb/bdi/min_ratio
5
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  56064 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  56704 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  84160 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  96960 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  94080 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  93120 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  93120 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  91520 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  89600 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  93696 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  93696 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  72960 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  90624 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  90624 KB
[over_bground_thresh]: wakeup flush-8:0 : BDI_RECLAIMABLE =  90688 KB

As mentioned in the above logs, once cache is reserved for Process B,
and patch is applied there is less writeback cache thrashing on sdb
by frequent forced writeback by flush-8:16 in over_bground_thresh.

After all, small cached data will be flushed by periodic writeback
once every dirty_writeback_interval.

Suggested-by: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Signed-off-by: Namjae Jeon <namjae.jeon@samsung.com>
Signed-off-by: Vivek Trivedi <t.vivek@samsung.com>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
writeback: Fix occasional slow sync(1)

In case when system contains no dirty pages, wakeup_flusher_threads()
will submit WB_SYNC_NONE writeback for 0 pages so wb_writeback() exits
immediately without doing anything. Thus sync(1) will write all the
dirty inodes from a WB_SYNC_ALL writeback pass which is slow.

Fix the problem by using get_nr_dirty_pages() in
wakeup_flusher_threads() instead of calculating number of dirty pages
manually. That function also takes number of dirty inodes into account.

CC: stable@vger.kernel.org
Reported-by: Paul Taysom <taysom@chromium.org>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Cristoforo Cataldo <cristoforo.cataldo@gmail.com>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm:page-writeback: increase bdi_min_ratio to 5

Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm: fix calculation of dirtyable memory

The system uses global_dirtyable_memory() to calculate number of
dirtyable pages/pages that can be allocated to the page cache.  A bug
causes an underflow thus making the page count look like a big unsigned
number.  This in turn confuses the dirty writeback throttling to
aggressively write back pages as they become dirty (usually 1 page at a
time).  This generally only affects systems with highmem because the
underflowed count gets subtracted from the global count of dirtyable
memory.

The problem was introduced with v3.2-4896-gab8fabd

Fix is to ensure we don't get an underflowed total of either highmem or
global dirtyable memory.

Signed-off-by: Sonny Rao <sonnyrao@chromium.org>
Signed-off-by: Puneet Kumar <puneetster@chromium.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Tested-by: Damien Wyart <damien.wyart@free.fr>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
block: account iowait time when waiting for completion of IO request

Using wait_for_completion() for waiting for a IO request to be executed
results in wrong iowait time accounting. For example, a system having
the only task doing write() and fdatasync() on a block device can be
reported being idle instead of iowaiting as it should because
blkdev_issue_flush() calls wait_for_completion() which in turn calls
schedule() that does not increment the iowait proc counter and thus does
not turn on iowait time accounting.

The patch makes block layer use wait_for_completion_io() instead of
wait_for_completion() where appropriate to account iowait time
correctly.

Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
sched: add wait_for_completion_io[_timeout]

The only difference between wait_for_completion[_timeout]() and
wait_for_completion_io[_timeout]() is that the latter calls
io_schedule_timeout() instead of schedule_timeout() so that the caller
is accounted as waiting for IO, not just sleeping.

These functions can be used for correct iowait time accounting when the
completion struct is actually used for waiting for IO (e.g. completion
of a bio request in the block layer).

Modified to compile with 3.4.0 on the Nexus 4 (Mako)

Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>