[PATCH 2/3] Optimize copy_page for modern ARM platforms

The existing implementation of copy_page for ARM appears to be
optimized for older platforms. Benchmark testing in a sandbox
environment shows suboptimal performance on modern platforms
like armv6 and armv7, with speed-ups ranging from 10% (Cortex A8)
to 80% (armv6 used in Raspberry Pi) being achievable.

This commit optimizes copy_page and introduces the new compile-time
constant PREFETCH_DISTANCE, defined in cache.h, which when
multiplied by L1_CACHE_BYTES is equal to the offset used for
prefetches performed with the PLD instruction. For platforms where
L1_CACHE_BYTES is 32 (armv5 and armv6), copy_page processes 32 bytes
at a time while doing one prefetch per iteration, while for armv7
(with L1_CACHE_BYTES equal to 64), 64 bytes are processed at at time
with one prefetch per iteration. When no preload instruction is
available (platforms earlier than armv5), no preload instructions
are generated and 32 bytes are processed at at time.

To facilitate specifying instructions for architectures with no
preload instruction, the NO_PLD macro is added to assembler.h,
augmenting the PLD macro.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: 7301/1: Rename the T() macro to TUSER() to avoid namespace conflicts

This macro is used to generate unprivileged accesses (LDRT/STRT) to user
space.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
[PATCH 1/3] Rename ARM assembler push/pull macros

The ARM assembler library functions use a macro called "push" that
along with a macro called "pull" is used to shift bytes around in a
word in an endian-independent way. However, the modern unified
ARM assembler syntax also defines the instruction "push" to push
data onto the stack, which has specific encodings in the Thumb2
instruction set. For prevent possible conflicts going forward, and
to allow the use of the more transparent "push" instruction along
with the modern unified assembler syntax, this patch renames all
occurrences of the "push" macro to "pushbits", as well as renaming
the macro argument, when also called "push", to "pushshift".

For consistency, the macro called "pull" with its argument name
"pull" are also renamed to "pullbits" and "pullshift", respectively.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
[PATCH 3/3] Optimize ARM memset and memzero functions

The ARM memset and memzero functions are optimized with lower overhead
for small requests, generation of more 16-bit Thumb2 instructions when
compiled in Thumb2 mode, and configurable destination alignment before
the main block-copying loop.

The new compile-time constant MEMSET_WRITE_ALIGN_BYTES is introduced
in assembler.h, to augment the CALGN macro that previously regulated
32-byte write alignment but was only used on the Feroceon platform.
MEMSET_WRITE_ALIGN_BYTES can have values of 0 (no write alignment),
8, or 32. Apart from Feroceon, memset write alignment of 32 bytes
appears to benefit the armv6 platform, while the armv7 platform
seems benefit from alignment to 8 bytes for memset/memzero.

The CALGN macro is renamed to MEMSET_CALGN for memset and memzero; the
original CALGN macro is reserved for the memcpy family of functions
(memcpy, copy_from_user, copy_to_user) currently implemented in
copy_template.S, and the associated compile time constant
WRITE_ALIGN_BYTES defines the write alignment for memcpy-related
functions that will be utilized in subsequent contributions. Because
the current CALGN implementation in copy_template.S only implements
32-byte write alignment and is broken on Thumb2, it is only enabled
when WRITE_ALIGN_BYTES is equal to 32 and Thumb2 mode is not enabled.

Finally, memset and memzero now include a directive to enable unified
ARM assembler syntax.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: assembler.h: use same WRITE_ALIGN_BYTES on omap4 as on armv6 platforms

Not all armv7 have 64 kbytes of l1 cache, for example the
Cortex-A9, which has 32 kbytes. As all omap4 platforms also
run Cortex-A9, adjust WRITE_ALIGN_BYTES accordingly.

Still needs testing.

Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: implement optimized percpu variable access

From: Rob Herring <rob.herring@calxeda.com>

Use the previously unused TPIDRPRW register to store percpu offsets.
TPIDRPRW is only accessible in PL1, so it can only be used in the kernel.

This replaces 2 loads with a mrc instruction for each percpu variable
access. With hackbench, the performance improvement is 1.4% on Cortex-A9
(highbank). Taking an average of 30 runs of "hackbench -l 1000" yields:

Before: 6.2191
After: 6.1348

Will Deacon reported similar delta on v6 with 11MPCore.

The asm "memory" constraints are needed here to ensure the percpu offset
gets reloaded. Testing by Will found that this would not happen in
__schedule() which is a bit of a special case as preemption is disabled
but the execution can move cores.

Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: keep __my_cpu_offset consistent with generic one

Commit 14318efb(ARM: 7587/1: implement optimized percpu variable access)
introduces arm's __my_cpu_offset to optimize percpu vaiable access,
which really works well on hackbench, but will cause __my_cpu_offset
to return garbage value before it is initialized in cpu_init() called
by setup_arch, so accessing percpu variable before setup_arch may cause
kernel hang. But generic __my_cpu_offset always returns zero before
percpu area is brought up, and won't hang kernel.

So the patch tries to clear __my_cpu_offset on boot CPU early
to avoid boot hang.

At least now percpu variable is accessed by lockdep before
setup_arch(), and enabling CONFIG_LOCK_STAT or CONFIG_DEBUG_LOCKDEP
can trigger kernel hang.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Nicolas Pitre <nico@linaro.org>
Cc: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ext4: Speed up FITRIM by recording flags in ext4_group_info

In ext4, when FITRIM is called every time, we iterate all the
groups and do trim one by one. It is a bit time wasting if the
group has been trimmed and there is no change since the last
trim.

So this patch adds a new flag in ext4_group_info->bb_state to
indicate that the group has been trimmed, and it will be cleared
if some blocks is freed(in release_blocks_on_commit). Another
trim_minlen is added in ext4_sb_info to record the last minlen
we use to trim the volume, so that if the caller provide a small
one, we will go on the trim regardless of the bb_state.

A simple test with my intel x25m ssd:
df -h shows:
/dev/sdb1              40G   21G   17G  56% /mnt/ext4
Block size:               4096

run the FITRIM with the following parameter:
range.start = 0;
range.len = UINT64_MAX;
range.minlen = 1048576;

without the patch:
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.505s
user	0m0.000s
sys	0m1.224s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.359s
user	0m0.000s
sys	0m1.178s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.228s
user	0m0.000s
sys	0m1.151s

with the patch:
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.625s
user	0m0.000s
sys	0m1.269s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m0.002s
user	0m0.000s
sys	0m0.001s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m0.002s
user	0m0.000s
sys	0m0.001s

A big improvement for the 2nd and 3rd run.

Even after I delete some big image files, it is still much
faster than iterating the whole disk.

[root@boyu-tm test]# time ./ftrim /mnt/ext4/a
real	0m1.217s
user	0m0.000s
sys	0m0.196s

Cc: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Andreas Dilger <adilger.kernel@dilger.ca>
Signed-off-by: Tao Ma <boyu.mt@taobao.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Dan Trifan <jimsth@gmail.com>

Conflicts:
	fs/ext4/mballoc.c

Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm: remove compressed copy from zram in-memory

Swap subsystem does lazy swap slot free with expecting the page
would be swapped out again so we can avoid unnecessary write.

But the problem in in-memory swap(ex, zram) is that it consumes
memory space until vm_swap_full(ie, used half of all of swap device)
condition meet. It could be bad if we use multiple swap device,
small in-memory swap and big storage swap or in-memory swap alone.

This patch makes swap subsystem free swap slot as soon as swap-read
is completed and make the swapcache page dirty so the page should
be written out the swap device to reclaim it.
It means we never lose it.

I tested this patch with kernel compile workload.

1. before

compile time : 9882.42
zram max wasted space by fragmentation: 13471881 byte
memory space consumed by zram: 174227456 byte
the number of slot free notify: 206684

2. after

compile time : 9653.90
zram max wasted space by fragmentation: 11805932 byte
memory space consumed by zram: 154001408 byte
the number of slot free notify: 426972

Cc: Hugh Dickins <hughd@google.com>
Cc: Seth Jennings <sjenning@linux.vnet.ibm.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Cc: Konrad Rzeszutek Wilk <konrad@darnok.org>
Cc: Shaohua Li <shli@kernel.org>
Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>

boype: added necessary swap_info_struct to mm/swapfile.c
and include/linux/swap.h

Signed-off-by: Dan Trifan <jimsth@gmail.com>

Conflicts:
	mm/page_io.c