[PATCH 2/3] Optimize copy_page for modern ARM platforms

The existing implementation of copy_page for ARM appears to be
optimized for older platforms. Benchmark testing in a sandbox
environment shows suboptimal performance on modern platforms
like armv6 and armv7, with speed-ups ranging from 10% (Cortex A8)
to 80% (armv6 used in Raspberry Pi) being achievable.

This commit optimizes copy_page and introduces the new compile-time
constant PREFETCH_DISTANCE, defined in cache.h, which when
multiplied by L1_CACHE_BYTES is equal to the offset used for
prefetches performed with the PLD instruction. For platforms where
L1_CACHE_BYTES is 32 (armv5 and armv6), copy_page processes 32 bytes
at a time while doing one prefetch per iteration, while for armv7
(with L1_CACHE_BYTES equal to 64), 64 bytes are processed at at time
with one prefetch per iteration. When no preload instruction is
available (platforms earlier than armv5), no preload instructions
are generated and 32 bytes are processed at at time.

To facilitate specifying instructions for architectures with no
preload instruction, the NO_PLD macro is added to assembler.h,
augmenting the PLD macro.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: 7301/1: Rename the T() macro to TUSER() to avoid namespace conflicts

This macro is used to generate unprivileged accesses (LDRT/STRT) to user
space.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
[PATCH 1/3] Rename ARM assembler push/pull macros

The ARM assembler library functions use a macro called "push" that
along with a macro called "pull" is used to shift bytes around in a
word in an endian-independent way. However, the modern unified
ARM assembler syntax also defines the instruction "push" to push
data onto the stack, which has specific encodings in the Thumb2
instruction set. For prevent possible conflicts going forward, and
to allow the use of the more transparent "push" instruction along
with the modern unified assembler syntax, this patch renames all
occurrences of the "push" macro to "pushbits", as well as renaming
the macro argument, when also called "push", to "pushshift".

For consistency, the macro called "pull" with its argument name
"pull" are also renamed to "pullbits" and "pullshift", respectively.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
[PATCH 3/3] Optimize ARM memset and memzero functions

The ARM memset and memzero functions are optimized with lower overhead
for small requests, generation of more 16-bit Thumb2 instructions when
compiled in Thumb2 mode, and configurable destination alignment before
the main block-copying loop.

The new compile-time constant MEMSET_WRITE_ALIGN_BYTES is introduced
in assembler.h, to augment the CALGN macro that previously regulated
32-byte write alignment but was only used on the Feroceon platform.
MEMSET_WRITE_ALIGN_BYTES can have values of 0 (no write alignment),
8, or 32. Apart from Feroceon, memset write alignment of 32 bytes
appears to benefit the armv6 platform, while the armv7 platform
seems benefit from alignment to 8 bytes for memset/memzero.

The CALGN macro is renamed to MEMSET_CALGN for memset and memzero; the
original CALGN macro is reserved for the memcpy family of functions
(memcpy, copy_from_user, copy_to_user) currently implemented in
copy_template.S, and the associated compile time constant
WRITE_ALIGN_BYTES defines the write alignment for memcpy-related
functions that will be utilized in subsequent contributions. Because
the current CALGN implementation in copy_template.S only implements
32-byte write alignment and is broken on Thumb2, it is only enabled
when WRITE_ALIGN_BYTES is equal to 32 and Thumb2 mode is not enabled.

Finally, memset and memzero now include a directive to enable unified
ARM assembler syntax.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: assembler.h: use same WRITE_ALIGN_BYTES on omap4 as on armv6 platforms

Not all armv7 have 64 kbytes of l1 cache, for example the
Cortex-A9, which has 32 kbytes. As all omap4 platforms also
run Cortex-A9, adjust WRITE_ALIGN_BYTES accordingly.

Still needs testing.

Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: implement optimized percpu variable access

From: Rob Herring <rob.herring@calxeda.com>

Use the previously unused TPIDRPRW register to store percpu offsets.
TPIDRPRW is only accessible in PL1, so it can only be used in the kernel.

This replaces 2 loads with a mrc instruction for each percpu variable
access. With hackbench, the performance improvement is 1.4% on Cortex-A9
(highbank). Taking an average of 30 runs of "hackbench -l 1000" yields:

Before: 6.2191
After: 6.1348

Will Deacon reported similar delta on v6 with 11MPCore.

The asm "memory" constraints are needed here to ensure the percpu offset
gets reloaded. Testing by Will found that this would not happen in
__schedule() which is a bit of a special case as preemption is disabled
but the execution can move cores.

Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: keep __my_cpu_offset consistent with generic one

Commit 14318efb(ARM: 7587/1: implement optimized percpu variable access)
introduces arm's __my_cpu_offset to optimize percpu vaiable access,
which really works well on hackbench, but will cause __my_cpu_offset
to return garbage value before it is initialized in cpu_init() called
by setup_arch, so accessing percpu variable before setup_arch may cause
kernel hang. But generic __my_cpu_offset always returns zero before
percpu area is brought up, and won't hang kernel.

So the patch tries to clear __my_cpu_offset on boot CPU early
to avoid boot hang.

At least now percpu variable is accessed by lockdep before
setup_arch(), and enabling CONFIG_LOCK_STAT or CONFIG_DEBUG_LOCKDEP
can trigger kernel hang.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Rob Herring <rob.herring@calxeda.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Nicolas Pitre <nico@linaro.org>
Cc: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Ming Lei <tom.leiming@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ext4: Speed up FITRIM by recording flags in ext4_group_info

In ext4, when FITRIM is called every time, we iterate all the
groups and do trim one by one. It is a bit time wasting if the
group has been trimmed and there is no change since the last
trim.

So this patch adds a new flag in ext4_group_info->bb_state to
indicate that the group has been trimmed, and it will be cleared
if some blocks is freed(in release_blocks_on_commit). Another
trim_minlen is added in ext4_sb_info to record the last minlen
we use to trim the volume, so that if the caller provide a small
one, we will go on the trim regardless of the bb_state.

A simple test with my intel x25m ssd:
df -h shows:
/dev/sdb1              40G   21G   17G  56% /mnt/ext4
Block size:               4096

run the FITRIM with the following parameter:
range.start = 0;
range.len = UINT64_MAX;
range.minlen = 1048576;

without the patch:
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.505s
user	0m0.000s
sys	0m1.224s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.359s
user	0m0.000s
sys	0m1.178s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.228s
user	0m0.000s
sys	0m1.151s

with the patch:
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m5.625s
user	0m0.000s
sys	0m1.269s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m0.002s
user	0m0.000s
sys	0m0.001s
[root@boyu-tm linux-2.6]# time ./ftrim /mnt/ext4/a
real	0m0.002s
user	0m0.000s
sys	0m0.001s

A big improvement for the 2nd and 3rd run.

Even after I delete some big image files, it is still much
faster than iterating the whole disk.

[root@boyu-tm test]# time ./ftrim /mnt/ext4/a
real	0m1.217s
user	0m0.000s
sys	0m0.196s

Cc: Lukas Czerner <lczerner@redhat.com>
Reviewed-by: Andreas Dilger <adilger.kernel@dilger.ca>
Signed-off-by: Tao Ma <boyu.mt@taobao.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Signed-off-by: Dan Trifan <jimsth@gmail.com>

Conflicts:
	fs/ext4/mballoc.c

Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm: remove compressed copy from zram in-memory

Swap subsystem does lazy swap slot free with expecting the page
would be swapped out again so we can avoid unnecessary write.

But the problem in in-memory swap(ex, zram) is that it consumes
memory space until vm_swap_full(ie, used half of all of swap device)
condition meet. It could be bad if we use multiple swap device,
small in-memory swap and big storage swap or in-memory swap alone.

This patch makes swap subsystem free swap slot as soon as swap-read
is completed and make the swapcache page dirty so the page should
be written out the swap device to reclaim it.
It means we never lose it.

I tested this patch with kernel compile workload.

1. before

compile time : 9882.42
zram max wasted space by fragmentation: 13471881 byte
memory space consumed by zram: 174227456 byte
the number of slot free notify: 206684

2. after

compile time : 9653.90
zram max wasted space by fragmentation: 11805932 byte
memory space consumed by zram: 154001408 byte
the number of slot free notify: 426972

Cc: Hugh Dickins <hughd@google.com>
Cc: Seth Jennings <sjenning@linux.vnet.ibm.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Cc: Konrad Rzeszutek Wilk <konrad@darnok.org>
Cc: Shaohua Li <shli@kernel.org>
Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>

boype: added necessary swap_info_struct to mm/swapfile.c
and include/linux/swap.h

Signed-off-by: Dan Trifan <jimsth@gmail.com>
Linux v3.0.96

This is the 3.0.96 stable release.

Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm: Dynamic management of dirty page writebacks

This feature allows to have two different intervals for dirty page
writebacks and to dynamically manage them when the system has been
resumed (it's active) or when has been suspended.

Three new procfs parameters are exposed inside /proc/sys/vm path:
- "dynamic_dirty_writeback" is the activation status of this feature,
  set 1 to enable it, set 0 to disable it and use the standard behaviour
- "dirty_writeback_active_centisecs" is the interval for the dirty page
  writebacks when the system is active (screen on)
- "dirty_writeback_suspend_centisecs" is the interval for the dirty page
  writebacks when the system is suspended (screen off)

This feature is in part inspired on Francisco Franco's patch:
https://github.com/franciscofranco/mako/commit/34d79548cf16b69c7d0154836018e7f258a22947

Signed-off-by: Cristoforo Cataldo <cristoforo.cataldo@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
defconfig: enable DYNAMIC_PAGE_WRITEBACK

Signed-off-by: Dan Trifan <jimsth@gmail.com>
mm: finetune the filesystem dynamic read-ahead

Signed-off-by: Dan Trifan <jimsth@gmail.com>
kernel: cgroup_timer_slack: Dynamic management of timer slack

This feature allows to use different minimal timer slack values and to
dynamically manage them when the system has been resumed (it's active)
or when has been suspended.

A new file is provided in addition to the two files provided by the
original timer slack controller:
- "timer_slack.min_slack_suspend_ns" is the minimal timer slack value
  to use when the system is suspended (screen off)

For further info, please refers to the timer slack documentation:
Documentation/cgroupd/timer_slack.txt

Signed-off-by: Cristoforo Cataldo <cristoforo.cataldo@gmail.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
defconfig: enable dynamic CGROUP_TIMER_SLACK

Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: 7746/1: mm: lazy cache flushing on non-mapped pages

Currently flush_dcache_page() thinks pages as non-mapped if
mapping_mapped(mapping) return false. This approach is very
coase:
	- mmap on part of file may cause all pages backed on
	the file being thought as mmaped

	- file-backed pages aren't mapped into user space actually
	if the memory mmaped on the file isn't accessed

This patch uses page_mapped() to decide if the page has been
mapped.

From the attached test code, I find there is much performance
improvement(>25%) when accessing page caches via read under this
situations, so memcpy benefits a lot from not flushing cache
under this situation.

No.   read time without the patch	No. read time with the patch
================================================================
No. 0, time  22615636 us		No. 0, time  22014717 us
No. 1, time  4387851 us 		No. 1, time  3113184 us
No. 2, time  4276535 us 		No. 2, time  3005244 us
No. 3, time  4259821 us 		No. 3, time  3001565 us
No. 4, time  4263811 us 		No. 4, time  3002748 us
No. 5, time  4258486 us 		No. 5, time  3004104 us
No. 6, time  4253009 us 		No. 6, time  3002188 us
No. 7, time  4262809 us 		No. 7, time  2998196 us
No. 8, time  4264525 us 		No. 8, time  3007255 us
No. 9, time  4267795 us 		No. 9, time  3005094 us

1), No.0. is to read the file from storage device, and others are
to read the file from page caches basically.
2), file size is 512M, and is on ext4 over usb mass storage.
3), the test is done on Pandaboard.

unsigned int  sum = 0;
unsigned long sum_val = 0;

static unsigned long tv_diff(struct timeval *tv1, struct timeval *tv2)
{
	return (tv2->tv_sec - tv1->tv_sec) * 1000000 +
		(tv2->tv_usec - tv1->tv_usec);
}

int main(int argc, char *argv[])
{
	char *mbuf, fbuf;
	int fd;
	int i;
	unsigned long page_size, size;
	struct stat stat;
	struct timeval t1, t2;
	unsigned char *rbuf = malloc(32 * page_size);

	if (!rbuf) {
		printf("	%sn", "malloc failed");
		exit(-1);
	}

	page_size = getpagesize();
	fd = open(argv[1], O_RDWR);
	assert(fd >= 0);

	fstat(fd, &stat);
	size = stat.st_size;
	printf("%s: file %s, size %lu, page size %lun",
		argv[0],
		argv[1], size, page_size);

	gettimeofday(&t1, NULL);
	mbuf = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
	if (!mbuf) {
		printf("	%sn", "mmap failed");
		exit(-1);
	}

	for (i = 0 ; i < size ; i += (page_size * 32)) {
		int rcnt;
		lseek(fd, i, SEEK_SET);
		rcnt = read(fd, rbuf, page_size * 32);
		if (rcnt != page_size * 32) {
			printf("%s: read faildn", __func__);
			exit(-1);
		}
	}
	free(rbuf);
	munmap(mbuf, size);
	gettimeofday(&t2, NULL);
	printf("tread mmaped time: %luusn", tv_diff(&t1, &t2));

	close(fd);
}

Cc: Michel Lespinasse <walken@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Ming Lei <ming.lei@canonical.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
ARM: 6998/2: kernel: use proper memory barriers for bitops

Improve scalability by avoiding costly and unnecessary L2 cache sync
in handling bitops.

Signed-off-by: Heechul Yun <hyun@nvidia.com>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
sync: allow async waits to be canceled

In order to allow drivers to cleanly handled teardown we need to allow them
to cancel pending async waits.  To do this cleanly, we move allocation of
sync_fence_waiter to the driver calling sync_async_wait().

Change-Id: Ifcd95648be6ec07026d67f810070a4310f099989
Signed-off-by: Erik Gilling <konkers@android.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
sync: export sync API symbols

This is needed to allow modules to link against the sync subsystem

Change-Id: I15c1818de329f24e4113ef1d0923413b22fd0eff
Signed-off-by: Erik Gilling <konkers@android.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>
sw_sync: export sw_sync API

Needed to let modules link against sw_sync.

Change-Id: I71d3e52677ef21b6e1ecdb84f831491be1f4fbe6
Signed-off-by: Erik Gilling <konkers@android.com>
Signed-off-by: Dan Trifan <jimsth@gmail.com>