Makefile: optimize

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
netfilter Makefile: fix build

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
arch: arm: lib: optimize memcpy for cortex-A15

The LDRD/STRD instruction is faster than LDM/STM on Cortex-A15.
Also optimized preload cache size for Cortex-A15.

Added USE_LDRDSTRD_OVER_LDMSTM to turn on LDRD/STRD optimization.
Added ARM_PLD_SIZE, default 32. Should set to 64 for Cortex-A15.

Bug 1185248

Signed-off-by: Chandler Zhang <chazhang@nvidia.com>
Reviewed-by: Krishna Reddy <vdumpa@nvidia.com>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
Apply arch: arm: lib: optimize memcpy for cortex-A15 to memutils

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>

Conflicts:
	arch/arm/mach-msm/memutils/copy_from_user.S
	arch/arm/mach-msm/memutils/copy_template.S
	arch/arm/mach-msm/memutils/copy_to_user.S
	arch/arm/mach-msm/memutils/memcpy.S
msm: msm_rq_stats: remove is_smp() API

is_smp() API is unavailable on few target codes.
Until this API is defined use CONFIG_SMP to know
whether target supports SPM are not.

CRs-fixed: 579034
Change-Id: I5bd5c208b18aae27d7f36e77e9d5b38639c2c00e
Signed-off-by: Murali Nalajala <mnalajal@codeaurora.org>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm:rq_stats: Register for Cpufreq policy notification

Currently the load of the system is calculated using the max
frequency that is read during startup(It does not get updated
when the policy changes). As a result, the average load of the
system is incorrectly calculated and the system might not bring
the cpus online to manage the additional load. This
reduces the performance of the system.

To fix this issue, register rq_stats to receive cpufreq policy
notifications. Update the policy's max frequency when a
notification is received and use that value to calculate the load.

Change-Id: Icc78e28c736c170e198f723fd96c13dfb2dafe8a
Signed-off-by: Archana Sathyakumar <asathyak@codeaurora.org>
Signed-off-by: Anji Jonnala <anjir@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm: rq_stats: Use cpufreq API to get the current frequency

Since the acpuclock APIs are deprecated, use the cpufreq to
get the current frequency of CPUs to calculate the cpu
normalized load value.

Signed-off-by: Archana Sathyakumar <asathyak@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm: rq_stats: Fix wrong usage of cpufreq_quick_get()

cpufreq_quick_get() returns frequency in KHz, which is exactly what
cur_freq expect. Remove the wrong multiplication.

Signed-off-by: Junjie Wu <junjiew@codeaurora.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm: rq_stats: Calculate load based on current freq limit

Prior to this patch, load has been calculated based on an initial
value of policy.cpuinfo.max_freq. This value may change for several
reasons: userspace settings, thermal throttling (hello hammerhead!)
or extended frequency tables where the absolute maximum level is
often not used during regular operations. This results in unreliable
calculations. Consider current policy.max to fix this behaviour.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm: msm_rq_stats: Tone down mpdecision

Add a new Kconfig to enable more conservative computations sent
to mpdecision:

 * Do not make IO busy
 * Do a time window weighted load computation instead of just a sum
 * Do an sliding window averaged time computation for number runnable

The purpose of these changes is to make mpdecision less likely to
keep extra cores online.  It currently has a tendancy to run more
cores running than needed and this can even cause some lag issues.

Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
msm: rq_stats: small change to current cpu policy

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched: Compute avg_nr_running for RQ Stats

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched_avg: add run queue averaging

Add code to calculate the run queue depth of a cpu and iowait
depth of the cpu.

The scheduler calls in to sched_update_nr_prod whenever there
is a runqueue change. This function maintains the runqueue average
and the iowait of that cpu in that time interval.

Whoever wants to know the runqueue average is expected to call
sched_get_nr_running_avg periodically to get the accumulated
runqueue and iowait averages for all the cpus.

Change-Id: Id8cb2ecf0ed479f090a83ccb72dd59c53fa73e0c
Signed-off-by: Jeff Ohlstein <johlstei@codeaurora.org>
(cherry picked from commit 0299fcaaad80e2c0ac9aa583c95107f6edc27750)
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>

Conflicts:
	kernel/sched/Makefile
	kernel/sched/sched.h

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>

Conflicts:
	kernel/sched/Makefile
defconfig: update

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched: Implement smarter wake-affine logic

The wake-affine scheduler feature is currently always trying to pull
the wakee close to the waker. In theory this should be beneficial if
the waker's CPU caches hot data for the wakee, and it's also beneficial
in the extreme ping-pong high context switch rate case.

Testing shows it can benefit hackbench up to 15%.

However, the feature is somewhat blind, from which some workloads
such as pgbench suffer. It's also time-consuming algorithmically.

Testing shows it can damage pgbench up to 50% - far more than the
benefit it brings in the best case.

So wake-affine should be smarter and it should realize when to
stop its thankless effort at trying to find a suitable CPU to wake on.

This patch introduces 'wakee_flips', which will be increased each
time the task flips (switches) its wakee target.

So a high 'wakee_flips' value means the task has more than one
wakee, and the bigger the number, the higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention to
the wakee with a high 'wakee_flips', pulling such a task may benefit
the wakee. Also imply that the waker will face cruel competition later,
it could be very cruel or very fast depends on the story behind
'wakee_flips', waker therefore suffers.

Furthermore, if waker also has a high 'wakee_flips', that implies that
multiple tasks rely on it, then waker's higher latency will damage all
of them, so pulling wakee seems to be a bad deal.

Thus, when 'waker->wakee_flips / wakee->wakee_flips' becomes
higher and higher, the cost of pulling seems to be worse and worse.

The patch therefore helps the wake-affine feature to stop its pulling
work when:

	wakee->wakee_flips > factor &&
	waker->wakee_flips > (factor * wakee->wakee_flips)

The 'factor' here is the number of CPUs in the current CPU's NUMA node,
so a bigger node will lead to more pulling since the trial becomes more
severe.

After applying the patch, pgbench shows up to 40% improvements and no regressions.

Tested with 12 cpu x86 server and tip 3.10.0-rc7.

The percentages in the final column highlight the areas with the biggest wins,
all other areas improved as well:

	pgbench		    base	smart

	| db_size | clients |  tps  |	|  tps  |
	+---------+---------+-------+   +-------+
	| 22 MB   |       1 | 10598 |   | 10796 |
	| 22 MB   |       2 | 21257 |   | 21336 |
	| 22 MB   |       4 | 41386 |   | 41622 |
	| 22 MB   |       8 | 51253 |   | 57932 |
	| 22 MB   |      12 | 48570 |   | 54000 |
	| 22 MB   |      16 | 46748 |   | 55982 | +19.75%
	| 22 MB   |      24 | 44346 |   | 55847 | +25.93%
	| 22 MB   |      32 | 43460 |   | 54614 | +25.66%
	| 7484 MB |       1 |  8951 |   |  9193 |
	| 7484 MB |       2 | 19233 |   | 19240 |
	| 7484 MB |       4 | 37239 |   | 37302 |
	| 7484 MB |       8 | 46087 |   | 50018 |
	| 7484 MB |      12 | 42054 |   | 48763 |
	| 7484 MB |      16 | 40765 |   | 51633 | +26.66%
	| 7484 MB |      24 | 37651 |   | 52377 | +39.11%
	| 7484 MB |      32 | 37056 |   | 51108 | +37.92%
	| 15 GB   |       1 |  8845 |   |  9104 |
	| 15 GB   |       2 | 19094 |   | 19162 |
	| 15 GB   |       4 | 36979 |   | 36983 |
	| 15 GB   |       8 | 46087 |   | 49977 |
	| 15 GB   |      12 | 41901 |   | 48591 |
	| 15 GB   |      16 | 40147 |   | 50651 | +26.16%
	| 15 GB   |      24 | 37250 |   | 52365 | +40.58%
	| 15 GB   |      32 | 36470 |   | 50015 | +37.14%

Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Cc: Mike Galbraith <efault@gmx.de>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Pranav Vashi <neobuddy89@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched/cpuidle: reduce IPI store. Backport upstream 3.16 scheduler updates

Signed-off-by: Paul Reioux <reioux@gmail.com>
Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched/fair: Optimize find_busiest_queue()

Use for_each_cpu_and() and thereby avoid computing the capacity for
CPUs we know we're not interested in.

Reviewed-by: Paul Turner <pjt@google.com>
Reviewed-by: Preeti U Murthy <preeti@linux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/n/tip-lppceyv6kb3a19g8spmrn20b@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>

backported for Linux 3.4

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>

Conflicts:
	kernel/sched/fair.c

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>
sched/fair: Optimize cgroup pick_next_task_fair()

Since commit 2f36825b1 ("sched: Next buddy hint on sleep and preempt
path") it is likely we pick a new task from the same cgroup, doing a put
and then set on all intermediate entities is a waste of time, so try to
avoid this.

Measured using:

  mount nodev /cgroup -t cgroup -o cpu
  cd /cgroup
  mkdir a; cd a
  mkdir b; cd b
  mkdir c; cd c
  echo $$ > tasks
  perf stat --repeat 10 -- taskset 1 perf bench sched pipe

PRE :      4.542422684 seconds time elapsed   ( +-  0.33% )
POST:      4.389409991 seconds time elapsed   ( +-  0.32% )

Which shows a significant improvement of ~3.5%

Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tejun Heo <tj@kernel.org>
Link: http://lkml.kernel.org/r/1328936700.2476.17.camel@laptop
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul Reioux <reioux@gmail.com>

Backported for Linux 3.4 kernel

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>

Conflicts:
	kernel/sched/fair.c

Signed-off-by: Stefanescu Cristian <howerkraft@gmail.com>