cpufreq: Resolve CPUFREQ_NOTIFY issue

When the cpufreq_stat receives the frequency policy notification,
it checks if the frequency table and stats table are present for the cpu.
If both are present, it returns with -EBUSY error to the notification.
Due to this, none of the other registered modules receive this
notification.

If the stat table and sysfs nodes are already created, return 0 instead
of returning EBUSY error.

Change-Id: I1fd4a46e6a65edfc6253c4271288f10912ae2ad7
Signed-off-by: Archana Sathyakumar <asathyak@codeaurora.org>
msm:rq_stats: Register for Cpufreq policy notification

Currently the load of the system is calculated using the max
frequency that is read during startup(It does not get updated
when the policy changes). As a result, the average load of the
system is incorrectly calculated and the system might not bring
the cpus online to manage the additional load. This
reduces the performance of the system.

To fix this issue, register rq_stats to receive cpufreq policy
notifications. Update the policy's max frequency when a
notification is received and use that value to calculate the load.

Change-Id: Icc78e28c736c170e198f723fd96c13dfb2dafe8a
Signed-off-by: Archana Sathyakumar <asathyak@codeaurora.org>
Signed-off-by: Anji Jonnala <anjir@codeaurora.org>
dynamic fsync: add reboot notifier to force flush outstanding data

this should further prevent data corruption from kernel panics and
forced reboots

bump version to 1.2

Signed-off-by: Paul Reioux <reioux@gmail.com>
dynamic fsync: add kernel panic notifier to force flush outstanding data

more paranoia safety checks

Signed-off-by: Paul Reioux <reioux@gmail.com>
workqueue: don't use WQ_HIGHPRI for unbound workqueues

Unbound wqs aren't concurrency-managed and try to execute work items
as soon as possible.  This is currently achieved by implicitly setting
%WQ_HIGHPRI on all unbound workqueues; however, WQ_HIGHPRI
implementation is about to be restructured and this usage won't be
valid anymore.

Add an explicit chain-wakeup path for unbound workqueues in
process_one_work() instead of piggy backing on %WQ_HIGHPRI.

Change-Id: Iecd17a9935ee28f856d8b726bb4c296762922bed
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 974271c485a4d8bb801decc616748f90aafb07ec
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit 5b92cea36d8328c17380d7997744284707134397)
workqueue: factor out worker_pool from global_cwq

Move worklist and all worker management fields from global_cwq into
the new struct worker_pool.  worker_pool points back to the containing
gcwq.  worker and cpu_workqueue_struct are updated to point to
worker_pool instead of gcwq too.

This change is mechanical and doesn't introduce any functional
difference other than rearranging of fields and an added level of
indirection in some places.  This is to prepare for multiple pools per
gcwq.

v2: Comment typo fixes as suggested by Namhyung.

Change-Id: Iefae84798c2af580f425b92ed79117935d99f21f
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Git-commit: bd7bdd43dcb81bb08240b9401b36a104f77dc135
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit e3676a67902c18021bd571aa3fcefc74b377b65f)
workqueue: use @pool instead of @gcwq or @cpu where applicable

Modify all functions which deal with per-pool properties to pass
around @pool instead of @gcwq or @cpu.

The changes in this patch are mechanical and don't caues any
functional difference.  This is to prepare for multiple pools per
gcwq.

Change-Id: I4be6727e1cce6f9aa2a0057b96bdc725c84f1ea8
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 63d95a9150ee3bbd4117fcd609dee40313b454d9
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit ae73f6e6eae63e64d552227a109b6ccb802cdd04)
workqueue: separate out worker_pool flags

GCWQ_MANAGE_WORKERS, GCWQ_MANAGING_WORKERS and GCWQ_HIGHPRI_PENDING
are per-pool properties.  Add worker_pool->flags and make the above
three flags per-pool flags.

The changes in this patch are mechanical and don't caues any
functional difference.  This is to prepare for multiple pools per
gcwq.

Change-Id: I1824fd1c509d8ac6b0619536621a22b15b316256
Signed-off-by: Tejun Heo <tj@kernel.org>
Git-commit: 11ebea50dbc1ade5994b2c838a096078d4c02399
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit 1ab4f069d6d1255625a42209a42cf9d6773480d9)
workqueue: introduce NR_WORKER_POOLS and for_each_worker_pool()

Introduce NR_WORKER_POOLS and for_each_worker_pool() and convert code
paths which need to manipulate all pools in a gcwq to use them.
NR_WORKER_POOLS is currently one and for_each_worker_pool() iterates
over only @gcwq->pool.

Note that nr_running is per-pool property and converted to an array
with NR_WORKER_POOLS elements and renamed to pool_nr_running.  Note
that get_pool_nr_running() currently assumes 0 index.  The next patch
will make use of non-zero index.

The changes in this patch are mechanical and don't caues any
functional difference.  This is to prepare for multiple pools per
gcwq.

v2: nr_running indexing bug in get_pool_nr_running() fixed.

v3: Pointer to array is stupid.  Don't use it in get_pool_nr_running()
    as suggested by Linus.

Change-Id: I46e9488601d764d25e4a6c707de129ab68f7064c
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Git-commit: 4ce62e9e30cacc26885cab133ad1de358dd79f21
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit ca6a47d36f578a3c9ae47037413abca8319fd99f)
workqueue: reimplement WQ_HIGHPRI using a separate worker_pool

WQ_HIGHPRI was implemented by queueing highpri work items at the head
of the global worklist.  Other than queueing at the head, they weren't
handled differently; unfortunately, this could lead to execution
latency of a few seconds on heavily loaded systems.

Now that workqueue code has been updated to deal with multiple
worker_pools per global_cwq, this patch reimplements WQ_HIGHPRI using
a separate worker_pool.  NR_WORKER_POOLS is bumped to two and
gcwq->pools[0] is used for normal pri work items and ->pools[1] for
highpri.  Highpri workers get -20 nice level and has 'H' suffix in
their names.  Note that this change increases the number of kworkers
per cpu.

POOL_HIGHPRI_PENDING, pool_determine_ins_pos() and highpri chain
wakeup code in process_one_work() are no longer used and removed.

This allows proper prioritization of highpri work items and removes
high execution latency of highpri work items.

v2: nr_running indexing bug in get_pool_nr_running() fixed.

v3: Refreshed for the get_pool_nr_running() update in the previous
    patch.

Change-Id: Id843c0a425f51f84083786fbf413d999d35771b7
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Josh Hunt <joshhunt00@gmail.com>
LKML-Reference: <CAKA=qzaHqwZ8eqpLNFjxnO2fX-tgAOjmpvxgBFjv6dJeQaOW1w@mail.gmail.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fengguang Wu <fengguang.wu@intel.com>
Git-commit: 3270476a6c0ce322354df8679652f060d66526dc
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Matt Wagantall <mattw@codeaurora.org>
(cherry picked from commit f2a9a769bf4fa185757115007bf94c36c2f29aca)
msm: rpm-smd: Configure WQ for higer priority

When the workqueue runs at a lower priority, it gets starved by higher
priority threads. Bump the WQ priority to high to ensure rpm smd work
queue gets a fair chance

Change-Id: I06b864611cf45afe6931d6030327806032894663
Signed-off-by: Mahesh Sivasubramanian <msivasub@codeaurora.org>
(cherry picked from commit e0eccf8f6dece61cdb639c4b43374c34c3acf3ea)
sched: smart wake-affine

wake-affine stuff is always trying to pull wakee close to waker, by theory,
this will bring benefit if waker's cpu cached hot data for wakee, or the
extreme ping-pong case.

And testing show it could benefit hackbench 15% at most.

However, the whole stuff is somewhat blindly and time-consuming, some
workload therefore suffer.

And testing show it could damage pgbench 50% at most.

Thus, wake-affine stuff should be smarter, and realise when to stop
it's thankless effort.

This patch introduced per task 'nr_wakee_switch', which will be increased
each time the task switch it's wakee.

So a high 'nr_wakee_switch' means the task has more than one wakee, and
less the wakee number, higher the wakeup frequency.

Now when making the decision on whether to pull or not, pay attention on
the wakee with a high 'nr_wakee_switch', pull such task may benefit wakee,
but that imply waker will face cruel competition later, it could be very
crule or very fast depends on the story behind 'nr_wakee_switch', whatever,
waker therefore suffer.

Furthermore, if waker also has a high 'nr_wakee_switch', that imply multiple
tasks rely on it, waker's higher latency will damage all those tasks, pull
wakee in such cases seems to be a bad deal.

Thus, when 'waker->nr_wakee_switch / wakee->nr_wakee_switch' become higher
and higher, the deal seems to be worse and worse.

This patch therefore help wake-affine stuff to stop it's work when:

	wakee->nr_wakee_switch > factor &&
	waker->nr_wakee_switch > (factor * wakee->nr_wakee_switch)
The factor here is the online cpu number, so more cpu will lead to more pull
since the trial become more severe.

After applied the patch, pgbench show 42% improvement at most.

Test:
	Test with 12 cpu X86 server and tip 3.10.0-rc1.

				base	smart

	| db_size | clients |  tps  | |  tps  |
	+---------+---------+-------+ +-------+
	| 21 MB   |       1 | 10749 | | 10337 |
	| 21 MB   |       2 | 21382 | | 21391 |
	| 21 MB   |       4 | 41570 | | 41808 |
	| 21 MB   |       8 | 52828 | | 58792 |
	| 21 MB   |      12 | 48447 | | 54553 |
	| 21 MB   |      16 | 46246 | | 56726 |	+22.66%
	| 21 MB   |      24 | 43850 | | 56853 |	+29.65%
	| 21 MB   |      32 | 43455 | | 55846 |	+28.51%
	| 7483 MB |       1 |  9290 | |  8848 |
	| 7483 MB |       2 | 19347 | | 19351 |
	| 7483 MB |       4 | 37135 | | 37511 |
	| 7483 MB |       8 | 47310 | | 50210 |
	| 7483 MB |      12 | 42721 | | 49396 |
	| 7483 MB |      16 | 41016 | | 51826 |	+26.36%
	| 7483 MB |      24 | 37540 | | 52579 |	+40.06%
	| 7483 MB |      32 | 36756 | | 51332 |	+39.66%
	| 15 GB   |       1 |  8758 | |  8670 |
	| 15 GB   |       2 | 19204 | | 19249 |
	| 15 GB   |       4 | 36997 | | 37199 |
	| 15 GB   |       8 | 46578 | | 50681 |
	| 15 GB   |      12 | 42141 | | 48671 |
	| 15 GB   |      16 | 40518 | | 51280 |	+26.56%
	| 15 GB   |      24 | 36788 | | 52329 |	+42.24%
	| 15 GB   |      32 | 36056 | | 50350 | +39.64%

CC: Ingo Molnar <mingo@kernel.org>
CC: Peter Zijlstra <peterz@infradead.org>
CC: Mike Galbraith <efault@gmx.de>
Signed-off-by: Michael Wang <wangyun@linux.vnet.ibm.com>
Signed-off-by: Paul Reioux <reioux@gmail.com>
CPU hotplug, writeback: Don't call writeback_set_ratelimit() too often during hotplug

The CPU hotplug callback related to writeback calls writeback_set_ratelimit()
during every state change in the hotplug sequence. This is unnecessary
since num_online_cpus() changes only once during the entire hotplug operation.

So invoke the function only once per hotplug, thereby avoiding the
unnecessary repetition of those costly calculations.

Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>