mutex: Make more scalable by doing less atomic operations

Date	Thu, 4 Apr 2013 10:54:16 -0400

In the __mutex_lock_common() function, an initial entry into
the lock slow path will cause two atomic_xchg instructions to be
issued. Together with the atomic decrement in the fast path, a total
of three atomic read-modify-write instructions will be issued in
rapid succession. This can cause a lot of cache bouncing when many
tasks are trying to acquire the mutex at the same time.

This patch will reduce the number of atomic_xchg instructions used by
checking the counter value first before issuing the instruction. The
atomic_read() function is just a simple memory read. The atomic_xchg()
function, on the other hand, can be up to 2 order of magnitude or even
more in cost when compared with atomic_read(). By using atomic_read()
to check the value first before calling atomic_xchg(), we can avoid a
lot of unnecessary cache coherency traffic. The only downside with this
change is that a task on the slow path will have a tiny bit
less chance of getting the mutex when competing with another task
in the fast path.

The same is true for the atomic_cmpxchg() function in the
mutex-spin-on-owner loop. So an atomic_read() is also performed before
calling atomic_cmpxchg().

The mutex locking and unlocking code for the x86 architecture can allow
any negative number to be used in the mutex count to indicate that some
tasks are waiting for the mutex. I am not so sure if that is the case
for the other architectures. So the default is to avoid atomic_xchg()
if the count has already been set to -1. For x86, the check is modified
to include all negative numbers to cover a larger case.

The following table shows the scalability data on an 8-node 80-core
Westmere box with a 3.7.10 kernel. The numactl command is used to
restrict the running of the high_systime workloads to 1/2/4/8 nodes
with hyperthreading on and off.

+-----------------+------------------+------------------+----------+
|  Configuration  | Mean Transaction | Mean Transaction | % Change |
|		  |  Rate w/o patch  | Rate with patch  |	   |
+-----------------+------------------------------------------------+
|		  |              User Range 1100 - 2000		   |
+-----------------+------------------------------------------------+
| 8 nodes, HT on  |      36980       |      148590      | +301.8%  |
| 8 nodes, HT off |      42799       |      145011      | +238.8%  |
| 4 nodes, HT on  |	 61318       |      118445      |  +51.1%  |
| 4 nodes, HT off |     158481       |      158592      |   +0.1%  |
| 2 nodes, HT on  |     180602       |      173967      |   -3.7%  |
| 2 nodes, HT off |     198409       |      198073      |   -0.2%  |
| 1 node , HT on  |     149042       |      147671      |   -0.9%  |
| 1 node , HT off |     126036       |      126533      |   +0.4%  |
+-----------------+------------------------------------------------+
|		  |              User Range 200 - 1000		   |
+-----------------+------------------------------------------------+
| 8 nodes, HT on  |      41525       |      122349      | +194.6%  |
| 8 nodes, HT off |      49866       |      124032      | +148.7%  |
| 4 nodes, HT on  |	 66409       |      106984      |  +61.1%  |
| 4 nodes, HT off |     119880       |      130508      |   +8.9%  |
| 2 nodes, HT on  |     138003       |      133948      |   -2.9%  |
| 2 nodes, HT off |     132792       |      131997      |   -0.6%  |
| 1 node , HT on  |     116593       |      115859      |   -0.6%  |
| 1 node , HT off |     104499       |      104597      |   +0.1%  |
+-----------------+------------------+------------------+----------+
AIM7 benchmark run has a pretty large run-to-run variance due to random
nature of the subtests executed. So a difference of less than +-5%
may not be really significant.

This patch improves high_systime workload performance at 4 nodes
and up by maintaining transaction rates without significant drop-off
at high node count.  The patch has practically no impact on 1 and 2
nodes system.

The table below shows the percentage time (as reported by perf
record -a -s -g) spent on the __mutex_lock_slowpath() function by
the high_systime workload at 1500 users for 2/4/8-node configurations
with hyperthreading off.

+---------------+-----------------+------------------+---------+
| Configuration | %Time w/o patch | %Time with patch | %Change |
+---------------+-----------------+------------------+---------+
|    8 nodes    |      65.34%     |      0.69%       |  -99%   |
|    4 nodes    |       8.70%	  |      1.02%	     |  -88%   |
|    2 nodes    |       0.41%     |      0.32%       |  -22%   |
+---------------+-----------------+------------------+---------+
It is obvious that the dramatic performance improvement at 8
nodes was due to the drastic cut in the time spent within the
__mutex_lock_slowpath() function.

The table below show the improvements in other AIM7 workloads (at 8
nodes, hyperthreading off).

+--------------+---------------+----------------+-----------------+
|   Workload   | mean % change | mean % change  | mean % change   |
|              | 10-100 users  | 200-1000 users | 1100-2000 users |
+--------------+---------------+----------------+-----------------+
| alltests     |     +0.6%     |   +104.2%      |   +185.9%       |
| five_sec     |     +1.9%     |     +0.9%      |     +0.9%       |
| fserver      |     +1.4%     |     -7.7%      |     +5.1%       |
| new_fserver  |     -0.5%     |     +3.2%      |     +3.1%       |
| shared       |    +13.1%     |   +146.1%      |   +181.5%       |
| short        |     +7.4%     |     +5.0%      |     +4.2%       |
+--------------+---------------+----------------+-----------------+
Signed-off-by: Waiman Long <Waiman.Long@hp.com>
Reviewed-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
mutex: restrict mutex spinning to only one task per mutex

Date	Thu, 4 Apr 2013 10:54:17 -0400

The current mutex spinning code allow multiple tasks to spin on a
single mutex concurrently. There are two major problems with this
approach:

 1. This is not very energy efficient as the spinning tasks are not
    doing useful work. The spinning tasks may also block other more
    important or useful tasks from running as preemption is disabled.
    Only one of the spinners will get the mutex at any time. The
    other spinners will have to wait for much longer to get it.

 2. The mutex data structure on x86-64 should be 32 bytes. The spinning
    code spin on lock->owner which, in most cases, should be in the same
    64-byte cache line as the lock->wait_lock spinlock. As a result,
    the mutex spinners are contending the same cacheline with other
    CPUs trying to get the spinlock leading to increased time spent
    on the spinlock as well as on the mutex spinning.

These problems are worse on system with large number of CPUs. One way
to reduce the effect of these two problems is to allow only one task
to be spinning on a mutex at any time.

This patch adds a new spinner field in the mutex.h to limit the
number of spinner to only one task. That will increase the size of
the mutex by 8 bytes in a 64-bit environment (4 bytes in a 32-bit
environment).

The AIM7 benchmarks were run on 3.7.10 derived kernels to show the
performance changes with this patch on a 8-socket 80-core system
with hyperthreading off.  The table below shows the mean % change
in performance over a range of users for some AIM7 workloads with
just the less atomic operation patch (patch 1) vs the less atomic
operation patch plus this one (patches 1+2).

+--------------+-----------------+-----------------+-----------------+
|   Workload   | mean % change   | mean % change   | mean % change   |
|              | 10-100 users    | 200-1000 users  | 1100-2000 users |
+--------------+-----------------+-----------------+-----------------+
| alltests     |     -0.2%       |     -3.8%       |    -4.2%        |
| five_sec     |     -0.6%       |     -2.0%       |    -2.4%        |
| fserver      |     +2.2%       |    +16.2%       |    +2.2%        |
| high_systime |     -0.3%       |     -4.3%       |    -3.0%        |
| new_fserver  |     +3.9%       |    +16.0%       |    +9.5%        |
| shared       |     -1.7%       |     -5.0%       |    -4.0%        |
| short        |     -7.7%       |     +0.2%       |    +1.3%        |
+--------------+-----------------+-----------------+-----------------+
It can be seen that this patch improves performance for the fserver and
new_fserver workloads while suffering some slight drop in performance
for the other workloads.

Signed-off-by: Waiman Long <Waiman.Long@hp.com>
Reviewed-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
mutex: dynamically disable mutex spinning at high load
Date	Thu, 4 Apr 2013 10:54:18 -0400

The Linux mutex code has a MUTEX_SPIN_ON_OWNER configuration
option that was enabled by default in major distributions like Red
Hat. Allowing threads waiting on mutex to spin while the mutex owner
is running will theoretically reduce latency on the acquisition of
mutex at the expense of energy efficiency as the spinning threads
are doing no useful work.

This is not a problem on a lightly loaded system where the CPU may
be idle anyway. On a highly loaded system, the spinning tasks may be
blocking other tasks from running even if they have higher priority
because the spinning was done with preemption disabled.

This patch will disable mutex spinning if the current load is high
enough. The load is considered high if there are 2 or more active tasks
waiting to run on the current CPU. If there is only one task waiting,
it will check the average load at the past minute (calc_load_tasks).
If it is more than double the number of active CPUs, the load is
considered high too.  This is a rather simple metric that does not
incur that much additional overhead.

The AIM7 benchmarks were run on 3.7.10 derived kernels to show the
performance changes with this patch on a 8-socket 80-core system
with hyperthreading off.  The table below shows the mean % change
in performance over a range of users for some AIM7 workloads with
just the less atomic operation patch (patch 1) vs the less atomic
operation patch plus this one (patches 1+3).

+--------------+-----------------+-----------------+-----------------+
|   Workload   | mean % change   | mean % change   | mean % change   |
|              | 10-100 users    | 200-1000 users  | 1100-2000 users |
+--------------+-----------------+-----------------+-----------------+
| alltests     |      0.0%       |     -0.1%       |    +5.0%        |
| five_sec     |     +1.5%       |     +1.3%       |    +1.3%        |
| fserver      |     +1.5%       |    +25.4%       |    +9.6%        |
| high_systime |     +0.1%       |      0.0%       |    +0.8%        |
| new_fserver  |     +0.2%       |    +11.9%       |   +14.1%        |
| shared       |     -1.2%       |     +0.3%       |    +1.8%        |
| short        |     +6.4%       |     +2.5%       |    +3.0%        |
+--------------+-----------------+-----------------+-----------------+
It can be seen that this patch provides some big performance
improvement for the fserver and new_fserver workloads while is still
generally positive for the other AIM7 workloads.

Signed-off-by: Waiman Long <Waiman.Long@hp.com>
Reviewed-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
modified for tuna kernel from LKML

Signed-off-by: Paul Reioux <reioux@gmail.com>
OMAP4: PM: Handle LPDDR CLK IO errata during suspend/idle

There is a HW bug in CMD PHY which gives ISO signals as same for
both PADn and PADp on differential IO pad, because of which IO
leaks higher as pull controls are differential internally and
pull value does not match A value.
Though there is no functionality impact due to this bug, it is
seen that by disabling the pulls there is a savings ~500uA in
OSWR, but draws ~300uA more during OFF mode.

To save power during both idle/suspend following approach taken:
 1) Enable WA during boot-up.
 2) Disable WA while attempting suspend and enable during resume.

CDDS no: OMAP4460-1.0BUG00291
(OMAP official errata ID yet to be available).

Change-Id: Ic39f17be50ff602c71e714b05af3811b35c1a67f
Reported-by: Logan Lee <hklee@ti.com>
Signed-off-by: Shashi Shekar B.S. <shashi.shekar@ti.com>
Signed-off-by: Ambresh K <ambresh@ti.com>
drivers/gpu/pvr/pvr_governor: add a simple PVR GPU governor

this simple PVR GPU governor will scale between 3 speeds:
  idle -- GPU not active
  nominal -- GPU active with low graphical load
  turbo -- GPU active and boosted to max frequency

This should further save battery by intelligently selecting GPU frequency
based on GPU loading.

Signed-off-by: Paul Reioux <reioux@gmail.com>
drivers/gpu/pvr/pvr_governor: add sysfs entry to switch governor type

0 - "performance" ==> always return system turbo speed
1 - "ondemand" ==> return based on GPU loading (by default "ondemand"
governor is active)

Signed-off-by: Paul Reioux <reioux@gmail.com>
drivers/gpu/pvr/pvr_governor: add "powersave" governor as another option

    0 - "performance" ==> always return system turbo speed
    1 - "ondemand" ==> return based on GPU loading (by default "ondemand"
                       governor is active)
    2 - "powersave" ==> run alway at nominal speed

Signed-off-by: Paul Reioux <reioux@gmail.com>
drivers/gpu/pvr/pvr_governor: fix sysfs to show "powersave" governor

Signed-off-by: Paul Reioux <reioux@gmail.com>
drivers/gpu/pvr/pvr_governor: add sysfs interface for current GPU freqeuncy

Signed-off-by: Paul Reioux <reioux@gmail.com>
Enable PVR GOVERNOR and tweaks
OMAP4430: TWL data: update retention voltages to spec rev 0.6

Update retention voltages from Operating Conditions Data Manual
Addendum Revision 0.6 (August 2011).

Note: vdd_iva is used as TWL's vcore2 is common between OMAP4430
and OMAP4460 - hence the update is done for 4460 as well(follow
on patch updates rest of the domain voltages to the same voltage
for OMAP4460).

Change-Id: I68a561049f3faf54b877a4d6293682829ee99b62
Signed-off-by: Nishanth Menon <nm@ti.com>

Conflicts:
	arch/arm/mach-omap2/omap_twl.c

Conflicts:
	arch/arm/mach-omap2/omap_twl.c
OMAP4460: TWL/TPS data: update retention voltages to spec rev 0.3

Update retention voltages from Operating Conditions Data Manual
Addendum Revision 0.3 (August 2011).

OFF voltage is also updated as per requirement of 4460 OFF mode
conditions.

Change-Id: I558f0a483911b2616d0a19f9272a03258e35904a
Signed-off-by: Nishanth Menon <nm@ti.com>

Conflicts:
	arch/arm/mach-omap2/omap_twl.c
ARM: cache: assume 64-byte L1 cachelines for ARMv7 CPUs

To ensure correct alignment of cacheline-aligned data, the maximum
cacheline size needs to be known at compile time.

Since Cortex-A8 and Cortex-A15 have 64-byte cachelines (and it is likely
that there will be future ARMv7 implementations with the same line size)
then it makes sense to assume that CPU_V7 implies a 64-byte L1 cacheline
size. For CPUs with smaller caches, this will result in some harmless
padding but will help with single zImage work and avoid hitting subtle
bugs with misaligned data structures.

Signed-off-by: Will Deacon <will.deacon@xxxxxxx>

Conflicts:
	arch/arm/mach-omap2/Kconfig
Enable Sync RM_L1_CACHE_SHIFT =6
OMAP: DSS2: Fix callback completion issue in mgr->blank

Uncommitted callbacks were not completed yet eclipsed in mgr->blank.

Also fixed mgr index in ECLIPSED_CACHE callback.
This patch is co-authored with Muralidhar Dixit <murali.dixit@ti.com>

Change-Id: Ia20b79e9a6308441423d9900469e0c4adfc30eee
Signed-off-by: Lajos Molnar <lajos@ti.com>
Signed-off-by: Muralidhar Dixit <murali.dixit@ti.com>
OMAP4: DSS: Fix to correct the manager index for VSYNC irq handler

Patch to fix the Manager id in dss_completion_irq_handler.
DSSCOMP is expecting the manager id 1 to be mapped to HDMI display.
This patch fixes the DSSCOMP_STATE_DISPLAYED callback in DSSCOMP.

Change-Id: I0aab3e494cb305d75a64fabc4cd6919def636285
Signed-off-by: Muralidhar Dixit <murali.dixit@ti.com>
Signed-off-by: Varadarajan Charulatha <charu@ti.com>
OMAPDSS: DSSCOMP: export needed functions for PVR SGX

OMAPDSS: DSSCOMP: export needed functions for PVR SGX

Make dsscomp symbol visible to kernel modules. The SGX DDK
in modular form requires this symbol to be exported.

Change-Id: I5d45b953379f1ae7c05d10271427a9cd611248a6
Signed-off-by: Rogelio Garcia <rgarcia@ti.com>
OMAPDSS: DSSCOMP: Remove PVR_SGX dependency.

OMAPDSS: DSSCOMP: Remove PVR_SGX dependency.

In order to disable the in-kernel version of the DDK
and be able to use it as a module we need to remove
this dependency.
No harm to DSSCOMP, it will just wait until a DDK
is installed to execute.

Change-Id: Ie1436a45fb20044d0121eb196c6155b50baa7915
Signed-off-by: Rogelio Garcia <rgarcia@ti.com>
OMAPDSS: drop small YUV encoded frames on OMAP44xx

OMAPDSS: drop small YUV encoded frames on OMAP44xx

OMAP44xx has hardware limitation: in stall mode, when the frame pixel size
is less than output SyncFifo depth(16), DISPC hangs without sending any data.
Simple workaround is to drop such frame, as cases when YUV frames of this size
are very rare.

Change-Id: I31dc262b07a52b66209c093f136ad2bac66f9439
Signed-off-by: Sergiy Kibrik <sergiy.kibrik@globallogic.com>
hrtimer: Don't reinitialize a cpu_base's lock on CPU_UP

The current code makes the assumption that a cpu_base lock cannot
be held if a CPU is offline, which is tragically wrong.

If a hrtimer is not queued, then it will not be migrated by
migrate_hrtimers() when a CPU is offlined. Therefore, the
hrtimer's cpu_base may still point to a CPU which has
subsequently gone offline if the timer wasn't enqueued at the
time the CPU went down.

Normally this wouldn't be a problem, but a cpu_base's lock is
blindly reinitialized each time a CPU is brought up. If a CPU is
brought online during the period that another thread is
performing a hrtimer operation on a stale hrtimer, then the lock
will be reinitialized under its feet, and a SPIN_BUG() like the
following will be observed:

 [  448.212369] BUG: spinlock already unlocked on CPU#3,
 swapper/3/0
[  448.217368]  lock: 0xc47b1a40, .magic: dead4ead, .owner:
swapper/3/0, .owner_cpu: -1
[  448.217398] [<c0014398>] (unwind_backtrace+0x0/0x120) from
[<c0269ab0>] (do_raw_spin_unlock+0x44/0xdc)
[  448.217419] [<c0269ab0>] (do_raw_spin_unlock+0x44/0xdc) from
[<c071bbb0>] (_raw_spin_unlock_irqrestore+0xc/0x38)
[  448.217438] [<c071bbb0>]
(_raw_spin_unlock_irqrestore+0xc/0x38) from [<c00aa498>]
(hrtimer_try_to_cancel+0xf8/0x10c)
[  448.217455] [<c00aa498>] (hrtimer_try_to_cancel+0xf8/0x10c)
from [<c00aa4bc>] (hrtimer_cancel+0x10/0x20)
[  448.217475] [<c00aa4bc>] (hrtimer_cancel+0x10/0x20) from
[<c00e6934>] (rcu_idle_exit_common+0x5c/0xe0)
[  448.217494] [<c00e6934>] (rcu_idle_exit_common+0x5c/0xe0) from
[<c00e6ae8>] (rcu_idle_exit+0x9c/0xb4)
[  448.217514] [<c00e6ae8>] (rcu_idle_exit+0x9c/0xb4) from
[<c000f2dc>] (cpu_idle+0x98/0xf0)
[  448.217529] [<c000f2dc>] (cpu_idle+0x98/0xf0) from
[<000081ac>] (0x81ac)

CRs-Fixed: 460670
Change-Id: I588f6a864054abe6d4c249aad815c6d2f5a54077
Signed-off-by: Michael Bohan <mbohan@codeaurora.org>
Signed-off-by: Francisco Franco <franciscofranco.1990@gmail.com>