mm: change initial readahead window size calculation

Change the logic which determines the initial readahead window size
such that for small requests (one page) the initial window size
will be x4 the size of the original request, regardless of the
VM_MAX_READAHEAD value. This prevents a rapid ramp-up
that could be caused due to increasing VM_MAX_READAHEAD.

Change-Id: I93d59c515d7e6c6d62348790980ff7bd4f434997
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
mm: pass readahead info down to the i/o scheduler

Some i/o schedulers (i.e. row-iosched, cfq-iosched) deploy an idling
algorithm in order to be better synced with the readahead algorithm.
Idling is a prediction algorithm for incoming read requests.

In this patch we mark pages which are part of a readahead window, by
setting a newly introduced flag. With this flag, the i/o scheduler can
identify a request which is associated with a readahead page. This
enables the i/o scheduler's idling mechanism to be en-sync with the
readahead mechanism and, in turn, can increase read throughput.

Change-Id: I0654f23315b6d19d71bcc9cc029c6b281a44b196
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
arm/crypto: Add optimized AES and SHA1 routines

Add assembler versions of AES and SHA1 for ARM platforms.  This has provided
up to a 50% improvement in IPsec/TCP throughout for tunnels using AES128/SHA1.

Platform   CPU SPeed    Endian   Before (bps)   After (bps)   Improvement

IXP425      533 MHz      big     11217042        15566294        ~38%
KS8695      166 MHz     little    3828549         5795373        ~51%

Signed-off-by: David McCullough <ucdevel@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Change-Id: I821aa8126e8ab0540f5cbc20ffee3a2e7a384d12
ARM: 7626/1: arm/crypto: Make asm SHA-1 and AES code Thumb-2 compatible

This patch fixes aes-armv4.S and sha1-armv4-large.S to work
natively in Thumb.  This allows ARM/Thumb interworking workarounds
to be removed.

I also take the opportunity to convert some explicit assembler
directives for exported functions to the standard
ENTRY()/ENDPROC().

For the code itself:

  * In sha1_block_data_order, use of TEQ with sp is deprecated in
    ARMv7 and not supported in Thumb.  For the branches back to
    .L_00_15 and .L_40_59, the TEQ is converted to a CMP, under the
    assumption that clobbering the C flag here will not cause
    incorrect behaviour.

    For the first branch back to .L_20_39_or_60_79 the C flag is
    important, so sp is moved temporarily into another register so
    that TEQ can be used for the comparison.

  * In the AES code, most forms of register-indexed addressing with
    shifts and rotates are not permitted for loads and stores in
    Thumb, so the address calculation is done using a separate
    instruction for the Thumb case.

The resulting code is unlikely to be optimally scheduled, but it
should not have a large impact given the overall size of the code.
I haven't run any benchmarks.

Signed-off-by: Dave Martin <dave.martin@linaro.org>
Tested-by: David McCullough <ucdevel@gmail.com> (ARM only)
Acked-by: David McCullough <ucdevel@gmail.com>
Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

Change-Id: I819f921198fd6cabfe0ae422607a5392d49040f4
ARM: 7723/1: crypto: sha1-armv4-large.S: fix SP handling

Make the SHA1 asm code ABI conformant by making sure all stack
accesses occur above the stack pointer.

Origin:
http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=1a9d60d2

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: Nicolas Pitre <nico@linaro.org>
Cc: stable@vger.kernel.org
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

Change-Id: Ia7fbf87db0cba99cb22cb432748a0e7f84192177
Revert "cpufreq: fix interactive bug during hotplug."

This reverts commit c5d82e843dc210898d67edc3362cfece50016c77.
cpufreq: interactive: Bring in line with Google's 3.4

cpufreq: interactive: take idle notifications only when active

Register an idle notifier only when the governor is active.  Also
short-circuit work of idle end if the governor is not enabled.

Signed-off-by: Sam Leffler <sleffler@chromium.org>
Change-Id: I4cae36dd2e7389540d337d74745ffbaa0131870f

cpufreq: interactive: keep freezer happy when not current governor

Fix a problem where the hung task mechanism was deeming the interactive
clock boost thread as hung.  This was because the thread is created at
module init but never run/woken up until needed.  If the governor is not
being used this can be forever.  To workaround this explicitly wake up
the thread once all the necessary data structures are initialized.  The
latter required some minor code shuffle.

Signed-off-by: Sam Leffler <sleffler@chromium.org>
Change-Id: Ie2c058dd75dcb6460ea10e7ac997e46baf66b1fe

cpufreq: interactive: handle speed up and down in the realtime task

Not useful to have a separate, non-realtime workqueue for speed down
events, avoid priority inversion for speed up events.

Change-Id: Iddcd05545245c847aa1bbe0b8790092914c813d2
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: remove input_boost handling

Now handled in userspace Power HAL instead.

Change-Id: I78a4a2fd471308bfcd785bbefcc65fede27314cf
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: always limit initial speed bump to hispeed

First bump speed up to hispeed_freq whenever the current speed is below
hispeed_freq, instead of only when the current speed is the minimum speed.
The previous code made it too difficult to use hispeed_freq as a common
intermediate speed on systems that frequently run at speeds between
minimum and hispeed_freq.

Change-Id: I04ec30bafabf5741e267ff289209b8c2d846824b
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: run at fraction of hispeed_freq when load is low

When load is below go_hispeed_load, apply the percentage of CPU load to
a max frequency of hispeed_freq instead of the max speed.  This avoids
jumping too quickly to hispeed_freq when it is a relatively low
percentage of max speed.  This also allows go_hispeed_load to be set to
a high percentage relative to hispeed_freq (as a percentage of max speed,
again useful when hispeed_freq is a low fraction of max speed), to cap
larger loads at hispeed_freq.  For example, a load of 60% will typically
move to 60% of hispeed_freq, not 60% of max speed.  This causes the
governor to apply two different speed caps, depending on whether load is
below or above go_hispeed_load.

Also fix the type of hispeed_freq, which was u64, to match other
speed data types (and avoid overhead and allow division).

Change-Id: Ie2d0668be161c074aaad77db2037505431457b3a
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: pin timers to associated CPU

Helps avoid waking up other CPUs to react to activity on the local CPU.

Change-Id: Ife272aaa7916894a437705d44521b1a1693fbe8e
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: use deferrable timer by default

Avoid wakeups only to handle the governor timer when the system is otherwise
idle.

For platforms where the power cost of remaining in idle at higher CPU
speed may outweigh the cost of a governor wakeup from idle to lower the speed,
set parameter cpufreq_interactive.governidle=1.

Change-Id: Id6c43eb35caecf9b0574fcdd5b769711bc7e6de6
Signed-off-by: LianWei WANG <a22439@motorola.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: kick timer on idle exit past expiry

The deferrable timer list isn't checked on all idle exits, such as when
hi-res timers expire or ISRs schedule workers.  If the idle loop is
exited and it's past time to run the governor load polling timer,
run it immediately.  This ensures we handle load spikes caused by actvity
that does not run the normal timer list.

Rename the field that timestamps the "time_in_idle" value to be more
accurate.

Change-Id: Ied590ecbefc83c9a9ec5eb9e31903557f6fa1614
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: trace actual speed in target speed decisions

Tracing adds actual speed since this is expected to be key to the
choice of target speed.

Change-Id: Iec936102d0010c4e9dfa143c38a9fd0d551189c3
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: change speed according to current speed and target load

Add a target_load attribute that specifies how aggressively the governor is
to adjust speed to meet the observed load.  New target speed is calculated
as the current actual speed (may be higher than target speed on SMP) times
the CPU load (as a fraction) divided by target load (fraction).

cpufreq_frequency_table_target() call use CPUFREQ_RELATION_L to set
the next higher speed rather than next lower speed.

Change-Id: If432451da82f5fed12e15c9421d7d27792376150
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: apply above_hispeed_delay to each step above hispeed

Apply above_hispeed_delay whenever increasing speed to a new speed above
hispeed (not just the first step above hispeed).

Change-Id: Ibb7add7db47f2a4306a9458c4e1ebabb60698636
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: allow arbitrary speed / target load mappings

Accept a string of target loads and speeds at which to apply the
target loads, per the documentation update in this patch.  For example,
"85 1000000:90 1700000:99" targets CPU load 85% below speed 1GHz,  90%
at or above 1GHz, until 1.7GHz and above, at which load 99% is targeted.

Attempt to avoid oscillations by evaluating the current speed
weighted by current load against each new choice of speed, choosing a
higher speed if the current load requires a higher speed.

Change-Id: Ie3300206047c84eca5a26b0b63ea512e5207550e
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: remove load since last speed change

The longer-term load since last speed change isn't terribly useful,
may delay recognition of dropping load, and would need forthcoming
changes to adjust load for changing CPU speeds.  Drop it.

Change-Id: Ic3cbb0542cc3484617031787e03ed9bdd632dec1
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: adjust load for changes in speed

Add notifier for speed transitions.  Keep a count of CPU active
microseconds times current frequency, converted to a percentage relative
to the current frequency when load is evaluated.

Change-Id: I5c27adb11081c50490219784ca57cc46e97fc28c
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: specify duration of CPU speed boost pulse

Sysfs attribute boostpulse_duration specifies the duration of boosting CPU
speed in response to bootpulse events.  Duration is specified in usecs,
default 80ms.

Change-Id: Ifd41625574891a44f1787a4e85d1e7b4f2afb52b
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: add timer slack to limit idle at speed > min

Always use deferrable timer for load sampling.

Set a non-deferrable timer to an additional slack time to allow prior to
waking up from idle to drop speed when not at minimum speed.  Slack value
-1 avoids wakeups to drop speed.  Default is 80ms.

Remove the governidle module param and its timer management in idle.  For
platforms on which holding speed above mimum in idle costs power, use the
new timer slack to select how long to wait before waking up to drop speed.

Change-Id: I270b3980667e2c70a68e5bff534124b4411dbad5
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: fix boosting logic

35a84de cpufreq: interactive: apply above_hispeed_delay to each step above hispeed

caused the speed choice logic to osciallate between boosting and not boosting.
Add back code to ensure speed does not drop below boost frequency while
boosting.

Change-Id: Id420068480fcc7f5c4989ff523e2a8d22e2f4db2
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: fix racy timer stopping

When stopping the governor, del_timer_sync() can race against an
invocation of the idle notifier callback, which has the potential
to reactivate the timer.

To fix this issue, a read-write semaphore is used. Multiple readers are
allowed as long as pcpu->governor_enabled is true.  However it can be
moved to false only after taking a write semaphore which would wait for
any on-going asynchronous activities to complete and prevent any more of
those activities to be initiated.

[toddpoynor@google.com: cosmetic and commit text changes]
Change-Id: Ib51165a735d73dcf964a06754c48bdc1913e13d0
Signed-off-by: Nicolas Pitre <nicolas.pitre@linaro.org>

cpufreq: interactive: fix race on timer restart on governor start

Starting the governor, or restarting on a hotplugged-in CPU, can race
with the timer start in idle, triggering a BUG on timer already pending.
Start the timer before setting the enable flag, and use enable_sem to
protect the sequence (and ensure correct order of the update to the
enable flag).  Delete any existing timer for safety.

Change-Id: Ife77cf9fe099e8fd8543224cbf148c6722c2ffb0
Reported-by: Francisco Franco <francisco.franco@cloudcar.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: default go_hispeed_load 99%, doc updates

Update default go_hispeed_load from 85% to 99%.  Recent changes to the
governor now use a default target_load of 90%.  go_hispeed_load should
not be lower than the target load for hispeed_freq, which could lead
to oscillating speed decisions.  Other recent changes reduce the need
to dampen speed jumps on load spikes, while input event boosts from
userspace are the preferred method for anticipating load spikes with
UI impacts.

General update to the documentation to reflect recent changes.

Change-Id: I1b92f3091f42c04b10503cd1169a943b5dfd6faf
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: init default values at compile time

Change-Id: Ia4966e949a6c24c34fdbd4a6e522cd7c37e4108e
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: don't handle transition notification if not enabled

If multiple governors are in use then avoid processing frequency transition
notifications for CPUs on which the interactive governor is not enabled.

Change-Id: Ibd75255b921d887501a64774a8c4f62302f2d4e4
Reported-by: Francisco Franco <francisco.franco@cloudcar.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: fix deadlock on spinlock in timer

Need to use irqsave/restore spinlock calls to avoid a deadlock in calls
from the timer.

Change-Id: I15b6b590045ba1447e34ca7b5ff342723e53a605
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: fix race on governor start/stop

There is race condition when both two cpu do CPUFREQ_GOV_STOP and one cpu
do CPUFREQ_GOV_START soon. The sysfs_remove_group is not done yet on one
cpu, but sysfs_create_group is called on another cpu, which cause governor
start failed and then kernel panic in timer callback because the policy and
cpu mask are all kfree in cpufreq driver.

Replace atomic with mutex to lock the whole START/STOP sequence.

Change-Id: I3762b3d44315ae021b8275aca84f5ea9147cc540
Signed-off-by: Lianwei Wang <a22439@motorola.com>

cpufreq: interactive: Enable CPU utilization statistics

 * Notify of CPU utilization on the speedchange task. This is required
   for MPDecision to properly consider hotplug scenarios.

Change-Id: If01a2efb7a7630544b16c782989406ac44865f61

cpufreq: interactive: allow arbitrary speed / delay mappings

Accept a string of delays and speeds at which to apply the delay before
raising each step above hispeed. For example, "80000 1300000:200000
1500000:40000" means that the delay at or above 1GHz, until 1.3GHz is 80 msecs,
the delay until 1.5GHz is 200 msecs and the delay at or above 1.5GHz is 40
msecs when hispeed_freq is 1GHz.

[toddpoynor@google.com: add documentation]
Change-Id: Ifeebede8b1acbdd0a53e5c6916bccbf764dc854f
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>

cpufreq: interactive: add io_is_busy interface

Previously the idle time returned from get_cpu_idle_time_us included the
iowait time. So the iowait time was always calculated as idle time.

But now the idle time returned from get_cpu_idle_time_us does not include
the iowait time anymore because of below commit which cause the iowait time
always calculated as busy time:
    6beea0c nohz: Fix update_ts_time_stat idle accounting

Add the io_is_busy interface, as does the ondemand governor, and let the user
configure the iowait time as busy or idle through the io_is_busy sysfs
interface.

By default, io_is_busy is disabled.

[toddpoynor@google.com: minor updates]
Change-Id: If7d70ff864c43bc9c8d7fd7cfc66f930d339f9b4
Signed-off-by: Lianwei Wang <lian-wei.wang@motorola.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: fix crash on error paths in get_tokenized_data

Use separate variable for error code, free proper pointer.

Change-Id: Ia83cccb195997789ac6afbf5b8761f7b278196d6
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>

cpufreq: interactive: base above_hispeed_delay on target freq, not current

Time to wait should be based on the intended target speed, not the
actual speed (which may be held high by another CPU).

Change-Id: Ifc5bb55d06adddb9a02af90af05398a78f282272
Reported-by: Arve Hjønnevåg <arve@android.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
cpufreq: interactive: fix uninitialized spinlock

Add missing spinlock init

Backtrace:
[<c0011ce4>] (dump_backtrace+0x0/0x10c) from [<c0662a68>] (dump_stack+0x18/0x1c)
 r6:00000032 r5:c0bd09ec r4:e6848000 r3:00000000
[<c0662a50>] (dump_stack+0x0/0x1c) from [<c06670b0>] (spin_dump+0x80/0x94)
[<c0667030>] (spin_dump+0x0/0x94) from [<c06670f0>] (spin_bug+0x2c/0x30)
 r5:c08f91fc r4:c0bd09ec
[<c06670c4>] (spin_bug+0x0/0x30) from [<c0245f74>] (do_raw_spin_unlock+0x88/0xcc)
 r5:e547bac0 r4:c0bd09ec
[<c0245eec>] (do_raw_spin_unlock+0x0/0xcc) from [<c066c9cc>] (_raw_spin_unlock_irqrestore+0x14/0x40)
 r5:e547bac0 r4:60000013
[<c066c9b8>] (_raw_spin_unlock_irqrestore+0x0/0x40) from [<c044b884>] (store_above_hispeed_delay+0x6c/0x80)
 r4:c0b4cf78 r3:00000007
[<c044b818>] (store_above_hispeed_delay+0x0/0x80) from [<c0235d24>] (kobj_attr_store+0x1c/0x28)
 r7:e68ff000 r6:00000032 r5:e58137c0 r4:e61cde80
[<c0235d08>] (kobj_attr_store+0x0/0x28) from [<c0156b78>] (sysfs_write_file+0x104/0x184)
[<c0156a74>] (sysfs_write_file+0x0/0x184) from [<c0100680>] (vfs_write+0xb0/0x140)
[<c01005d0>] (vfs_write+0x0/0x140) from [<c0100900>] (sys_write+0x44/0x70)
 r8:00000000 r7:00000004 r6:00000032 r5:bee43c90 r4:e5600300
[<c01008bc>] (sys_write+0x0/0x70) from [<c000e400>] (ret_fast_syscall+0x0/0x30)
 r9:e6842000 r8:c000e584 r6:00000032 r5:bee43c90 r4:00000009

Change-Id: I80a1e0b3fecb24adba501ff44f568479deeff7fa
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
cpufreq: interactive: handle errors from cpufreq_frequency_table_target

Add checks for error return from cpufreq_frequency_table_target, and be
less noisy on the existing call with an error check.  CPU hotplug and
system shutdown may cause this call to return -EINVAL.

Bug: 8613560
Change-Id: Id78d8829920462c0db1c7e14e717d91740d6cb44
Signed-off-by: Todd Poynor <toddpoynor@google.com>
cpufreq: interactive: reduce chance of zero time delta on load eval

Reschedule load sampling timer after timestamp of sample start taken,
hold spinlock across entire sequence to avoid preemption.  Avoid the
WARN for zero time delta in the load sampling timer function.

Change-Id: Idc10a756f09141decb6df92669521a1ebf0dbc10
Signed-off-by: Todd Poynor <toddpoynor@google.com>
cpufreq: interactive: avoid underflow on active time calculation

Check for idle time delta less than elapsed time delta, avoid
underflow computing active time.

Change-Id: I3e4c6ef1ad794eec49ed379c0c50fa727fd6ad28
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
cpufreq: interactive: fix race on cpufreq TRANSITION notifier

The cpufreq TRANSTION notifier callback does not check the
governor_enabled state on affected CPUS, which will case
kernel panic in update_load because the policy object maybe
NULL or invalid when governor_enabled is false.

Change-Id: Ie0f1718124f61e2f9b5da57abc6981ada5b83908
Signed-off-by: Lianwei Wang <a22439@motorola.com>
cpufreq: interactive: resched timer if max freq raised

When the policy max freq is raised, and before the timer is
rescheduled in idle callback, the cpu freq may stuck at a
lower freq.

The target_freq shall be updated too, else on a high load
situation, the new_freq is always equal to target_freq and
which will cause freq stuck at a lower freq too.

Reschedule the timer on gov limits callback.

Change-Id: I6c187001ab43e859731429b64f75a74eebc37a24
Signed-off-by: Lianwei Wang <a22439@motorola.com>
drivers: cpufreq: Send a uevent when governor changes

* Useful so userspace tools can reconfigure.
cpufreq: interactive: fix show_target_loads and show_above_hispeed_delay

Remove a trailing whitespace from target_loads and above_hispeed_delay. Problem
happens when user-space program tried to restore parameters that saved before
changing parameters. In this case was returned error(EINVAL).

Change-Id: I5a74e3824602cd6f2b74651adda5ec1b627e61e9
Signed-off-by: Minsung Kim <ms925.kim@samsung.com>
cpufreq: interactive: Add a sampling_down_factor for max frequencies

Change min_sample_time to 100ms when running at max

Change-Id: Ia7e35b0625bedf20e7ef3a1f52e5828ffbfed93e
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Allow 1 ms error in above_hispeed_delay comparisons

Allow for an error of 1 ms while taking into account
above_hispeed_delay for a frequency

Change-Id: I744e44387152e4efb5978df4f2b9533bf79d4582
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: sync freq feature for interactive governor

1) Add load info to cpufreq_interactive_cpuinfo
2) If load on any other online cpu exceeds sync_freq_load_threshold,
   do not allow the frequency to drop below sync_freq

Change-Id: I3617e10f87b85178914a18bcf04ac2a31a4f1ec1
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
(cherry picked from commit c26e25944543c61b34b1830938bb6f353a2580ac)
cpufreq: interactive: Reset floor_validate_time if busy at max for 100ms

When the interactive governor selects to run at max frequency it doesn't
re-schedule the timer until it hits an idle. This change checks if the CPU
has been continuously busy for last 100ms on hitting an idle start. If yes,
then floor_validate_time is reset so that the CPU stays at max frequency
for at least another 100 ms before stepping down.
This is an important feature for detecting CPU intensive workloads which
require high frequencies for achieving better performance.

Change-Id: I7d48ffbc3d50a80af9be3bf94667ee3d0120b763
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>
cpufreq: interactive: Remove trace event from idle_start handler

Removed the trace_cpufreq_interactive_idle_start.
Also fix a crash resulting from accessing NULL policy before taking
the pcpu->enable_sem lock. The policy can be NULL if the core is
hotplugged out before the enable_sem lock is taken.

Change-Id: I7e2809cc016b3b383a44cdf3c697013e2d2b5417
Signed-off-by: Rohit Gupta <rohgup@codeaurora.org>