ARM: Optimize memset/memcpy (Credits to: Harm Hanemaaijer)

[PATCH 1/3] Rename ARM assembler push/pull macros

      The ARM assembler library functions use a macro called "push" that
      along with a macro called "pull" is used to shift bytes around in a
      word in an endian-independent way. However, the modern unified
      ARM assembler syntax also defines the instruction "push" to push
      data onto the stack, which has specific encodings in the Thumb2
      instruction set. For prevent possible conflicts going forward, and
      to allow the use of the more transparent "push" instruction along
      with the modern unified assembler syntax, this patch renames all
      occurrences of the "push" macro to "pushbits", as well as renaming
      the macro argument, when also called "push", to "pushshift".

      For consistency, the macro called "pull" with its argument name
      "pull" are also renamed to "pullbits" and "pullshift", respectively.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>

[PATCH 2/3] Optimize copy_page for modern ARM platforms

      The existing implementation of copy_page for ARM appears to be
      optimized for older platforms. Benchmark testing in a sandbox
      environment shows suboptimal performance on modern platforms
      like armv6 and armv7, with speed-ups ranging from 10% (Cortex A8)
      to 80% (armv6 used in Raspberry Pi) being achievable.

      This commit optimizes copy_page and introduces the new compile-time
      constant PREFETCH_DISTANCE, defined in cache.h, which when
      multiplied by L1_CACHE_BYTES is equal to the offset used for
      prefetches performed with the PLD instruction. For platforms where
      L1_CACHE_BYTES is 32 (armv5 and armv6), copy_page processes 32 bytes
      at a time while doing one prefetch per iteration, while for armv7
      (with L1_CACHE_BYTES equal to 64), 64 bytes are processed at at time
      with one prefetch per iteration. When no preload instruction is
      available (platforms earlier than armv5), no preload instructions
      are generated and 32 bytes are processed at at time.

      To facilitate specifying instructions for architectures with no
      preload instruction, the NO_PLD macro is added to assembler.h,
      augmenting the PLD macro.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>

[PATCH 3/3] Optimize ARM memset and memzero functions

      The ARM memset and memzero functions are optimized with lower overhead
      for small requests, generation of more 16-bit Thumb2 instructions when
      compiled in Thumb2 mode, and configurable destination alignment before
      the main block-copying loop.

      The new compile-time constant MEMSET_WRITE_ALIGN_BYTES is introduced
      in assembler.h, to augment the CALGN macro that previously regulated
      32-byte write alignment but was only used on the Feroceon platform.
      MEMSET_WRITE_ALIGN_BYTES can have values of 0 (no write alignment),
      8, or 32. Apart from Feroceon, memset write alignment of 32 bytes
      appears to benefit the armv6 platform, while the armv7 platform
      seems benefit from alignment to 8 bytes for memset/memzero.

      The CALGN macro is renamed to MEMSET_CALGN for memset and memzero; the
      original CALGN macro is reserved for the memcpy family of functions
      (memcpy, copy_from_user, copy_to_user) currently implemented in
      copy_template.S, and the associated compile time constant
      WRITE_ALIGN_BYTES defines the write alignment for memcpy-related
      functions that will be utilized in subsequent contributions. Because
      the current CALGN implementation in copy_template.S only implements
      32-byte write alignment and is broken on Thumb2, it is only enabled
      when WRITE_ALIGN_BYTES is equal to 32 and Thumb2 mode is not enabled.

      Finally, memset and memzero now include a directive to enable unified
      ARM assembler syntax.

      Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
[PATCH] Speed up console framebuffer imageblit function

Especially on platforms with a slower CPU but a relatively high
framebuffer fill bandwidth, like current ARM devices, the existing
console monochrome imageblit function used to draw console text is
suboptimal for common pixel depths such as 16bpp and 32bpp. The existing
code is quite general and can deal with several pixel depths. By creating
special case functions for 16bpp and 32bpp, by far the most common pixel
formats used on modern systems, a significant speed-up is attained
which can be readily felt on ARM-based devices like the Raspberry Pi
and the Allwinner platform, but should help any platform using the
fb layer.

The special case functions allow constant folding, eliminating a number
of instructions including divide operations, and allow the use of an
unrolled loop, eliminating instructions with a variable shift size,
reducing source memory access instructions, and eliminating excessive
branching. These unrolled loops also allow much better code optimization
by the C compiler. The code that selects which optimized variant is used
is also simplified, eliminating integer divide instructions.

The speed-up, measured by timing 'cat file.txt' in the console, varies
between 40% and 70%, when testing on the Raspberry Pi and Allwinner
ARM-based platforms, depending on font size and the pixel depth, with
the greater benefit for 32bpp.

Signed-off-by: Harm Hanemaaijer <fgenfb@yahoo.com>
ARM: bitops: prefetch the destination word for write prior to strex

The cost of changing a cacheline from shared to exclusive state can be
significant, especially when this is triggered by an exclusive store,
since it may result in having to retry the transaction.

This patch prefixes our atomic bitops implementation with prefetchw,
to try and grab the line in exclusive state from the start. The testop
macro is left alone, since the barrier semantics limit the usefulness
of prefetching data.

Acked-by: Nicolas Pitre <nico@linaro.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>