Partial revert of ARM optimiations

These probably don't improve on performance much and might cause
issues with audio quality
Signed-off-by: Paul Beeler <sparksco@gmail.com>