Remove some optimizations

The probably won't increase performance much and might cause some
issues with audio quality
Signed-off-by: Paul Beeler <sparksco@gmail.com>