Remove some optimizations that didn't help, and add @inbounds where it does