Inner loop optimizations

Surprising speedup from combining array and pointer access. The inner
loop is less of a bottleneck lately though so overall gains are minimal.