Swap value function arrays instead of reallocating them

This brings the relative runtime (compared to your master) to 0.89 or so on my machine.
Agreed, I just pushed a new commit.