Switch path stuff to vectors

This should be more CPU cache friendly compared to lists so should
be faster. (profiling looked promising)