Octree: improve performance by removing intermediate vectors

Also removes vector-based prim storage