VertexLoader: do not prepare for vertices if we need to skip them

Fixes a recent regression from #716. Not a big fan of doing the DataSkip in VertexLoaderManager, but tbh it isn't much worse than doing it in VertexLoader. It also removes the dependency of VertexLoader on bpmem, which makes me happy.
cc @neobrain / @degasus, is this a potential bug, or is the comment out of date?