VertexLoader: do not prepare for vertices if we need to skip them

Fixes a recent regression from #716. Not a big fan of doing the DataSkip in VertexLoaderManager, but tbh it isn't much worse than doing it in VertexLoader. It also removes the dependency of VertexLoader on bpmem, which makes me happy.
This is moved code, I have no idea. Assuming 3 is CULL_NONE and primitives < 5 are triangles, quads, triangle strips, etc. (!lines && !points).