VertexLoader: cache NativeVertexFormat

This fix a performance regression of PR #672.
(Maybe rename to map_entry here too, if you feel like it)