Only update buffers when new verts are added