Fix vbo full

Fix a bug that renderer will have render artifact when exceeding VBO size, Also adding a test for it.