Add some more asserts for catching invalid VPs when build without WITH_VERIFY_PTR