Avoid touching GL at all if a buffer resize isn't required