Ensure bo is set should we panic in VBO_DerefBusyObj