SG_REF/SG_UNREF in wrong order can cause segfaults when setting the same mode again.