Use a more generic solution by tracking previous mode and exit only if we're back to previous mode.