macosx: better check if we can open the new vout within the main window

fixes #9919
macosx: improve readability for main window check