macosx: reset float-on-top when native fullscreen has already finished

This fixes issues where video window just disappears when exiting fullscreen,
only on Mavericks.

close #9814
macosx: fix float-on-top-behaviour if setting is changed while in fullscreen
macosx: fix typo and set path for open dialog correctly