libgui: Improving window mode switches

More systematical approach to handling all of the possible combinations.