Fix layout mess-ups when toggling visibility