Change PlaybackBox group and password popup handling to be more sane.

Backport from master to fixes/0.26.

If an invalid password is entered, the password popup is reshown.  If
the password popup is aborted and was entered from the group popup or
is the initial group, the group popup is reshown.  If the group popup
is aborted and was entered because of the QueryInitialFilter setting,
the PlaybackBox screen is closed.

Refs #11231