Make sure the dialog shows somewhere.
Make sure we do not configure more times than needed.