Wait for the Display to finish to get rid of the old PAM handle