Make sure hitting the close button does not trigger double prevent navigation warnings

(cherry picked from commit 64502a19b044e96c19c6b7666d1dfcb33acd7b72)