Don't remove event listener until after closeBlur event is triggered

This is for #5239.