Kill annoying alert that won A/B