Adds button that hides affected urls list