Only show remove all button when there are more than 1 user