Only show remove user on hover for admins

https://trello.com/c/uORsGmAQ