Correctly removing invite button if not owner