Allow user to decline an invitation