You can remove someone’s access from a specific conversation by using the sharing menu at the top of the conversation view. You can use the same method to give access to a conversation to a user.