Revert "Accept IRCConversation types in add/remove channel"

This reverts commit 8b39680f67e7dfd00cb2818ee27e2b6b7e0516c0.