Fixing small issue with duplicate users

Users are sometimes not removed from channel user lists, resulting in duplicates when they join again. A set is more appropriate for this, anyway.