Remove channel information after being kicked or parting.

Also fixed _destroy_channel() to remove any users not in common anymore as a result of leaving the channel.