Clean up parting users that have no more common channels with the bot