Add contact even if vcard fetching didn't work.

Also fixed a bug. We DO have to listen to the 'remove' event for roster items.