[libcontacts] Allow listeners to be removed