[libcommhistory] Simplify unknown address comparison

libcontacts address resolution makes this unnecessary.