Switched from using PersonNumbes (which not all contacts have) to using PhoneNumbers as the basis for conversations. Speed has slowed down but all conversations now show up