Improve callstats number matching code.

Make it a little more efficient, and make sure the ContactInfo and
CallStatsDetails instances always stay in sync.

Change-Id: I0b01e6fc036ce70dccbccf1d9002843331304d50