[performance] get all extra metadata at once if possible

Doing multiple queries for this also meant multiple model updates,
and multiple contact field parsing on the sqlite side.