Improve logic, exit early if we don't have a change timestamp.
Only query once for mailmap entries as well.

Probably some optimization to be done here, but these
change arrays are typically only 1 or 2 elements, so
not really an issue.