Cache addresses and only add those which are new.

This also changes the approach for getting imap headers by using a sorting
method which avoids going over all emails each time (except for first time
run).