Don't allocate a Set when there are not attachments.