Don't bother reconciling external visible decls against our current set of
declarations if we didn't have a lookup map when the external decls were added.