Use a defaultdict for annotations