I think this could be simplified a lot using `dict.setdefault()` and/or `collections.defaultdict()`