Account for diffs deeper in the tree

@theresaanna found a bug when testing dummy data. This resolves by making sure all labels in the tree are accounted for.