``defaultdict`` pattern here almost, though it may be annoying to deal with ``labels``