Fix up tracking stuff

Did you know: in python, dict() as a default arg means that it's only
evaluated once? Thus, EVERYTHING gets the same dict.

This bit me in the ass hard.