Unify ref/unref tracking, make it work as one would expect