Actually add classes to the seen_classes set