Refactor Dirty to keep custom serializers from running every time dirty run