Fix a bug in FactorLoader, which makes validation pass mistakenly.