Remove unneeded check in the CardinalityFactor