Restrict factor checks to the existing list of prime numbers