Remove divisors which are not primes.