Reduce search space for valid fd's
Be more explicit about which sets we handle