Larger zygotes have greater chance of surviving because they have more stored food reserves. Large gametes produce large zygotes. This produces immediate selection for small gametes to find and fuse with large gametes. Otherwise, they have no chance. Larger gametes should try to avoid fuzing with small ones, but the penalty is not as great.
Intermediates lose out.
Given anisogamy, females become a scarce resource because males can make sperm and fertilize eggs faster than females can produce them. This is costly and becomes the limiting factor. The argument holds whenever females invest more than males in each offspring whether in just eggs or in eggs plus subsequent care of young. In many species, males are capable of producing sufficient sperm to fertilize the eggs of females whenever they encounter a receptive female while females take time to secure resources, produce eggs, and care for young. A good example of the difference that we often encounter in male and female receptivity is the Coolidge Effect .