Frequency-dependent selection is an evolutionary process by which the fitness of a phenotype or genotype depends on the phenotype or genotype composition of a given population.
 In positive frequency-dependent selection, the fitness of a phenotype or genotype increases as it becomes more common.
 In negative frequency-dependent selection, the fitness of a phenotype or genotype decreases as it becomes more common. This is an example of balancing selection.
 More generally, frequency-dependent selection includes when biological interactions make an individual's fitness depend on the frequencies of other phenotypes or genotypes in the population.

Frequency-dependent selection is usually the result of interactions between species (predation, parasitism, or competition), or between genotypes within species (usually competitive or symbiotic), and has been especially frequently discussed with relation to anti-predator adaptations. Frequency-dependent selection can lead to polymorphic equilibria, which result from interactions among genotypes within species, in the same way that multi-species equilibria require interactions between species in competition (e.g. where αij parameters in Lotka-Volterra competition equations are non-zero).  Frequency-dependent selection can also lead to dynamical chaos when some individuals' fitnesses become very low at intermediate allele frequencies.

Negative

The first explicit statement of frequency-dependent selection appears to have been by Edward Bagnall Poulton in 1884, on the way that predators could maintain color polymorphisms in their prey.

Perhaps the best known early modern statement of the principle is Bryan Clarke's 1962 paper on apostatic selection (a synonym of negative frequency-dependent selection). Clarke discussed predator attacks on polymorphic British snails, citing Luuk Tinbergen's classic work on searching images as support that predators such as birds tended to specialize in common forms of palatable species. Clarke later argued that frequency-dependent balancing selection could explain molecular polymorphisms (often in the absence of heterosis) in opposition to the neutral theory of molecular evolution.

Another example is plant self-incompatibility alleles. When two plants share the same incompatibility allele, they are unable to mate. Thus, a plant with a new (and therefore, rare) allele has more success at mating, and its allele spreads quickly through the population.

A similar example is the csd alleles of the honey bee. A larva that is homozygous at csd is inviable. Therefore rare alleles spread through the population, pushing the gene pool toward an ideal equilibrium where every allele is equally common.

The major histocompatibility complex (MHC) is involved in the recognition of foreign antigens and cells. Frequency-dependent selection may explain the high degree of polymorphism in the MHC.

In behavioral ecology, negative frequency-dependent selection often maintains multiple behavioral strategies within a species. A classic example is the Hawk-Dove model of interactions among individuals in a population. In a population with two traits A and B, being one form is better when most members are the other form.  As another example, male common side-blotched lizards have three morphs, which either defend large territories and maintain large harems of females, defend smaller territories and keep one female, or mimic females in order to sneak matings from the other two morphs. These three morphs participate in a rock paper scissors sort of interaction such that no one morph completely outcompetes the other two. Another example occurs in the scaly-breasted munia, where certain individuals become scroungers and others become producers.

A common misconception is that negative frequency-dependent selection causes the genetic diversity of influenza haemagglutinin (HA) glycoproteins. This is not an example of negative frequency-dependent selection. This is because the rate at which a particular influenza strain will spread is linked to absolute abundance, not relative abundance.

Positive

Positive frequency-dependent selection gives an advantage to common phenotypes. A good example is warning coloration in aposematic species.  Predators are more likely to remember a common color pattern that they have already encountered frequently than one that is rare. This means that new mutants or migrants that have color patterns other than the common type are eliminated from the population by differential predation.  Positive frequency-dependent selection provides the basis for Müllerian mimicry, as described by Fritz Müller, because all species involved are aposematic and share the benefit of a common, honest signal to potential predators.

Another, rather complicated example occurs in the Batesian mimicry complex between a harmless mimic, the scarlet kingsnake (Lampropeltis elapsoides), and the model, the eastern coral snake (Micrurus fulvius), in locations where the model and mimic were in deep sympatry, the phenotype of the scarlet kingsnake was quite variable due to relaxed selection. But where the pattern was rare, the predator population was not 'educated', so the pattern brought no benefit. The scarlet kingsnake was much less variable on the allopatry/sympatry border of the model and mimic, most probably due to increased selection since the eastern coral snake is rare, but present, on this border. Therefore, the coloration is only advantageous once it has become common.

See also
 Apostatic selection
 Evolutionary game theory
 Evolutionarily stable strategy
 Frequency-dependent foraging by pollinators
 Fluctuating Selection
 Mimicry
 Tit for tat

References

Bibliography
 Robert H. Tamarin (2001) Principles of Genetics.  7th edition, McGraw-Hill.

Evolutionary biology concepts
Population dynamics
Selection