Vestigial organs are organs of the body which are smaller and simpler than those in related species. They have lost, or almost lost, their original feature.

Vestigiality is evidence for evolution, since they only make sense if evolution has occurred. They were one of the puzzles of pre-Darwinian natural history. The puzzle vanished once biologists realised they once were working adaptations, in the ancestors of present-day animals.

They occur in animals (and plants) which have changed their style of life from their ancestors. Thus snakes lost their legs as their system of movement changed. But one type of snake – the boas – have vestigial rear legs and pelvis. The human vermiform appendix is another example. That was much larger, and stored microbes which produced cellulase to break down plant cell walls. Leaves are the main diet of apes, but they are not a main part of man's diet. Cellulose cannot be digested by our species.

An unused organ usually degenerates, and becomes smaller or vanishes altogether. Thus amphibia living in dark caves lose their sight and their body colour. Mutations pile up. All structures need energy for their development, maintenance, and weight. This, and the risk of disease in the part (e.g., infection, cancer), provide some selection for the removal of parts which no longer help an organism's fitness.

The human appendix had selection against it because of deaths from appendicitis. Appendicitis was untreatable until modern times. The selection pressure was probably weak, because most cases occur after the age of peak fertility.

Observations on the cave-dwelling amphipod Gammarus minus showed active selection against their vestigial eyes. When mating, the males remain on their females for a week or so after fertilisation, guarding them against other males. The eye sizes of paired males were compared with those of unpaired males. The eye sizes of paired males were smaller than the unpaired ones. Obviously, on average, the paired males would have higher fertility than the unpaired males. The selection gradient was estimated as –0.30, which means the selection for small eyes was quite strong. The researchers supposed that the loss of sight freed more of the nervous system to process other sensory inputs.p310

Occasionally, selection turns vestigial organs to a new function. Thus the mammalian ear ossicles were once bones in the jaw of early proto-mammals, the Therapsida. The halteres of flies is a perfect example. The word exaptation was coined for this phenomenon.

Fitness 
All functions and structures develop and change because they affect the biological fitness of the organism. They make it more or less likely that individuals reproduce and contribute genes to the next generation. So, for example, an ability of a frog to survive the bite of its local venomous snake increases its fitness. But if it lives in an area where the snake does not, then it reduces its fitness. This is because it bears the "cost" of a complicated mechanism without getting any advantage in survival and reproduction. It works the other way round, too. Snakes which eat toxic prey have to use ways to survive the prey's toxins. The snake's digestion and metabolism is slower, and they move around more slowly. See rough-skinned newt and garter snake.

Everything has a cost: either it uses up energy or its structure occupies space that might be used differently.

References 

Evolutionary biology