The general rule is to use the JPEG format for photographic type images and to use GIF for illustration style images. Photographic images are basically just that, they have the full range of colors that tend to blend into each other. Illustration style images are ones with a limited number of colors, or areas of solid color. Many banner ads are of this type. However, general rules tend to break down rather quickly on the web.
Every choice involves a trade-off of some sort.
This brief article is just a basic introduction to choosing the best format for website images. For a really great, in-depth, book on this subject check out the link at the bottom of this article. The important concept here is to understand the basic differences between the two major formats and how they affect your images. Compression is achieved by selectively removing parts of the image. A photographic image can be subjected to a very high degree of compression without noticable degradation.