Fixed a problem where urls weren't being joined properly if they started with '//'
Prevent crawler from requesting media URLs or mailto: URLs
Added zip, eps, and pdf to list of media extensions.
Get rid of duplicate URLs on the frontier.