FIX: Keep inferred URLs separate do reduce duplicates.

Before this change, there was a bug, that if a parent URL didn't exist prior to processing, but did exist after it, a duplicate would be created.

This change keeps so-called "inferred" URLs separate from directly spidered ones, and recalculates the necessary inferred URLs after a processing function is applied.
FIX: Don't get stuck in 404 infinite loops.

If you hit a 404 page, don't follow outbound links.  This can avoid infinite loops due to badly coded pages.
FIX: Drop trailing / from base url when making absolute URLs.
FIX: Fixed extension dropping url processor when ? is present.