NEW: Drop trailing slashes as well as extensions when cleaning URLs.
FIX: Edge-case fix when creating import rules.
FIX: Avoid duplicates introduced by URL processing.

Sometimes, multiple spidered URLs will map to the same physical URL.  In this case, duplicates shouldn't
be created.  Generally speaking the difference will due to something trivial like trailing slashes.

The first URL discovered will be used, and the other ignored.