Fixes issue in crawler where urls are not properly formed for crawling