fix for picking up URL:s with wrong content type
Merge branch 'master' of github.com:soulgalore/crawler