add options for curl settings
add options for crawling 404 pages and using effective urls