set the curl timeout to 5 seconds

this is more reasonable than 30 seconds
add options for curl settings
add options for crawling 404 pages and using effective urls
use curl_setopt_array to set options
Merge pull request #7 from mfairchild365/master

Options, options and more options