spec tests done...
updated to use options from trollop in crawler directly