add useful indexes
refacto crawl status info to fasten collect
make sure booleans passed in shell are correctly interpretd