added ability to set a time limit on crawler