searchbot crawler improvements