improving crawler fault tolerance