crawler configuration finalizing