adjustments times for crawler urls