updates for crawler