done for distributed crawler