need to rework crawler to have single thread for certain repositories