a configureable distributed crawler