Multithreaded one level crawler