Distributed and specialized web crawler