A flexible, modular web crawler