a distributed page crawler and parser worked in a highly customized way