A distributed webpage crawler and parser worked in a highly configurable way