crawler: URL preprocessing handling extracted to separate function
index_files: example for building and processing webgraph added