web crawler (find temporary files, parse robots.txt and search some folders)