Added support to fetch robots.txt and some code reenginering