add a default robots.txt file just in case we can't access the online version
make use of the local robots.txt file when we can't get the online version
improve the source code documentation for solution one
remove unnecessary logic for the initial url