refactoring
add robots.txt functionality