implemented parser for the robot.txt and modified main.py to print the robot.txt content of the target host