Added more functionality to the robcrawler. Added more attributes to the website class. Refactored code with correct usage of variable names and tabbed spaces.