Added robots and site map generation, clean up