Removed old stuff from robots.txt file.
Allow asset files to be spidered (i.e. for the Internet Archive).