The folks at Archive.org said that robots.txt files don't serve the purpose of an archive site.