Make Sphider obey robots.txt
Added SEO friendly dynamic page titles