This is just research at the moment. Too early for any recommendations. Danger with recommendations in indexing, that people may use them, and when new development comes along we will need another short term solution. What will the user want to do when searching for info. What do current search tools do eg.
Robots have some fundamental problems, but are liked.
Worm, Web Crawler , Lycos etc., are private bits of software. Harvest is more public and freely available and can be configured as a robot. UK database and let it soft through it. Would be more acceptable to a robot visit your site that was being funded to provide information to the community. Definite need for indexing.