Consider re-generating static pages on a daily basis instead. I run a couple of engines based on SFgate. SFgate is a perl CGI interface to freeWAIS-sf. WAIS stands for Wide Area Information Service. It's a set of protocols for creating a hierarchy of searchable databases.
Implementations also contain an indexing program to create the database.
It does this by taking every word in a document, stemming it to strip off pluralization and such, then applying a soundex algorithm. It stores this hash and a pointer to the original document in a database. There are three freely available implementations. The original WAIS, freeWAIS, and freeWAIS-sf . The original was written to demonstrate the technology.