spideyagent (pypi): Each user can run their own threaded search engine and contribute to a global search database searching only the sites they want
Keywords: search|engine|agent|crawler