Added infrastructure for DB search queries.
Also duplicated search form.
Basic search from the DB.
Added infrastructure for crawler thread
Added throttling, page columns
Infrastructure to pick a page, crawl, and index.
Also refactored stuff into separate files by responsibility.
Processing and real indexing of sites.