Fix for issue where a crash during downloading/indexing would result (after the restart) in documents being downloaded again that had previously been download before the crash.
The problem with that was that both documents would later be returned in the result list for some query, wasting one of the 100 top results that are reported (although the duplicated document would be filtered out of the report so from a presentation point of view it was okay).
removed commments, some unused code
added in some licence information
when generating report lines are now treated as phrases when search (rather than a bag-of-word)
early work on synonym expansion
now single word lines are converted to two clause queries: one with the word appended to the previous line, and the other with it prefixing the subsequent line