blacklist disabled
double quotes in url ar url encoded prior to download
An exception that occurs in MainHeartBeat when attempting to kill a process will not cause aborting of processes to be aborted
Increased ping timeout for heartbeat
now avoids downloading the same link more than once in the same session
changed Log.esc to accept Object instances
Indexer.add now accepts a variable number of args
renamed Store.has to Store.addExisting to indicate that it can cause side effects
Fixes how timeouts are handled during download.
Mutex-lock fix for downloader
Concurrency fixes in downloader
fixed issue where filter frequently prevented BingSearchEngine from downloading 2nd page of results
results / observervations regarding run without blacklist
refactoring: created a com.hourglassapps.cpi_ii.lucene package for general indexing and search classes
Early work on indexer for downloaded files.
In downloader when all results have previously been downloaded we now populate the __types.txt file.
File indexer for downloaded files
Improved exception handling in MainIndexDownloaded.
Tika no longer uses server provided mime types to determine types of downloaded files -- now it uses its own automagical method
fixes memory leak in indexer for downloaded files
indexer for downloaded files now doesn't need to index everything from the beginning if it's interrupted or crashes
Initial work on moving from multi-process downloader to a multi-threaded one.
The reason for the change is that it'll help me integrate the indexing component into the downloader, for faster combined downloading/indexing (hopefully).
The downside is slightly less redundancy in case a crash occurs during download.
Threading fixes, including a change to ensure that if a QueryThread hangs or crashes it won't disrupt downloading from other threads
Fixes expansion of stemmed terms
itegration of indexer with downloading code
work on integrating indexer with downloader
CREATE_APPEND is the OPEN_MODE used for IndexWriter during indexing now while downloading
ExpansionDistributor now has responsibilty for closing QueryThreads (ensuring that its deferred isn't resolved until all downloads have been handled)
now can force quit downloader / indexer cleanly
fixed handling of the indexer's feeder journals so that we don't skip over unindexed directories after a crash or force quit
downloaded documents and index are now save to same directory
added the lucene demo SearchFiles to test index