Issue 4078: Reorganize indexing and add unit tests

Issue 4078: Reorganize indexing and add unit tests, with fixes for BOT issues

https://code.google.com/p/otwarchive/issues/detail?id=4078