fix bug in index verification test code; add more tests for indexing; fix bug in indexer when encountering duplicate terms as we are ignoring those for now