apache/lucenenet — pull request #29 by laimis (2014-12-27 16:48:11+00:00)
Title: change implementation so that Read overrides behave as expected

Discovered this issue while running Lucene.Net.Analysis\TestGraphTokenizers tests. From time to time an assert would fail at this location:

https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.TestFramework/Analysis/BaseTokenStreamTestCase.cs#L964

The idea here is to randomly simulate an exception being thrown when token stream is being iterated. Instead of "evil exception" being thrown, the following assert would fail:

https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.TestFramework/Analysis/MockReaderWrapper.cs#L92

The underlying problem is the implementation of MockReaderWrapper in C#. It was not exactly a 1-to-1 map to the Java version due to framework differences and bug was introduced in MockReaderWrapper constructor:

https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.TestFramework/Analysis/MockReaderWrapper.cs#L43

StringReader's ReadToEnd() gets called advancing the iterator to the end. When the actual token stream iteration tries to read the stream, 0 is returned as the underlying text stream has been already iterated by the "ReadToEnd" call in the constructor. Furthermore, MockReaderWrapper should override Read() method as to make sure that all the possible ways that the reader is being used are covered (see https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.TestFramework/Analysis/MockTokenizer.cs#L242 how it can call Read with no params).

Also changed the way MockReader is initialized, instead of new StringReader, text string is passed to it directly. That seemed to make the most sense as there does not appear to be the need to create new stream reader on the same text just to extract that text again.

TestGraphTokenizers still occassionally fail but that appears to be due to another bug that gets invoked radomly and is yet to be determined what is the cause of it. Taking care of this "evil exception" branch gets us moving forward in that investigation.