apache/lucenenet — pull request #30 by laimis (2014-12-27 18:57:55+00:00)
Title: return -1 if reader returns 0 or less

This fixes failures that can be observed in Lucene.Net.Analysis tests and possible other locations with test failure messages like these:

Expected string length 9 but was 13. Strings differ at index 1.
Expected: "npffoeijg"
But was: "n\0pf\0\0\0foeijg"

Expected string length 2 but was 1. Strings differ at index 1.
Expected: "[\0"
But was: "["

Expected string length 6 but was 8. Strings differ at index 6.
Expected: "432794"
But was: "432794\0\0"

MockTokenizer implementation uses equals comparison to 0 to check for an end of stream and rewrites it to -1 as Java would return and the calling classes expect. However some of the implementations (e.g. MockCharFilter) return -1 already and MockTokenizer should handle that. Without that check it ends up using values in the byte array which in "end of stream" cases contains \0 chars.

I tried checking other locations for cases of the same bug and could not find any. Although it does not mean it does not exist. It is so subtle and hard to find, took me forever just to figure out what was going on. Hopefully we don't have this situation anywhere else, and if we do, tests will lead us to such locations.