OpenGrok/OpenGrok — pull request #23 by kahatlen (2013-03-26 19:41:56+00:00)
Title: Combine Hash2TokenStream and List2TokenStream

Hash2TokenStream and List2TokenStream are almost identical.

One of the differences is that List2TokenStream has a bug that prevents it from seeing more than one token in the last element of the list. For example, with the following list

Arrays.asList("abc.def", "ghi.jkl")

List2TokenStream returns only three tokens: "abc", "def", "ghi"

Hash2TokenStream, on the other hand, returns all four: "abc", "def", "ghi", "jkl"

Another difference is that List2TokenStream only splits the elements in the list if they contain dots. That is, if the list contains "class$java$lang$NullPointerException", the token stream returns that as a single token. On the other hand, "java.lang.NullPointer" would have been split into three tokens. Hash2TokenStream would have split both strings into multiple tokens.

In both cases, I find the behaviour of Hash2TokenStream more reasonable. I therefore propose that we create a new token stream based on Hash2TokenStream and make it accept a java.lang.Iterable instead of a java.util.Set. Since both Set and List implement Iterable, the new class can replace both Hash2TokenStream and List2TokenStream.