BaseXdb/basex — Issue #305: Make Token.MAXLEN configurable
Author: malamut2
Created: 2011-12-06 16:00:23+00:00
State: open

I ran across the constant Token.MAXLEN which defines the maximal string length up to which hash codes will be calculated, and up to which strings will even appear in indices. This constant is currently of type byte. I suggest that you change this to int, and make the value configurable. This could be achieved with minimal additional code by using a variable optionally defined at the command line with -D. The default should stay at 96.

Background: in some cases, IDs may be longer than 96 characters. In our specific use case, we need to use URLs as IDs which have a typical length of some 120 chars. This makes even simple xqueries like /tag[id=$longID] very slow if there are lots of /tag entries. What's worse: while the query is understandably slow if no text index exists, it becomes even slower (much slower!) if a text index is established. I suspect that is because BaseX optimizes the query for index use, but since the key cannot be in the index now *all* data has to be searched, not just the /tag elements.

I attempted simply setting MAXLEN = 192 in the BaseX sources and did not experience any problems. Do you agree? In that case, it would be great if you could make this configurable.