olivernn/lunr.js — pull request #79 by cambridgemike (2014-04-01 05:43:47+00:00)
Title: Creates a TokenMetadataStore to return startPosition of tokens in results

I took a stab at prototyping a solution for https://github.com/olivernn/lunr.js/issues/25, and took your comments in https://github.com/olivernn/lunr.js/pull/58 into consideration.

I think the long term goal should be to refactor lunr.js, and replace existing tokens with a `lunr.Token`. Unfortunately this seems a bit overwhelming, as it presents a lot of complexities with previously serialized dataStores and complicates the TokenStore. As a step in the right direction, I propose a `lunr.TokenMetadataStore`, which maps doc refs and indexed tokens to a `lunr.Token`. In this case, by "indexed token" I mean present day token (string) and a `lunr.Token`, which is an object I introduce to encapsulate metadata about tokens (like StartPosition). This MetadataStore lives on the sidelines, and is only added to when a document is indexed, and data is only retrieved when a document is surfaced in search results.

So now you'll get back a result set that looks like

```javascript
// idx.add({ id: 2,
// body: "Some are born great, some achieve greatness, and some have greatness thrust upon them."
// })
// idx.search("greatness")

[{
"ref": 2,
"score": 0.12345
"tokens": [
{indexedAs: "great", raw: "great", startPos: 15, field: "body"},
{indexedAs: "great", raw: "greatness", startPos: 35, field: "body"},
{indexedAs: "great", raw: "greatness", startPos: 60, field: "body"}
]
}]
```

Overall, I created three new object types:
1. `lunr.Token` which is a container for metadata like startPosition
2. `lunr.TokenList` is essentially an array of `lunr.Tokens`, but has some helper method to extract the indexed tokens and the raw tokens.
3. `lunr.TokenMetadataStore` is a dataStore as described above.

The only changes I had to make to the existing codebase are outlined as follows:

**pipeline.js#run**

This is probably the most substantial change I made. Running the pipeline now returns a lunr.TokenList instead of an array of strings. This works with old tokenizers that return strings or new tokenizers that return `lunr.Token`s. The pipeline passes string tokens to the stack, so backwards compatibility is preserved for 3rd party pipeline functions and 3rd party tokenizers.

**index.js#add**

Updated this method to store the `lunr.Tokens` returned by the pipeline in the `lunr.tokenMetadataStore`. I added a configuration variable on the `lunr.Index.prototype` called `useTokenMetadata` which controls this behavior.

**index.js#search**

Once a list of documents is found, return a list of `lunr.Tokens` that are associated with the docment and were in the query string.

**tokenizer.js**

This was a pretty big change, and I did a quick and dirty job. I tried to keep the runtime relatively sane, but didn't worry too much about code organization. This should definitely be cleaned up if/when a merge happens.

**Notes:**

- Drawbacks: This will use up a lot of memory, since we create an individual token for every single string that gets indexed.
- Nomenclature: I started using the term "index token" to describe a present day "token", i.e a string that has been run through the pipeline (and will eventually end up in the index). A "raw token" is a String with the value of how the token appeared in the original document (with the exception of being lowercased, since we currently lowercase int he tokenizer).
- Tests: I updated a few things in the tests, but otherwise they all passed. I wrote a few tests just to smokescreen my additions. If you think this is headed in the right direction then I can sure it up with more tests.