basho/riak_kv — pull request #568 by martinsumner (2013-05-27 21:49:16+00:00)
Title: Hscic 2i applyregex

Follow discussions with Russell Brown et al, this is the regular expression feature for 2i performance improvements written as a branch to extend the other 2i changes (streaming, pagination etc). This has been written to support the requirements of the NHS Spine project in England.

The idea is that a regular expression can be passed as a parameter in a 2i query (using the HTTP protocol) - with the key 'apply_regexp' used with the value being a regular expression. If the regular expresion is invalid (which means it won't compile when erlang attempts to re:compile/1 it with no options), then a a bad request response is returned with details of the compilation failure. The regular expression will be applied to all index terms which are returned in a 2i range query (and will be ignored if it is anything but a range query) - and only passing terms will be added to the buffer of results. This allows for index terms to be overloaded with concatenated pieces of information, and results filtered by this information when the ordering of the information in the term prevents the filtering being applied to the range. As this reduces the information being sent between vnodes, and horizontally distributes the process of the regex across the cluster, it is expected that this may often be 50% faster than using return_terms and filtering terms in the application.

The feature should work independently of return_terms - return_terms can be either true or false, and the expression will still be applied. With max_results set, only those results within the range and passing the regular expression will be returned and count towards the maximum results. It works with streaming (although you may get repeated sequences of empty result sets) if there is a low pass rate on the regular expression.

Implementation has been done by extending the v2 query record to support a term_regexp field. A skip_result atom has been added as a potential alternative Val to be passed to the accumulator/buffer, when there is nomatch for the regular expression on the term - and that accumulator/buffer then doesn't add the result to the set.

I've run a set of local tests to prove counts match between sending the regexp to the cluster, applying it in the client after using return_terms. Also basic functional tests that behaviour with different other parameter combinations is as expected. I've not forked riak_test, and added any tests here, due to understanding issues on my part.

I've only considered the eleveldb backend, as I believe this is the only one to support v2 queries.

Martin