basho/riak_kv — pull request #535 by russelldb (2013-04-25 09:05:03+00:00)
Title: Add pagination to 2i results

Adds a `max_results` option to PB and HTTP 2i endpoints for limiting
the number of results returned for a 2i query. Will return an opaque
`continuation` if there are at least `max_results` results returned. A
subsequent query that includes the `continuation` as a parameter will
return the next `max_results` results.

The results will be returned sorted ascending. In the case of $key
and $bucket queries, and equals queries, the results will be sorted by
primary key In the case of range queries the results will be sorted by
index value. Since the values must be returned from the backend for
sorting, there is a further option `return_terms` that will cause the
indexed terms to be returned to the user also, where relevant (range
queries on index fields.)

Sorting is performed by the module `sms.erl` originally written by Reid
Draper. It is used as a buffer by `riak_kv_index_fsm`.

A new vnode function `stop_fold` has been added so the fsm can tell
the backend to break out of the fold when enough results have been
gathered.

The option to stream 2i results has also been added. Adding the
`stream` option to either HTTP or PB queries will cause the API
endpoints to return chunks of results as they are received from the
fsm This can lead to much lower perceived latencies as the application
recieves the first results of a large set much sooner.

An example HTTP query: `curl
"http://127.0.0.1:10018/buckets/2ibucket/index/field2_int/1/100?return_terms=true"`
Will return the first 100 {value, key} pairs for the `field2_int`
index, sorted by value. If there are 100 results returned, it will
also return an opaque `continuation` that may be sumbitted to continue
the query: `curl
"http://127.0.0.1:10018/buckets/2ibucket/index/field2_int/1/100?return_terms=true&continuation=some_gibberish_looking_value"`
A further `continuation` maybe returned with the results.

-----
Riak tests can be found at https://github.com/basho/riak_test/tree/pt29-2i-pagination
Riak-Erlang-Client support at https://github.com/basho/riak-erlang-client/tree/pt29-2i-pagination

NOTE: requires
https://github.com/basho/riak_core/tree/pt29-2i-pagination
https://github.com/basho/riak_pb/tree/pt29-2i-pagination