basho/riak_cs — pull request #600 by reiddraper (2013-06-25 21:14:05+00:00)
Title: Improve object listing performance and efficiency

When using Riak CS with Riak 1.4 or greater, we can take advantage of paginated
object listing. This allows us to no longer have to do list-keys on the entire
bucket and do a multi-get of all of the objects. This ends up being faster and
cause less load on the cluster. For backward compatibility, this is implemented
as a new FSM, `riak_cs_list_objects_fsm_v2`, which is only used if explicitly
set in the application. The default is false. This commit adds the option to
the `app.config`, so that it's easy for users to see the new flag. It can also
be set in the console like this:

`application:set_env(riak_cs, fold_objects_for_list_keys, true).`

Furthermore, the built in statistics for the new FSM can be seen by turning on
debug logging. This can be done just for the module like this:

`lager:trace_console([{module, riak_cs_list_objects_fsm_v2}], debug).`

There is an EQC test that is added for some pure, helper functions in the new
FSM. There is also a new `riak_test` test that runs the current list objects
test, but uses the new FSM.

* A bunch of duplicated effort in the two FSMs has been put into a shared
module
* Whether or not to use the new or old FSM is abstracted by the helper module
* The full set of S3 list objects options are supported