basho/rebar — pull request #317 by kevsmith (2012-09-20 14:23:14+00:00)
Title: Added retry logic to error handling for shelled out commands

This is an attempt to work around recent github flakiness. Our automated
rebar builds have been failing quite often due to clones of deps erroring
when github has a flaky episode. We considered hosting our own repos
to workaround the problem but decided teaching rebar to retry on errors
was less work and generally more useful.

I added the option {error_handler, {retry, Fun}} to the list options
rebar_utils.sh/2 understands. The retry function is called with the
command and error, as rebar does with existing error handlers, and
a third argument representing the number of retries attempted. The
retry function must return a boolean indicating whether or not the
failing operation should be retried.

The git cloning logic was then modified to use the retry handler.
Failing clone operations will be retried for a hardcoded maximum of
3 times. rebar will sleep for an increasing interval before each retry:
250ms, 500ms, 750ms. The dep directory containing the failed clone
is deleted before each retry, too.