ArchiveTeam/hyves-grab — pull request #2 by yipdw (2013-11-20 14:20:18+00:00)
Title: Move wait time to httploop_result.

This PR moves the wait determination (and execution) to `httploop_result`.

httploop_result is the Lua hook that's guaranteed to be called after every request (https://github.com/alard/wget-lua/blob/lua/src/http.c#L3099); download_child_p isn't (see https://github.com/alard/wget-lua/blob/lua/src/recur.c#L406-L427). This means that you can run into behavior like this:

1. On a non-leaf node (i.e. HTML page with links), wait some amount of time after looking at non-image links.
2. On a set of N leaf nodes (e.g. pages with no unconsidered links or images), fetch them all without waiting.

Step 2 is, in my experience, prone to triggering Hyves rate-limits. Moving the wait to an after-fetch hook means that the wait is no longer dependent on whether we do a leaf or non-leaf request.

I've been running this code for a few hours on nodes that previously got stuck in a rate-limit cycle. They still occasionally get rate-limited, but now recover much faster.