ArchiveTeam/formspring-grab — pull request #1 by nsapa (2013-03-26 21:59:08+00:00)
Title: Back off for 30 seconds

timbart of Formsping asks us to reduce the current speed of crawling because their servers are overloaded.
They will send us 500 errors and we will wait for 30 seconds when we receive them.

2:45 timbart> I'd really appreciate if you could throttle your scraper.
22:45 timbart> I write backend services
22:46 timbart> or add a header and we'll throttle on our side
22:46 @chronomex> sure, I'll diddle the numbers on my end
22:46 @chronomex> what would be reasonable?
22:46 nico_32> timbart: send 500 when you are overloaded and the scraper stop
for 30s
22:46 @chronomex> yeah, that's an easy start
22:47 @chronomex> we backoff and retry on 500-series errors
22:47 @chronomex> iirc
22:47 timbart> a 403 would be better
22:47 @chronomex> 403 goes in the permanent archive presently, we can change
that but it's not instant
22:47 @chronomex> again iirc
22:48 timbart> let's stick with 500 then
22:48 timbart> if you could scrape at half the speed I'd be very grateful