mozilla/socorro — pull request #1493 by peterbe (2013-09-11 22:42:55+00:00)
Title: fixes bug 914912 - Improve timeouts on crontabber ftpscraper?

@rhelmer @selenamarie r?

The idea here is simply that we re-try 3 times on all HTTPErrors (with error code `>= 500`) and all URLErrors and sleep for 30 seconds in between.

Note that we set the timeout to 60 seconds on urllib2 so if the "ftp site" is severely clogged up, it'll take our cron job a total of 60 + 30 + 60 + 30 + 60 + 30 + 60 (=5m30s) before it just plainly gives up.

If you're wondering why I'm using both HTTPError and URLError is because because HTTPErrors happen when, for example, you have an apache or something that is health but that passes on the request to another app server that is failing or timing out. The URLErrors happen if you can't even reach that Apache server. Ie. the network to the destination is clogged.