Stanford-Online/class2go — pull request #2131 by sefk (2013-02-05 09:09:56+00:00)
Title: Add retry logic and broader error handling to the interactive autograder (#1730)

We need to be more robust to various kinds of interactive autograder failures. I try to handle failures more generally. Features.

- **Retry Logic.** when a failure occurs, log and retry. Will retry four times before giving up.

- **Exponential backoff on retries.** Simple backoff with jitter

- **More broad error handling.** We were seeing 500's from exceptions that we weren't catching. Now catching all exceptions in this area and treating equally. This is a bit of a blunt weapon, but tough to be more precise since there are *so many* types of errors that we can see. For example, the most prevalent source of errorrs now, "BadStatusLine" comes from parsing in the HTTP stack, not in connection handling.

- **Error watchwords.** We've seen remote graders fail with successful error codes, but a return value of "Time Out." So we watch for those now with an explicit blacklist. Since false negatives here are scary, we only consider the first ten characters in the response.

A fair number of unit tests written to exercise this code. This will slow down our unit test runs a bit, unfortunately. We could consider excluding these with an attribute later if we want (attr=slow)

You now see stuff like this in the unit test runs:

```
INFO interactive grader "SQL_Grader_schroot" returned in 0:00:00.000218
INFO interactive grader "SQL_Grader_schroot" attempt 1/4, retrying in 1.32 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" attempt 2/4, retrying in 2.43 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" returned in 0:00:00.000220
INFO interactive grader "SQL_Grader_schroot" attempt 1/4, retrying in 1.83 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" attempt 2/4, retrying in 2.45 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" attempt 3/4, retrying in 6.38 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" returned in 0:00:00.000245
INFO interactive grader "SQL_Grader_schroot" attempt 1/4, retrying in 1.52 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" attempt 2/4, retrying in 3.53 sec: HTTP Error 500: Server Error
INFO interactive grader "SQL_Grader_schroot" attempt 3/4, retrying in 6.92 sec: HTTP Error 500: Server Error
ERROR interactive grader "SQL_Grader_schroot" attempt 4/4, giving up: HTTP Error 500: Server Error
```