Repository: DMOJ/online-judge

Issue #376: RabbitMQ message routing
Created: 2016-01-10T02:26:58Z
The current implementation of #183 relies on DMOJ/judge-server#80 to reject unwanted submissions. Should there be no judges to handle the submission, the same submission will be rejected, requeued, and redelivered, only to be rejected again and again.
To resolve this issue, we must use a headers exchange, so that messages are routed based on available language and problem. However, RabbitMQ exchanges rejects all unroutable messages.
The only solution that seems practical is periodically dispatch all QU messages again. However, this suffers from a pretty big flaw in itself, as there is the potential for a race condition.
There is also an issue of backwards compatibility, such massive change without any trace of compatibility will result in significant downtime during implementation. A few hours of downtime (warning: this is the expected time. In reality, it may be days.) for a bug that most people probably won't even notice is not practical.
The suggested solution, hence, is using a fanout exchange to dispatch messages to the old submission queue, and to a headers exchange, which would dispatch to new clients. This headers exchange is allowed to send messages to /dev/null. There will be an additional function added to amqpd that receives submissions from the old submission queue, and schedule them to be processed a minute later. If the submission status is still QU, schedule it for rejudging. This would solve the judging race while also maintaining compatibility, as old judges will still receive submissions normally, perhaps with a minute delay, but it would still work.
However, this solution suffers the ABA problem. To prevent ABA, we will add a last_judge datetime field, serving as the judge attempt identification. Hence, we simply filter those submissions that are still queued after a minute, resulting in a maximum two minute wait before a rejudge is attempted.
Now we shall look back to the fanout solution, and realize that ultimately, the old queue is not necessary when MySQL can store that state, though the old queue can be used to minimize the wait time to almost always one minute. Hence, the fanout is ultimately unnecessary. (It is worth noting that backwards compatibility is an excuse to make that option look more attractive, but since there is a better solution, why bother?)
Despite this, attempts to rejudge old submissions may still cause chaos. To resolve this, we shall endeavour to only rejudge the submissions that could actually be rejudged, given the judges currently running, and fail other attempts. Since the admin already counts the amount of successes and failures in rejudging, this is an acceptable solution. This will result in an extra query per rejudge, but since the rejudge operation already runs multiple queries, it's not much worse.
In conclusion, we shall use a headers exchange and a cron to resolve this horrible issue with repetitive rejection.
/cc @Xyene
P.S. It is interesting to note that attempting to write an essay is a valid method of coming up with a solution, as an acceptable alternative to an intelligent conversation.

Duplicate issue #375: Hang on rejudging submission with no testdata
Created: 2016-01-10T00:29:13Z
AMQP just cycles through judges that are incapable of grading.

Closing comment (2016-01-10T02:27:40Z): I am tempted to mark this as a duplicate of #376.