zopefoundation/ZODB — pull request #15 by vpelletier (2013-10-25 13:26:34+00:00)
Title: Stop looking for revision when past it.

This fix comes from the following ZEO log error:

```
2013-10-25 09:05:38,248 ERROR ZODB.ConflictResolution Unexpected error
Traceback (most recent call last):
File "ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/ConflictResolution.py", line 243, in tryToResolveConflict
oldData = self.loadSerial(oid, oldSerial)
File "ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 434, in loadSerial
raise POSKeyError(oid)
POSKeyError: 0x803e9f
```

This means that conflict resolution was started, but for some reason oldSerial does not exist in ZODB for given object. This object has a *lot* of old versions, so although each revision is small a whole cluster was brought to a halt for 15 minutes, with disk loading data at 10MB/s (very probably limited by IOop/s), walking back this object's history while holding the commit lock - hence blocking all clients.

I believe this patch would have saved us from the downtime (I cannot be sure, as the looked up serial isn't logged - any idea on how to log this ?), and generally makes sense.

Then, any idea on why a conflict was resolved based on a revision which does not exist in ZODB (or more precisely "could not be found", although I trust revision-walking code enough to make both equivalent until proven guilty) ?
Looking deeper, we found another occurrence of this error, where it happened in a row for 5 consecutive commit-pending (as in "queue lock: transactions waiting: 11" with the 5 first unlocked transactions failing with the same error). Our clients have at most 2 threads, so I take this as a hint that whatever is causing this can cross process boundaries. We are checking more, and will update this pull request if we find more.

Lastly, I haven't signed the Zope contributor agreement, but feel free to take this patch as a trivial bug fix (which, IIRC, is exempted from agreement).