neo4j/neo4j — pull request #3181 by chrisvest (2014-09-30 07:36:35+00:00)
Title: [2.2] Fix rare corruption bug in the page cache

Here's the scenario we're now guarding against:

* We want to read a page that is not in the cache, so we do a page fault. This
is fine, the page gets loaded into the cache and added to our translation
table.
* Then it gets evicted and cleaned. This is also fine, though this is
inherently racy because we are taking locks in the wrong order. Because of
the lock-ordering problem, the translation table _may_ not get updated, and
is exactly what happens in this scenario. The page is now unbound, and in
both the translation table and in the freelist. This is fine, though, because
any thread that grabs it will check it with isBoundTo, which will return
false, and then that thread will do a new page fault.
* As part of the eviction, the filePageId field is set to -1, and the swapper
field is set to null. This is what makes the page "unbound."
* Then we want to access the page again.
* We first look in our translation table. We do find the page, but because the
page has been evicted (we can tell by the -1 filePageId and the null
swapper) we know that our translation table is out of date, so we are going
to do a page fault anyway.
* We grab a free page from the freelist, so we have a place to fault the file
page into. By pure chance, _this ends up being the same exact page object!_
* One of the first things that happens when we do a page fault, is that we set
the filePageId and the swapper fields. _These are now set to the same values
they had before the page was evicted!_
* The next thing that happens in a page fault is the IO to read the page
contents from the file into the page in memory. This can fail with various
IO-related errors, and in our scenario _it does!_
* We now have a page that is correctly placed in the translation table, is
bound to the right filePageId and swapper, but _has the wrong contents_
because the page fault ultimately failed.
* The next time we want to access the page, we will find it in the translation
table, see that it is correctly bound, and proceed to read the wrong
contents.

Since eviction will only consider pages that are "loaded" (aka. not already
free), the fix is the following: The isLoaded already only considered whether
the filePageId field was set or not, so it is possible to make a page appear
loaded for the purpose of eviction, while at the same time making the isBountTo
check always fail. Instead of assigning the filePageId and swapper _before_
doing the swapIn IO, we move the assignment of the swapper field down so it
happens _after_ the page has been successfully swapped in, while leaving the
assignment of the filePageId field where it is. This means that a page that
fails in swapIn will appear loaded for the purpose of eviction, while leaving
no room for it to be confused with a bound page.