neo4j/neo4j — pull request #2607 by chrisvest (2014-06-25 14:06:27+00:00)
Title: Fix problem that could lead to missing index constraints in HA

Here's the thing: When we insert a new record into a store, we do the write immediately to grab
the id associated with that particular location in the file. When we update a record, we keep
the change around in memory until the stores are flushed, which only happens on log rotation.

When we create a new constraint, we first create the associated constraint index, and start off
the index population job. At this point, the owner field for this constraint index is empty.
When the population completes and our transaction commits, we then create the related
constraint rule, and update the index constraint record such that its owner fields references
this newly created constraint rule. This is of course all written to the transaction log, and
applied to the store files with as little actual IO as possible. Like I said above, we only
flush the store files when we rotate the log.

Because we want really fast access to the small bits of information that is our schema store,
we load up and cache the entire schema store when the database starts up. This schema cache is
then kept in sync with the schema store, when schema-mutating transactions are committed and
applied.

When we start up, we do recovery (if needed) and then we do our post-recovery. In our post-
recovery, we seek out all constraint indexes that have no owning constraint, and then we drop
them. We do this because we might have crashed in between creating the index, and creating
and committing the associated constraint. If that happens, then the constraint ended up never
being committed, and so we don't need the index that was created for it.

When we do a mode switch, from leader to follower or follower to leader, we *partially* restart
the database. This means that we don't do any recovery, and we don't attempt to rotate the log.
We do still do our post-recovery, however, and this is look through all our constraint rules,
and drop indexes that don't have any owning constraints.

Now, it somehow happened, that we occasionally had schema-mutations applied to our schema store,
but not to the schema cache. Specifically, the schema cache would containt constraint indexes
that had an empty owner field, while the store had been correctly updated. The new leader
instance of the cluster would then inject a transaction that dropped all the constraint indexes,
and push that transaction out to the followers.

The constraint would then be rendered broken. So broken, in fact, that it couldn't even be
dropped, since dropping a constraint also implies dropping its index, but the index can no
longer be read from the schema store because it has been deleted.

This commit fixes this problem, by reloading the schema cache when we have a
follower-to-leader mode switch.