Fix to the potential race issues among rollback, transaction ,and compaction.

There are potential race conditions among rollback, transaction, and
compaction tasks. Through the lock on a file instance, this change makes
sure that transaction or compaction should not be scheduled if a rollback
operation is being executed, and vice-versa.

Change-Id: Ic27b6f5e0d92c7d5053481d50bc9c2fe61ecdb86