Project: Longhorn (Runtime / Cloud Native Storage)
File: 20200701-backupstore-file-locks.md

This enhancement will address backup issues that are the result of concurrently running backup operations, by implementing a synchronisation solution that utilizes files on the backup store as Locks. https://github.com/longhorn/longhorn/issues/612 https://github.com/longhorn/longhorn/issues/1393 https://github.com/longhorn/longhorn/issues/1392 https://github.com/longhorn/backupstore/pull/37 Identify and prevent backup issues caused as a result of concurrent backup operations. Since it should be safe to do backup creation & backup restoration at the same time, we should allow these concurrent operations. The idea is to implement a locking mechanism that utilizes the backupstore, to prevent the following dangerous cases of concurrent operations. prevent backup deletion during backup restoration prevent backup deletion while a backup is in progress prevent backup creation during backup deletion prevent backup restoration during backup deletion The locking solution shouldn't unnecessary block operations, so the following cases should be allowed. allow backup creation during restoration allow backup restoration during creation The locking solution should have a maximum wait time for lock acquisition, which will fail the backup operation so that the user does not have to wait forever. The locking solution should be self expiring, so that when a process dies unexpectedly, future processes are able to acquire the lock. The locking solution should guarantee that only a single type of lock is active at a time. The locking solution should allow a lock to be passed down into async running go routines. Before this enhancement, it is possible to delete a backup while a backup restoration is in progress. This would lead to an unhealthy restoration volume. After this enhancement, a backup deletion could only happen after the restoration has been completed. This way the backupstore continues to contain all the necessary blocks that are required for the restoration. After this enhancement, creation & restoration operations are mutually exclusive with backup deletion operations. Conceptually the lock can be thought of as a RW lock, it includes a `Type` specifier where different types are mutually exclusive. To allow the lock to be passed into async running go routines, we add a `count` field, that keeps track of the current references to this lock. ```go type FileLock struct { Name string Type LockType Acquired bool driver BackupStoreDriver volume string count int32 serverTime time.Time refreshTimer *time.Ticker } ``` To make the lock self expiring, we rely on `serverTime` updates which needs to be refreshed by a timer. We chose a `LOCKREFRESHINTERVAL` of 60 seconds, each refresh cycle a locks `serverTime` will be updated. A lock is considered expired once the current time is after a locks `serverTime` + `LOCKMAXWAIT_TIME` of 150 seconds. Once a lock is expired any currently active attempts to acquire that lock will

```go const ( LOCKS_DIRECTORY = "locks" LOCK_PREFIX = "lock" LOCK_SUFFIX = ".lck" LOCKREFRESHINTERVAL = time.Second * 60 LOCKMAXWAIT_TIME = time.Second * 150 LOCKCHECKINTERVAL = time.Second * 10 LOCKCHECKWAIT_TIME = time.Second * 2 ) ``` Lock Usage create a new lock instance via `lock := lock.New()` call `lock.Lock()` which will block till the lock has been acquired and increment the lock reference count. defer `lock.Unlock()` which will decrement the lock reference count and remove the lock once unreferenced. To make sure the locks are mutually exclusive, we use the following process to acquire a lock. create a lock file on the backupstore with a unique `Name`. retrieve all lock files from the backupstore order them by `Acquired` then by `serverTime` followed by `Name` check if we can acquire the lock, we can only acquire if there is no unexpired(i) lock of a different type(ii) that has priority(iii). Locks are self expiring, once the current time is after `lock.serverTime + LOCKMAXWAIT_TIME` we no longer need to consider this lock as valid. Backup & Restore Locks are mapped to compatible types while Delete Locks are mapped to a different type to be mutually exclusive with the others. Priority is based on the comparison order, where locks are compared by `lock.Acquired` then by `lock.serverTime` followed by `lock.Name`. Where acquired locks are always sorted before non acquired locks. if lock acquisition times out, return err which will fail the backup operation. once the lock is acquired, continuously refresh the lock (updates `lock.serverTime`) once the lock is acquired, it can be passed around by calling `lock.Lock()` once the lock is no longer referenced, it will be removed from the backupstore. It's very unlikely to run into lock collisions, since we use uniquely generated name for the lock filename. In cases where two locks have the same `lock.serverTime`, we can rely on the `lock.Name` as a differentiator between 2 locks. A number of integration tests will need to be added for the `longhorn-engine` in order to test the changes in this proposal: place an expired lock file into a backupstore, then verify that a new lock can be acquired. place an active lock file of Type `Delete` into a backupstore, then verify that backup/restore operations will trigger lock acquisition timeout. place an active lock file of Type `Delete` into a backupstore, then verify that a new `Delete` operation can acquire a lock. place an active lock file of Type `Backup/Restore` into a backupstore, then verify that delete operations will trigger lock acquisition timeout. place an active lock file of Type `Backup/Restore` into a backupstore, then verify that a new `Backup/Restore` operation can acquire a lock. No special upgrade strategy is necessary.