Add retry logic to applier (#16)

Also contains a test cleanup fix