Persistent key-value based distributed storage system with failure handling