raft: get rid of the using of defer in critical path

Or just go back to defer if that's not actually allocating.