amplab/tachyon — pull request #219 by aarondav (2014-06-23 02:29:02+00:00)
Title: Use JSON as the EditLog's serialization format

Using JSON as a serialization format brings a couple great benefits:
- Evolvability of the log format -- JSON allows us to easily add,
remove, or change fields in our serialization format and still
successfully read from old logs. This is infeasible today.
- Human readability of the log [1], enabling easier manual intervention
if something goes wrong during recovery and better error reports.

There are some potential downsides of using JSON over binary:
- Performance overheads of serializing more data and as JSON.
Deserialization costs are less important, as they are only during
recovery.
- Space overheads of human-readable String data overy binary data.

In order to help quantify the cost of JSON, I did a brief performance evaluation between this implementation and the prior binary format. The evaluation included simply continuously writing one type of log message for 10 seconds, then counting (A) how many entries were written and (B) how large the log had grown. Note that a flush() was inserted after every write.

The results are as follows:

**Large Entries [2]**
```
Version | Ops/sec | Bytes/op
JSON | 105,894 | 300
Binary | 5,410 | 155
```

**Small Entries [3]**
```
Version | Ops/sec | Bytes/op
JSON | 174,400 | 100
Binary | 124,267 | 10
```

Somewhat inexplicably, the JSON version always won in speed, perhaps because it buffers each entry before writing it out, whereas the binary format writes every field individually. Regardless of relative difference, the JSON format was able to sustain >100k operations per second, which seems sufficient.

There was certainly significant blowup in the actual log size, though, up to 10x for very small logs. Hopefully this will not be impactful on actual disk usage, though, as we continuously clean up old edit logs as we merge them into the durable image file.

### Notes
[1] Example snippet of new log format:
```json
{"type":"CREATE_FILE","transId":7,"parameters":{"blockSizeByte":1073741824,"directory":false,"recursive":true,"path":"/testfile","creationTimeMs":1403490397675}}
{"type":"ADD_CHECKPOINT","transId":8,"parameters":{"fileId":4,"path":"/home/aaron/tachyon/underfs/tachyon/data/4","length":15133965}}
{"type":"COMPLETE_FILE","transId":9,"parameters":{"fileId":4}}
```

[2] Benchmarked "large entry":

```java
editLog.createDependency(Lists.newArrayList(1,2,3), Lists.newArrayList(4,5,6), "commandPrefix",
Lists.newArrayList(ByteBuffer.wrap(new byte[4]), ByteBuffer.wrap(new byte[8])),
"comment", "framework", "frameworkVersion", DependencyType.getDependencyType(1),
1234, 34932482432L);
```

[3] Benchmarked "small entry":
```java
editLog.setPinned(123, true)
```