oaeproject/Hilary — pull request #615 by stuartf (2013-09-10 13:31:44+00:00)
Title: Intermittent test failure: verify leaf delete on a non-leaf node soft deletes the message

```
[0m 1) Messagebox #deleteMessage verify leaf delete on a non-leaf node soft deletes the message:
[0m[31m Uncaught AssertionError: {
"a1": {
"id": "msg-box-test-0.9950088497716933#1374853928293",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928293|",
"body": "A1",
"createdBy": "u:camtest:foo",
"created": "1374853928293",
"level": 0,
"replyTo": null
},
"a2": {
"id": "msg-box-test-0.9950088497716933#1374853928296",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928293#1374853928296|",
"body": "A2",
"createdBy": "u:camtest:foo",
"created": "1374853928296",
"level": 1,
"replyTo": "1374853928293"
},
"a3": {
"id": "msg-box-test-0.9950088497716933#1374853928299",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928296#1374853928299|",
"body": "A3",
"createdBy": "u:camtest:foo",
"created": "1374853928299",
"level": 1,
"replyTo": "1374853928296"
},
"a4": {
"id": "msg-box-test-0.9950088497716933#1374853928301",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928293#1374853928301|",
"body": "A4",
"createdBy": "u:camtest:foo",
"created": "1374853928301",
"level": 1,
"replyTo": "1374853928293"
},
"b1": {
"id": "msg-box-test-0.9950088497716933#1374853928296",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928296|",
"body": "B1",
"createdBy": "u:camtest:foo",
"created": "1374853928296",
"level": 0,
"replyTo": null
},
"c1": {
"id": "msg-box-test-0.9950088497716933#1374853928297",
"messageBoxId": "msg-box-test-0.9950088497716933",
"threadKey": "1374853928297|",
"body": "C1",
"createdBy": "u:camtest:foo",
"created": "1374853928297",
"level": 0,
"replyTo": null
}
}[0m[90m
at setupMessages (/home/travis/build/oaeproject/Hilary/node_modules/oae-messagebox/tests/test-messagebox.js:95:44)
at Function._.each._.forEach (/home/travis/build/oaeproject/Hilary/node_modules/underscore/underscore.js:87:24)
at setupMessages (/home/travis/build/oaeproject/Hilary/node_modules/oae-messagebox/tests/test-messagebox.js:93:35)
at module.exports.createMessage (/home/travis/build/oaeproject/Hilary/node_modules/oae-messagebox/lib/api.js:146:24)
at executeQuery (/home/travis/build/oaeproject/Hilary/node_modules/oae-util/lib/cassandra.js:734:9)
at onReturn (/home/travis/build/oaeproject/Hilary/node_modules/helenus/lib/connection.js:448:7)
at onReturn (/home/travis/build/oaeproject/Hilary/node_modules/helenus/lib/connection.js:379:7)
at exports.Connection.connection.addListener.self.transport.receiver.client._reqs.(anonymous function) (/home/travis/build/oaeproject/Hilary/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:80:11)
at Object.CassandraClient.recv_execute_cql_query (/home/travis/build/oaeproject/Hilary/node_modules/helenus/lib/cassandra/Cassandra.js:8295:12)
at exports.Connection (/home/travis/build/oaeproject/Hilary/node_modules/helenus/node_modules/helenus-thrift/lib/thrift/connection.js:83:37)
```

This happens when multiple comments with the same comment created time get created due to fast creation.

One fix is to include nanoseconds in the created time -- although there will still be a race condition. Another solution is to "lock" a timestamp in Redis for ~1 second on each creation. If the timestamp is already locked by another request, increment the millis by 1 and try again. Continue trying until you get a timestamp lock. That should ensure you never have conflicts.