av9ash/GitBugs huggingface dataset:
Bug: RTE during re-serialization for message filtering during 3.0 -> 4.0 upgrade
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Oct/20 14:58
Resolved: 14/Oct/20 08:59

When trying to upgrade 3.0 to 4.0, we’re often running into a problem, if older node serves as a coordinator:

{code}
 15294 java.lang.RuntimeException: Can not deserialize message org.apache.cassandra.distributed.impl.MessageImpl@4c46aead
  15295         at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:299) ~[dtest-4.0-beta3.jar:?]
  15296         at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessage$7(Instance.java:315) ~[dtest-4.0-beta3.jar:?]
  15297         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_232]
  15298         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
  15299         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232]
  15300         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232]
  15301         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [dtest-4.0-beta3.jar:?]
  15302         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
  15303 Caused by: java.io.EOFException
  15304         at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:180) ~[dtest-4.0-beta3.jar:?]
  15305         at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:68) ~[dtest-4.0-beta3.jar:?]
  15306         at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:243) ~[dtest-4.0-beta3.jar:?]
  15307         at org.apache.cassandra.net.Message$Serializer.deserializeHeaderPost40(Message.java:694) ~[dtest-4.0-beta3.jar:?]
  15308         at org.apache.cassandra.net.Message$Serializer.deserializePost40(Message.java:765) ~[dtest-4.0-beta3.jar:?]
  15309         at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:625) ~[dtest-4.0-beta3.jar:?]
  15310         at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:295) ~[dtest-4.0-beta3.jar:?]
  15311         ... 7 more
{code}
Bug: Fix flaky test test_expiration_overflow_policy_cap - ttl_test.TestTTL
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Oct/20 20:51
Resolved: 16/Oct/20 19:40

https://app.circleci.com/pipelines/github/dcapwell/cassandra/622/workflows/adcd463c-156a-43c7-a9bc-7f3e4938dbe8/jobs/3514

{code}
>           assert warning, 'Log message should be print for CAP and CAP_NOWARN policy'
E           AssertionError: Log message should be print for CAP and CAP_NOWARN policy
E           assert []

ttl_test.py:410: AssertionError
{code}
Bug: Rename master branches to trunk in all repositories
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 09:06
Resolved: 27/Oct/20 10:35

Applies to the following repositories
* cassandra-builds
* cassandra-website
* cassandra-dtest
* cassandra-sidecar
* cassandra-diff
* cassandra-in-jvm-dtest-api
* cassandra-harry

This was discussed in https://lists.apache.org/thread.html/r54db4cd870d2d665060d5fb50d925843be4b4d54dc64f3d21f04c367%40%3Cdev.cassandra.apache.org%3E

The general preference there was trunk over main, so to match the cassandra repository.
Bug: BLOG - Review Changelog 01 post
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Oct/20 10:59
Resolved: 28/Oct/20 12:36

Review 2020-10-28-cass_changelog_1.markdown.

Dev ML thread - https://lists.apache.org/thread.html/rd9b53fa8d3e8c5b3c22032bcd798b4b4e281fd900d4c9e66dfd6f2c6%40%3Cdev.cassandra.apache.org%3E
Bug: ByteBufferAccessor cast exceptions are thrown when trying to query a virtual table
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Oct/20 14:26
Resolved: 12/Oct/20 08:20

Start a fresh trunk node, and try to run

SELECT * FROM system_views.local_read_latency ;

You’ll get: 
{code:java}
ERROR [Native-Transport-Requests-1] 2020-09-30 09:44:45,099 ErrorMessage.java:457 - Unexpected exception during request
 java.lang.ClassCastException: org.apache.cassandra.db.marshal.ByteBufferAccessor cannot be cast to java.lang.String
         at org.apache.cassandra.serializers.AbstractTextSerializer.serialize(AbstractTextSerializer.java:29)
         at org.apache.cassandra.db.marshal.AbstractType.decompose(AbstractType.java:131) {code}
Bug: NPE when calling broadcast address on unintialized node
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Oct/20 13:48
Resolved: 14/Oct/20 08:41

When trying to run upgrades, sometimes we’re calling broadcasts addrerss on an uninitialised new node:

{code}
java.lang.IllegalStateException: Can't use shut down instances, delegate is null
	at org.apache.cassandra.distributed.impl.AbstractCluster$Wrapper.delegate(AbstractCluster.java:163)
	at org.apache.cassandra.distributed.impl.DelegatingInvokableInstance.broadcastAddress(DelegatingInvokableInstance.java:53) 
	at org.apache.cassandra.distributed.impl.Instance$2.allowIncomingMessage(Instance.java:278) 
	at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:1031) ~[dtest-3.0.19.jar:?]
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:213) 
	at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:182) 
	at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93) 
{code}
Bug: SEPExecutor does not release blocked threads as it should
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Oct/20 09:48
Resolved: 10/Nov/20 09:56

While adding some tests for the {{ThreadPoolMetrics}}, I discovered that the {{SEPExecutor}} does not release the blocked threads as it should.

If the number of tasks added to a SEPExecutor exceed the max queue size. The threads adding those task will be block until enough space become available for all the blocked tasks. At this point all the blocked threads will released at once.
Bug: 'SSLEngine closed already' exception on failed outbound connection
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Nov/20 17:09
Resolved: 18/Nov/20 14:18

Occasionally Netty will invoke {{OutboundConnectionInitiator#exceptionCaught()}} handler to process an exception of the following kind:
{code}
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
{code}

When we invoke {{ctx.close()}} later in that method, the listener, set up in {{channelActive()}}, might be
failed with an {{SSLException("SSLEngine closed already”)}} by Netty, and {{exceptionCaught()}} will be invoked
once again, this time to handle the {{SSLException}} triggered by {{ctx.close()}}.

The exception at this stage is benign, and we shouldn't be double-logging the failure to connect.
Bug: Remove duplicate code used by CQLTester and SchemaLoader
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Nov/20 16:36
Resolved: 02/Dec/20 09:57

While digging into the cause behind CASSANDRA-16302, we discovered that there is some duplicate code in {{SchemaLoader}} and {{CQLTester}}. Over time that code has evolved slightly differently in both classes leading to some different behavior. 

Having the same code being used by both class will ensure that if the code needs to be change the behavior will be correct same one for both classes.
Bug: SSTableLoader will fail if encryption parameters are used due to CASSANDRA-16144
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Nov/20 14:35
Resolved: 17/Nov/20 19:15

CASSANDRA-16144 recently introduced [repeated calls |https://github.com/apache/cassandra/compare/trunk...dcapwell:commit_remote_branch/CASSANDRA-16144-trunk-209E2350-3A50-457E-A466-F2661CD0D4D1#diff-b87acacbdc34464d327446f7a7e64718dbf843d70f5fbc9e5ddcd1bafca0f441R478]to _clientEncOptions.applyConfig()_ for each encryption parameter passed to the sstableloader command line.
This consistently fails because _applyConfig()_ can be called only once due to the _ensureConfigNotApplied()_ check at the beginning of the method.

This call is not necessary since the _with...()_ methods will invoke _applyConfig()_ each time:
{code:java}
public EncryptionOptions withTrustStore(String truststore)
{
    return new EncryptionOptions(keystore, keystore_password, truststore, truststore_password, cipher_suites,
                                        protocol, algorithm, store_type, require_client_auth, require_endpoint_verification,
                                        enabled, optional).applyConfig();
}
{code}
I'll build a patch for this with the appropriate unit test.
Bug: Cannot replace_address /X because it doesn't exist in gossip
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Oct/20 19:11
Resolved: 15/Dec/20 22:55

We see this exception around nodes crashing and trying to do a host replacement; this error appears to be correlated around multiple node failures.

A simplified case to trigger this is the following

*) Have a N node cluster
*) Shutdown all N nodes
*) Bring up N-1 nodes (at least 1 seed, else replace seed)
*) Host replace the N-1th node -> this will fail with the above

The reason this happens is that the N-1th node isn’t gossiping anymore, and the existing nodes do not have its details in gossip (but have the details in the peers table), so the host replacement fails as the node isn’t known in gossip.

This affects all versions (tested 3.0 and trunk, assume 2.2 as well)
Bug: upgrade from C* 3.11.9 to 4.0-beta3 fails if 3.11.9 is configured with OldNetworkTopologyStrategy
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Nov/20 03:08
Resolved: 26/Nov/20 13:08

When we upgrade Cassandra from 3.11.9 to 4.0-beta3, if the old cluster (3.11.9) is configured with OldNetworkTopologyStrategy, 4.0-beta3 Cassandra fails to start with the following exception: 

 
{code:java}
ERROR [main] 2020-11-24 22:49:33,423 CassandraDaemon.java:278 - Error while loading schema:
org.apache.cassandra.exceptions.ConfigurationException: Unable to find replication strategy class 'org.apache.cassandra.locator.OldNetworkTopologyStrategy'
        at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:720)
        at org.apache.cassandra.locator.AbstractReplicationStrategy.getClass(AbstractReplicationStrategy.java:422)
        at org.apache.cassandra.schema.ReplicationParams.fromMapWithDefaults(ReplicationParams.java:90)
        at org.apache.cassandra.schema.ReplicationParams.fromMap(ReplicationParams.java:82)
        at org.apache.cassandra.schema.KeyspaceParams.create(KeyspaceParams.java:64)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaceParams(SchemaKeyspace.java:971)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:956)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:949)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:859)
        at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:100)
        at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:89)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.locator.OldNetworkTopologyStrategy
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.cassandra.utils.FBUtilities.classForName(FBUtilities.java:716)
        ... 13 common frames omitted
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unable to find replication strategy class 'org.apache.cassandra.locator.OldNetworkTopologyStrategy'{code}
Bug: Drain and/or shutdown might throw because of slow messaging service shutdown
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Nov/20 16:05
Resolved: 18/Nov/20 14:06

If we invoke nodetool drain before shutdown, it sometimes fails to shut down messaging service in time (in this case - timing out the shutdown of the eventloopgroup by Netty). But, not before we manage to set isShutdown of StorageService to true, despite aborting further drain logic (including shutting down mutation stages).

Then, via on shutdown hook, we invoke drain() method again, implicitly. We see that the mutation stage is not shutdown and proceed to assert that isShutdown == false, failing that assertion and triggering a second error log message.

The patch merely ensures that any exception thrown by MS shutdown is captured so that drain logic can complete the first time around.
Bug: cqlsh row_id resets on page boundaries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Oct/20 00:36
Resolved: 17/Dec/20 17:19

When you run a query such as

{code}
expand on; 
select * from table_with_clustering_keys where token(partition_key) = 1192326969048244361;
{code}

We print out a header for each row that looks like the following

@ Row 1

In 3.0 all values printed were uniq, but in 4.0 they are no longer unique

{code}
$ grep Row 3.0-rows.results | sort | uniq -c | sort -k1 -h -r | head -n 10
  1 @ Row 999
  1 @ Row 998
  1 @ Row 997
  1 @ Row 996
  1 @ Row 995
  1 @ Row 994
  1 @ Row 993
  1 @ Row 992
  1 @ Row 991
  1 @ Row 990
{code}

{code}
$ grep Row 4.0-rows.results | sort | uniq -c | sort -k1 -h -r | head -n 10
 10 @ Row 9
 10 @ Row 8
 10 @ Row 7
 10 @ Row 6
 10 @ Row 5
 10 @ Row 48
 10 @ Row 47
 10 @ Row 46
 10 @ Row 45
 10 @ Row 44
{code}
Bug: Make sure OOM errors are rethrown on truncation failure
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Nov/20 14:22
Resolved: 15/Dec/20 09:57

Seems we don't rethrow OOM errors in [truncation verb handler|https://github.com/apache/cassandra/blob/609876275738589fdfb9a3e20cb2f594aa404037/src/java/org/apache/cassandra/db/TruncateVerbHandler.java#L44]

We also send both success and failure messages if the failure reason is not an FSError
Bug: Circle CI python upgrade test build includes non-upgrade tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Dec/20 22:15
Resolved: 10/Dec/20 21:51

https://app.circleci.com/pipelines/github/dcapwell/cassandra/841/workflows/bcadf6e6-8d04-4010-8a47-99f7f9b5ac1d/jobs/4949/parallel-runs/23?filterBy=ALL

Jenkins uses the following for finding tests

{code}
export RUN_STATIC_UPGRADE_MATRIX=true
./run_dtests.py \
    --execute-upgrade-tests-only \
    --dtest-print-tests-only \
    --dtest-print-tests-output=${WORKSPACE}/test_list.txt \
    --cassandra-dir=$CASSANDRA_DIR
{code}

Where as circle ci uses

{code}
export RUN_STATIC_UPGRADE_MATRIX=true
./run_dtests.py \
    --execute-upgrade-tests \
    --dtest-print-tests-only \
    --dtest-print-tests-output=/tmp/all_dtest_tests_j8_upgradetests_without_vnodes_raw \
    --cassandra-dir=../cassandra
{code}

The difference here is that —execute-upgrade-tests-only will block non-upgrade tests, where —execute-upgrade-tests will include upgrade tests; for this reason non-upgrade tests are scheduled as part of the build.

Tests such as the following were found in the allow list: 
transient_replication_test.py::TestTransientReplicationSpeculativeQueries::test_custom_speculate
ttl_test.py::TestTTL::test_collection_set_ttl
auth_test.py::TestAuth::test_alter_ks_auth

many more were found
Bug: GossipTest hangs until timeout, then fails.
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Nov/20 11:56
Resolved: 06/Nov/20 19:25

A couple of recent updates appear to have broken {{o.a.c.d.t.GossipTest}}

* There seems to have been a merge/commit race between CASSANDRA-16146 ([{{fee7a108}}|https://github.com/apache/cassandra/commit/fee7a10823da1e29bd0e6504fea9679389180c9e]) and CASSANDRA-15935 ([{{41952a2f}}|https://github.com/apache/cassandra/commit/41952a2f73ba5198250f64beba8f7ff1203204ab]). The former adds a ByteBuddy interception on {{StorageService::bootstrap}}, but the latter changed the method signature, so this never actually gets injected. This causes a latch in the test not to be counted down and it hangs until timeout.
* After fixing the test code, it still hangs due to changes to {{server_encryption_options}} initialization in CASSANDRA-16144 ([{{f293376a}}|https://github.com/apache/cassandra/commit/f293376aa8dd315a208ef2f03bdcb7a84dcc675c]). It appears to be causing an incorrect keystore location to be specified, which causes instance startup to fail, again leading to the test hanging until it times out. I don't have the cycles to dig into this further right now, but reverting that commit (and making the test fix above) restores the green bar.
Bug: Fix documentation to reflect real defaults for commit logs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Nov/20 10:11
Resolved: 09/Nov/20 20:55

About the documentation on commit logs:
 * Commit logs default is every 10 sec - *periodic*, not batch [https://github.com/apache/cassandra/blob/cassandra-3.11.8/conf/cassandra.yaml#L385-L386] (same in 3.11.9 or trunk). It can lead to data losses but still makes sense to me because fsync-ing every write is very expensive, and (even in batches) should be reserved for the super paranoid where handling multiple power-out (or sudden disk failures) simultaneously on multiple nodes needs to be handled (as [~mck] explained to me).

  * Nonetheless, apart from the theory behind it, in a more practical perspective, *the doc is wrong* and says *Default Value: batch* which mislead me and this could happen to more people: [https://cassandra.apache.org/doc/latest/architecture/storage_engine.html#commit-log]

 

I think this is important enough for us to quickly fix the documentation in trunk. As this information is fully missing in the 3.11 branch documentation, I suggest we add it as well on that branch (based on information about commit logs from trunk).
Bug: Change Dynamic Snitch Default Badness Threshold to 1.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Nov/20 16:31
Resolved: 19/Nov/20 00:05

With the removal of compaction and IO from the DynamicEndpointSnitch score calculation, the default badness threshold of 10% (0.1) is too small of a margin from experience with production clusters. When compaction and IO values were included, the resulting scores were dominated by them and 10% was a much more noticeable difference. When relying solely on latency, the DynamicEndpointSnitch can rely on nodes that are performing only marginally better than their peers. This results in a lopsided request distribution among the replicas despite similar performance. 

Some graphs are attached from a production cluster showing the read count and latency among the nodes with the default of 0.1 and with the badness threshold set to 1.
Bug: Potential NPE in JVMStabilityInspector
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Nov/20 16:56
Resolved: 23/Nov/20 15:37

On either a FileNotFoundException or SocketException, JVMStabilityInspector checks the error message for the string "Too many open files". However, both of these exceptions have a constructor which sets a null message, which can lead to NPE if handled.
Bug: TableMetrics are exposed before ColumnFamilyStore is fully initialized
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Oct/20 14:46
Resolved: 10/Nov/20 13:06

The {{ColumnFamilyStore}} exposes the {{TableMetrics}} before it is fully initialized, due to that it is possible to perform a call via the metrics that access uninitialized part of the {{ColumnFamilyStore}}. 

The following test can be added to ColumnFamilyMetricTest to show the issue:
{code:java}
@Test
public void testStartupRaceConditionOnMetricListeners()
{
 // Since the ColumnFamilyStore instance reference escapes during the construction
 // we have a race condition and listeners can see an instance that is in an unknown state.
 // This test just check that all callbacks can access the data without throwing any exception.
 registerMetricListener();
 SchemaLoader.createKeyspace("Keyspace2",
 KeyspaceParams.simple(1),
 SchemaLoader.standardCFMD("Keyspace2", "Standard2"));
}

private void registerMetricListener()
{
 CassandraMetricsRegistry.Metrics.addListener(new MetricRegistryListener.Base()
 {
 @Override
 public void onGaugeAdded(String name, Gauge<?> gauge)
 {
 gauge.getValue();
 }

 @Override
 public void onGaugeRemoved(String name)
 {

 }

 @Override
 public void onCounterAdded(String name, Counter counter)
 {
 counter.getCount();
 }

 @Override
 public void onCounterRemoved(String name)
 {

 }

 @Override
 public void onHistogramAdded(String name, Histogram histogram)
 {
 histogram.getCount();
 }

 @Override
 public void onHistogramRemoved(String name)
 {

 }

 @Override
 public void onMeterAdded(String name, Meter meter)
 {
 meter.getCount();
 }

 @Override
 public void onMeterRemoved(String name)
 {

 }

 @Override
 public void onTimerAdded(String name, Timer timer)
 {
 timer.getCount();
 }

 @Override
 public void onTimerRemoved(String name)
 {

 }
 });{code}

While looking into that ticket we also discovered a problem with the used of {{Metered}} in {{CacheMetrics}}.
Metrics reporter looks for metrics classes that are instance of the standard codahale classes. Due to that, other Metered implementations are not be exposed through the reporter. This ticket will also address that issue.
Bug: Add tests to cover ClientRequest metrics 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Oct/20 08:48
Resolved: 11/Nov/20 11:53

We do not have test that covers the ClientRequest metrics.
* ClientRequestMetrics
* CASClientRequestMetrics
* CASClientWriteRequestMetrics
* ClientWriteRequestMetrics
* ViewWriteMetrics
Bug: Lightweight transaction linearizability violations when using list append and prepend operations
Status: Triage Needed
Priority: Normal
Created: 23/Dec/20 12:01

This is an equivalent of [https://github.com/scylladb/scylla/issues/7611]

Cassandra's "list" type is a CRDT which allows a write-only "append" operation, not requiring a read. This is implemented by using the current time on the coordinator node as the key of the new item, so that a second append operation will put the new item after the item from the first append operation.

However, in case of lightweight transactions, uncoordinated use of node-local timestamp may lead to duplicate and non-monotonic timestamp values. 
 This leads to loss of LWT linearizability.
 The issue is reproducible with a Jepsen test for list append, prepend in presence of time and topology nemeses.

Relevant links: [https://github.com/scylladb/scylla/issues/7116]

{{Code generating an independent timestamp is in [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/Lists.java#L499]}}

{\{ ByteBuffer uuid = ByteBuffer.wrap(UUIDGen.getTimeUUIDBytes());}}
 \{{ params.addCell(column, CellPath.create(uuid), buffer);}}

 
A related problem is that this approach ignores user-supplied statement timestamps.