av9ash/GitBugs huggingface dataset:
Bug: Bump snakeyaml from 1.26 to 2.0
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 17/Jul/23 14:53
Resolved: 17/Jul/23 15:07

snakeyaml 1.26 has CVEs. Bump version for snakeyaml from 1.26 to 2.0

To see the CVEs, goto [https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/4.1.0] and seach for [org.yaml|https://mvnrepository.com/artifact/org.yaml] » [snakeyaml|https://mvnrepository.com/artifact/org.yaml/snakeyaml] under compile dependencies.Vulnerabilites are listed thusly:

Direct vulnerabilities:
[CVE-2022-41854|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-41854]
[CVE-2022-38752|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-38752]
[CVE-2022-38751|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-38751]
[View 4 more ...|https://mvnrepository.com/artifact/org.yaml/snakeyaml/1.26#]
Vulnerabilities from dependencies:
[CVE-2022-22971|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-22971]
[CVE-2022-22970|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-22970]
[CVE-2022-22968|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-22968]

GitHub Issue:

https://github.com/apache/cassandra/pull/2455
Bug: Generate / add Gradle project to be imported by IDE instead of generating IDE specific project files
Status: Triage Needed
Priority: Normal
Created: 14/Jul/23 05:23

Many IDEs (including IntelliJ) support importing a project from the Gradle script. Having such a definition, we would not have to generate IDE project files after changing a branch or setting up a new work tree.


It is handy for working on the Cassandra project in IntelliJ because it allows running tests with multiple flavors - that is, currently, when we run a test, the default configuration is used. However, in real life, we use a different configuration for regular unit tests, compression, CDC, oa, etc. I found no way to configure that directly in IntelliJ, but when you use a Gradle project, it is possible to configure multiple test tasks and select which one you want to run - see the attached screenshot.
 

!image-2023-07-14-07-19-57-787.png!
Bug: When starting up after a failed repair was detected Cassandra should clean up automatically
Status: Triage Needed
Priority: Normal
Created: 05/Jul/23 15:11

When a repair fails and is not restarted using "resume" the operator has to manually clear out the data directory before restarting. Instead Cassandra could detect this case (or does detect this case) and can clean up itself. This removes one error prone step for the operator or control plane.
Bug: Docs : 3.11  Versioned Docs list connectedNativeClientsByUser and connections as an available metric
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 25/Jul/23 08:26
Resolved: 02/Jan/25 15:58

https://issues.apache.org/jira/browse/CASSANDRA-13665 introduced 2 new metrics on the client metrics, connections and connectedNativeClientsByUser - the 3.11 docs list these metrics as being available when they are not.
Bug: rename the java doc about FlushLargestColumnFamily to flushLargestMemtable
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 29/Aug/23 03:48
Resolved: 02/Jan/25 15:50

_*FlushLargestColumnFamily*_ did not exist anymore. The java doc should be improved like this:
{code:java}
/**
 * A thread that reclaims memory from a MemtablePool on demand.  The actual reclaiming work is delegated to the
 * cleaner Runnable, e.g., MemtableCleaner {@link AbstractAllocatorMemtable#flushLargestMemtable()}.
 */
public class MemtableCleanerThread<P extends MemtablePool> implements Interruptible
{ {code}
Bug: Modernize CQLSH datetime conversions
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 25/Sep/23 11:26
Resolved: 06/Mar/24 15:48

Python 3.x introduced many updates to datetime conversion which allows simplified conversions.

1. For example, tracing.py defines a function datetime_from_utc_to_local() but datetime now has a native function astimezone() which will convert UTC to local time.

Review the following users of datetime which apply conversions:
 * cqlshmain.py
 * formatting.py 
 * tracing.py

Example: 
{code:java}
>>> from dateutil import tz
>>> import datetime
>>> a = datetime.datetime.now().astimezone(tz.tzutc())
>>> a
datetime.datetime(2023, 9, 25, 11, 22, 36, 251705, tzinfo=tzutc())
>>> b = a.astimezone()
>>> b
datetime.datetime(2023, 9, 25, 14, 22, 36, 251705, tzinfo=datetime.timezone(datetime.timedelta(seconds=10800), 'EST')) {code}
See [[PEP 495|http://example.com]]]
Bug: cassandra-stress in simplenative mode and prepared fails with exceptions
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 11/Aug/23 03:40
Resolved: 29/Nov/23 17:18

% cassandra-stress write n=10 -mode simplenative cql3 prepared

...

ap'; org.apache.cassandra.transport.messages.ResultMessage$Prepared is in unnamed module of loader 'app')

java.lang.ClassCastException: class [B cannot be cast to class org.apache.cassandra.transport.messages.ResultMessage$Prepared ([B is in module java.base of loader 'bootstrap'; org.apache.cassandra.transport.messages.ResultMessage$Prepared is in unnamed module of loader 'app')

java.io.IOException: Operation x10 on key(s) [4e334f364c4c4b373530]: Error executing: (ClassCastException): class [B cannot be cast to class org.apache.cassandra.transport.messages.ResultMessage$Prepared ([B is in module java.base of loader 'bootstrap'; org.apache.cassandra.transport.messages.ResultMessage$Prepared is in unnamed module of loader 'app')

 

 at org.apache.cassandra.stress.Operation.error(Operation.java:127)

 at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:242)

 at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:471)

java.io.IOException: Operation x10 on key(s) [373038504b3436363830]: Error executing: (ClassCastException): class [B cannot be cast to class org.apache.cassandra.transport.messages.ResultMessage$Prepared ([B is in module java.base of loader 'bootstrap'; org.apache.cassandra.transport.messages.ResultMessage$Prepared is in unnamed module of loader 'app')

 

 at org.apache.cassandra.stress.Operation.error(Operation.java:127)

 at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)

 at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:242)

 at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:471)

java.lang.RuntimeException: Failed to execute warmup

 at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:128)

 at org.apache.cassandra.stress.StressAction.run(StressAction.java:71)

 at org.apache.cassandra.stress.Stress.run(Stress.java:101)

 at org.apache.cassandra.stress.Stress.main(Stress.java:54)
Bug: GCInspector "Error accessing field of java.nio.Bits" under java17
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 13/Sep/23 13:47
Resolved: 14/Sep/23 17:33

Running under java17, {{GCInspector}} throws the following exception:
{noformat}
DEBUG [main] 2023-09-13 09:35:28,031 GCInspector.java:85 - Error accessing field of java.nio.Bits
java.lang.reflect.InaccessibleObjectException: Unable to make field private static final java.util.concurrent.atomic.AtomicLong java.nio.Bits.TOTAL_CAPACITY accessible: module java.base does not "opens java.nio" to unnamed module @503d687a
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
        at org.apache.cassandra.service.GCInspector.<clinit>(GCInspector.java:80)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:328)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:729)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:853)
INFO  [main] 2023-09-13 09:35:28,039 PaxosUncommittedTracker.java:236 - enabling PaxosUncommittedTracker {noformat}
This is because {{GCInspector}} uses reflection to read the {{TOTAL_CAPACITY}} from {{{}java.nio.Bits{}}}. Access was restricted somewhere between 11 and 17.

Note: this is a rather harmless error, as we only look at {{Bits.totalCapacity}} for metrics collection on how much direct memory is being used by \{{ByteBuffer}}s. If we fail to read the field, we simply return -1 for the metric value.
Bug: Replace obsolete Python functions
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 22/Aug/23 18:31
Resolved: 01/Sep/23 19:48

# Python's stdlib datetime.timezone.utc can replace the custom UTC timezone in pylib/cqlshlib/util.py.
 # Removed unused attributes and functions:
 ## field_size _limit 
 ## trim_if_present()
 ## maybe_ensure_text()
 # Simplify cql_version handling into a settor

Functionality should remain the same while simplifying the code by eliminating ~60 lines.
Bug: Update the How to Commit page after CASSANDRA-18618
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 09/Aug/23 19:08
Resolved: 13/Aug/23 20:44

In CASSANDRA-18618, checks like rat and checkstyle were moved to a separate ant task - "check".

We need to update our How to commit page that on branches Cassandra-5.0 + we need to run now ant realclean && ant jar build-test check pre-commit. While on that, we already have the cassandra-5.0 branch, so that we can add that too.
Bug: Move deb/rpm repositories from dist/downloads .a.o to apache.jfrog.io
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 12/Jul/22 10:38
Resolved: 16/Aug/22 20:03

Move our official debian and redhat repositories from downloads.apache.org to Apache's JFrog Artifactory server at apache.jfrog.io 

That is, the following URLs would be moved from
```
https://downloads.apache.org/cassandra/debian/
https://downloads.apache.org/cassandra/redhat/
```
to
```
https://apache.jfrog.io/artifactory/cassandra-deb/
https://apache.jfrog.io/artifactory/cassandra-rpm/
```

The rationale to do this is to avoid the strict opinionated checksum and signature requirements on downloads.a.o (dist.a.o), as the debian and redhat repositories have their own system for integrity and signing (which we already do).

Furthermore, as these repositories and their binaries are "convenience binaries" and not the official Cassandra source binaries, they do not need to be on downloads.a.o and can be served from apache.jfrog.io. This is similar to maven binaries (and docker images). Apache Arrow is already taking this approach: https://arrow.apache.org/install/ 

An advantage to using apache.frog.io is that these repositories maintain all past patch versions on each repo series (major/minor). This has been requested by users a number of times, for the sake of rolling back to a previous patch version. downloads.a.o can only contain the latest version.

This will BREAK everyone's existing `/etc/apt/sources.list.d/cassandra.sources.list` and `/etc/yum.repos.d/cassandra.repo` files. Folk will need to update these files to point to the new repo URLs. This would require an announcement to both users@ and dev@. I do not know how we can avoid this breakage. We could put in a simple README.md in the original URL locations explaining the breakage and how to fix.
Bug: Flaky org.apache.cassandra.tools TopPartitionsTest testServiceTopPartitionsSingleTable
Status: In Progress
Priority: Normal
Created: 03/Aug/22 21:39

h3.  
{code:java}
Error Message
If this failed you probably have to raise the beginLocalSampling duration expected:<1> but was:<0>

Stacktrace
junit.framework.AssertionFailedError: If this failed you probably have to raise the beginLocalSampling duration expected:<1> but was:<0> at org.apache.cassandra.tools.TopPartitionsTest.testServiceTopPartitionsSingleTable(TopPartitionsTest.java:83) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Standard Output
INFO [main] 2022-08-02 01:49:49,333 YamlConfigurationLoader.java:104 - Configuration location: file:////home/cassandra/cassandra/build/test/cassandra.cdc.yaml DEBUG [main] 2022-08-02 01:49:49,339 YamlConfigurationLoader.java:124 - Loading settings from file:////home/cassandra/cassandra/build/test/cassandra.cdc.yaml INFO [main] 2022-08-02 01:49:49,642 Config.java:1167 - Node configuration:[allocate_tokens_for_keyspace=null; allocate_tokens_for_local_replication_factor=null; allow_extra_insecure ...[truncated 50809 chars]... lizing counter cache with capacity of 2 MiBs INFO [MemtableFlushWriter:1] 2022-08-02 01:49:53,519 CacheService.java:163 - Scheduling counter cache save to every 7200 seconds (going to save all keys). DEBUG [MemtableFlushWriter:1] 2022-08-02 01:49:53,575 ColumnFamilyStore.java:1330 - Flushed to [BigTableReader(path='/home/cassandra/cassandra/build/test/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/nb-1-big-Data.db')] (1 sstables, 4.915KiB), biggest 4.915KiB, smallest 4.915KiB
{code}
Bug: DOC - Fix typo in UNLOGGED BATCH section of CQL DML page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Aug/22 15:24
Resolved: 02/Jan/25 15:32

Typo in https://cassandra.apache.org/doc/trunk/cassandra/cql/dml.html#unlogged-batches

bq. If the {{UNLOGGED}} option is used, a failed batch might leave the {color:#DE350B}patch{color} only partly applied.

should be:

bq. If the {{UNLOGGED}} option is used, a failed batch might leave the {color:#00875A}batch{color} only partly applied.
Bug: Rate-limit new client connection setup to avoid overwhelming bcrypt
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Aug/22 17:31
Resolved: 15/Aug/22 14:33

A flood of reconnects can cause a ton of pain at the bcrypt phase of validating incoming connections. While this shouldn't happen during normal operations, we need a rate limit server side - if there's a bad client out there (version and/or configuration) that misbehaves, a way to cap the pain on a server is quite useful. Right now the only option is to cap the total connections which has other issues that aren't an ideal tradeoff (inability to connect, etc).

Moving authentication requests to a small, separate pool will prevent starvation handling all other requests. If the authExecutor pool backs up it may cause authentication timeouts, but the clients should back off and retry while the rest of the system continues to make progress.
Bug: Remediate CVE-2022-25857 - org.yaml_snakeyaml version 1.26 has vulnerabilities
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Sep/22 06:48
Resolved: 21/Sep/22 10:58

|org.yaml_snakeyaml|[CVE-2022-25857|https://nvd.nist.gov/vuln/detail/CVE-2022-25857]|Fixed in: 1.31
21 days ago| |6|Impacted versions: <1.31
Discovered: a day ago
Published: 21 days ago
The package org.yaml:snakeyaml from 0 and before 1.31 are vulnerable to Denial of Service (DoS) due missing to nested depth limitation for collections.|
Bug: LWT IF col = NULL is inconsistent with SQL NULL
Status: Open
Priority: Normal
Created: 20/Jul/22 12:09

In SQL, any comparison with NULL is NULL, which is interpreted as FALSE in a condition. To test for NULLness, you use IS NULL or IS NOT NULL.

But LWT uses IF col = NULL as a NULLness test. This is likely to confuse people coming from SQL and hamper attempts to extend the dialect.
Bug: WEBSITE - Replace Plausible with Matomo (website analytics)
Status: Open
Priority: Normal
Created: 10/Aug/22 09:46

Our hosted Plausible can and should be replaced with ASF's hosted Matomo.

See https://privacy.apache.org/
Specifically
- https://privacy.apache.org/policies/privacy-policy-public.html
- https://privacy.apache.org/faq/committers.html
- https://privacy.apache.org/matomo/
Bug: Add Cassandra to endoflife.date
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jul/22 12:19
Resolved: 11/Jun/24 19:39

A few Apache projects are featured on [https://endoflife.date/] which is a community-sourced tracker of end-of-life dates and support timelines:
 * [zookeeper|https://endoflife.date/zookeeper]
 * [solr|https://endoflife.date/solr]

Instructions on how to add a new project can be found on [https://endoflife.date/contribute#new-adding-a-new-product] .

Cassandra EOL info can be found on [https://cassandra.apache.org/_/download.html,] and possibly on dev mailing list threads.
Bug: Test failure: org.apache.cassandra.distributed.test.SchemaTest.schemaReset
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/22 11:35
Resolved: 16/Sep/22 08:04

The test {{{}org.apache.cassandra.distributed.test.SchemaTest.schemaReset{}}}, recently introduced by CASSANDRA-17658, is flaky on 4.1 and trunk:
 * 4.1: [https://ci-cassandra.apache.org/job/Cassandra-4.1/134/testReport/org.apache.cassandra.distributed.test/SchemaTest/schemaReset_2/]
 * trunk: [https://ci-cassandra.apache.org/job/Cassandra-trunk/1265/testReport/org.apache.cassandra.distributed.test/SchemaTest/schemaReset_2/]

{code:java}
Error Message
Condition with lambda expression in org.apache.cassandra.distributed.test.SchemaTest that uses org.apache.cassandra.distributed.Cluster was not fulfilled within 1 minutes.
Stacktrace
org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in org.apache.cassandra.distributed.test.SchemaTest that uses org.apache.cassandra.distributed.Cluster was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:165)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:895)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:864)
	at org.apache.cassandra.distributed.test.SchemaTest.schemaReset(SchemaTest.java:115)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Standard Output
INFO  [main] <main> 2022-08-15 15:02:14,783 Reflections.java:219 - Reflections took 1873 ms to scan 8 urls, producing 1754 keys and 6912 values
INFO  [main] <main> 2022-08-15 15:02:16,407 Reflections.java:219 - Reflections took 1561 ms to scan 8 urls, producing 1754 keys and 6912 values
Node id topology:
node 1: dc = datacenter0, rack = rack0
node 2: dc = datacenter0, rack = rack0
Configured node count: 2, nodeIdTopology size: 2
DEBUG [main] node1 2022-08-15 15:02:17,554 InternalLoggerFactory.ja
...[truncated 1761288 chars]...
cutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
INFO  [node2_isolatedExecutor:3] node2 2022-08-15 15:03:52,096 MessagingService.java:519 - Waiting for messaging service to quiesce
{code}
Bug: Optionally avoid hint transfer during decommission
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Aug/22 16:20
Resolved: 08/Sep/22 21:39

Both because they aren’t strictly necessary to maintain consistency, and because throttling induced by their rate-limiter (see {{hinted_handoff_throttle}}) may stall progress, transferring hints during decommission (specifically unbootstrap) rather than just pausing, disabling, and truncating them probably doesn’t make sense. The only other concern would be the BatchLog, which nominally depends on hint delivery to maintain its "guarantees". However, during BatchLog replay on unbootstrap, {{ReplayingBatch}} ignores batches older the gcgs anyway.

Here's a proposal from [~aleksey] that might strike a reasonable balance:

1.) We continue to transfer hints by default during decommission, but at a higher rate. We could, for instance, stop having {{DispatchHintsTask}} divide its effective rate by the number of nodes in the cluster.

{noformat}
int nodesCount = Math.max(1, StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
double throttleInBytes = DatabaseDescriptor.getHintedHandoffThrottleInKiB() * 1024.0 / nodesCount;
this.rateLimiter = RateLimiter.create(throttleInBytes == 0 ? Double.MAX_VALUE : throttleInBytes);
{noformat}

2.) We provide an option to simply avoid transferring hints during unbootstrap. Even this would only take the BatchLog from "best effort" to "slightly less effort" ;)
Bug: Add interim prioritization logic for CompactionTasks
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Aug/22 19:18
Resolved: 07/Sep/22 16:54

While ultimately we could serve to have a real scheduler and single entry point for compactions, for now some moderate progress seems in order. There are cases where, during expansions, you can end up in a state where you can't finish upgrading sstables as they keep getting aborted by Incremental Repair and other operations that ultimately should be lower priority and not have the ability to cancel these operations.

 The idea here is that tasks like {{upgradestables}} and {{cleanup}} (and other tasks that have the same priority) will get serialised since {{markAllCompacting}} is synchronising on data, and higher priority tasks will be able to still cancel others via {{runWithCompactionsDisabled}}.
Bug: Full repair on a keyspace with a large amount of tables causes OOM
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 29/Jul/22 20:47
Resolved: 08/Feb/24 16:02

Running a nodetool repair -pr --full on a keyspace with a few hundred tables will cause a direct memory OOM, or lots of heap pressure with use_offheap_merkle_trees: false.  Adjusting repair_session_space_in_mb does not seem to help.  From an initial look at a heap dump, it appears to node is holding many _remote_ trees in memory.
Bug: Secondary index building should not block compaction progress
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jul/22 16:55
Resolved: 03/Aug/22 18:54

While CASSANDRA-11218 should also address this same problem, that ticket's a pretty big beautiful hammer and there's a fairly surgical modification we can make to the 2i building process to mitigate 2i building's ability to block compaction.

While 11218 approaches this from a "lots of compactions can starve 2i build and things never finish", what I'd like to address here is the inverse that building multiple secondary indices in parallel can occupy all available compactors to a point where we don't have any left to do actual compactions. Completing 2i build can take several hours and the backlog of compactions from that time may result in too many uncompacted sstables left to catch up.

A fairly simple fix is to break 2i building out to its own threadpool w/a hot proppable param you can use to change the # of threads for that build in the extreme cases where the default of 2 is indufficient.