Project: CockroachDB (App Definition and Development / Database)
File: cluster-observability.md

Last update: May 2023 This document provides an architectural overview of the observability layer in CockroachDB. Original author: j82w Table of contents: UI will automatically call CockroachDB endpoint when users went to the page. The gateway node sends the query to the system statistics table and in memory crdb_internal table. The gateway node joins all the results and return all of them to frontend. Frontend would auto-refresh the results, and do all the sorting in the browser. SQL Activity page now display only persisted stats when selecting to view fingerprints. This means data just recently executed might take up to 10min to show on the Console. UI will automatically call CockroachDB endpoint when users went to the page. The gateway node sends the query to only the system statistics table. Avoiding the in memory join overhead. Frontend would auto-refresh the results, and do all the sorting in the browser. master & 22.2 backport SQL Activity page adds new search criteria which requires a limit and sort to be specified. User must specify which column to sort by and click the apple button for the UI to call the http endpoint on CockroachDB. The gateway node sends the query to only the system statistics table. Avoiding the in memory join overhead. The UI limit the results to the top 100 queries by default with max of 500. User must do a manual refresh to get new results. & Add new system activity tables and update job User must specify which column to sort by and click the apple button for the UI to call the http endpoint on CockroachDB. The gateway node sends the query to the system activity table. If no results are found it falls back to the system statistics table. The UI limit the results to the top 100 queries by default with max of 500. User must do a manual refresh to get new results. The results below were calculated using a test cluster with 9 nodes and 100k-115k rows on our statistics tables, making a request to return the results for the past 1h. | Version information | Latency to load SQL Activity page | ||--| | Before any change | 1.7 minutes | | Changes on 22.1 & 22.2 | 9.9 seconds | | Changes on 23.1 | ~500ms | The performance gains are from: Avoiding the fan-out to all nodes. Avoiding the virtual table which filters and limits do not get pushed down to. Limiting the results to the top 100 by default. In 23.1 using new activity table which caches the top 500 by the 6 most popular columns. Contains the in memory statistics that have not been flushed to the system statistics tables Expensive operation GRPC fan-out to all nodes. Filters are not pushed down. All the rows are returned to gateway node, a virtual table is created, and then filters are applied. system.statementstatistics [PRIMARY KEY (aggregatedts, fingerprintid, transactionfingerprintid, planhash, appname, nodeid) USING HASH WITH (bucket_count=8)](https://github.com/cockroachdb/cockroach/blob/8f10f78aed7606edd454e886183bf22c74a3153e/pkg/sql/catalog/systemschema/system.go#LL552C26-L553C39) system.transaction_statistics Each node has background that collects stats and it to statistics table every 10

Statistics tables are aggregated by 1 hour and there is a row per a node to avoid contention. Separate runs to delete the rows when table grows above the default . system.statement_activity system.transaction_activity Created to improve the performance of the ui pages. UI groups by fingerprint. Have to aggregate all the rows from based on node id. Users focus on top queries. Computing top queries is expensive and slow. Activity tables have top 500 based on 6 different properties. Limited to 6 properties because there is currently 19 properties total. Calculating and storing all 19 properties is to expensive. Activity tables are updated via the Job coordinates running it on a single node. Provide fault tolerance if a node fails. Activity update job is triggered when the local node . This guarantees no contention with the local node flush which has the overhead of doing the activity update job. Other nodes can have contention with activity update job. Activity table does not have node id as a primary key which aligns with UI and reduces cardinality. Activity update job has 2 queries. Number of rows are less than max top * number of columns to cache. The is a more efficient query by avoiding doing the top part of the query which is not necessary since we know there are less rows. Number of rows are greater than max top * number of columns to cache. The has to do the necessary sort and limits to get the top of each column. ```mermaid sequenceDiagram box less than max top * num columns cached participant sqlActivityUpdater.transferAllStats end box greater than max top * num columns cached participant sqlActivityUpdater.transferTopStats end PersistedSQLStats->>sqlActivityUpdater: Statistics flush is done via channel sqlActivityUpdater->>sqlActivityUpdater.compactActivityTables: Removes rows if limit is hit sqlActivityUpdater->>sqlActivityUpdater.transferAllStats: if less than 3000 rows sqlActivityUpdater->>sqlActivityUpdater.transferTopStats: if greater than 300 rows ``` sql/contention kv/kvserver/concurrency/ TxnIdCache EventStore Resolver sql.contention.eventstore.resolutioninterval: 30 sec sql.contention.eventstore.durationthreshold: default 0; ```mermaid sequenceDiagram title KV layer creates trace SQL layer->>KV layer: Initial call to KV layer KV layer->>locktablewaiter: Transaction hit contention locktablewaiter->>contentionEventTracer.notify: verify it's new lock contentionEventTracer.notify->>contentionEventTracer.emit: Adds ContentionEvent to trace span KV layer->>SQL layer: Return the results of the query SQL layer->>KV layer: if tracing is enabled then network call to get trace KV layer->>SQL layer: returns traces ``` <br> ```mermaid sequenceDiagram title Transaction id cache connExecutor.recordTransactionStart->>txnidcache.Record: Add to cache with default fingerprint id connExecutor.recordTransactionFinish->>txnidcache.Record: Replace the cache with actual fingerprint id ``` <br> ```mermaid sequenceDiagram title Contention events insert process executorstatementmetrics->>contention.registry: AddContentionEvent(ExtendedContentionEvent) contention.registry->>event_store: addEvent(ExtendedContentionEvent) event_store->>ConcurrentBufferGuard: buffer with eventBatch with 64 events ConcurrentBufferGuard->>eventBatchChan: Flush to batch to channel when full ``` <br> ```mermaid sequenceDiagram title Background task 'contention-event-intake' event_store.eventBatchChan->>resolver.enqueue: Append to unresolvedEvents eventstore.eventBatchChan->>eventstore.upsertBatch: Add to unordered cache (blockingTxnId, waitingTxnId, WaitingStmtId) ``` <br> ```mermaid sequenceDiagram title Background task 'contention-event-resolver' 30s with jitter event_store.flushAndResolve->>resolver.dequeue: Append to unresolvedEvents resolver.dequeue->>resolver.resolveLocked: Batch by CoordinatorNodeID resolver.resolveLocked->>RemoteNode(RPCRequest): Batch blocking txn ids RemoteNode(RPCRequest)->>txnidcache.Lookup: Lookup the id to get fingerprint RemoteNode(RPCRequest)->>resolver.resolveLocked: Return blocking txn id & fingerprint results resolver.resolveLocked->>LocalNode(RPCRequest): Batch waiting txn ids LocalNode(RPCRequest)->>txnidcache.Lookup: Lookup the id to get fingerprint LocalNode(RPCRequest)->>resolver.resolveLocked: Return waiting txn id & fingerprint results resolver.resolveLocked->>resolver.resolveLocked: Move resolved events to resolved queue resolver.dequeue->>event_store.flushAndResolve: Return all resolved txn fingerprints eventstore.flushAndResolve->>eventstore.upsertBatch: Replace existing unresolved with resolved events ```