Project: TiKV (App Definition and Development / Database)
File: CHANGELOG.md

All notable changes to this project are documented in this file. See also and . Improvements Enhance disk space protection to improve storage stability Simplify the algorithm of L0 flow control Improve the error log report in the raft client module Improve logging threads to avoid them becoming a performance bottleneck Add more statistics types of write queries Bug Fixes Fix the issue of unavailable TiKV caused by Raftstore deadlock when migrating Regions. The workaround is to disable the scheduling and restart the unavailable TiKV. Fix the issue that CDC adds scan retries frequently due to the Congest error Fix the issue that the Raft connection is broken when the channel is full Fix the issue that batch messages are too large in Raft client implementation Fix the issue that some coroutines leak in resolved_ts Fix a panic issue that occurs to the coprocessor when the size of response exceeds 4 GiB Fix the issue that snapshot Garbage Collection (GC) misses GC snapshot files when snapshot files cannot be garbage collected Fix a panic issue caused by timeout when processing Coprocessor requests Fix a memory leak caused by monitoring data of statistics threads Fix a panic issue caused by getting the cgroup information from some platforms Bug Fixes Fix the issue that the GcKeys task does not work when it is called by multiple keys. Caused by this issue, compaction filer GC might not drop the MVCC deletion information. Improvements Simplify the algorithm of L0 flow control Improve the error log report in raft client module Make the slow log of TiKV coprocessor only consider the time spent on processing requests Drop log instead of blocking threads when the slogger thread is overloaded and the queue is filled up Add more statistics types of write queries Bug Fixes Fix the issue that CDC add scan retries frequently due to Congest error Fix that the raft connection is broken when the channel is full Fix the issue that batch messages are too large in Raft client implementation Fix the issue that concurrent leaks in resolved_ts Fix a panic issue that occurs to coprocessor when response size exceeds 4 GiB Fix the issue that snapshot Garbage Collection (GC) misses GC snapshot files when snapshot files cannot be garbage collected Fix a panic issue that occurs when processing coprocessor requests times out Bug Fixes Fix the issue of unavailable TiKV caused by Raftstore deadlock when migrating Regions. The workaround is to disable the scheduling and restart the unavailable TiKV. Bug Fixes Fix the wrong tikvraftstorehibernatedpeerstate metric Fix the wrong arguments type of the json_unquote() function in the coprocessor Skip clearing callback during graceful shutdown to avoid breaking ACID in some cases Fix a bug that the read index is shared for replica reads on a Leader Fix the wrong function that casts DOUBLE to DOUBLE Bug Fixes Fix the issue that the GcKeys task does not work when it is called by multiple keys. Caused by this issue, compaction filer GC might not drop the MVCC deletion information. Improvements Support dynamically modifying TiCDC configurations Reduce the size of Resolved TS message to save network bandwidth Limit the counts of peer stats in the heartbeat message reported by a single store Bug Fixes Fix a bug that some files are missed to be imported during the process of importing snapshot files when upgrading TiKV from v3.x to v4.x or

Fix the issue that the GC (Garbage Collection) failure (such as file corrupted) of a single snapshot file stops the GC process of all other GC-able files The slow log of TiKV coprocessor only considers the time spent on processing requests Drop log instead of blocking threads when the slogger thread is overloaded and the queue is filled up Fix a bug of the panic caused by timeout when processing Coprocessor requests Fix the TiKV panic issue that occurs when upgrading from a pre-5.0 version with Titan enabled Fix the issue that TiKV of a newer version cannot be rolled back to v5.0.x Fix the issue that TiKV might delete files before it ingests to RocksDB Fix the parsing failure caused by the left pessimistic locks Improvements Make the prewrite requests as idempotent as possible to reduce the chance of undetermined errors Prevent the risk of stack overflow when handling many expired commands Avoid excessive commit request retrying by not using the Stale Read request's startts to update maxts Handle read ready and write ready separately to reduce read latency Reduce the impact on data import speed when the I/O rate limiting is enabled Improve the load balance between Raft gRPC connections Bug Fixes Fix the issue that the duration calculation might panic on certain platforms Fix the issue that Load Base Split mistakenly uses the unencoded keys of batchgetcommand Fix the issue that changing the resolved-ts.advance-ts-interval configuration online cannot take effect immediately Fix the issue of follower metadata corruption in rare cases with more than 4 replicas Fix the panic issue that occurs when building a snapshot twice if encryption is enabled Fix the wrong tikvraftstorehibernatedpeerstate metric Fix the wrong arguments type of the json_unquote() function in the coprocessor Fix a bug that the index keys in a pessimistic transaction might be repeatedly committed Fix the issue that the ReadIndex request returns stale result right after the leader is transferred Improvements Use zstd to compress Region snapshots, preventing large space differences between nodes in case of heavy scheduling or scaling Solve OOM issues in multiple cases Add memory usage tracking for each module Solve the OOM issue caused by oversized Raft entries cache Solve the OOM issue caused by stacked GC tasks Solve the OOM issue caused by fetching too many Raft entries from the Raft log to memory at one time Split Regions more evenly to mitigate the issue that the growth of Region size exceeds the splitting speed when there are hotspot writes Bug Fixes Fix the issue that the coprocessor fails to properly handle the signed or unsigned integer types in the IN expression Fix the issue of many empty Regions after batch ingesting SST files Fix a bug that TiKV cannot start up after the file dictionary file is damaged Fix a TiCDC OOM issue caused by reading old values Fix the issue of empty value in the secondary index for the clustered primary key column when collation is latin1_bin Add the abort-on-panic configuration, which allows TiKV to generate the core dump file when panic

Users still need to correctly configure the environment to enable core dump Fix the performance regression issue of point get queries that occurs when TiKV is not busy Improvements Increase the speed of inserting SST files by moving the verification process to the Import thread pool from the Apply thread pool Add more metrics for the garbage collection module of Raft logs to locate performance problems in the module Collapse some uncommon storage-related metrics in Grafana dashboard Bug Fixes Fix the issue that a down TiKV node causes the resolved timestamp to lag Fix the issue that TiKV cannot detect the memory lock when TiKV perform a reverse table scan Fix the issue that the accumulation of GC tasks might cause TiKV to be OOM (out of memory) Fix the issue of TiKV panic that occurs when the files do not exist when TiDB Lightning imports data Fix the issue that the node of a TiKV replica is down after the node gets snapshots because TiKV cannot modify the metadata accurately Fix the leak issue of the backup thread pool Fix the issue of casting illegal strings into floating-point numbers Bug Fixes Fix the issue that the GcKeys task does not work when it is called by multiple keys. Caused by this issue, compaction filer GC might not drop the MVCC deletion information. Improvements Limit the TiCDC sink's memory consumption Add the memory-bounded upper limit for the TiCDC old value cache Bug Fixes Fix the wrong tikvraftstorehibernatedpeerstate metric Fix the wrong arguments type of the json_unquote() function in the coprocessor Skip clearing callback during graceful shutdown to avoid breaking ACID in some cases Fix a bug that the read index is shared for replica reads on a Leader Fix the wrong function that casts DOUBLE to DOUBLE Improvements Limit the TiCDC sink's memory consumption Add the memory-bounded upper limit for the TiCDC old value cache Bug Fixes Fix the wrong tikvraftstorehibernatedpeerstate metric Fix the wrong arguments type of the json_unquote() function in the coprocessor Skip clearing callback during graceful shutdown to avoid breaking ACID in some cases Fix a bug that the read index is shared for replica reads on a Leader Fix the wrong function that casts DOUBLE to DOUBLE New Features Enable the Hibernate Region feature by default Improvements BR now supports the S3-compatible storage using the virtual-host addressing mode Support the back pressure for TiCDC's scan speed Reduce the memory usage of TiCDC's initial scan Improve the cache hit ratio of the TiCDC's Old Value feature in the pessimistic transaction Split Regions more evenly Bug Fixes Fix a TiCDC OOM issue caused by reading old values Fix the issue of empty value in the secondary index for the clustered primary key column when collation is latin1_bin Add the abort-on-panic configuration, which allows TiKV to generate the core dump file when panic occurs. Users still need to correctly configure the environment to enable core dump Fix the performance regression issue of point get queries that occurs when TiKV is not busy Improvements Use `zstd` to compress the Region snapshot Bug Fixes Fix the issue that the coprocessor fails to properly handle the signed or unsigned integer types in the `IN` expression Fix the issue of many empty Regions after batch ingesting SST files Fix the potential panic that occurs when the input of `caststringas_time` is invalid UTF-8 bytes Fix a bug that TiKV cannot start up after the file dictionary file is damaged Compatibility Changes Replace the `rocksdb.auto-tuned` configuration item with Delete the `raftstore.sync-log` configuration item. By default, written data is forcibly spilled to the disk. Before v5.0, you can explicitly disable `raftstore.sync-log`. Since v5.0, the configuration value is forcibly set to `true` Change the default value of the `gc.enable-compaction-filter` configuration item from `false` to `true` Change the default value of the configuration item from `false` to `true` New features Support log redaction to desensitize the output log

The configuration item `security.redact-info-log`. Its default value is `false`, which means that desensitization is disabled. To enable desensitization for tikv-server logs, set the variable value to `true` Support transaction async commit Support Raft joint consensus Improvements Enable the system to automatically adjust the data compaction speed by default to balance the contention for I/O resources between background tasks and foreground reads and writes Enable the GC Compaction Filter feature by default to reduce GCs consumption of CPU and I/O resources Compatibility Changes Before v4.0.16, when TiDB converts an illegal UTF-8 string to a Real type, an error is reported directly. Starting from v4.0.16, TiDB processes the conversion according to the legal UTF-8 prefix in the string Improvements Reduce disk space consumption by adopting the zstd algorithm to compress SST files when restoring data using Backup & Restore or importing data using Local-backend of TiDB Lightning Bug Fixes Fix a panic issue that occurs when Region merge, ConfChange, and Snapshot happen at the same time in extreme conditions Fix the issue of negative sign when the decimal divide result is zero Fix the issue that the average latency of the by-instance gRPC requests is inaccurate in TiKV metrics Fix the issue of TiCDC panic that occurs when the downstream database is missing Fix the issue that the Raft connection is broken when the channel is full Fix the issue that TiDB cannot correctly identify whether the Int64 types in Max/Min functions are a signed integer or not, which causes the wrong calculation result of Max/Min Fix the issue that CDC adds scan retries frequently due to the Congest error Feature Enhancement Support changing TiCDC configurations dynamically Improvements Handle read ready and write ready separately to reduce read latency The slow log of TiKV coprocessor only considers the time spent on processing

Drop log instead of blocking threads when the slogger thread is overloaded and the queue is filled up Reduce the size of Resolved TS messages to save network bandwidth Bug Fixes Fix the issue that BR reports the "file already exists" error when TDE is enabled during data restore Fix the issue that TiKV deletes stale Regions too frequently Fix the issue that TiKV frequently reconnects the PD client Check stale file information from the encryption file dictionary Compatibility Changes Change the default value of merge-check-tick-interval from 10 to 2 to speed up the Region merge process Feature Enhancements Add a metric pending to monitor the number of pending PD heartbeats, which helps locate the issue of slow PD threads Support using the virtual-host addressing mode to make BR support the S3-compatible storage Improvements Shutdown the status server first to make sure that the client can correctly check the shutdown status Always respond to stale peers to make sure that these peers are cleared quicker Limit the TiCDC sink's memory consumption When a Region is too large, use the even split to speed up the split process Bug Fixes Fix the issue that the duration calculation might panic on certain platforms Fix the wrong function that casts DOUBLE to DOUBLE Fix the issue that the panic log might be lost when using the async logger Fix the panic issue that occurs when building a snapshot twice if encryption is enabled Fix the wrong arguments type of the json_unquote() function in the coprocessor Fix the issues of suspicious warnings during shutdown and the non-deterministic response from Raftstore Fix the issue of backup threads leak Fix the issue that Region split might panic and corrupt the metadata if the split process is too slow and Region merge is on-going Fix the issue that the Region heartbeats prevent TiKV from splitting large Regions in some situations Fix the wrong statistics caused by the format inconsistency of CM Sketch between TiKV and TiDB Fix the wrong statistics of the apply wait duration metric Fix the "Missing Blob" error after using deletefilesin_range in Titan Improvements Make the calculation process of store used size more precise Set more Regions in the EpochNotMatch message to reduce Region misses Speed up freeing the memory accumulated in the long-running cluster Bug Fixes Fix a bug that TiKV cannot start if the file_dict file is not fully written into the disk that has been full Limit TiCDC's scan speed at 128MB/s by default Reduce the memory usage of TiCDC's initial scan Support the back pressure for TiCDC's scan speed Fix a potential OOM issue by avoiding unnecessary reads to get TiCDC old values Fix a TiCDC OOM issue caused by reading old values Add a timeout mechanism for S3 storages to avoid the client hanging without responses Improvements Prevent a large number of reconnections in a short period of time Optimize the write operations in the scenarios of many tombstones Change the default value of `leader-transfer-max-log-lag` to `128` to increase the success rate of leader transfer Bug Fixes Fix the issue that the `IN` expression does not properly handle unsigned/signed integers Fix the issue that the ingest operation is not re-entrant Fix the issue that the space is missed when converting JSON to string in TiKV coprocessor New Features Support the `utf8mb4unicodeci` collation Support the `castyearas_time` collation Improvements Add metrics of server information for DBaaS Support multiple clusters in Grafana dashboards Report RocksDB metrics to TiDB Record the suspension time for Coprocessor tasks Add thresholds of key counts and key size for Load Base Split Check whether the file exists before data import Improve Fast Tune panels Bug Fixes Fix the issue that TiKV is failed to build with `PROST=1` Fix the unmatched memory diagnostics Fix the issue that the end key of a partial RawKV-restore range is inclusive Fix the issue that TiKV might panic when loading the old value of a key of a rolled-back transaction during TiCDC's incremental scan Fix the configuration glitch of old values when changefeeds with different settings connect to one Region Fix a crash issue that occurs when running a TiKV cluster on a machine with a network interface that lacks the MAC address (introduced in v4.0.9) Fix the issue of TiKV OOM when backing up a huge Region Fix the issue that `region-split-check-diff` cannot be customized Fix the issue of TiKV panic when the system time goes back Bug Fixes Fix the wrong mapping between ready and peer Fix the issue that some logs are not redacted when `security.redact-info-log` is set to `true` Improvements Add the tag to trace the source of the `split` command Support dynamically changing the

configuration Reduce the impact on performance when running Backup & Restore and TiDB Lightning Add monitoring metrics for the ingesting SST errors Prevent hibernation when some peers are still catching up with logs Increase the success rate of the pipelined pessimistic locking Change the default value of `apply-max-batch-size` and `store-max-batch-size` to `1024` Add the `max-background-flushes` configuration item Enable the unified read pool for the storage module by default Disable `force-consistency-checks` by default to improve performance Bug Fixes Fix the issue that Coprocessor might return wrong results when there are more than 255 columns Fix the issue that Region Merge might cause data loss during network partition Fix the issue that the `ANALYZE` statement might cause panic when using the `latin1` character set Fix the wrong results returned when converting the numeric type to the time type Fix a bug that TiDB Lightning fails to ingest SST files to TiKV with the Importer-backend or Local-backend when Transparent Data Encryption (TDE) is enabled Fix the invalid `advertise-status-addr` value (`0.0.0.0`) Fix the issue that an error is returned indicating that a key exists when this key is locked and deleted in a committed transaction Fix the issue that the RocksDB cache mapping error causes data corruption Fix a bug that Follower Read might return stale data after the leader is transferred Improvements Add the Fast-Tune panel page to assist performance diagnostics Add the `security.redact-info-log` configuration item, which redacts user data from logs Reformat the metafile of error codes Enable dynamically changing the `pessimistic-txn.pipelined` configuration Enable the memory profiling features by default Bug Fixes Fix the bug that the mutex conflict in encryption causes pd-worker to process heartbeats slowly Fix the issue that the memory profile is mistakenly generated Fix the failure to back up databases on GCS when the storage class is specified Fix the bug that a learner cannot find a leader when the Region is restarted or newly split Improvements Support the JSON log format Bug Fixes Fix the issue of unavailable Status API when TLS handshake fails Fix the potential undefined behaviors Fix the possible panic caused by generating snapshots when executing `UnsafeDestroyRange` Improvements Reduce QPS drop when `DropTable` or `TruncateTable` is being executed Support generating metafile of error codes Add performance statistics for cf scan detail Add the `rocksdb perf context` panel in the Grafana default template Bug Fixes Fix the estimation error for a non-index column when collation is enabled Fix the issue that Green GC might miss locks during the process of Region transfer Fix a panic issue that occurs when TiKV runs very slowly during Raft membership change Fix the deadlock issue that occurs between the PD client thread and other threads when calling PD sync requests Upgrade jemalloc to v5.2.1 to address the issue of memory allocation in huge page Fix the issue that the unified thread pool hangs for long-running queries New Features Define error code for errors Bug Fixes Speed up leader election when Hibernate Region is enabled Fix the memory leak issue during scheduling Add the `hibernate-timeout` configuration item to prevent the leader from becoming hibernate too fast Improvements Introduce the new

configuration to control the size of the backup thread pool Do not send store heartbeats when receiving snapshots Support dynamically changing the shared block cache's capacity Bug Fixes Fix the issue that reads might get stale data during merging Fix the issue that collation does not work on the `min`/`max` function when aggregation is pushed down to TiKV Bug Fixes Fix a memory safety issue for the status server Fix the issue of lost precision in JSON numeric comparison Fix the wrong query slow log Fix the issue that a peer cannot be removed when its store is isolated during multiple merge processes Fix the issue that `tikv-ctl recover-mvcc` does not remove invalid pessimistic locks Fix the issue that some Titan histogram metrics are missing Fix the issue that TiKV returns `duplicated error` to TiCDC New Features Support the `encryption-meta` command in TiKV Control Add a perf context metric for `RocksDB::WriteImpl` Bug Fixes Fix the issue that the `use-unified-pool` configuration in the startup log is incorrectly printed Fix the issue that the tikv-ctl does not support relative path Fix the bug that the monitoring metric of Point Selects is inaccurate Fix the issue that a peer might not be destroyed after the network isolation disappears Fix the issue that a `read index` request may get out-of-date commit index Improve the reliability of backup and restore with S3 and GCS storages New Features Add the `--advertise-status-addr` start flag to specify the status address to advertise Bug Fixes Fix the issue of backup fails with DefaultNotFound error. Tolerate out-of-order read states from raft ready. Handles the case when the callback is dropped by mistake when getting snapshot. Fix the issue that `asciibin` and `latin1bin` is not supported by TiKV for new collation framework. Don't clean snapshot files when shutting down. Encryption: fix master key not being able to rotate. Encryption: fix snapshot apply failure caused lock CF snapshot file not being encrypted on

Compatibility Changes Move the encryption-related configuration to the security-related configuration, which means changing ` New Features Support encryption debugging for tikv-ctl, so that tikv-ctl can be used to operate and manage the cluster when the encryption storage is enabled Support encrypting the lock column family in snapshots Use the heatmap in the Grafana dashboard for Raftstore latency summary to better diagnose the jitter issue Support setting the upper limit for the size of the gRPC message Add in Grafana dashboard the encryption-related monitoring metrics Support Application-Layer Protocol Negotiation (ALPN) Add more statistics about Titan Support using the task ID provided by the client as the identifier in the unified read pool to avoid that the priority of a task is lowered by another task in the same transaction Improve the performance of the `batch insert` request Bug Fixes Fix the issue that many empty Regions are generated after restoration Fix the panic issue of Raftstore when receiving out-of-order read index responses Fix the issue that an invalid storage or coprocessor read pool configuration might not be rejected when the unified thread pool is enabled Fix the panic issue of the `join` operation when the TiKV server is shut down Fix the issue that no result is returned when searching TiKV slow logs via diagnostics API Fix the issue that notable memory fragmentation is generated when the TiKV node is running for a long time Fix the issue that the SQL statement fails to execute when an invalid date is stored Fix the issue that the backup data cannot be restored from GCS Fix the issue that KMS key ID is not validated during encryption at rest Fix the underlying correctness issue of the Coprocessor in compilers of different architecture Fix the `snapshot ingestion` error when encrytion is enabled Fix the `Invalid cross-device link` error when rewriting the configuration file Fix the issue of wrong toml format when writing the configuration file to an empty file Fix the issue that a destroyed peer in Raftstore can still process requests Compatibility Changes Disable the Hibernate Region feature by default Important Bug Fixes Fix the deadlock issue caused by the probe request from TiDB Fix the issue that the minimum commit timestamp of a transaction might overflow which affects data correctness New Features Support using the user-owned KMS key for the server-side encryption when backing up data to S3 Enable the load-based `split region` operation Support validating common names Add the file lock check to avoid starting multiple TiKV instances that are bound to the same address Support AWS KMS in encryption at rest Bug Fixes Address the OpenSSL security issue: CVE-2020-1967 Avoid protecting rollback records written by `BatchRollback` to improve performance when many write conflicts exist in optimistic transactions Fix the issue that the needless wake-up of transactions results in useless retry and performance reduction in heavy lock-race workloads Fix the issue that the Region might be stuck in the multi-time merging Fix the issue that the learner is not deleted when deleting the learner Fix the issue that follower read might cause panic in raft-rs Fix the bug that a SQL operation might fail because of the `group by constant` error Fix the issue that an optimistic lock might block reads if the corresponding primary lock is a pessimistic lock Compatibility Changes Support the `pipelined` feature in pessimistic transactions, which improves the TPC-C performance by 20%. The risk is that the transaction commit might fail because of lock failure during the execution Enable the `unify-read-pool` configuration item in new clusters by default and use the previous setting of this item in old clusters New Features Support the `pipelined` feature in pessimistic transactions, which improves the TPC-C performance by 20%. The risk is that the transaction commit might fail because of lock failure during the execution Support TLS in the HTTP port Enable the `unify-read-pool` configuration item in new clusters by default and use the previous setting of this item in old clusters Bug Fixes Fix the possible panic caused by transferring the leader when the Follower Read feature is enabled New Features Support the configuration of persistent dynamic update Bug Fixes Fix the panic issue caused by empty short values during backup Fix the issue that Hibernate Regions might not be woken up correctly Compatibility Changes Add the `readpool.unify-read-pool` configuration item (`True` by default) to control whether point queries use the same threads with Coprocessor New Features Support fetching configuration items from the status port via HTTP API Optimize the performance of `Chunk Encoder` in Coprocessor Bug Fixes Fix the inconsistent behaviors of the `CAST` function in TiDB and TiKV Upgrade the RocksDB version to 6.4.6 Fix the issue that the system cannot perform the compaction task normally when the disk space is used up by automatically creating a 2GB empty file when TiKV is started Support quick backup and restoration + + + Support reading data from Follower replicas + Improve the performance of TiDB reading data through index Fix the issue that the `CAST` function behaves inconsistently in TiKV and in TiDB + + + + backup Change the name of the backup file from `startkey` to the hash value of `startkey` to reduce the file name's length for easy reading

Disable RocksDB's `forceconsistencychecks` check to avoid false positives in the consistency check Add the incremental backup feature sst_importer Fix the issue that the SST file does not have MVCC properties during restoring Add the monitoring items such as `tikvimportdownloadduration`, `tikvimportdownloadbytes`, `tikvimportingestduration`, `tikvimportingestbytes`, and `tikvimporterror_counter` to observe the overheads of downloading and ingesting SST files raftstore Fix the issue of Follower Read that the follower reads stale data when the leader changes, thus breaking transaction isolation Support the distributed backup and restore feature Support the Follower Read feature Fix the issue that an error is returned indicating that a key exists when this key is locked and deleted in a committed transaction Add the `endpointslowlogthreshold` configuration item Fix the bug that TiKV panics when parsing responses with missing reason phrases Set `sync-log` to `true` as an nonadjustable value Change gc failure log to warning Fix a bug that might read stale data during region merging Fix memory leak during scheduling Add the `hibernate-timeout` configuration that delays region hibernation to improve rolling update performance Fix the potential wrong result read from ingested files Fix the issue that a peer can not be removed when its store is isolated during multiple merge processes Avoid sending store heartbeats to PD after snapshots are received Improve the PD client log Fix a panic issue that Titan GC may delete an already deleted blob file Fix the issue that clean snapshot files which were in used after restarting Change schedule tick failure log to debug level to make logs less verbose Make grpc message size limit configurable Fix the issue that the memory defragmentation will not be very effective after running for a long time Improve the performance when many conflicts and the `BatchRollback` condition exist in optimistic transactions Fix the issue of decreased performance that occurs because the pessimistic lock `waiter` is frequently awakened when many conflicts exist in pessimistic transactions Fix the issue that the node cannot be deleted correctly after the isolation recovery in some cases Fix the issue of data loss during network isolation caused by the Region Merge operation Fix the issue that learner cannot be removed correctly in some cases Fix the issue that the scanning result of raw key-value pairs might be out of order Fix the issue of reconnection when the batch of Raft messages is too large Fix the issue of gRPC thread deadlock caused by the empty request Fix the issue that the processing logic of restarting the learner is incorrect during the merge process Fix the issue that repeated requests on the cleanup of lock might destroy the atomicity of the transaction Fix the issue of conflict detection failure or data index inconsistency caused by inserting an existing key into a transaction and then deleting it immediately when disabling the consistency check parameter Introduce a flow control mechanism in Raftstore to solve the problem that without flow control, it might lead to too slow tracking and cause the cluster to be stuck, and the transaction size might cause frequent reconnection of TiKV connections , Optimize the log output by removing unnecessary logs Fix the panic that might occur when the peer is removed under high loads Fix the issue that Hibernate Regions are not waken up in some cases Raftstore Fix the system panic issue #6460 or data loss

#5981 caused by Region merge failure Support `yield` to optimize scheduling fairness, and support pre-transferring the leader to improve leader scheduling stability Raftstore Speed up the configuration change to speed up the Region scattering Transaction Add the `tikvlockmanagerwaiterlifetimeduration`, `tikvlockmanagerdetectduration`, and `tikvlockmanagerdetect_duration` monitoring metrics to monitor `waiter`s lifetime, the time cost of detecting deadlocks, and the status of `Wait` table Optimize the following configuration items to reduce transaction execution latency caused by changing Region leader or the leader of deadlock detector in extreme situations Change the default value of `wait-for-lock-time` from `3s` to `1s` Change the default value of `wake-up-delay-duration` from `100ms` to `20ms` Fix the issue that the leader of the deadlock detector might be incorrect during the Region Merge process Coprocessor Modify the level of the output log from `error` to `warn` when an error occurs in Coprocessor Modify the update behavior of statistics sampling data from directly updating the row to deleting before inserting, to keep consistency with the update behavior of tidb-server Raftstore Fix the panic caused by repeatedly sending the `destroy` message to `peerfsm` and `peerfsm` being destroyed multiple times Update the default value of `split-region-on-table` from `true` to `false` to disable splitting Regions by table by default Engine Fix the issue that empty data might be returned because RocksDB iterator errors are not correctly processed in extreme conditions Transaction Fix the issue that TiKV fails to write data into keys and GC is blocked because the pessimistic locks are incorrectly cleaned up Optimize the pessimistic lock waiting mechanism to improve the performance in scenarios where the lock conflict is severe Update the default value of `tikvalloc` from `tikvalloc/default` to `jemalloc` Update grpc to fix a potential memory leak issue Deadlock: only observe valid region in order to make sure the manager is in the valid region Pessimistic Transaction: keep lock's ttl when receive a smaller ttl rust-rocksdb: fix titan options for cf when create cf Fix TiKV panic when aggregation expr type is not valid Pessimistic Transaction: reduce clean up requests in lock_manager Fix a region merge bug which may cause panic: set is_merging flag after restart in raftstore Generate flamegraph at runtime Support to change the config gc io limit dynamically Limit the speed of write for GC Engine: update rocksdb and titan rocksdb: Fix OnFlushCompleted fired before flush result write to MANIFEST titan: Fix status overrided by mistake Makefile: add a new rule for CI test Add metrics for commit log duration Pessimistic Transaction: Add support for lock wait timeout LockManager: make has_waiter accurate Fix wrong txn_size when acquire pessimistic lock Fix the problem that split check is always scanning caused by updating approximate in pd-worker Update rust-rocksdb to avoid intra_L0 compaction issue Fix the bug which may break atomicity: product primary locks of pessimistic transactions from being collapsed Enable rocksdb forceconsistencychecks and handle background error Fix the bug in raftstore that painic when getting value encouters an error Fix the bug that do not return the right tso when checking lock Reduce the overhead of region's heartbeat Reduce message flush in raftstore Check Lock's TTL when doing clean up Fix the issue that the approximate keys is not correct when region is empty Make the config support rocksdb doubly skiplist to optimize `reverse-scan` Optimize point-get in coprocessor Support batch-split command and empty batch command Fix `PointGetter` performance issue when there are concurrent write Fix the output on short version flag Support the pessimistic transaction API: txn-heart-beat `titan` GC and monitoring improvement Update `grpcio` to

Support GRPC memory quota Fix commit index is not forwarded when merge entry is empty Fix a resource leak bug in batch grpc Fix the issue that ReadIndex might fail to respond to requests because of duplicate context Fix potential scheduling jitters caused by premature `PutStore` Fix incorrect timestamps reported from Region heartbeats Fix potential TiKV panics during region merge Speed up leader change check for the dead lock detector Support using `grpc env` to create deadlock clients Add `config-check` to check whether the configuration is correct Fix the issue that ReadIndex does not return anything when there is no leader Exclude shared block cache from core dump Fix the bug that TiKV panics if the Raft Log is not written in time Fix the bug that the panic information is not written into the log file after TiKV panics Fix the bug that the insert operation might be incorrectly performed in the pessimistic transaction Lower the output level of some logs that require no manual intervention to INFO Improve the accuracy of monitoring the storage engine size Improve the accuracy of the Region size in TiKV Control Improve the performance of the deadlock detector for pessimistic locks Improve the performance of GC in the Titan storage engine Add the statistics of the size of blob files in statistics information Fix the core dump issue caused by the incorrectly cleaned memory resources when the process exits Add all monitoring metrics related to the Titan engine , Add the number of open file handles for Titan when counting the number of open file handles to avoid the issue that no file handle is available because of inaccurate statistics of file handles Set `blobrunmode` to decide whether to enable the Titan engine on a specific CF Fix the issue that the read operations cannot get the commit information of pessimistic transactions Add the `blob-run-mode` configuration parameter to control the running mode of the Titan engine, and its value can be `normal`, `read-only` or `fallback` Improve the performance of detecting deadlocks Engine Introduce Titan, a key-value plugin that improves write performance for scenarios with value sizes greater than 1KiB, and relieves write amplification in certain degrees Optimize memory management to reduce memory allocation and copying for `Iterator Key Bound Option` Support `block cache` sharing among different column families Server Support reversed `rawscan` and `rawbatch_scan` Support batch receiving and sending Raft messages, improving TPS by 7% for write intensive scenarios Support getting monitoring information via HTTP Support Local Reader in RawKV to improve performance Reduce context switch overhead from `batch commands` Raftstore Support Multi-thread Raftstore and Multi-thread Apply to improve scalabilities, concurrency capacity, and resource usage within a single node. Performance improves by 70% under the same level of pressure Support checking RocksDB Level 0 files before applying snapshots to avoid write stall Support Hibernate Regions to optimize CPU consumption from RaftStore (Experimental) Remove the local reader thread Transaction Support distributed GC and concurrent lock resolving for improved GC performance Support the pessimistic transaction model (Experimental) Modify the semantics of `Insert` to allow Prewrite to succeed only when there is no Key Remove `txn scheduler ` Add monitoring items related to `read index` and `GC worker` Coprocessor Refactor the computation framework to implement vector operators, computation using vector expressions, and vector aggregations to improve performance Support providing operator execution status for the `EXPLAIN ANALYZE` statement in TiDB Switch to the `work-stealing` thread pool model to reduce context switch cost Misc Develop a unified log format specification with restructured log system to facilitate collection and analysis by tools Add performance metrics related to configuration information and key bound crossing. Engine Check iterator status when scanning. Fix the issue that ingested files and directory are not synchronized. Server Sanitize block size configuration. Support replicating the `delete_range` request without deleting the data when applying. Add read index related metrics. Add GC worker related metrics. Raftstore Fix the issue that local reader cache is not cleared correctly. Fix request latency jetter when transferring leader and conf changes. Remove invalid empty

Clear stale reads after role change. Synchronize all CF files for the received snapshots. Fix missing fsync calls for snapshots. Coprocessor Improve coprocessor batch executor. Transaction Support `ResolveLockLite` to allow only resolving specified lock keys. Improve pessimistic lock transaction. Tikv-ctl Improve `bad-regions` and `tombstone` subcommands. Misc Add dist_release. Engine Support multiple column families sharing a block cache Server Remove `TxnScheduler` Support pessimistic lock transactions Raftstore Support hibernate Regions to reduce the consumption of the raftstore CPU Fix the issue that the leader does not reply to the `ReadIndex` requests for the learner Fix the issue of transferring leader failure in some cases Fix the possible dirty read issue in some cases Fix the issue that a snapshot lacks data in some cases Coprocessor Add more RPN functions `LogicalOr` `LTReal` `LEReal` `GTReal` `GEReal` `NEReal` `EQReal` `IsNull` `IsTrue` `IsFalse` Support comparison arithmetic for `Int` Support comparison arithmetic for `Decimal` Support comparison arithmetic for `String` Support comparison arithmetic for `Time` Support comparison arithmetic for `Duration` Support comparison arithmetic for `Json` Support plus arithmetic for `Int` Support plus arithmetic for `Real` Support plus arithmetic for `Decimal` Support MOD functions for `Int` Support MOD functions for `Real` Support MOD functions for `Decimal` Support minus arithmetic for `Int` Support minus arithmetic for `Real` Support minus arithmetic for `Decimal` Engine Fix the issue that may cause incorrect statistics on read traffic Fix the issue that may cause prefix extractor panic when deleting a range Optimize memory management to reduce memory allocation and copying for `Iterator Key Bound Option` Fix the issue that failing to consider learner log gap may in some cases cause panic Support `block cache` sharing among different `column families` Server Reduce context switch overhead of `batch commands` Check the validity of seek iterator status RaftStore Support configurable `properties index distance` Coprocessor Add batch index scan executor Add vectorized evaluation framework Add execution summary framework for batch executors Check the maximum column when constructing the RPN expression to avoid invalid column offset that may cause evaluation panic Add `BatchLimitExecutor` Replace the original `futures-cpupool` with `tokio-threadpool` in ReadPool to reduce context switch Add batch aggregation framework Add `BatchSelectionExecutor` Add batch aggression function `AVG` Add RPN function `LogicalAnd` Misc Support `tcmalloc` as a memory allocator Optimize the Coprocessor calculation execution framework and implement the TableScan section, with the Single TableScan performance improved by 5% ~ 30% Implement the definition of the `BatchRows` row and the `BatchColumn` column Implement `VectorLike` to support accessing encoded and decoded data in the same way Define the `BatchExecutor` to interface and implement the way of converting requests to `BatchExecutor` Implement transforming the expression tree into the RPN format Implement the `BatchTableScanExecutor` vectorization calculation operator Unify the log format for easy collection and analysis by tools Support using the Local Reader to read in the Raw Read interface Add metrics about configuration information Add metrics about key exceeding bound Add an option to control panic or return an error when encountering the key exceeding bound error Add support for the `INSERT` operation, make prewrite succeed only when keys do not exist, and eliminate `Batch Get` Use more fair batch strategy in the Batch System Support Raw scan in tikv-ctl Support hibernating regions Support distributed GC Check RocksDB Level 0 files before applying snapshots to avoid Write Stall Support reverse `rawscan` and `rawbatch_scan` Support using HTTP to obtain monitoring information Support DST better Support receiving and sending Raft messages in batch Introduce a new storage engine Titan Upgrade gRPC to

Support receiving the client requests and sending replies in batch Support multi-thread Apply Support multi-thread Raftstore RaftstoreFix the panic occurred when restarting TiKV and `is_merging` is given an incorrect value in the process of merging Regions and applying the Compact log ImporterRemove the limit on the gRPC message length Fix the incorrect result of counting keys in a Region in some cases Add the `config-check` option in TiKV to check whether the TiKV configuration item is valid Optimize the starting process to reduce jitters caused by restarting nodes Optimize the resolving locking process in some cases to speed up resolving locking for transactions Optimize the `gettxncommit_info` process to speed up committing transactions Simplify Raft-related logs Resolve the issue that TiKV exits abnormally in some cases Return region error when TiKV is closing Support reverse `rawscan` and `rawbatch_scan` Unify the log format Improve the accuracy of Region's approximate size or keys in extreme cases to improve the accuracy of scheduling Optimize processing the empty callback when processing the Raftstore message to avoid sending unnecessary message Fix the issue that incomplete snapshots are generated in the system caused by the iterator not checking the status Add a feature to check the validity for the `block-size` configuration Fix the issue that Regions are not available during the leader transfer process in extreme conditions Fix the issue that TiKV loses data when the power of the machine fails abnormally, caused by delayed data flush to the disk when receiving snapshots Fix the issue that the learner reads an empty index when there is only one leader and one learner Process `ScanLock` and `ResolveLock` in the thread pool with a high priority to reduce their impacts on commands with a normal priority Synchronize all CF files for received snapshots Reject transfer leader when the region recently changed config Add priority label to coprocessor metrics Fix the issue that read index may read stale data during transferring leader Fix the issue that `CommitMerge` may cause TiKV unable to restart Fix unknown logs Fix potential quorum changes when transferring leader (https://github.com/pingcap/raft-rs/issues/221) Fix the Importer bug that some SST files fail to be imported but it still returns successful import result Support setting a speed limit in Importer when uploading SST files to TiKV Change Importer RocksDB SST default compression method to `lz4` to reduce CPU consumption Fix the issue of wrong statistics of the read traffic Fix the raftstore performance issue when checking to decide whether to process pending snapshots when many Regions exist Do not ingest files when the number of level 0 SST files exceeds `levelzeroslowdownwritestrigger/2` Fix the `StoreNotMatch` issue caused by decoding protobuf error in some cases Improve import speed by increasing default region-split-size to 512 MiB Fix OOM issue by storing the intermediate SST files on disk instead of memory Restrict memory usage by RocksDB Fix the issue that scattering Region doesn't take effect Fix the panic issue caused by Region merge in some cases Fix the issue that Importer fails to import data in some cases Fix the `KeyNotInRegion` error in some cases Add the detailed `StoreNotMatch` error message Fix the abnormal result issue of the event listener in some cases #4126 Fix the duplicate write issue when closing TiKV #4146 Support obtaining the monitoring information using the HTTP method #3855 Fix the NULL issue of data_format #4075 Add verifying the range for scan

#4124 Support the configuration format in the unit of `DAY` (`d`) and fix the configuration compatibility issue Fix the possible panic issue caused by `Approximate Size Split` Fix two issues about Region merge , Avoid transferring the leader to a newly created peer, to optimize the possible delay Coprocessor Add more built-in functions - Fix the time function parsing issue and the time zone related issues Optimize the memory usage for pushdown aggregation computing Transaction Optimize the read logic and memory usage of MVCC to improve the performance of the scan operation and the performance of full table scan is 1 time better than that in TiDB 2.0 Fold the continuous Rollback records to ensure the read performance - Separate the GC module to reduce the impact on write Add the`upper bound` support in the `kv_scan` command Raftstore Improve the snapshot writing process to avoid RocksDB stall - Support `Region Split` according to statistics to reduce the I/O overhead Support `Region Split` according to the number of keys to improve the concurrency of index scan Improve the Raft message process to avoid unnecessary delay brought by `Region Split` Enable the `PreVote` feature by default to reduce the impact of network isolation on services Storage Engine Fix the `CompactFiles` bug in RocksDB and reduce the impact on importing data using Lightning Upgrade RocksDB to

to fix the possible issue of snapshot file corruption Improve `IngestExternalFile` to avoid the issue that flush could block write tikv-ctl - The `compact` command supports specifying whether to compact data in the bottommost level Tools Fast full import of large amounts of data: Support new Improve the error message of `WriteConflict` Add the panic mark file Downgrade grpcio to avoid the segment fault issue caused by the new version of gRPC Add the upper limit to the `kv_scan` interface Optimize the RocksDB Write stall issue caused by applying snapshots Add raftstore `tick` metrics Upgrade RocksDB and fix the Write block issue and that the source file might be damaged by the Write operation when performing `IngestExternalFile` Upgrade grpcio and fix the issue that too many pings is wrongly reported Optimize the concurrency for coprocessor requests Add the support for Log functions Add the support for the `sha1` function Add the support for the `truncate_int` function Add the support for the `year` function Add the support for the `truncate_real` function Fix the reporting error behavior related to time functions , Fix the issue that the time parsed from string is inconsistent with that in TiDB Support splitting Regions based on statistics estimation to reduce the I/O cost Reduce clone in the transaction scheduler Add the pushdown support for a large number of built-in functions Add the `leader-transfer-max-log-lag` configuration to fix the failure issue of leader scheduling in specific scenarios Add the `max-open-engines` configuration to limit the number of engines opened by `tikv-importer` simultaneously Limit the cleanup speed of garbage data to reduce the impact on `snapshot apply` Broadcast the commit message for crucial Raft messages to avoid unnecessary delay Fix the leader election issue caused by discarding the `PreVote` message of the newly split Region Fix follower related statistics after merging Regions Fix the issue that the local reader uses obsolete Region information Support UnsafeDestroyRange API to speedup garbage data cleaning after table/index has been truncated/dropped Support `batch split` to avoid too large Regions caused by the Write operation on hot Regions Support splitting Regions based on the number of rows to improve the index scan efficiency Use `LocalReader` to separate the Read operation from the raftstore thread to lower the Read latency Refactor the MVCC framework, optimize the memory usage and improve the scan Read performance Support splitting Regions based on statistics estimation to reduce the I/O usage Optimize the issue that the Read performance is affected by continuous Write operations on the rollback record Reduce the memory usage of pushdown aggregation computing Add the pushdown support for a large number of built-in functions and better charset support Optimize the GC workflow, improve the GC speed and decrease the impact of GC on the system Enable `prevote` to speed up service recovery when the network is abnormal Add the related configuration items of RocksDB log files Adjust the default configuration of `scheduler_latch` Support setting whether to compact the data in the bottom layer of RocksDB when using tikv-ctl to compact data manually Add the check for environment variables when starting TiKV Support dynamically configuring the `dynamiclevelbytes` parameter based on the existing data Support customizing the log format Integrate tikv-fail in tikv-ctl Add I/O metrics of threads Fix decimal related issues Fix the issue that `gRPC maxsendmessage_len` is set mistakenly Fix the issue caused by misconfiguration of `region_size` Upgrade Rust to the `nightly-2018-06-14` version Provide a `Raft PreVote` configuration to avoid leader reelection generated when network recovers after network isolation Add a metric to display the number of files and `ingest` related information in each layer of RocksDB Print `key` with too many versions when GC works Use `static metric` to optimize multi-label metric performance (YCSB `raw get` is improved by 3%) Remove `box` in multiple modules and use patterns to improve the operating performance (YCSB `raw get` is improved by 3%) Use `asynchronous log` to improve the performance of writing logs Add a metric to collect the thread status Decease memory copy times by decreasing `box` used in the application to improve the performance Fix two issues about Region merge #4003 and #4004 Avoid transferring the leader to a newly created peer, to optimize the possible delay #3929 Fix redundant Region heartbeats #3930 Add the end-key limit to the kv_scan interface #3749 Abandon the max-tasks-xxx configuration and add max-tasks-per-worker-xxx #3093 Fix the CompactFiles issue in

#3789 Fix the issue that the memory consumed by Raftstore EntryCache keeps increasing when a node goes down 3529 Enable dynamic-level-bytes by default to reduce space amplification Update Region's approximate size and approximate keys count after Region merge Enlarge schedulers default slots to reduce false conflicts Reduce continuous records of rollback transactions, to improve the Read performance when conflicts are extremely severe Limit the size and number of RocksDB log files, to reduce unnecessary disk usage in long-running condition Fix the crash issue when converting the data type from string to decimal Fix the potential overflow issue in decimal operations Fix the dirty read issue that might occur in the process of merge Add the RocksDB `PerfContext` interface for debugging Add the `region-properties` command for `tikv-ctl` Make GC record the log when GC encounters many versions of data Remove the `import-mode` parameter Fix the issue that `reverse-seek` is slow when many RocksDB tombstones exist Fix the crash issue caused by `do_sub` Correct wrong peer meta for learners Report an error instead of getting a result if divisor/dividend is 0 in dodivmod Support configuring more gRPC related parameters Support configuring the timeout range of leader election Fix the issue that the Raft log is not printed Fix the issue that obsolete learner is not deleted Fix the issue that the snapshot intermediate file is mistakenly deleted Reduced number of `thread_yield` calls Fix the issue that `SELECT FOR UPDATE` prevents others from reading More verbose logs for slow query Speed up delete range Fix the bug that raftstore is accidentally blocked when generating the snapshot Fix the issue that Learner cannot be successfully elected in special conditions Fix the issue that split might cause dirty read in extreme conditions Correct the default value of the read thread pool configuration Protect critical configuration from incorrect modification Support `Region Merge` [experimental] Add the `Raw DeleteRange` API Add the `GetMetric` API Add `Raw Batch Put`, `Raw Batch Get`, `Raw Batch Delete` and `Raw Batch Scan` Add Column Family options for the RawKV API and support executing operation on a specific Column Family Support Streaming and Streaming Aggregation in Coprocessor Support configuring the request timeout of Coprocessor Carry timestamps with Region heartbeats Support modifying some RocksDB parameters online, such as `block-cache-size` Support configuring the behavior of Coprocessor when it encounters some warnings or errors Support starting in the importing data mode to reduce write amplification during the data importing process Support manually splitting Region in halves Improve the data recovery tool `tikv-ctl` Return more statistics in Coprocessor to guide the behavior of TiDB Support the `ImportSST` API to import SST files [experimental] Add the TiKV Importer binary to integrate with TiDB Lightning to import data quickly [experimental] Optimize read performance using `ReadPool` and increase the `rawget/get/batchget` by 30% Improve metrics performance Inform PD immediately once the Raft snapshot process is completed to speed up balancing Solve performance jitter caused by RocksDB flushing Optimize the space reclaiming mechanism after deleting data Speed up garbage cleaning while starting the server Reduce the I/O overhead during replica migration using `DeleteFilesInRanges` Fix the issue that gRPC call does not returned when the PD leader switches Fix the issue that it is slow to offline nodes caused by snapshots Limit the temporary space usage consumed by migrating replicas Report the Regions that cannot elect a leader for a long time Update the Region size information in time according to compaction events Limit the size of scan lock to avoid request timeout Limit the memory usage when receiving snapshots to avoid OOM Increase the speed of CI test Fix the OOM issue