Repository: pingcap/tiflash
Pull request #10055: KVStore: Fix fail to restore tikv value larger than 16MiB
Author: login: JaySon-Huang
Labels: name: release-note
description: Denotes a PR that will be considered when it comes time to generate release notes.
name: size/M
description: Denotes a PR that changes 30-99 lines, ignoring generated files.
name: approved
name: lgtm
Created: 2025-03-28T10:02:23Z | Merged: 2025-03-31T10:06:08Z

### What problem does this PR solve?

Issue Number: close https://github.com/pingcap/tiflash/issues/10052

Problem Summary:

```
# tidb.toml
[performance]
txn-entry-size-limit = 125829120
# tiup edit
#    performance.txn-entry-size-limit: 125829120
```
```
# tikv.toml
[raftstore]
raft-entry-max-size = '120MiB'
# tiup edit
#     raftstore.raft-entry-max-size: '120MiB'
```

```
set global max_allowed_packet = 1073741824;
CREATE TABLE `t` (
  `id` bigint(21) unsigned NOT NULL,
  `text` longtext DEFAULT NULL
);
```

TiDB could allow a raft-log with 120MiB at max, while TiFlash `Region::deserialize` use `readBinary2<std::string>` only allow 16MiB at max. So if there are some raftlog larger than 16MiB and flushed to `RegionData`, then TiFlash could fail to restart when restoring Region data from disk.
If the raftlog is committed, the rows are decoded into `Block` and write down to `IStorage`. Reading/writing rows larger than `16MiB` in the `IStorage` layer does not cause any error.


### What is changed and how it works?

```commit-message
KVStore: Fix restoring large tikv value
Add function `readTiKVStringBinary` for `TiKVValue::deserialize` and `TiKVKey::deserialize` to allow restoring tikv key/values at most 120MiB
```

### Check List

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code

Side effects

- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [ ] Breaking backward compatibility

Documentation

- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Changes MySQL compatibility

### Release note

<!-- bugfix or new feature needs a release note -->

```release-note
Fix an issue that TiFlash may fail to restart when user write rows larger than 16MiB
```



Diff:
diff --git a/dbms/src/IO/ReadHelpers.h b/dbms/src/IO/ReadHelpers.h
index 1983b60a685..e878065471a 100644
--- a/dbms/src/IO/ReadHelpers.h
+++ b/dbms/src/IO/ReadHelpers.h
@@ -34,7 +34,11 @@
 #include <iterator>
 #include <type_traits>
 
-#define DEFAULT_MAX_STRING_SIZE 0x00FFFFFFULL
+static constexpr UInt64 DEFAULT_MAX_STRING_SIZE = 0x00FFFFFFULL; // 16777215, 16MiB-1
+
+// According to `txn-entry-size-limit` in TiDB, the max size of a TiKV key/value is 120MB.
+// https://docs.pingcap.com/tidb/stable/tidb-configuration-file/#txn-entry-size-limit-new-in-v4010-and-v500
+static constexpr UInt64 TIKV_MAX_VALUE_SIZE = 125829120;
 
 namespace DB
 {
diff --git a/dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp b/dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp
index 3a81960df6b..14a110328de 100644
--- a/dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp
+++ b/dbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp
@@ -181,6 +181,8 @@ ColumnFilePersisteds deserializeSavedColumnFiles(
     {
         dtpb::DeltaLayerMeta meta;
         String data;
+        // Note: if the data is too large (DEFAULT_MAX_STRING_SIZE), this may cause exception when restore,
+        // but it is OK so far
         readStringBinary(data, buf);
         RUNTIME_CHECK_MSG(
             meta.ParseFromString(data),
@@ -217,6 +219,8 @@ ColumnFilePersisteds createColumnFilesFromCheckpoint( //
     {
         dtpb::DeltaLayerMeta meta;
         String data;
+        // Note: if the data is too large (DEFAULT_MAX_STRING_SIZE), this may cause exception when restore,
+        // but it is OK so far
         readStringBinary(data, buf);
         RUNTIME_CHECK_MSG(
             meta.ParseFromString(data),
diff --git a/dbms/src/Storages/KVStore/TiKVHelpers/TiKVKeyValue.h b/dbms/src/Storages/KVStore/TiKVHelpers/TiKVKeyValue.h
index be506f57650..36a3df466b1 100644
--- a/dbms/src/Storages/KVStore/TiKVHelpers/TiKVKeyValue.h
+++ b/dbms/src/Storages/KVStore/TiKVHelpers/TiKVKeyValue.h
@@ -32,7 +32,7 @@ struct StringObject : std::string
         : Base(std::move(str_))
     {}
     StringObject(StringObject && obj)
-        : Base((Base &&) obj)
+        : Base((Base &&)obj)
     {}
     StringObject(const char * str, const size_t len)
         : Base(str, len)
@@ -54,12 +54,13 @@ struct StringObject : std::string
         if (this == &a)
             return *this;
 
-        (Base &)* this = (Base &&) a;
+        (Base &)* this = (Base &&)a;
         return *this;
     }
 
     const std::string & getStr() const { return *this; }
     size_t dataSize() const { return Base::size(); }
+    size_t size() const = delete;
     std::string toString() const { return *this; }
 
     // Format as a hex string for debugging. The value will be converted to '?' if redact-log is on
@@ -67,15 +68,14 @@ struct StringObject : std::string
 
     explicit operator bool() const { return !empty(); }
 
-    size_t serialize(WriteBuffer & buf) const { return writeBinary2((const Base &)*this, buf); }
+    size_t serialize(WriteBuffer & buf) const { return writeBinary2(static_cast<const Base &>(*this), buf); }
 
-    static StringObject deserialize(ReadBuffer & buf) { return StringObject(readBinary2<Base>(buf)); }
+    static StringObject deserialize(ReadBuffer & buf) { return StringObject(readTiKVStringBinary(buf)); }
 
 private:
     StringObject(const Base & str_)
         : Base(str_)
     {}
-    size_t size() const = delete;
 };
 
 using TiKVKey = StringObject<true>;
@@ -90,4 +90,4 @@ struct hash<DB::StringObject<is_key>>
 {
     size_t operator()(const DB::StringObject<is_key> & k) const { return std::hash<std::string>()(k); }
 };
-} // namespace std
\ No newline at end of file
+} // namespace std
diff --git a/dbms/src/Storages/KVStore/Utils/SerializationHelper.h b/dbms/src/Storages/KVStore/Utils/SerializationHelper.h
index 2004c6320bd..a573c4bb2f2 100644
--- a/dbms/src/Storages/KVStore/Utils/SerializationHelper.h
+++ b/dbms/src/Storages/KVStore/Utils/SerializationHelper.h
@@ -60,6 +60,14 @@ inline size_t writeBinary2(const std::string & s, WriteBuffer & buf)
     return 4 + s.size();
 }
 
+inline std::string readStringWithLength(ReadBuffer & buf, size_t length)
+{
+    std::string s;
+    s.resize(length);
+    buf.readStrict(&s[0], length);
+    return s;
+}
+
 template <>
 inline std::string readBinary2<std::string>(ReadBuffer & buf)
 {
@@ -72,18 +80,23 @@ inline std::string readBinary2<std::string>(ReadBuffer & buf)
             "Too large string size, size={} max_size={}",
             size,
             DEFAULT_MAX_STRING_SIZE);
-    std::string s;
-    s.resize(size);
-    buf.readStrict(&s[0], size);
-    return s;
+
+    return readStringWithLength(buf, size);
 }
 
-inline std::string readStringWithLength(ReadBuffer & buf, size_t length)
+inline std::string readTiKVStringBinary(ReadBuffer & buf)
 {
-    std::string s;
-    s.resize(length);
-    buf.readStrict(&s[0], length);
-    return s;
+    UInt32 size = 0;
+    readIntBinary(size, buf);
+
+    if (size > TIKV_MAX_VALUE_SIZE)
+        throw Exception(
+            ErrorCodes::LOGICAL_ERROR,
+            "Too large tikv key/value size, size={} max_size={}",
+            size,
+            TIKV_MAX_VALUE_SIZE);
+
+    return readStringWithLength(buf, size);
 }
 
 size_t writeBinary2(const metapb::Peer & peer, WriteBuffer & buf);
diff --git a/dbms/src/Storages/KVStore/tests/gtest_region_persister.cpp b/dbms/src/Storages/KVStore/tests/gtest_region_persister.cpp
index 59698f77dd5..13c6d3e8650 100644
--- a/dbms/src/Storages/KVStore/tests/gtest_region_persister.cpp
+++ b/dbms/src/Storages/KVStore/tests/gtest_region_persister.cpp
@@ -257,24 +257,38 @@ try
 {
     TableID table_id = 100;
     auto region = makeTmpRegion();
+
     TiKVKey key = RecordKVFormat::genKey(table_id, 323, 9983);
     region->insertDebug("default", TiKVKey::copyFrom(key), TiKVValue("value1"));
     region->insertDebug("write", TiKVKey::copyFrom(key), RecordKVFormat::encodeWriteCfValue('P', 0));
     region->insertDebug("lock", TiKVKey::copyFrom(key), RecordKVFormat::encodeLockCfValue('P', "", 0, 0));
 
+    TiKVKey large_value_key = RecordKVFormat::genKey(table_id, 324, 9983);
+    region->insertDebug(
+        "default",
+        TiKVKey::copyFrom(large_value_key),
+        // slightly less than `TIKV_MAX_VALUE_SIZE` for other key-values
+        TiKVValue(String(static_cast<size_t>(TIKV_MAX_VALUE_SIZE - 1024), 'v')));
+
     region->updateRaftLogEagerIndex(1024);
 
-    const auto path = dir_path + "/region.test";
-    WriteBufferFromFile write_buf(path, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_CREAT);
-    size_t region_ser_size = std::get<0>(region->serialize(write_buf));
-    write_buf.next();
-    write_buf.sync();
-    ASSERT_EQ(region_ser_size, (size_t)Poco::File(path).getSize());
+    String serialized_str;
+    {
+        WriteBufferFromOwnString write_buf;
+        size_t region_ser_size = std::get<0>(region->serialize(write_buf));
+        write_buf.next();
+        write_buf.finalize();
+        serialized_str = write_buf.releaseStr();
+        ASSERT_EQ(region_ser_size, serialized_str.size());
+        LOG_INFO(Logger::get(), "region_ser_size={}", region_ser_size);
+    }
 
-    ReadBufferFromFile read_buf(path, DBMS_DEFAULT_BUFFER_SIZE, O_RDONLY);
-    auto new_region = Region::deserialize(read_buf);
-    ASSERT_REGION_EQ(*new_region, *region);
     {
+        ReadBufferFromString read_buf(serialized_str);
+        auto new_region = Region::deserialize(read_buf);
+        ASSERT_REGION_EQ(*new_region, *region);
+
+        // eager_truncated_index
         const auto & [eager_truncated_index, applied_index] = new_region->getRaftLogEagerGCRange();
         ASSERT_EQ(eager_truncated_index, 1024);
     }


Reviews:
user: login: CalvinNeo
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T02:42:56Z
user: login: JaySon-Huang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T09:34:21Z
user: login: JinheLin
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T09:36:33Z
user: login: Lloyd-Pottiger
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T09:42:25Z
user: login: CalvinNeo
body: lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-31T09:44:54Z

Review comments:
diff_hunk: @@ -72,18 +80,25 @@ inline std::string readBinary2<std::string>(ReadBuffer & buf)
             "Too large string size, size={} max_size={}",
             size,
             DEFAULT_MAX_STRING_SIZE);
-    std::string s;
-    s.resize(size);
-    buf.readStrict(&s[0], size);
-    return s;
+
+    return readStringWithLength(buf, size);
 }
 
-inline std::string readStringWithLength(ReadBuffer & buf, size_t length)
+inline std::string readTiKVStringBinary(ReadBuffer & buf)
 {
-    std::string s;
-    s.resize(length);
-    buf.readStrict(&s[0], length);
-    return s;
+    UInt32 size = 0;
+    readIntBinary(size, buf);
+
+    // According to `txn-entry-size-limit` in TiDB, the max size of a TiKV key/value is 120MB.
+    // https://docs.pingcap.com/tidb/stable/tidb-configuration-file/#txn-entry-size-limit-new-in-v4010-and-v500
path: dbms/src/Storages/KVStore/Utils/SerializationHelper.h
user: login: CalvinNeo
body: Maybe move this to where `TIKV_MAX_VALUE_SIZE` is defined?
created_at: 2025-03-31T02:42:56Z
author_association: MEMBER
diff_hunk: @@ -72,18 +80,25 @@ inline std::string readBinary2<std::string>(ReadBuffer & buf)
             "Too large string size, size={} max_size={}",
             size,
             DEFAULT_MAX_STRING_SIZE);
-    std::string s;
-    s.resize(size);
-    buf.readStrict(&s[0], size);
-    return s;
+
+    return readStringWithLength(buf, size);
 }
 
-inline std::string readStringWithLength(ReadBuffer & buf, size_t length)
+inline std::string readTiKVStringBinary(ReadBuffer & buf)
 {
-    std::string s;
-    s.resize(length);
-    buf.readStrict(&s[0], length);
-    return s;
+    UInt32 size = 0;
+    readIntBinary(size, buf);
+
+    // According to `txn-entry-size-limit` in TiDB, the max size of a TiKV key/value is 120MB.
+    // https://docs.pingcap.com/tidb/stable/tidb-configuration-file/#txn-entry-size-limit-new-in-v4010-and-v500
path: dbms/src/Storages/KVStore/Utils/SerializationHelper.h
user: login: JaySon-Huang
body: fixed
created_at: 2025-03-31T09:34:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: ti-chi-bot[bot]
created_at: 2025-03-31T09:42:32Z
body: [LGTM Timeline notifier]
---
Timeline:

- `2025-03-31 09:36:39.230697813 +0000 UTC m=+1471492.914933908`: :ballot_box_with_check: agreed by [JinheLin](https://github.com/JinheLin).
- `2025-03-31 09:42:31.666629692 +0000 UTC m=+1471845.350865788`: :ballot_box_with_check: agreed by [Lloyd-Pottiger](https://github.com/Lloyd-Pottiger).
author_association: CONTRIBUTOR
user: login: ti-chi-bot[bot]
created_at: 2025-03-31T09:44:58Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/pingcap/tiflash/pull/10055#pullrequestreview-2728752627" title="Approved">CalvinNeo</a>*, *<a href="https://github.com/pingcap/tiflash/pull/10055#pullrequestreview-2728725403" title="Approved">JinheLin</a>*, *<a href="https://github.com/pingcap/tiflash/pull/10055#pullrequestreview-2728742332" title="Approved">Lloyd-Pottiger</a>*

The full list of commands accepted by this bot can be found [here](https://prow.tidb.net/command-help?repo=pingcap%2Ftiflash).

The pull request process is described [here](https://book.prow.tidb.net/#/workflows/pr)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/pingcap/tiflash/blob/master/OWNERS)~~ [CalvinNeo,JinheLin,Lloyd-Pottiger]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: ti-chi-bot
created_at: 2025-04-01T05:34:35Z
body: In response to a cherrypick label: new pull request created to branch `release-8.5`: #10067.
author_association: MEMBER
user: login: ti-chi-bot
created_at: 2025-04-01T05:34:41Z
body: In response to a cherrypick label: new pull request created to branch `release-8.1`: #10068.
But this PR has conflicts, please resolve them!
author_association: MEMBER
user: login: ti-chi-bot
created_at: 2025-04-01T05:34:46Z
body: In response to a cherrypick label: new pull request created to branch `release-7.5`: #10069.
But this PR has conflicts, please resolve them!
author_association: MEMBER