Project: GreptimeDB (App Definition and Development / Database)
File: 2023-07-06-table-engine-refactor.md

Feature Name: table-engine-refactor Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/1869 Date: 2023-07-06 Author: "Yingwen <realevenyag@gmail.com>" Refactor Table Engine Refactor table engines to address several historical tech debts. Both `Frontend` and `Datanode` have to deal with multiple regions in a table. This results in code duplication and additional burden to the `Datanode`. Before: ```mermaid graph TB subgraph Frontend["Frontend"] subgraph MyTable A("region 0, 2 -> Datanode0") B("region 1, 3 -> Datanode1") end end MyTable --> Metasrv Metasrv --> ETCD MyTable-->TableEngine0 MyTable-->TableEngine1 subgraph Datanode0 Procedure0("procedure") TableEngine0("table engine") region0 region2 mytable0("my_table") Procedure0-->mytable0 TableEngine0-->mytable0 mytable0-->region0 mytable0-->region2 end subgraph Datanode1 Procedure1("procedure") TableEngine1("table engine") region1 region3 mytable1("my_table") Procedure1-->mytable1 TableEngine1-->mytable1 mytable1-->region1 mytable1-->region3 end subgraph manifest["table manifest"] M0("my_table") M1("regions: [0, 1, 2, 3]") end mytable1-->manifest mytable0-->manifest RegionManifest0("region manifest 0") RegionManifest1("region manifest 1") RegionManifest2("region manifest 2") RegionManifest3("region manifest 3") region0-->RegionManifest0 region1-->RegionManifest1 region2-->RegionManifest2 region3-->RegionManifest3 ``` `Datanodes` can update the same manifest file for a table as regions are assigned to different nodes in the cluster. We also have to run procedures on `Datanode` to ensure the table manifest is consistent with region manifests. "Table" in a `Datanode` is a subset of the table's regions. The `Datanode` is much closer to `RegionServer` in `HBase` which only deals with regions. In cluster mode, we store table metadata in etcd and table manifest. The table manifest becomes redundant. We can remove the table manifest if we refactor the table engines to region engines that only care about regions. What's more, we don't need to run those procedures on `Datanode`. After: ```mermaid graph TB subgraph Frontend["Frontend"] direction LR subgraph MyTable A("region 0, 2 -> Datanode0") B("region 1, 3 -> Datanode1") end end MyTable --> Metasrv Metasrv --> ETCD MyTable-->RegionEngine MyTable-->RegionEngine1 subgraph Datanode0 RegionEngine("region engine") region0 region2 RegionEngine-->region0 RegionEngine-->region2 end subgraph Datanode1 RegionEngine1("region engine") region1 region3 RegionEngine1-->region1 RegionEngine1-->region3 end RegionManifest0("region manifest 0") RegionManifest1("region manifest 1") RegionManifest2("region manifest 2") RegionManifest3("region manifest 3") region0-->RegionManifest0 region1-->RegionManifest1 region2-->RegionManifest2 region3-->RegionManifest3 ``` This RFC proposes to refactor table engines into region engines as a first step to make the `Datanode` acts like a `RegionServer`. We plan to refactor the `TableEngine` trait into `RegionEngine` gradually. This RFC focuses on the `mito` engine as it is the default table engine and the most complicated engine. Currently, we built `MitoEngine` upon `StorageEngine` that manages regions of the `mito`

Since `MitoEngine` becomes a region engine, we could combine `StorageEngine` with `MitoEngine` to simplify our code structure. The chart below shows the overall architecture of the `MitoEngine`. ```mermaid classDiagram class MitoEngine~LogStore~ { -WorkerGroup workers } class MitoRegion { +VersionControlRef version_control -RegionId region_id -String manifest_dir -AtomicI64 lastflushmillis +region_id() RegionId +scan() ChunkReaderImpl } class RegionMap { -HashMap&lt;RegionId, MitoRegionRef&gt; regions } class ChunkReaderImpl class WorkerGroup { -Vec~RegionWorker~ workers } class RegionWorker { -RegionMap regions -Sender sender -JoinHandle handle } class RegionWorkerThread~LogStore~ { -RegionMap regions -Receiver receiver -Wal~LogStore~ wal -ObjectStore object_store -MemtableBuilderRef memtable_builder -FlushSchedulerRef~LogStore~ flush_scheduler -FlushStrategy flush_strategy -CompactionSchedulerRef~LogStore~ compaction_scheduler -FilePurgerRef file_purger } class Wal~LogStore~ { -LogStore log_store } class MitoConfig MitoEngine~LogStore~ o-- MitoConfig MitoEngine~LogStore~ o-- MitoRegion MitoEngine~LogStore~ o-- WorkerGroup MitoRegion o-- VersionControl MitoRegion -- ChunkReaderImpl WorkerGroup o-- RegionWorker RegionWorker o-- RegionMap RegionWorker -- RegionWorkerThread~LogStore~ RegionWorkerThread~LogStore~ o-- RegionMap RegionWorkerThread~LogStore~ o-- Wal~LogStore~ ``` We replace the `RegionWriter` with `RegionWorker` to process write requests and DDL requests. We also merge region's metadata with table's metadata. It should make metadata much easier to maintain. ```mermaid classDiagram class VersionControl { -CowCell~Version~ version -AtomicU64 committed_sequence } class Version { -RegionMetadataRef metadata -MemtableVersionRef memtables -LevelMetasRef ssts -SequenceNumber flushed_sequence -ManifestVersion manifest_version } class MemtableVersion { -MemtableRef mutable -Vec~MemtableRef~ immutables +mutable_memtable() MemtableRef +immutable_memtables() &[MemtableRef] +freezemutable(MemtableRef newmutable) MemtableVersion } class LevelMetas { -LevelMetaVec levels -AccessLayerRef sst_layer -FilePurgerRef file_purger -Option~i64~ compactiontimewindow } class LevelMeta { -Level level -HashMap&lt;FileId, FileHandle&gt; files } class FileHandle { -FileMeta meta -bool compacting -AtomicBool deleted -AccessLayerRef sst_layer -FilePurgerRef file_purger } class FileMeta { +RegionId region_id +FileId file_id +Option&lt;Timestamp, Timestamp&gt; time_range +Level level +u64 file_size } VersionControl o-- Version Version o-- RegionMetadata Version o-- MemtableVersion Version o-- LevelMetas LevelMetas o-- LevelMeta LevelMeta o-- FileHandle FileHandle o-- FileMeta class RegionMetadata { +RegionId region_id +VersionNumber version +SchemaRef table_schema +Vec~usize~ primarykeyindices +Vec~usize~ value_indices +ColumnId nextcolumnid +TableOptions region_options +DateTime~Utc~ created_on +RegionSchemaRef region_schema } class RegionSchema { -SchemaRef user_schema -StoreSchemaRef store_schema -ColumnsMetadataRef columns } class Schema class StoreSchema { -Vec~ColumnMetadata~ columns -SchemaRef schema -usize rowkeyend -usize usercolumnend } class ColumnsMetadata { -Vec~ColumnMetadata~ columns -HashMap&lt;String, usize&gt; nametocol_index -usize rowkeyend -usize timestampkeyindex -usize usercolumnend } class ColumnMetadata RegionMetadata o-- RegionSchema RegionMetadata o-- Schema RegionSchema o-- StoreSchema RegionSchema o-- Schema RegionSchema o-- ColumnsMetadata StoreSchema o-- ColumnsMetadata StoreSchema o-- Schema StoreSchema o-- ColumnMetadata ColumnsMetadata o-- ColumnMetadata ``` This is a breaking change. Rename `TableEngine` to `RegionEngine` Simplify schema relationship in the `mito` engine Refactor the `Datanode` into a `RegionServer`.