Project: YDB (App Definition and Development / Database)
File: ROADMAP.md

We use the following symbols as abbreviations: - feature appeared in the Roadmap for 2023; - feature appeared in the Roadmap for 2024; - feature has been released; - feature is partially available and is under development; - feature has been refused; - not yet released, but we are in rush. Unique secondary indexes Apply indexes automatically to optimize data fetching Default values for table columns Asynchronous LLVM JIT query compilation Parameters in DECLARE clause are becoming optional, better SQL compatibility Cost-based optimizer for join order and join algorithm selection ``INSERT INTO table FROM SELECT`` for large datasets Support for transactional writes into both row and column tables Support for computed columns in a table Support for temporary tables Support for VIEW SQL clause Data Spilling in case there is issufient amount of RAM TPC-H, TPC-DS for 100TB dataset Support for Snapshot Readonly transactions mode Better resource management for KQP Resource Manager (share information about nodes resources, avoid OOMs) Switch to New Engine for OLTP queries Support `not null` for PK (primary key) table columns Aggregates and predicates push down to column-oriented tables Optimize data formats for data transition between query phases Index Rename/Rebuild KQP Session Actor as a replacement for KQP Worker Actor (optimize to reduce CPU usage) PostgreSQL compatibility Support PostgreSQL datatypes serialization/deserialization* in YDB Public API PostgreSQL compatible query execution* (TPC-C, TPC-H queries should work) Support for PostgreSQL wire protocol* Support a single Database connection string instead of multiple parameters Support constraints in query optimizer Query Processor 3.0 (a set of tasks to be more like traditional database in case of query execution functionality) Support for Streaming Lookup Join* via MVCC snapshots (avoid distributed transactions, scalability is better) Universal API call for DML, DDL with unlimited results size for OLTP/OLAP workload* (aka ExecuteQuery) Support for secondary indexes in ScanQuery* Transaction can see its own updates* (updates made during transaction execution are not buffered in RAM anymore, but rather are written to disk and available to read by this transaction) Computation graphs caching (compute/datashard programs) (optimize CPU usage) RPC Deadline & Cancellation propagation (smooth timeout management) DDL for column-oriented tables Volatile transactions. YDB Distributed transactions

minimize network round trips in happy path Table statistics for cost-based optimizer Memory optimization for row tables (avoid full SST index loading, dynamic cache adjusting) Reduce minimum requirements for the number of cores to 2 for YDB node Incremental backup and Point-in-time recovery ``ALTER CHANGEFEED`` Async Replication between YDB databases (column tables, topics) Async Replication between YDB databases (schema changes) Support for Debezium format Topics autoscaling (increase/decrease number of partitions in the topic automatically) Extended Kafka API protocol to YDB Topics support (balance reads, support for v19) Schema for YDB Topics Message-level parallelism in YDB Topics Get YDB topics (aka pers queue, streams) ready for production Turn on MVCC support by default Enable Snapshot read mode by default (take and use MVCC snapshot for reads instead of running distributed transaction for reads) Change Data Capture (be able to get change feed of table updates) Async Replication between YDB databases (first version, row tables, w/o schema changes) Background compaction for DataShards Compressed Backups. Add functionality to compress backup data Process of Extending State Storage without cluster downtime. If a cluster grows from, say, 9 nodes to 900 State Storage configuration stays the same (9 nodes), it leads to a performance bottleneck. Split/Merge DataShards *BY LOAD* by default. Most users require this feature turned on by default Support PostgreSQL datatypes in tablet local database Basic histogram for DataShards (first step towards cost based optimizations) Transaction can see its own updates (updates made during transaction execution are not buffered in RAM anymore, but rather are written to disk and available to read by this transaction) Data Ingestion from topic to table (implement built-in compatibility to ingest data to YDB tables from topics) Support snapshot read over read replicas (consistent reads against read replicas) Transactions between topics and tables Support for Kafka API compatible protocol to YDB Topics Tracing capabilities Automatically balance tablet channels via BlobStorage groups Datashard iterator reads via MVCC (refused) Switch to TRope (or don't use TString/std::string directly, provide zero-copy data passing between components) Avoid Node Broker as SPF (NBS must work without Node Broker under emergency conditions) Subscriptions in SchemeBoard (optimize interaction with SchemeBoard via subsription to updates) Basic LDAP Support Support for OpenID Connect Authentication via KeyCloack Support for SASL framework BlobStorage latency optimization (p999), less CPU consumption ActorSystem performance optimizations Optimize ActorSystem for ARM processors Effortless initial cluster deployment (provide only nodes and disks description) Reduce number of BlobStorage groups for a database (add ability to remove unneeded groups) "One leg" storage migration without downtime (migrate 1/3 of the cluster from one AZ to another for mirror3-dc erasure encoding) ActorSystem 1.5 (dynamically reassign threads in different thread pools) Publish an utility for BlobStorage management (it's called ds_tool for now, improve it and open) Self-heal for degrated BlobStorage groups (automatic self-heal for groups with two broken disks, get VDisk Donors production ready) BlobDepot (a component for smooth blobs management between groups) Avoid BSC (BlobStorage Controller) as SPF (be able to run the cluster without BSC in emergency cases) BSC manages static group (reconfiguration of the static BlobStorage group must be done BlobStorage Controller as for any other group) (Semi-)Hard disk space separation (Better guarantees for disk space usage by VDisks on a single PDisk) Reduce space amplification (Optimize storage layer) Storage nodes decommission (Add ability to remove storage nodes) Backup for column tables Column tables autosharding Log Store (log friendly column-oriented storage which allows to create 1+ million tables for logs storing) Column-oriented Tables (introduce a Column-oriented tables in additon to Row-orinted tables) Tiered Storage for Column-oriented Tables (with the ability to store the data in S3) Run the