Project: StarRocks (App Definition and Development / Database)
File: release-2.3.md

displayed_sidebar: "English" Release date: October 11, 2023 Fixed the following issues: The bug in the third-party library librdkafka causes the load tasks of Routine Load jobs to get stuck during data loading, and newly created load tasks also fail to execute. Spark or Flink connectors may fail to export data because of inaccurate memory statistics. BEs crash when a Stream Load job uses the keyword `if`. An error `"get TableMeta failed from TNetworkAddress"` is reported when data is loaded into partitioned StarRocks external tables. Release date: September 4, 2023 Fixed the following issue: A Routine Load job fails to consume data. Release date: August 4, 2023 Fixed the following issue: FE memory leak caused by blocked LabelCleaner threads. Release date: July 31, 2023 Optimized tablet scheduling logic to prevent tablets remaining pending for a long period or an FE crashing under certain circumstances. Optimized the scheduling logic of TabletChecker to prevent the checker from repeatedly scheduling tablets that are not repaired. The partition metadata records visibleTxnId, which corresponds to the visible version of tablet replicas. When the version of a replica is inconsistent with others, it is easier to trace the transaction that created this version. Fixed the following issues: Incorrect table-level scan statistics in the FEs cause inaccurate metrics related to table queries and loading. BEs may crash if the Join key is a large BINARY column. An aggregate operator may trigger thread safety issues in certain scenarios, causing BEs to crash. The version number for a tablet is inconsistent between the BE and FE after data is restored by using . Partitions cannot be automatically created after the table is recovered by using . The loading transaction is stuck in the Pending state and DDL statements are hung if data to be loaded using INSERT INTO does not meet quality requirements and the strict mode is enabled for data loading. Some INSERT jobs return ` In certain cases, the INSERT INTO SELECT operation times out when the Pipeline is not enabled. The query returns no data when the query condition is `WHERE partition_column < xxx` and the value in `xxx` is only accurate to the hour, not to minute and second, for example, `2023-7-21 22`. Release date: June 28, 2023 Optimized the error message returned when CREATE TABLE times out and added parameter tuning tips. Optimized the memory usage for the Primary Key tables with a large number of accumulated tablet versions. The synchronization of StarRocks external table metadata has been changed to occur during data loading. Removed the dependency of NetworkTime on system clocks to fix incorrect NetworkTime caused by inconsistent system clocks across servers. Fixed the following issues: When low cardinality dictionary optimization is applied for small tables that undergo frequent TRUNCATE operations, queries may encounter errors. BEs may crash when a view that contains UNION whose first child is a constant NULL is queried. In some cases, a query based on Bitmap Index may return an error. The result of rounding a DOUBLE or FLOAT value into a DECIMAL value in BEs is inconsistent with the result in FEs. A schema change sometimes may be hung if data loading occurs simultaneously with the schema

When you load a Parquet file into StarRocks by using Broker Load, Spark connector or Flink connector, BE OOM issues may occur. The error message `unknown error` is returned when a constant is specified in the ORDER BY clause and there is a LIMIT clause in the query. Release date: June 1, 2023 Optimized the error message reported when INSERT INTO ... SELECT expires due to a small `thriftservermaxworkerthread` value. Reduced memory consumption and optimized performance for multi-table joins that use the `bitmap_contains` function. The following bugs are fixed: Truncating partitions fails because the TRUNCATE operation is case-sensitive to partition names. Loading int96 timestamp data from Parquet files causes data overflow. Decommissioning a BE fails after a materialized view is dropped. When the execution plan for a query includes a Broadcast Join followed by a Bucket Shuffle Join, such as `SELECT * FROM t1 JOIN When the execution plan for a query includes a Cross Join followed by a Hash Join, and the right table of the Hash Join within a fragment instance is empty, the returned result may be incorrect. Decommissioning a BE fails due to failures in creating temporary partitions for materialized views. If a SQL statement contains a STRING value with multiple escape characters, the SQL statement cannot be parsed. Querying the data with the maximum value in the partition column fails. Load jobs fail after StarRocks rolls back from v2.4 to v2.3. Issues related to column pruning and reuse. Release date: April 25, 2023 Supports implicit conversion if the returned value of an expression can be converted to a valid Boolean value. The following bugs are fixed: If a user's LOAD_PRIV is granted at the table level, an error message `Access denied; you need (at least one of) the LOAD privilege(s) for this operation` is returned at transaction rollback in the event of a load job failure. After ALTER SYSTEM DROP BACKEND is executed to drop a BE, the replicas of tables whose replication number is set to 2 on that BE cannot be repaired. In this situation, data loads into these tables fail. NPE is returned when an unsupported data type is used in CREATE TABLE. The shortcircut logic of the Broadcast Join is abnormal, leading to incorrect query results. Disk usage may increase significantly after materialized views are used. The Audit Loader plugin cannot be completely uninstalled. The number of rows displayed in the result of `INSERT INTO XXX SELECT` may not match the result of `SELECT COUNT(*) FROM XXX`. If a subquery uses window functions and its parent query uses the GROUP BY clause, the query result cannot be aggregated. When a BE is started, the BE process exists but all the BEs ports cannot be open. If the disk I/O is exceedingly high, transactions on Primary Key tables are slowly committed, and consequently queries on these tables may return an error "backend not found". Release date: March 28, 2023 Executing complex queries that contain many expressions usually generates a large number of `ColumnRefOperators`. Originally, StarRocks uses `BitSet` to store `ColumnRefOperator::id`, which consumes a large amount of memory. In order to reduce memory usage, StarRocks now uses `RoaringBitMap` to store

A new I/O scheduling strategy is introduced to reduce the performance impact of large queries on small queries. To enable the new I/O scheduling strategy, configure the BE static parameter `pipelinescanqueue_mode=1` in be.conf and then restart BEs. The following bugs are fixed: A table whose expired data is not properly recycled occupies a relatively large portion of disk space. The error message displayed in the following scenario is not informative: A Broker Load job loads Parquet files into StarRocks and a `NULL` value is loaded into a NOT NULL column. Frequently creating a large number of temporary partitions to replace existing partitions leads to memory leaks and Full GC on the FE nodes. For Colocation tables, the replica status can be manually specified as `bad` by using statements like `ADMIN SET REPLICA STATUS PROPERTIES ("tabletid" = "10003", "backendid" = "10001", "status" = "bad");`. If the number of BEs is less than or equal to the number of replicas, the corrupted replica cannot be repaired. When the request `INSERT INTO SELECT` is sent to a Follower FE, the parameter `parallelfragmentexecinstancenum` does not take effect. When the operator `<=>` is used to compare a value with a `NULL` value, the comparison result is incorrect. The query concurrency metric decreases slowly when the concurrency limit of a resource group is continuously reached. Highly concurrent data load jobs may cause the error `"get database read lock timeout, database=xxx"`. Release date: March 9, 2023 Optimized the inference of `storagemedium`. When BEs use both SSD and HDD as storage devices, if the property `storagecooldowntime` is specified, StarRocks sets `storagemedium` to `SSD`. Otherwise, StarRocks sets `storage_medium` to `HDD`. The following bugs are fixed: A query may fail if ARRAY data from Parquet files in data lakes is queried. The Stream Load job initiated by a program is hung and the FE does not receive the HTTP request sent by the program. An error may occur when an Elasticsearch external table is queried. BEs may crash if an expression encounters an error during initialization. A query may fail if the SQL statement uses an empty array literal ` After StarRocks is upgraded from version 2.2 and later to version 2.3.9 and later, an error `No match for <expr> with operand types xxx and xxx` may occur when a Routine Load job is created with a calculation expression specified in the `COLUMN` parameter. A load job is hung after a BE restarts. When a SELECT statement uses an OR operator in the WHERE clause, extra partitions are scanned. Release date: February 20, 2023 During a schema change, if a tablet clone is triggered and the BE nodes on which the tablet replicas reside change, the schema change fails. The string returned by the group_concat() function is truncated. When you use Broker Load to load data from HDFS through Tencent Big Data Suite (TBDS), an error `invalid hadoop.security.authentication.tbds.securekey` occurs, indicating that StarrRocks cannot access HDFS by using the authentication information provided by TBDS. In some cases, CBO may use incorrect logic to compare whether two operators are equivalent. When you connect to a non-Leader FE node and send the SQL statement `USE

the non-Leader FE node forwards the SQL statement, with `<catalogname>` excluded, to the Leader FE node. As a result, the Leader FE node chooses to use the `defaultcatalog` and eventually fails to find the specified database. Release date: February 2, 2023 The following bugs are fixed: When resources are released after a large query finishes, there is a low probability that other queries are slowed down. This issue is more likely to occur if resource groups are enabled or the large query ends unexpectedly. For a Primary Key table, if a replica's metadata version falls behind, StarRocks incrementally clones the missing metadata from other replicas to this replica. In this process, StarRocks pulls a large number of versions of metadata, and if too many versions of metadata accumulate without timely GC, excessive memory may be consumed and consequently the BEs may encounter OOM exceptions. If an FE sends an occasional heartbeat to a BE, and the heartbeat connection times out, the FE considers the BE unavailable, leading to transaction failures on the BE. When you use a StarRocks external table to load data between StarRocks clusters, if the source StarRocks cluster is in an earlier version and the target StarRocks cluster is in a later version (2.2.8 ~ 2.2.11, 2.3.4 ~ 2.3.7, 2.4.1 or 2.4.2), the data loading fails. BEs crash when multiple queries run concurrently and memory usage is relatively high. When dynamic partitioning is enabled for a table and some partitions are dynamically deleted, if you execute TRUNCATE TABLE, an error `NullPointerException` is returned. Meanwhile, if you load data into the table, the FEs crash and can not restart. Release date: December 30, 2022 The following bugs are fixed: The column that is allowed to be NULL in a StarRocks table is incorrectly set to NOT NULL in a view created from that table. A new tablet version is generated when data is loaded into StarRocks. However, the FE may not yet detect the new tablet version and still requires BEs to read the historical version of the tablet. If the garbage collection mechanism removes the historical version, the query cannot find the historical version and an error "Not found: getappliedrowsets(version xxxx) failed tablet:xxx #version:x FE takes up too much memory when data is frequently loaded. For aggregate queries and multi-table JOIN queries, the statistics are not collected accurately and CROSS JOIN occurs in the execution plans, resulting in long query latency. Release date: December 22, 2022 The Pipeline execution engine supports INSERT INTO statements. To enable it, set the FE configuration item `enablepipelineloadforinsert` to `true`. The memory used by Compaction for the Primary Key table is reduced. Deprecated the FE parameter `defaultstoragemedium`. The storage medium of a table is automatically inferred by the system. The following bugs are fixed: BEs may hang up when the resource group feature is enabled and multiple resource groups run queries at the same time. When you create a materialized view by using CREATE MATERIALIZED VIEW AS SELECT, if the SELECT clause does not use aggregate functions, and uses GROUP BY, for example `CREATE MATERIALIZED VIEW test_view AS SELECT a,b from test group by b,a order by a;`, then the BE nodes all

If you restart the BE immediately after you use INSERT INTO to frequently load data into the Primary Key table to make data changes, the BE may restart very slowly. If only JRE is installed on the environment and JDK is not installed, queries fail after FE restarts. After the bug is fixed, FE cannot restart in that environment and it returns error `JAVA_HOME can not be jre`. To successfully restart FE, you need to install JDK on the environment. Queries cause BE crashes. `execmemlimit` cannot be set to an expression. You cannot create a sync refreshed materialized view based on subquery results. The comments for columns are deleted after you refresh the Hive external table. During a correlated JOIN, the right table is processed before the left table and the right table is very large. If compaction is performed on the left table while the right table is being processed, the BE node crashes. If the Parquet file column names are case-sensitive, and the query condition uses upper-case column names from the Parquet file, the query returns no result. During bulk loading, if the number of connections to Broker exceeds the default maximum number of connections, Broker is disconnected and the loading job fails with an error message `list path error`. When BEs are highly loaded, the metric for resource groups `starrocksberesourcegrouprunning_queries` may be incorrect. If the query statement uses OUTER JOIN, it may cause the BE node to crash. After you create an asynchronous materialized view by using StarRocks 2.4, and you roll back it to 2.3, you may find FE fails to start. When the Primary Key table uses delete_range, and the performance is not good, it may slow down data reading from RocksDB and cause high CPU usage. Release date: November 30, 2022 Colocate Join supports Equi Join. Fix the problem that primary key index files are too large due to continuously appending WAL records when data is frequently loaded. FE scans all tablets in batches so that FE releases db.readLock at scanning intervals in case of holding db.readLock for too long. The following bugs are fixed: When a view is created based directly on the result of UNION ALL, and the UNION ALL operator's input columns include NULL values, the schema of the view is incorrect since the data type of columns is NULL_TYPE rather than UNION ALL's input columns. The query result of `SELECT FROM ...` and `SELECT FROM ... LIMIT ...` is inconsistent. External tablet metadata synchronized to FE may overwrite local tablet metadata, which causes data loading from Flink to fail. BE nodes crash when null filter in Runtime Filter handles literal constants. An error is returned when you execute CTAS. The metrics `ScanRows` collected by pipeline engine in audit log may be wrong. The query result is incorrect when you query compressed HIVE data. Queries are timeout and StarRocks responds slowly after a BE node crashes. The error of Kerberos authentication failure occurs when you use Broker Load to load data. Too many OR predicates cause statistics estimation to take too long. BE node crashes if Broker Load loads ORC files (Snappy compression) contain uppercase column

An error is returned when unloading or querying Primary Key table takes more than 30 minutes. The backup task fails when you back up large data volumes to HDFS by using a broker. The data StarRocks read from Iceberg may be incorrect, which is caused by the `parquetlatematerialization_enable` parameter. An error `failed to init view stmt` is returned when a view is created. An error is returned when you use JDBC to connect StarRock and execute SQL statements. The query is timeout because the query involves too many buckets and uses tablet hint. A BE node crashes and cannot be restarted, and in the meantime, the loading job into a newly built table reports an error. All BE nodes crash when a materialized view is created. When you execute ALTER ROUTINE LOAD to update the offset of consumed partitions, an error `The specified partition 1 is not in the consumed partitions`may be returned, and followers eventually crash. Release date: November 10, 2022 The error message provides a solution when StarRocks fails to create a Routine Load job because the number of running Routine Load job exceeds the limit. The query fails when StarRocks queries data from Hive and fails to parse CSV files. The following bugs are fixed: The query may fail if HDFS files paths contain `()`. The result of ORDER BY ... LIMIT ... OFFSET is incorrect when the subquery contains LIMIT. StarRocks is case-insensitive when querying ORC files. BE may crash when RuntimeFilter is closed without invoking the prepare method. BE may crash because of memory leak. The query result may be incorrect after you add a new column and immediately delete data. BE may crash because of sorting data. If StarRocks and MySQL client are not on the same LAN, the loading job created by using INSERT INTO SELECT can not be terminated successfully by executing KILL only once. The metrics `ScanRows` collected by pipeline engine in audit log may be wrong. Release date: September 27, 2022 The following bugs are fixed: Query result may be inaccurate when you query an Hive external table stored as a text file. Nested arrays are not supported when you query Parquet files. Queries or a query may time out if concurrent queries that read data from StarRocks and external data sources are routed to the same resource group, or a query reads data from StarRocks and external data sources. When the Pipeline execution engine is enabled by default, the parameter parallelfragmentexecinstancenum is changed to 1. It will cause data loading by using INSERT INTO to be slow. BE may crash if there are mistakes when a expression is initialized. The error heap-buffer-overflow may occur if you execute ORDER BY LIMIT. Schema change fails if you restart Leader FE in the meantime. Release date: September 7, 2022 Late materialization is supported to accelerate range filter-based queries on external tables in Parquet format. The SHOW AUTHENTICATION statement is added to display user authentication-related information. A configuration item is provided to control whether StarRocks recursively traverses all data files for the bucketed Hive table from which StarRocks queries data. The resource group type `realtime` is renamed as `short_query`. StarRocks no longer distinguishes between uppercase letters and lowercase letters in Hive external tables by

The following bugs are fixed: Queries on an Elasticsearch external table may unexpectedly exit when the table is divided into multiple shards. StarRocks throws errors when sub-queries are rewritten as common table expressions (CTEs). StarRocks throws errors when a large amount of data is loaded. When the same Thrift service IP address is configured for multiple catalogs, deleting one catalog invalidates the incremental metadata updates in the other catalogs. The statistics of memory consumption from BEs are inaccurate. StarRocks throws errors for queries on Primary Key tables. Queries on logical views are not allowed even when you have SELECT permissions on these views. StarRocks does not impose limits on the naming of logical views. Now logical views need to follow the same naming conventions as tables. Add BE configuration `maxlengthforbitmapfunction` with a default value 1000000 for bitmap function, and add `maxlengthfortobase64` with a default value 200000 for base64 to prevent crash. Release date: August 22, 2022 Broker Load supports transforming the List type in Parquet files into non-nested ARRAY data type. Optimized the performance of JSON-related functions (jsonquery, getjsonstring, and getjson_int). Optimized the error message: During a query on Hive, Iceberg, or Hudi, if the data type of the column to query is not supported by StarRocks, the system throws an exception on the column. Reduced the scheduling latency of resource groups to optimize resource isolation performance. The following bugs are fixed: Wrong result is returned from the query on Elasticsearch external tables due to incorrect pushdown of the `limit` operator. Query on Oracle external tables fails when the `limit` operator is used. BE is blocked when all Kafka Brokers are stopped during a Routine Load. BE crashes during a query on a Parquet file whose data type mismatches that of the corresponding external table. Query times out because the scan range of external tables is empty. The system throws an exception when the ORDER BY clause is included in a sub-query. Hive Metastore hangs when Hive metadata is reloaded asynchronously. Release date: July 29, 2022 The Primary Key table supports complete DELETE WHERE syntax. For more information, see . The Primary Key table supports persistent primary key indexes. You can choose to persist the primary key index on disk rather than in memory, significantly reducing memory usage. For more information, see . Global dictionary can be updated during real-time data ingestion, optimizing query performance and delivering 2X query performance for string data. The CREATE TABLE AS SELECT statement can be executed asynchronously. For more information, see . Support the following resource group-related features: Monitor resource groups: You can view the resource group of the query in the audit log and obtain the metrics of the resource group by calling APIs. For more information, see . Limit the consumption of large queries on CPU, memory, and I/O resources: You can route queries to specific resource groups based on the classifiers or by configuring session variables. For more information, see . JDBC external tables can be used to conveniently query data in Oracle, PostgreSQL, MySQL, SQLServer, ClickHouse, and other databases. StarRocks also supports predicate pushdown, improving query performance. For more information, see .

Added the following functions: - , , , The compaction mechanism can merge large volume of metadata more quickly. This prevents metadata squeezing and excessive disk usage that can occur shortly after frequent data updates. Optimized the performance of loading Parquet files and compressed files. Optimized the mechanism of creating materialized views. After the optimization, materialized views can be created at a speed up to 10 times faster than before. Optimized the performance of the following operators: TopN and sort operators Equivalence comparison operators that contain functions can use Zone Map indexes when these operators are pushed down to scan operators. Optimized Apache Hive external tables. When Apache Hive tables are stored in Parquet, ORC, or CSV format, schema changes caused by ADD COLUMN or REPLACE COLUMN on Hive can be synchronized to StarRocks when you execute the REFRESH statement on the corresponding Hive external table. For more information, see . `hive.metastore.uris` can be modified for Hive resources. For more information, see . Optimized the performance of Apache Iceberg external tables. A custom catalog can be used to create an Iceberg resource. For more information, see . Optimized the performance of Elasticsearch external tables. Sniffing the addresses of the data nodes in an Elasticsearch cluster can be disabled. For more information, see . When the sum() function accepts a numeric string, it implicitly converts the numeric string. The year(), month(), and day() functions support the DATE data type. Fixed the following bugs: CPU utilization surges due to an excessive number of tablets. Issues that cause "fail to prepare tablet reader" to occur. The FEs fail to restart. The CTAS statement cannot be run successfully when the statement includes a JSON function. StarGo, a cluster management tool, can deploy, start, upgrade, and roll back clusters and manage multiple clusters. For more information, see . The pipeline engine is enabled by default when you upgrade StarRocks to version 2.3 or deploy StarRocks. The pipeline engine can improve the performance of simple queries in high concurrency scenarios and complex queries. If you detect significant performance regressions when using StarRocks 2.3, you can disable the pipeline engine by executing the `SET GLOBAL` statement to set `enablepipelineengine` to `false`. The statement is compatible with the MySQL syntax and displays the privileges assigned to a user in the form of GRANT statements. It is recommended that the memorylimitationperthreadforschemachange ( BE configuration item) use the default value 2 GB, and data is written to disk when data volume exceeds this limit. Therefore, if you have previously set this parameter to a larger value, it is recommended that you set it to 2 GB, otherwise a schema change task may take up a large amount of memory. To roll back to the previous version that was used before the upgrade, add the `ignoreunknownlogid` parameter to the **fe.conf** file of each FE and set the parameter to `true`. The parameter is required because new types of logs are added in StarRocks v2.2.0. If you do not add the parameter, you cannot roll back to the previous version. We recommend that you set the `ignoreunknownlogid` parameter to `false` in the fe.conf file of each FE after checkpoints are created. Then, restart the FEs to restore the FEs to the previous configurations.