Project: StarRocks (App Definition and Development / Database)
File: release-3.2.md

displayed_sidebar: "English" Release date: April 18, 2024 Fixed the following issue: The privileges of external tables cannot be found due to incompatibility issues. Release date: April 12, 2024 :::tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. Problem: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. Impact scope: This problem only affects queries on external tables in external catalogs. Other queries are not affected. Temporary workaround: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ::: Supports the column property, which can significantly facilitate the loading process during the construction of a global dictionary, accelerating the exact COUNT DISTINCT calculation. When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). Optimized the column type unionization rules for automatic schema detection in the FILES table function. When columns with the same name but different types exist in separate files, FILES will attempt to merge them by selecting the type with the larger granularity as the final type. For example, if there are columns with the same name but of types FLOAT and INT respectively, FILES will return DOUBLE as the final type. Primary Key tables support Size-tiered Compaction to reduce the I/O amplification. When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. Optimized the error messages for Routine Load. Optimized the error messages when the FILES table function is used to convert invalid data types. Fixed the following issues: FEs fail to start after system-defined views are dropped. Dropping system-defined views is now prohibited. BEs crash when duplicate sort key columns exist in Primary Key tables. Duplicate sort key columns are now prohibited. An error, instead of NULL, is returned when the input value of the to_json() function is NULL. In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. In shared-data mode, an error is returned when users modify the `enablepersistentindex` property of a Primary Key table. In shared-data mode, NULL values are given to columns that are not supposed to be changed when users update a Primary Key table with partial updates in column mode. Queries cannot be rewritten with asynchronous materialized views created on logical views. CNs crash when the Cross-cluster Data Migration Tool is used to migrate Primary Key tables to a shared-data cluster. The partition ranges of the external catalog-based asynchronous materialized views are not consecutive. Release date: March 12, 2024 :::tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and

Problem: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. Impact scope: This problem only affects queries on external tables in external catalogs. Other queries are not affected. Temporary workaround: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ::: Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce the write I/O amplification. Added the date function `milliseconds_diff`. Added the session variable `catalog`, which specifies the catalog to which the session belongs. Supports . Supports CREATE TABLE LIKE in Hive catalogs. Added the view `informationschema.partitionsmeta`, which records detailed metadata of partitions. Added the view `sys.fememoryusage`, which records the memory usage for StarRocks. `cbodecimalcaststringstrict` is used to control how CBO converts data from the DECIMAL type to the STRING type. The default value `true` indicates that the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). The DECIMAL type is not strictly filled in earlier versions, causing different results when comparing the DECIMAL type and the STRING type. The default value of the Iceberg Catalog parameter `enableicebergmetadata_cache` has been changed to `false`. From v3.2.1 to v3.2.3, this parameter is set to `true` by default, regardless of what metastore service is used. In v3.2.4 and later, if the Iceberg cluster uses AWS Glue as metastore, this parameter still defaults to `true`. However, if the Iceberg cluster uses other metastore service such as Hive metastore, this parameter defaults to `false`. The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cboeqbasetype` to adjust the rule used for the comparison. For example, users can set `cboeqbasetype` to `decimal`, and StarRocks then compares the columns as numeric values. Shared-data StarRocks clusters support the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks stores the data into multiple, uniformly prefixed partitions (sub-paths) under the bucket. This improves the read and write efficiency on data files in S3-compatible object storages. StarRocks supports using the parameter `s3compatiblefslist` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallbacktohadoopfs_list` to specify non-S3-compatible object storages that require access via HDFS Schema (this method requires the use of vendor-provided JAR packages). Optimized compatibility with Trino. Supports syntax conversion from the following Trino functions: currentcatalog, currentschema, tochar, fromhex, todate, totimestamp, and index. Optimized the query rewrite logic of materialized views. StarRocks can rewrite queries with materialized views created upon logical views. Improved the efficiency of converting the STRING type to the DATETIME type by 35% to 40%. The `aggtype` of BITMAP-type columns in an Aggregate table can be set to `replaceifnotnull` in order to support updates only to a few columns of the table. Improved the Broker Load performance when loading small ORC files. The tables with hybrid row-column storage support Schema Change. The tables with hybrid row-column storage support complex types including BITMAP, HLL, JSON, ARRAY, MAP, and STRUCT. A new internal SQL log file is added to record log data related to statistics and materialized

Fixed the following issues: "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. In shared-data clusters, primary key index directories are deleted every 5 hours. After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. Retries of the Publish phase may hang for Primary Key tables. Release date: February 8, 2024 [Preview] Supports hybrid row-column storage for tables. It allows better performance for high-concurrency, low-latency point lookups against Primary Key tables and partial data updates. Currently, this feature does not support modification via ALTER TABLE, changing Sort Key, and partial updates in column mode. Supports backing up and restoring asynchronous materialized views. Broker Load supports loading JSON-type data. Supports query rewrite using asynchronous materialized views created upon views. Queries against a view can be rewritten based on materialized views that are created upon that view. Supports CREATE OR REPLACE PIPE. Added the session variable `enablestrictorder_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. Added the session variable `enablematerializedviewforinsert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. When a single query is executed within the Pipeline framework, its memory limit is now constrained by the variable `querymemlimit` instead of `execmemlimit`. Setting the value of `querymemlimit` to `0` indicates no limit. Added the FE configuration item `httpworkerthreads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. Added the BE configuration item `lakepkcompactionmaxinput_rowsets`, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. Added the session variable `connectorsinkcompression_codec`, which specifies the compression algorithm used for writing data into Hive tables or Iceberg tables, or exporting data with Files(). Valid algorithms include GZIP, BROTLI, ZSTD, and LZ4. Added the FE configuration item `routineloadunstablethresholdsecond`. Added the BE configuration item `pindexmajorcompactionlimitper_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. Added the BE configuration item `enablelazydeltacolumncompaction`. The default value is `true`, indicating that StarRocks does not perform frequent compaction operations on delta columns. Added the FE configuration item `defaultmvrefresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. Changed the default value of the FE configuration item `defaultmvrefreshpartitionnum`to `1`. This indicates that when multiple partitions need to be updated during a materialized view refresh, the task will be split in batches, refreshing only one partition at a time. This helps reduce resource consumption during each refresh. Changed the default value of the BE/CN configuration item `starletusestarcache` to

This indicates that block data cache is enabled by default in shared-data clusters. If, before upgrading, you have manually configured the BE/CN configuration item `starletcacheevicthighwater` to `X`, you must configure the BE/CN configuration item `starletstarcachedisksizepercent` to `(1.0 - X) * 100`. For example, if you have set `starletcacheevicthighwater` to `0.3` before upgrading, you must set `starletstarcachedisksize_percent` to `70`. This ensures that both file data cache and block data cache will not exceed the disk capacity limit. Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. Added Data Cache-related metrics to the monitoring API. Optimized BE log printing to prevent too many irrelevant logs. Added the field `storagemedium` to the view `informationschema.be_tablets`. Supports `SET_VAR` in multiple sub-queries. A new field `LatestSourcePosition` is added to the return result of SHOW ROUTINE LOAD to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. The default retention period of trash files is changed to 1 day from the original 3 days. Supports collecting statistics from Iceberg tables with Partition Transform. The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. Fixed the following issues: The execution of ANALYZE TABLE gets stuck occasionally. The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storagepagecache_limit` in certain circumstances. Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. In some cases, `bitmaptostring` may return incorrect results due to data type overflow. When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. In some cases, querying semi-structured data in tables may cause BEs to crash. Release date: December 30, 2023 Fixed the following issue: When StarRocks is upgraded from v3.1.2 or earlier to v3.2, FEs may fail to restart. Release date: December 21, 2023 Supports reading tables and file external tables in Avro, SequenceFile, and RCFile formats through Java Native Interface (JNI). Added a view `object_dependencies` to the database `sys`. It contains the lineage information of asynchronous materialized views. Supports creating synchronous materialized views with the WHERE clause. Supports partition-level incremental refresh for asynchronous materialized views created upon Iceberg catalogs. [Preview] Supports creating asynchronous materialized views based on tables in a Paimon catalog with partition-level refresh. Supports the prepared statement. It allows better performance for processing high-concurrency point lookup queries. It also prevents SQL injection effectively. Supports the following Bitmap functions: , , and . Supports the Array function . Added a new metric `maxtabletrowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". A new BE configuration item `enablestreamloadverboselog` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. Upgraded the default GC algorithm in JDK8 to G1. A new value option `GROUPCONCATLEGACY` is added to the session variable to provide compatibility with the implementation logic of the function in versions earlier than v2.5. The authentication information `aws.s3.accesskey` and `aws.s3.accesssecret` for are hidden in audit

The `betablets` view in the `informationschema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes. The result returned by the statement provides a new field `OtherMsg`, which shows information about the last failed task. Fixed the following issues: The BEs crash if users create persistent indexes in the event of data corruption. The function occasionally causes the BEs to crash. After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. Some S3-compatible object storage returns duplicate files, causing the BEs to crash. Release date: December 1, 2023 Supports persisting indexes of to local disks. Supports even distribution of Data Cache among multiple local disks. Asynchronous materialized view The Query Dump file can include information of asynchronous materialized views. The Spill to Disk feature is enabled by default for the refresh tasks of asynchronous materialized views, reducing memory consumption. Supports creating and dropping databases and managed tables in , and supports exporting data to Hive's managed tables using INSERT or INSERT OVERWRITE. Supports , with which users can access different table formats (Hive, Iceberg, Hudi, and Delta Lake) that share a common metastore like Hive metastore or AWS Glue. Supports collecting statistics of Hive and Iceberg tables using ANALYZE TABLE, and storing the statistics in StarRocks, thus facilitating optimization of query plans and accelerating subsequent queries. Supports Information Schema for external tables, providing additional convenience for interactions between external systems (such as BI tools) and StarRocks. Added the following features of loading with the table function : Loading Parquet and ORC format data from Azure or GCP. Extracting the value of a key/value pair from the file path as the value of a column using the parameter `columnsfrompath`. Loading complex data types including ARRAY, JSON, MAP, and STRUCT. Supports unloading data from StarRocks to Parquet-formatted files stored in AWS S3 or HDFS by using INSERT INTO FILES. For detailed instructions, see . Supports used in an existing table to optimize the query and loading performance. You can set a new bucket key, bucket number, or sort key for a table. You can also set a different bucket number for specific partitions. Supports continuous data loading from or using the PIPE method. When PIPE detects new or modifications in a remote storage directory, it can automatically load the new or modified data into the destination table in StarRocks. While loading data, PIPE automatically splits a large loading task into smaller, serialized tasks, enhancing stability in large-scale data ingestion scenarios and reducing the cost of error retries. Supports , enabling users to access StarRocks data via HTTP and execute SELECT, SHOW, EXPLAIN, or KILL operations. Supports Runtime Profile and text-based Profile analysis commands (SHOW PROFILELIST, ANALYZE PROFILE, EXPLAIN ANALYZE) to allow users to directly analyze profiles via MySQL clients, facilitating bottleneck identification and discovery of optimization opportunities. Added the following functions: String functions: substringindex, urlextractparameter, urlencode, url_decode, and translate Date functions: dayofweekiso, weekiso, quartersadd, quarterssub, millisecondsadd, millisecondssub, datediff, jodatimeformat, strtojodatime, toiso8601, toteradate, and totera_timestamp Pattern matching function: regexpextractall hash function: xxhash364 Aggregate functions: approxtopk Window functions: cumedist, percentrank and session_number Utility functions: getqueryprofile and isrolein_session StarRocks supports access control through , providing a higher level of data security and allowing the reuse of existing services of external data

After integrating with Apache Ranger, StarRocks enables the following access control methods: When accessing internal tables, external tables, or other objects in StarRocks, access control can be enforced based on the access policies configured for the StarRocks Service in Ranger. When accessing an external catalog, access control can also leverage the corresponding Ranger service of the original data source (such as Hive Service) to control access (currently, access control for exporting data to Hive is not yet supported). For more information, see . Optimized ORC Reader: Optimized the ORC Column Reader, resulting in nearly a two-fold performance improvement for VARCHAR and CHAR data reading. Optimized the decompression performance of ORC files in Zlib compression format. Optimized Parquet Reader: Supports adaptive I/O merging, allowing adaptive merging of columns with and without predicates based on filtering effects, thus reducing I/O. Optimized Dict Filter for faster predicate rewriting. Supports STRUCT sub-columns, and on-demand dictionary column decoding. Optimized Dict Decode performance. Optimized late materialization performance. Supports caching file footers to avoid repeated computation overhead. Supports decompression of Parquet files in lzo compression format. Optimized CSV Reader: Optimized the Reader performance. Supports decompression of CSV files in Snappy and lzo compression formats. Optimized the performance of the count calculation. Optimized Iceberg Catalog capabilities: Supports collecting column statistics from Manifest files to accelerate queries. Supports collecting NDV (number of distinct values) from Puffin files to accelerate queries. Supports partition pruning. Reduced Iceberg metadata memory consumption to enhance stability in scenarios with large metadata volume or high query concurrency. Asynchronous materialized view Supports automatic refresh for an asynchronous materialized view created upon views or materialized views when schema changes occur on the views, materialized views, or their base tables. Data consistency: Added the property `queryrewriteconsistency` for asynchronous materialized view creation. This property defines the query rewrite rules based on the consistency check. Add the property `forceexternaltablequeryrewrite` for external catalog-based asynchronous materialized view creation. This property defines whether to allow force query rewrite for asynchronous materialized views created upon external catalogs. For detailed information, see . Added a consistency check for materialized views' partitioning key. When users create an asynchronous materialized view with window functions that include a PARTITION BY expression, the partitioning column of the window function must match that of the materialized view. Optimized the persistent index for Primary Key tables by improving memory usage logic while reducing I/O read and write amplification. Supports data re-distribution across local disks for Primary Key tables. Partitioned tables support automatic cooldown based on the partition time range and cooldown time. Compared to the original cooldown logic, it is more convenient to perform hot and cold data management on the partition level. For more information, see . The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes. For more information, see . Supports Fast Schema Evolution, which is controlled by the table property . After this feature is enabled, the execution efficiency of adding or dropping columns is significantly improved. This mode is disabled by default (Default value is `false`). You cannot modify this property for existing tables using ALTER TABLE. according to cluster information and the size of the data for Duplicate Key tables created with the Radom Bucketing strategy. Optimized StarRocks' compatibility with Metabase and Superset. Supports integrating them with external catalogs. supports the keyword DISTINCT. INSERT, UPDATE, and DELETE operations now support `SET_VAR`. Added the session variable `largedecimalunderlying_type = "panic"|"double"|"decimal"` to set the rules to deal with DECIMAL type overflow. `panic` indicates returning an error immediately, `double` indicates converting the data to DOUBLE type, and `decimal` indicates converting the data to

Supports Trace Query Profile for asynchronous materialized views, which can be used to analyze its transparent rewrite. To be updated. Added the following FE configuration items: `catalogmetadatacache_size` `enablebackupmaterialized_view` `enablecolocatemv_index` `enablefastschema_evolution` `jsonfilesize_limit` `lakeenableingest_slowdown` `lakeingestslowdown_threshold` `lakeingestslowdown_ratio` `lakecompactionscoreupperbound` `mvautoanalyze_async` `primarykeydiskscheduletime` `statisticautocollectsmalltable_rows` `streamloadtaskkeepmax_num` `streamloadtaskkeepmax_second` Removed FE configuration item `enablepipelineload`. Default value modifications: The default value of `enablesyncpublish` is changed from `false` to `true`. The default value of `enablepersistentindexbydefault` is changed from `false` to `true`. Data Cache-related configuration changes. Added `datacacheenable` to replace `blockcache_enable`. Added `datacachememsize` to replace `blockcachemem_size`. Added `datacachedisksize` to replace `blockcachedisk_size`. Added `datacachediskpath` to replace `blockcachedisk_path`. Added `datacachemetapath` to replace `blockcachemeta_path`. Added `datacacheblocksize` to replace `blockcacheblock_size`. Added `datacachechecksumenable` to replace `blockcachechecksum_enable`. Added `datacachedirectioenable` to replace `blockcachedirectio_enable`. Added `datacachemaxconcurrentinserts` to replace `blockcachemaxconcurrent_inserts`. Added `datacachemaxflyingmemorymb`. Added `datacacheengine` to replace `blockcache_engine`. Removed `blockcachemaxparcelmemory_mb`. Removed `blockcachereport_stats`. Removed `blockcachelruinsertionpoint`. After renaming Block Cache to Data Cache, StarRocks has introduced a new set of BE parameters prefixed with `datacache` to replace the original parameters prefixed with `blockcache`. After upgrade to v3.2, the original parameters will still be effective. Once enabled, the new parameters will override the original ones. The mixed usage of new and original parameters is not supported, as it may result in some configurations not taking effect. In the future, StarRocks plans to deprecate the original parameters with the `blockcache` prefix, so we recommend you use the new parameters with the `datacache` prefix. Added the following BE configuration items: `spillmaxdirbytesratio` `streamingagglimitedmemorysize` `streamingaggchunkbuffersize` Removed the following BE configuration items: Dynamic parameter `tcusememory_min` Dynamic parameter `tcfreememory_rate` Dynamic parameter `tcgcperiod` Static parameter `tcmaxtotalthreadcache_byte` Default value modifications: The default value of `disablecolumnpool` is changed from `false` to `true`. The default value of `thrift_port` is changed from `9060` to `0`. The default value of `enableloadcolocate_mv` is changed from `false` to `true`. The default value of `enablepindexminor_compaction` is changed from `false` to `true`. Added the following session variables: `enableperbucket_optimize` `enablewritehiveexternaltable` `hivetempstaging_dir` `spillrevocablemax_bytes` `thriftplanprotocol` Removed the following session variables: `enablepipelinequery_statistic` `enabledeliverbatch_fragments` Renamed the following session variables: `enablescanblockcache` is renamed as `enablescan_datacache`. `enablepopulateblockcache` is renamed as `enablepopulate_datacache`. Added reserved keywords `OPTIMIZE` and `PREPARE`. Fixed the following issues: BEs crash when libcurl is invoked. Schema Change may fail if it takes an excessively long period of time, because the specified tablet version is handled by garbage collection. Failed to access the Parquet files in MinIO via file external tables. The ARRAY, MAP, and STRUCT type columns are not correctly displayed in `information_schema.columns`. An error is reported if specific path formats are used during data loading via Broker Load: `msg:Fail to parse columnsFromPath, expected: `DATATYPE` and `COLUMNTYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. Complex queries that involve many unions, expressions, and SELECT columns can result in a sudden surge in the bandwidth or CPU usage within an FE node. The refresh of asynchronous materialized view may occasionally encounter deadlock. Optimization on Random Bucketing is disabled by default. To enable it, you need to add the property `bucket_size` when creating tables. This allows the system to dynamically adjust the number of tablets based on cluster information and the size of loaded data. Please note that once this optimization is enabled, if you need to roll back your cluster to v3.1 or earlier, you must delete tables with this optimization enabled and manually execute a metadata checkpoint (by executing `ALTER SYSTEM CREATE IMAGE`). Otherwise, the rollback will fail. Starting from v3.2.0, StarRocks has disabled non-Pipeline queries. Therefore, before upgrading your cluster to v3.2, you need to globally enable the Pipeline engine (by adding the configuration `enablepipelineengine=true` in the FE configuration file fe.conf). Failure to do so will result in errors