Project: ClickHouse (App Definition and Development / Database)
File: v21.3.1.6185-prestable.md

sidebar_position: 1 sidebar_label: 2022 Now all case-insensitive function names will be lower-cased during query analysis. This is needed for projection query routing. (). Now it's not allowed to create MergeTree tables in old syntax with table TTL because it's just ignored. Attach of old tables is still possible. (). Add experimental `Replicated` database engine. It replicates DDL queries across multiple hosts. (). Distributed query deduplication is a followup to and partially resolves the proposal . (). Add the ability to backup/restore metadata files for DiskS3. (). - Included in the pull request. (). Added file() function to read file as a String. This close . (). Tables with `MergeTree` engine now have two new table-level settings for query concurrency control. Setting `maxconcurrentqueries` limits the number of concurrently executed queries which are related to this table. Setting `minmarkstohonormaxconcurrentqueries` tells to apply previous setting only if query reads at least this number of marks. (). - Mentioned in - add function `htmlOrxmlCoarseParse`; - support `<script></script>` parse; - support `<style></style>` parse; - support `< (). Add quota type QUERYSELECTS and QUERYINSERTS. (). ExecutableDictionarySource added implicit_key option. Fixes . (). Added Server Side Encryption Customer Keys (the `x-amz-server-side-encryption-customer-(key/md5)` header) support in S3 client. See . Closes . (). Function `reinterpretAs` updated to support big integers. Fixes . (). Add setting `insertshardid` to support insert data into specific shard from distributed table. (). Added timezoneOffset(datetime) function which will give the offset from UTC in seconds. This close . (). New `eventtimemicroseconds column` in `system.part_log` table. (). ... Add aggregate function `deltaSum` for summing the differences between consecutive rows. (). Add two settings to delay or throw error during insertion when there are too many inactive parts. This is useful when server fails to clean up parts quickly enough. (). Add file engine settings: `enginefileemptyifnotexists` and `enginefiletruncateon_insert`. (). Add parallel select final for one part with level>0 when `donotmergeacrosspartitionsselectfinal` setting is 1. (). Improved performance of bitmap columns during joins. (). Partially reimplement HTTP server to make it making less copies of incoming and outgoing data. It gives up to 1.5 performance improvement on inserting long records over HTTP. (). Improve performance of aggregate functions by more strict aliasing. (). Fix the case when DataType parser may have exponential complexity (found by fuzzer). This closes . (). Add `compress` setting for `Memory` tables. If it's enabled the table will use less RAM. On some machines and datasets it can also work faster on SELECT, but it is not always the case. This closes . Note: there are reasons why Memory tables can work slower than MergeTree: (1) lack of compression (2) static size of blocks (3) lack of indices and prewhere... (). Do not squash blocks too much on INSERT SELECT if inserting into Memory table. In previous versions inefficient data representation was created in Memory table after INSERT SELECT. This closes . (). Improved performance of aggregation by several fixed size fields (unconfirmed). (). Speed up reading from `Memory` tables in extreme cases (when reading speed is in order of 50 GB/sec) by simplification of pipeline and (consequently) less lock contention in pipeline scheduling. (). Improve performance of GROUP BY multiple fixed size keys. (). The setting `distributedaggregationmemory_efficient` is enabled by default. It will lower memory usage and improve performance of distributed queries. (). Slightly better code in aggregation. (). Add back intDiv/module vectorConstant specializations for better performance. This fixes . The regression was introduced in https://github.com/ClickHouse/ClickHouse/pull/18145 .

Fix creation of `TTL` in cases, when its expression is a function and it is the same as `ORDER BY` key. Now it's allowed to set custom aggregation to primary key columns in `TTL` with `GROUP BY`. Backward incompatible: For primary key columns, which are not in `GROUP BY` and aren't set explicitly now is applied function `any` instead of `max`, when TTL is expired. Also if you use TTL with `WHERE` or `GROUP BY` you can see exceptions at merges, while making rolling update. (). Hedged Requests for remote queries. When setting `usehedgedrequests` enabled (by default), allow to establish many connections with different replicas for query. New connection is enabled in case existent connection(s) with replica(s) were not established within `hedgedconnectiontimeout` or no data was received within `receivedatatimeout`. Query uses the first connection which send non empty progress packet (or data packet, if `allowchangingreplicauntilfirstdatapacket`); other connections are cancelled. Queries with `maxparallelreplicas > 1` are supported. (). Print inline frames for fatal stacktraces. (). Do not silently ignore write errors. (). Added support for `PREWHERE` when tables have row-level security expressions specified. (). Add IStoragePolicy interface. (). Add ability to throttle INSERT into Distributed based on amount of pending bytes for async send (`bytestodelayinsert`/`maxdelaytoinsert` and `bytestothrow_insert` settings for `Distributed` engine has been added). (). move Conditions that are not related to JOIN to where clause. . (). Add separate config directive for Buffer profile. (). Show MaterializeMySQL tables in `system.parts`. (). Initialize MaxDDLEntryID to the last value after restarting. Before this PR, MaxDDLEntryID will remain zero until a new DDLTask is processed. (). Add conversion of block structure for INSERT into Distributed tables if it does not match. (). If user calls `JSONExtract` function with `Float32` type requested, allow inaccurate conversion to the result type. For example the number `0.1` in JSON is double precision and is not representable in Float32, but the user still wants to get it. Previous versions return 0 for non-Nullable type and NULL for Nullable type to indicate that conversion is imprecise. The logic was 100% correct but it was surprising to users and leading to questions. This closes . (). The value of MYSQLOPTRECONNECT option can now be controlled by "opt_reconnect" parameter in the config section of mysql replica. (). Return `DiskType` instead of `String` in IDisk::getType() as in the rest of storage interfaces. (). Fix data race in executable dictionary that was possible only on misuse (when the script returns data ignoring its input). (). Show full details of MaterializeMySQL tables in `system.tables`. (). Supports system.zookeeper path IN query. (). SHOW TABLES is now considered as one query in the quota calculations, not two queries. 2. SYSTEM queries now consume quota. 3. Fix calculation of interval's end in quota consumption. (). - Fix toDateTime64(toDate()/toDateTime()) for DateTime64 - Implement DateTime64 clamping to match DateTime behaviour. (). The setting `accessmanagement` is now configurable on startup by providing `CLICKHOUSEDEFAULTACCESSMANAGEMENT`, defaults to disabled (`0`) which was the prior value. (). Updated `CacheDictionary`, `ComplexCacheDictionary`, `SSDCacheDictionary`, `SSDComplexKeyDictionary` to use LRUHashMap as underlying index. (). Support all native integer types in bitmap functions. (). Normalize count(constant), sum(1) to count(). This is needed for projection query routing. (). Perform algebraic optimizations of arithmetic expressions inside `avg` aggregate function. close . (). Lockless `SYSTEM FLUSH DISTRIBUTED`. (). Implicit conversion from integer to Dicimal type might succeeded if integer value doe not fit into Decimal type. Now it throws `ARGUMENTOUTOF_BOUND`. (). Do not allow early constant folding of explicitly forbidden functions. (). Make FQDN and other DNS related functions work correctly in alpine images.

Fixed race between execution of distributed DDL tasks and cleanup of DDL queue. Now DDL task cannot be removed from ZooKeeper if there are active workers. Fixes . (). Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the `Map` data type. Omitted values are now set by default. (). https://github.com/ClickHouse/ClickHouse/issues/20576. (). Function 'reinterpretAs(x, Type)' renamed into 'reinterpret(x, Type)'. (). When loading config for mysql source ClickHouse will now randomize the list of replicas with the same priority to ensure the round-robin logics of picking mysql endpoint. This closes . (). Do only merging of sorted blocks on initiator with distributedgroupbynomerge. (). - Fill only requested columns when querying system.parts & system.parts_columns. Closes . ... (). Usability improvement: more consistent `DateTime64` parsing: recognize the case when unix timestamp with subsecond resolution is specified as scaled integer (like `1111111111222` instead of `1111111111.222`). This closes . (). MySQL dictionary source will now retry unexpected connection failures (Lost connection to MySQL server during query) which sometimes happen on SSL/TLS connections. (). Forbid to drop a column if it's referenced by materialized view. Closes . (). Provide better compatibility for mysql clients. 1. mysql jdbc 2. mycli. (). Case-insensitive compression methods for table functions. Also fixed `LZMA` compression method which was checked in upper case. (). Background thread which executes `ON CLUSTER` queries might hang waiting for dropped replicated table to do something. It's fixed. (). Fix a bug that moving pieces to destination table may failed in case of launching multiple clickhouse-copiers. (). Fix clickhouse-client abort exception while executing only `select`. (). Fix starting the server with tables having default expressions containing dictGet(). Allow getting return type of dictGet() without loading dictionary. (). Deadlock was possible if system.text_log is enabled. This fixes . (). BloomFilter index crash fix. Fixes . (). - Fix a segfault in function `fromModifiedJulianDay` when the argument type is `Nullable(T)` for any integral types other than Int32. (). `EmbeddedRocksDB` is an experimental storage. Fix the issue with lack of proper type checking. Simplified code. This closes . (). Prevent "Connection refused" in docker during initialization script execution. (). MaterializeMySQL: Fix replication for statements that update several tables. (). Fix the case when calculating modulo of division of negative number by small divisor, the resulting data type was not large enough to accomodate the negative result. This closes . (). Bugfix in StorageJoin. (). The `MongoDB` table engine now establishes connection only when it's going to read data. `ATTACH TABLE` won't try to connect anymore. (). Fix server crash after query with `if` function with `Tuple` type of then/else branches result. `Tuple` type must contain `Array` or another complex type. Fixes . (). fix toMinute function to handle special timezone correctly. (). Fixes . (). Fix CTE when using in INSERT SELECT. This fixes , fixes . (). Fix rare server crash on config reload during the shutdown. Fixes . (). Fix exception during vertical merge for `MergeTree` table engines family which don't allow to perform vertical merges. Fixes . (). Fixed the behavior when in case of broken JSON we tried to read the whole file into memory which leads to exception from the allocator. Fixes . (). Restrict to `DROP` or `RENAME` version column of `CollapsingMergeTree` and `ReplacingMergeTree` table engines. (). Fix too often retries of failed background tasks for `ReplicatedMergeTree` table engines family. This could lead to too verbose logging and increased CPU load. Fixes .

Fix incorrect result of binary operations between two constant decimals of different scale. Fixes . (). Fix null dereference with `joinusenulls=1`. (). Avoid invalid dereference in RANGE_HASHED() dictionary. (). Check if table function `view` is used in expression list and throw an error. This fixes . (). Fix `LOGICALERROR` for `joinuse_nulls=1` when JOIN contains const from SELECT. (). Fix abnormal server termination when http client goes away. (). Fix infinite loop when propagating WITH aliases to subqueries. This fixes . (). Fix function `transform` does not work properly for floating point keys. Closes . (). Add proper checks while parsing directory names for async INSERT (fixes SIGSEGV). (). Fix crash which could happen if unknown packet was received from remove query (was introduced in ). (). Fix the number of threads for scalar subqueries and subqueries for index (after single thread was always used). Fixes , . (). Fixed inconsistent behavior of dictionary in case of queries where we look for absent keys in dictionary. (). Fix subquery with union distinct and limit clause. close . (). Fix usage of `-Distinct` combinator with `-State` combinator in aggregate functions. (). `USE database;` query did not work when using MySQL 5.7 client to connect to ClickHouse server, it's fixed. Fixes . (). Fix 'Empty task was returned from async task queue' on query cancellation. (). Closes . Fixed Brotli http compression error, which reproduced for large data sizes, slightly complicated structure and with json output format. Update Brotli to the latest version to include the "fix rare access to uninitialized data in ring-buffer". (). Fixed behaviour, when `ALTER MODIFY COLUMN` created mutation, that will knowingly fail. (). - Block parallel insertions into storage join. (). Out of bound memory access was possible when formatting specifically crafted out of range value of type `DateTime64`. This closes . This closes . (). fix default_replica_path and default_replica_name values are useless on Replicated()MergeTree engine when the engine needs specify other parameters. (). Fix type mismatch issue when using LowCardinality keys in joinGet. This fixes . (). Fix the metadata leak when the ReplicatedMergeTree with custom (non default) ZooKeeper cluster is dropped. (). fix bug related to cast tuple to map. Closes . (). Fix `inputformatnullasdefault` take effective when types are nullable. This fixes . (). Fixes . Fixed bug that could cause duplicates with insert query (if one of the callbacks came a little too late). (). Now mutations allowed only for table engines that support them (MergeTree family, Memory, MaterializedView). Other engines will report a more clear error. Fixes . (). Fix crash in `EXPLAIN` for query with `UNION`. Fixes , . (). Fix bug with `joinusenulls` and joining `TOTALS` from subqueries. This closes and . (). Fix redundant reconnects to ZooKeeper and the possibility of two active sessions for a single clickhouse server. Both problems introduced in . (). Now `ALTER MODIFY COLUMN` queries will correctly affect changes in partition key, skip indices, TTLs, and so on. Fixes . (). Fix error `Bad cast from type ... to DB::ColumnLowCardinality` while inserting into table with `LowCardinality` column from `Values` format. Fixes . (). Fix SIGSEGV for distributed queries on failures. (). Fix a deadlock in `ALTER DELETE` mutations for non replicated MergeTree table engines when the predicate contains the table itself. Fixes . (). Fixed port clash from teststoragekerberized_hdfs test. (). Fix some of the issues found by Coverity. See . (). Allow to build ClickHouse with AVX-2 enabled globally. It gives slight performance benefits on modern

Not recommended for production and will not be supported as official build for now. (). Add some checks in order by fix the bug https://clickhouse-test-reports.s3.yandex.net/20472/5bdc57004682a5e0236ec630546d20ad752c2fde/stresstest(thread)/stderr.log. (). Print `stdout` and `stderr` to log when failed to start docker in integration tests. Before this PR there was a very short error message in this case which didn't help to investigate the problems:. (). Remove most of `sleep` commands from kafka integration tests, waiting for log events instead. Add test for different compression methods. (). Allow to start up with modified binary under gdb. In previous version if you set up breakpoint in gdb before start, server will refuse to start up due to failed integrity check. (). Introduce experimental support for window functions, enabled with `allowexperimentalfunctions = 1`. This is a preliminary, alpha-quality implementation that is not suitable for production use and will change in backward-incompatible ways in future releases. Please see for the list of supported features. (). NO CL ENTRY: 'Revert "Fix access control manager destruction order"'. (). NO CL ENTRY: 'Update argmax.md '. (). Update Pytest check (). (). In memory coordination inside ClickHouse (). client: more suggestions (). Allow to run all style checks in one file (). Fix (). Minor code improvements around ThreadStatus (). continue of (). Fix UBSan report in intDiv (). Allow writing partial buffer (). Remove an always-false condition from query parser (). UNBOUNDED FOLLOWING frame end (). Fix logical error in INSERT VALUES (). Fix test keeper integration tests (). Fix build (). CURRENT ROW and offset for start of ROWS frame (). Fix UBSan report in geoHashesInBox (). Add MSan annotation for system.stack_trace (). Fix flaky integration tests (). Do not use inputs which values are known constants in ActionsDAG. (). update perf tests (). UBsan fixes (). Remove useless code (). update changelog (). Close input ports in DelayedPortsProcessor as soon as all outputs are finished (). ReadBuffer: check for unread data on next() (). ReadBuffer: check that buffer position is not set beyond end (). CURRENT ROW frame start for RANGE frame (). Fix log message with elapsed time while pushing to views (). Avoid UBSan report in pointInPolygon (). Fix tests .reference files (). ROWS OFFSET frame end (). Add more tests for modulo of division of negative number (). detect unmarked long tests in flaky check (). Fix removing of filter column when split filter actions. (). Remove AddingConstColumn step and transform. (). LimitReadBuffer: check that position always advances (). Add fuzzer for ColumnsDescription (). Build actions dag to evaluate missing defaults. (). Remove never existing insertinmemorypartstimeout (). RANGE OFFSET window frame (). Allow to drop readonly tables (). Add logging if Poco cannot allocate thread in tcp server (). Use fixed version of confluent-kafka library in integration tests (). Useless changes (). remove some useless code (). Allow using MergeTreeWhereOptimizer not only with MergeTree-based storages (). Fix build of utils (). Fix UBSan report in arrayCumSum (). Add changelog for 21.2 (). Fix ubsan error (). Suppress signed overflow in AggregateFunctionGroupArrayMoving (). Remove dead code (). Handle syntax error for ARRAY JOIN with no args (). Fix benign race in system.parts (). Add final to some classes (). refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part (). fix a problem in ArithmeticOperationsInAgrFuncOptimize (). Add retries to testaccesscontroloncluster (). Stable sort for test cases in testdictionariesalllayoutsseparate_sources (). Fix not closed ports in DelayedPortsProcessor (). print changed settings in fuzzer when the server dies (). Put windows in such an order that we can sort less

Aggregate function deltaSum use restrict keyword (). make window functions faster (). Accept arbitrary numeric types for numbers() arguments (for scientific notation) (). Fix 00738lockforinnertable flakiness (). Filter push down (). Common types template instantiations (). Add test for sign column drop (). Merging (). Fix 01700systemzookeeperpathin (). Print stack trace on SIGTRAP (). Fix UBSan report in arrayDifference (). Suppress UBSan report in Decimal comparison (). Improve backtrace catching on server failures in CI for stress tests (). Fix abnormal server terminations due to write failures (). Improve logging during reading from MergeTree (). Fix UBSan report in intDiv (). (unrealistic case, never possible in prod) HashTable fix potential bug during resize with nonstandard grower (). Add a test for (). Add a test for (). Disable testkeeper snapshots for tests (). Fix non-zero session reconnect in integration test (). Reduce test scale (). Quick build fix (). Replace null fields in tuple during parsing with default values (). rabbitmq: add missing format factory settings (). Persistent coordination log storage (). Add hung check to stress test (). ignore data store files (). Dictionary create source with functions crash fix (). Fix fasttest retry for failed tests (). Don't backport base commit of branch in the same branch (). Add test for already fixed odbc Postgres date type conversion (). Improve backtrace catching on server failures in CI for fast-tests (). Better postgres db engine numeric conversion (). Fix undefined-behavior in ReservoirSamplerDeterministic.h (). ccache 4.2+ does not requires any quirks for SOURCEDATEEPOCH (). Check for EINTR in epoll_wait (). Fix push to dockerhub (). Better tests for protobuf format. (). Function sumMap decimal fix (). test for decimal ( p , s) in dictionaries (). Fix uncaught exception when HTTP client goes away (). Increase buffer for uncaught exception / std::terminate (). Constraints complex types support (). Suppress signed overflow in AggregateFunctionGroupArrayMoving 2 (). Add log message when stacktrace cannot be obtained for thread (). Preserve mysql logs in testmaterializemysql_database (). Minor changes in Decimal (). yamllint style check (). Fix buffer size for trace collection (). Try fix MaterializeMySQL integration test (). Fix performance tests (by avoid sharing status file for right and left server) (). Revert "optimize aggfunc column data copy ()" (). Fix UBSan report in Decimal arithmetic (). Update lowcardinality.md (). Fix DateTime64 from Float (). Add test for (). Merging (). Add a test for (). Fix various issues in DDLWorker (SIGSEGV and others) (). Documentation low cardinality fix (). Fix has_feature macros under gcc (). test for window functions (). Fix replace (). Better tests for protobuf format #2. (). Fix ccache 4.2+ usage (RULELAUNCHCOMPILE/RULELAUNCHLINK was not set) (). Bump zookeeper version to 3.6.2 in tests (). StorageRabbitMQ added UVLoop (). Trying to make nukeeper better in single server mode (). A followup correction to (). Add tests for zstd and zlib http compression (). CheckConstraintsBlockOutputStream optimize nullable column case (). Rewrite extractTextFromHTML function (). Fix abnormal server termination for nested writers (). (). remove unused code in MergeTreeWriteAheadLog::restore (). IColunm::hasEqualValues() (). AggregateFunctionSumMap better comment message (). clickhouse stop: wait for the server to be killed (process exited) (). (). fix a rare false negative in perf tests (). test for issue (). Add a test for (). Add a test for (). Fix typo in setting name (). Fix flaky tests with distributed queries (). Fix ParsingException::displayText() (). Use path as default prefix for coordination logs (). Try fix perftests. (). avoid race in librdkafka (). test for (). Tiny fix (). Just another fix for DDLWorker ().