Project: ClickHouse (App Definition and Development / Database)
File: v20.10.2.20-stable.md

Add support for nested multiline comments `/ comment / comment / */` in SQL. This conforms to the SQL standard. (). Change default value of `formatregexpescaping_rule` setting (it's related to `Regexp` format) to `Raw` (it means - read whole subpattern as a value) to make the behaviour more like to what users expect. (). Make `multiplejoinsrewriter_version` obsolete. Remove first version of joins rewriter. (). Add new feature: format LineAsString that accepts a sequence of line separated by newlines, spaces and/or commas. (). Added `formatReadableQuantity` function. It is useful for reading big numbers by human. (). Add the ability to remove column properties and table TTLs. Introduced queries `ALTER TABLE MODIFY COLUMN colname REMOVE whatto_remove` and `ALTER TABLE REMOVE TTL`. Both operations are lightweight and executed at the metadata level. (). Introduce eventtimemicroseconds field to `system.textlog`, `system.tracelog`, `system.querylog` and `system.querythread_log` tables. (). Now we support `WITH <identifier> AS (subquery) ... ` to introduce named subqueries in the query context. This closes . This closes . (). Allow to omit arguments for Replicated table engine if defaults are specified in config. (). Add table function `null('structure')`. (). Added query obfuscation tool. It allows to share more queries for better testing. This closes . (). Added format `RawBLOB`. It is intended for input or output a single value without any escaping and delimiters. This closes . (). fix . (). Introduce `enableglobalwith_statement` setting which propagates the first select's `WITH` statements to other select queries at the same level, and makes aliases in `WITH` statements visible to subqueries. (). Add the `reinterpretAsUUID` function that allows to convert a big-endian byte string to UUID. (). Add parallel quorum inserts. This closes . (). Improve performance of 256-bit bytes using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. (). Only `mlock` code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. (). We used to choose fixed key method to group by one fixed string. It's unnecessary since we have StringHashTable which do the similar packedFix optimization for FixedString columns. And we should use lowcardinalitykeyfixedstring if possible. (). Fix `DateTime <op> DateTime` mistakenly choosing the slow generic implementation. This fixes . (). Use one S3 DeleteObjects request instead of multiple DeleteObject in cycle. No any functionality changes, so covered by existing tests like integration/testlogfamily_s3. (). Faster 256-bit multiplication. (). Improve `quantileTDigest` performance. This fixes . (). Explicitly use a temporary disk to store vertical merge temporary data. (). Add `QueryMemoryLimitExceeded` event. This closes . (). Fixed the backward-incompatible change by providing the options to build without debug info for functions. (). dynamic reload zookeeper config. (). Allow parallel execution of distributed DDL. (). Fix potential memory leak caused by zookeeper exists watch. (). Fixed "Packet payload is not fully read" error in `MaterializeMySQL` database engine. (). Fix crash in `bitShiftLeft()` when called with negative big integer. (). Add `merge_algorithm` to system.merges table to improve merging inspections. (). Less unneded code generated by DecimalBinaryOperation template in FunctionBinaryArithmetic.

Now columns can be used to wrap over a list of columns and apply column transformers afterwards. (). Support for disabling persistency for StorageJoin and StorageSet, this feature is controlled by setting `disablesetandjoinpersistency`. And this PR solved issue . (). Construct `querystarttime` and `querystarttime_microseconds` from the same timespec. (). Allow using multi-volume storage configuration in storage Distributed. (). Show subqueries for `SET` and `JOIN` in `EXPLAIN` result. (). Provide a `loadbalancingfirstoffset` query setting to explicitly state what the first replica is. It's used together with `FIRSTOR_RANDOM` load balancing strategy, which allows to control replicas workload. (). Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. (). Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes . (). Avoid deadlock when executing INSERT SELECT into itself from a table with `TinyLog` or `Log` table engines. This closes . (). Ignore key constraints when doing mutations. Without this pr, it's not possible to do mutations when `forceindexbydate = 1` or `forceprimary_key = 1`. (). Add option to disable TTL move on data part insert. (). Enable `Atomic` database engine by default. (). Proper exception message for wrong number of arguments of CAST. This closes . (). Add the ability to specify specialized codecs like `Delta`, `T64`, etc. for columns with subtypes. Implements , fixes , fixes . (). Added `optimize` setting to `EXPLAIN PLAN` query. If enabled, query plan level optimisations are applied. Enabled by default. (). Do not allow connections to ClickHouse server until all scripts in `/docker-entrypoint-initdb.d/` are executed. (). fix . (). Unfold `{database}`, `{table}` and `{uuid}` macros in `zookeeperpath` on replicated table creation. Do not allow `RENAME TABLE` if it may break `zookeeperpath` after server restart. Fixes . (). Add support for "Raw" column format for `Regexp` format. It allows to simply extract subpatterns as a whole without any escaping rules. (). Now it's possible to change the type of version column for `VersionedCollapsingMergeTree` with `ALTER` query. (). Wait for `DROP/DETACH TABLE` to actually finish if `NO DELAY` or `SYNC` is specified for `Atomic` database. (). Pass through foruser settings via Distributed with cluster-secure. (). Use experimental pass manager by default. (). Implement forcedataskipping_indices setting. (). Backported in : Now it's allowed to execute `ALTER ... ON CLUSTER` queries regardless of the `<internal_replication>` setting in cluster config. (). Fix wrong Decimal multiplication result caused wrong decimal scale of result column. (). Stuff the query into ASTFunction's argument list so that we don't break the presumptions of some AST visitors. This fixes . (). Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes , . (). Fixed missed default database name in metadata of materialized view when executing `ALTER ... MODIFY QUERY`. (). Replace column transformer should replace identifiers with cloned ASTs. This fixes . (). Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in . (). Fix unreleased bug for LineAsString Format.

Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes . (). Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes . (). Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. (). Fixed segfault in CacheDictionary . (). Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). (). Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes . (). Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. (). Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes . (). Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes . (). Update jemalloc to fix possible issues with percpu arena. (). Now settings `numberoffreeentriesinpooltoexecutemutation` and `numberoffreeentriesinpooltolowermaxsizeofmerge` can be equal to `backgroundpool_size`. (). Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. (). Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. (). If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes . (). We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes . (). Backported in : Update jemalloc to fix percpu_arena with affinity mask. (). Fixes `Data compressed with different methods` in `joinalgorithm='auto'`. Keep LowCardinality as type for left table join key in `joinalgorithm='partial_merge'`. (). Adjust decimals field size in mysql column definition packet. (). Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes . (). Fix instance crash when using joinGet with LowCardinality types. This fixes . (). Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. (). Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. (). Fix hang of queries with a lot of subqueries to same table of `MySQL` engine. Previously, if there were more than 16 subqueries to same `MySQL` table in query, it hang forever. (). Fix rare race condition on server startup when system.logs are enabled. (). Fix race condition during MergeTree table rename and background cleanup. (). Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. (). Fixed compression in S3 storage. (). Fix multiple occurrences of column transformers in a select query. (). fixes fix attach mysql database engine throw exception(no query context). (). Report proper error when the second argument of `boundingRatio` aggregate function has a wrong type. (). Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in `ON CLUSTER`. Introduced in . (). Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. (). Fix `Missing columns` errors when selecting columns which absent in data, but depend on other columns which also absent in

Fixes . (). Fix bug when `ILIKE` operator stops being case insensitive if `LIKE` with the same pattern was executed. (). Mutation might hang waiting for some non-existent part after `MOVE` or `REPLACE PARTITION` or, in rare cases, after `DETACH` or `DROP PARTITION`. It's fixed. (). Fix 'Database <db> doesn't exist.' in queries with IN and Distributed table when there's no database on initiator. (). Significantly reduce memory usage in AggregatingInOrderTransform/optimizeaggregationin_order. (). Prevent the possibility of error message `Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call`. This fixes . (). Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. (). Fixed `Element ... is not a constant expression` error when using `JSON` function result in `VALUES`, `LIMIT` or right side of `IN` operator. (). Fix the order of destruction for resources in `ReadFromStorage` step of query plan. It might cause crashes in rare cases. Possibly connected with . (). Backported in : Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. (). Proper error handling during insert into MergeTree with S3. (). Backported in : Fix error `Cannot add simple transform to empty Pipe` which happened while reading from `Buffer` table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes . (). Fix race condition in AMQP-CPP. (). Fix rare race condition in dictionaries and tables from MySQL. (). Fixed too low default value of `maxreplicatedlogstokeep` setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. (). Fix error `Cannot find column` which may happen at insertion into `MATERIALIZED VIEW` in case if query for `MV` containes `ARRAY JOIN`. (). Backported in : Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). (). Fix some cases of queries, in which only virtual columns are selected. Previously `Not found column _nothing in block` exception may be thrown. Fixes . (). Backported in : Fix `select count()` inaccuracy for MaterializeMySQL. (). Backported in : Fix exception `Block structure mismatch` in `SELECT ... ORDER BY DESC` queries which were executed after `ALTER MODIFY COLUMN` query. Fixes . (). Backported in : Now exception will be thrown when `ALTER MODIFY COLUMN ... DEFAULT ...` has incompatible default with column type. Fixes . (). Backported in : Fix possible deadlocks in RBAC. (). Backported in : fixes fix difference expressions with same alias when analyze queries again. (). Backported in : Fix incorrect empty result for query from `Distributed` table if query has `WHERE`, `PREWHERE` and `GLOBAL IN`. Fixes . (). Backported in : Fix a crash when database creation fails. (). Backported in : Fix ambiguity in parsing of settings profiles: `CREATE USER ... SETTINGS profile readonly` is now considered as using a profile named `readonly`, not a setting named `profile` with the readonly constraint. This fixes .

Backported in : Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). (). Backported in : Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. (). Backported in : Allow to use direct layout for dictionaries with complex keys. (). Backported in : Fix `ALTER MODIFY ... ORDER BY` query hang for `ReplicatedVersionedCollapsingMergeTree`. This fixes . (). Backported in : Fixes regression, e.g. indexOf( (). Backported in : Fix segfault in some cases of wrong aggregation in lambdas. (). Now ClickHouse uses gcc-10 for the release build. Fixes . (). Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with `madvise` to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. (). In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 -> `OFF`, 1 -> `ON` in `option`s default values. - Added some descriptions and links to docs to the options. - Replaced `FUZZER` option (there is another option `ENABLEFUZZING` which also enables same functionality). - Removed `ENABLEGTESTLIBRARY` option as there is `ENABLETESTS`. (). Speed up build a little by removing unused headers. (). Fix build failure in OSX. (). Attempt to speed up build a little. (). Now we use clang-11 to build ClickHouse in CI. (). #14809 fix MaterializeMySQL empty transaction unstable test case found in CI. (). Reformat and cleanup code in all integration test .py files. (). Fixing tests/integration/testdistributedoverliveview/test.py. (). Switch from `clang-tidy-10` to `clang-tidy-11`. (). Convert to python3. This closes . (). Make performance test more stable and representative by splitting test runs and profile runs. (). Maybe fix MSan report in base64 (on servers with AVX-512). This fixes . (). Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes . (). Now all test images use `llvm-symbolizer-11`. (). Splitted huge test `testdictionariesalllayoutsand_sources` into smaller ones. (). Added a script to perform hardware benchmark in a single command. (). Fix CMake options forwarding in fast test script. Fixes error in . (). Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. (). Now we use clang-11 for production ClickHouse build. (). Allow to run AArch64 version of clickhouse-server without configs. This facilitates . (). Fail early in functional tests if server failed to respond. This closes . (). fix bug for build error: . (). fix bug for building querydbgenerator.cpp. (). Allow to build with llvm-11. (). Switch binary builds(Linux, Darwin, AArch64, FreeDSD) to clang-11. (). NO CL ENTRY: 'Bump numpy from 1.19.1 to 1.19.2 in /docs/tools'. (). NO CL ENTRY: 'Modify the minimum version of the Clang compiler'. (). NO CL ENTRY: 'fix a syntax error bug while using copier'. (). NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.9 to 0.4.13 in /docs/tools'. (). NO CL ENTRY: 'Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka"'. (). NO CL ENTRY: 'Revert "Avoid deadlocks in Log/TinyLog"'. (). NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.13 to 0.4.17 in /docs/tools'. ().