Project: ClickHouse (App Definition and Development / Database)
File: v20.10.1.4881-prestable.md

Add support for nested multiline comments `/ comment / comment / */` in SQL. This conforms to the SQL standard. (). Change default value of `formatregexpescaping_rule` setting (it's related to `Regexp` format) to `Raw` (it means - read whole subpattern as a value) to make the behaviour more like to what users expect. (). Make `multiplejoinsrewriter_version` obsolete. Remove first version of joins rewriter. (). Supporting MySQL types: `decimal` (as ClickHouse `Decimal`) and `datetime` with sub-second precision (as `DateTime64`). ... (). Allow to turn on fsync on inserts, merges and fetches. (). Secure inter-cluster query execution (with initial_user as current query user). (). Add `mapPopulateSeries` function. (). Allow user to specify settings for `ReplicatedMergeTree` storage in `<replicatedmergetree>` section of config file. It works similarly to `<mergetree>` section. For `ReplicatedMergeTree*` storages settings from `<mergetree>` and `<replicatedmergetree>` are applied together, but settings from `<replicatedmergetree>` has higher priority. Added `system.replicatedmergetree_settings` table. (). Add new feature: format LineAsString that accepts a sequence of line separated by newlines, spaces and/or commas. (). New query complexity limit settings `max_rows_to_read_leaf`, `max_bytes_to_read_leaf` for distributed queries to limit max rows/bytes read on the leaf nodes. Limit is applied for local reads only, excluding* the final merge stage on the root node. (). Add JSONStrings formats which output data in arrays of strings. (). Now insert statements can have asterisk (or variants) with column transformers in the column list. (). Added a script to import git repository to ClickHouse. (). Add the ability to specify `TTL ... RECOMPRESS codec_name` for MergeTree table engines family. (). Add eventtimemicroseconds to `system.asynchronousmetriclog` & `system.metric_log` tables. (). Add new feature: SHOW DATABASES LIKE 'xxx'. (). Support decimal data type for MaterializedMySQL. (). Allow configurable NULL representation for TSV output format. It is controlled by the setting `outputformattsvnullrepresentation` which is `\N` by default. This closes . Note that the setting only controls output format and `\N` is the only supported NULL representation for TSV input format. (). Add new feature: format LineAsString that accepts a sequence of line separated by newlines, spaces and/or commas. (). Added `formatReadableQuantity` function. It is useful for reading big numbers by human. (). Add the ability to remove column properties and table TTLs. Introduced queries `ALTER TABLE MODIFY COLUMN colname REMOVE whatto_remove` and `ALTER TABLE REMOVE TTL`. Both operations are lightweight and executed at the metadata level. (). Introduce eventtimemicroseconds field to `system.textlog`, `system.tracelog`, `system.querylog` and `system.querythread_log` tables. (). Now we support `WITH <identifier> AS (subquery) ... ` to introduce named subqueries in the query context. This closes . This closes . (). Allow to omit arguments for Replicated table engine if defaults are specified in config. (). Add table function `null('structure')`. (). Added query obfuscation tool. It allows to share more queries for better testing. This closes . (). Added format `RawBLOB`. It is intended for input or output a single value without any escaping and delimiters. This closes . (). fix . (). Introduce `enableglobalwith_statement` setting which propagates the first select's `WITH` statements to other select queries at the same level, and makes aliases in `WITH` statements visible to subqueries. (). Add the `reinterpretAsUUID` function that allows to convert a big-endian byte string to UUID. (). Add parallel quorum inserts. This closes . (). Enable compact parts by default for small parts. This will allow to process frequent inserts slightly more efficiently (4..100 times). (). Improve performance of 256-bit bytes using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. (). Only `mlock` code segment when starting

In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. (). We used to choose fixed key method to group by one fixed string. It's unnecessary since we have StringHashTable which do the similar packedFix optimization for FixedString columns. And we should use lowcardinalitykeyfixedstring if possible. (). Fix `DateTime <op> DateTime` mistakenly choosing the slow generic implementation. This fixes . (). Use one S3 DeleteObjects request instead of multiple DeleteObject in cycle. No any functionality changes, so covered by existing tests like integration/testlogfamily_s3. (). Faster 256-bit multiplication. (). Improve `quantileTDigest` performance. This fixes . (). Explicitly use a temporary disk to store vertical merge temporary data. (). When duplicate block is written to replica where it does not exist locally (has not been fetched from replicas), don't ignore it and write locally to achieve the same effect as if it was successfully replicated. (). Support custom codecs in compact parts. (). Now joinGet supports multi-key lookup. Continuation of . (). For INSERTs with inline data in VALUES format, support semicolon as the data terminator, in addition to the new line. Closes . (). SYSTEM RELOAD CONFIG now throws an exception if failed to reload and continues using the previous users.xml. The background periodic reloading also continues using the previous users.xml if failed to reload. (). Add an option to skip access checks for DiskS3. (). ClickHouse treats partition expr and key expr differently. Partition expr is used to construct an minmax index containing related columns, while primary key expr is stored as an expr. Sometimes user might partition a table at coarser levels, such as `partition by i / 1000`. However, binary operators are not monotonic and this PR tries to fix that. It might also benifit other use cases. (). Fix some trailing whitespaces in query format. (). Add `QueryMemoryLimitExceeded` event. This closes . (). Fixed the backward-incompatible change by providing the options to build without debug info for functions. (). dynamic reload zookeeper config. (). Allow parallel execution of distributed DDL. (). Fix potential memory leak caused by zookeeper exists watch. (). Fixed "Packet payload is not fully read" error in `MaterializeMySQL` database engine. (). Fix crash in `bitShiftLeft()` when called with negative big integer. (). Add `merge_algorithm` to system.merges table to improve merging inspections. (). Less unneded code generated by DecimalBinaryOperation template in FunctionBinaryArithmetic. (). Now columns can be used to wrap over a list of columns and apply column transformers afterwards. (). Support for disabling persistency for StorageJoin and StorageSet, this feature is controlled by setting `disablesetandjoinpersistency`. And this PR solved issue . (). Construct `querystarttime` and `querystarttime_microseconds` from the same timespec. (). Allow using multi-volume storage configuration in storage Distributed. (). Show subqueries for `SET` and `JOIN` in `EXPLAIN` result. (). Provide a `loadbalancingfirstoffset` query setting to explicitly state what the first replica is. It's used together with `FIRSTOR_RANDOM` load balancing strategy, which allows to control replicas workload. (). Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. (). Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes . (). Avoid deadlock when executing INSERT SELECT into itself from a table with `TinyLog` or `Log` table engines. This closes . (). Ignore key constraints when doing mutations. Without this pr, it's not possible to do mutations when `forceindexbydate = 1` or `forceprimary_key = 1`. (). Add option to disable TTL move on data part insert.

Enable `Atomic` database engine by default. (). Proper exception message for wrong number of arguments of CAST. This closes . (). Add the ability to specify specialized codecs like `Delta`, `T64`, etc. for columns with subtypes. Implements , fixes , fixes . (). Added `optimize` setting to `EXPLAIN PLAN` query. If enabled, query plan level optimisations are applied. Enabled by default. (). Do not allow connections to ClickHouse server until all scripts in `/docker-entrypoint-initdb.d/` are executed. (). fix . (). Unfold `{database}`, `{table}` and `{uuid}` macros in `zookeeperpath` on replicated table creation. Do not allow `RENAME TABLE` if it may break `zookeeperpath` after server restart. Fixes . (). Add support for "Raw" column format for `Regexp` format. It allows to simply extract subpatterns as a whole without any escaping rules. (). Now it's possible to change the type of version column for `VersionedCollapsingMergeTree` with `ALTER` query. (). Wait for `DROP/DETACH TABLE` to actually finish if `NO DELAY` or `SYNC` is specified for `Atomic` database. (). Pass through foruser settings via Distributed with cluster-secure. (). Use experimental pass manager by default. (). Implement forcedataskipping_indices setting. (). Fix `currentDatabase()` function cannot be used in `ON CLUSTER` ddl query. (). Fixed the incorrect sorting order of `Nullable` column. This fixes . (). Fix executable dictionary source hang. In previous versions, when using some formats (e.g. `JSONEachRow`) data was not feed to a child process before it outputs at least something. This closes . This closes . (). Fix a bug when converting Nullable String to Enum. Introduced by https://github.com/ClickHouse/ClickHouse/pull/12745 . This fixes . (). Fix rare segfaults in functions with combinator -Resample, which could appear in result of overflow with very large parameters. (). Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. (). Added the checker as neither calling `lc->isNullable()` nor calling `ls->getDictionaryPtr()->isNullable()` would return the correct result. (). Fix wrong Decimal multiplication result caused wrong decimal scale of result column. (). Stuff the query into ASTFunction's argument list so that we don't break the presumptions of some AST visitors. This fixes . (). Fix bug when `ALTER UPDATE` mutation with Nullable column in assignment expression and constant value (like `UPDATE x = 42`) leads to incorrect value in column or segfault. Fixes , . (). Fixed missed default database name in metadata of materialized view when executing `ALTER ... MODIFY QUERY`. (). Replace column transformer should replace identifiers with cloned ASTs. This fixes . (). Fix wrong monotonicity detection for shrunk `Int -> Int` cast of signed types. It might lead to incorrect query result. This bug is unveiled in . (). Fix unreleased bug for LineAsString Format. (). Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the `from_zk` include option). This fixes . (). Fix rare error in `SELECT` queries when the queried column has `DEFAULT` expression which depends on the other column which also has `DEFAULT` and not present in select query and not exists on disk. Partially fixes . (). Fixed bug in parsing MySQL binlog events, which causes `Attempt to read after eof` and `Packet payload is not fully read` in `MaterializeMySQL` database engine. (). Fixed segfault in CacheDictionary . (). Fix SIGSEGV for an attempt to INSERT into StorageFile(fd). (). Fix the issue when some invocations of `extractAllGroups` function may trigger "Memory limit exceeded" error. This fixes . (). Fixed `.metadata.tmp File exists` error when using `MaterializeMySQL` database engine. (). Publish CPU frequencies per logical core in `system.asynchronous_metrics`. This fixes .

Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes . (). Update jemalloc to fix possible issues with percpu arena. (). Now settings `numberoffreeentriesinpooltoexecutemutation` and `numberoffreeentriesinpooltolowermaxsizeofmerge` can be equal to `backgroundpool_size`. (). Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. (). Fixed `Cannot rename ... errno: 22, strerror: Invalid argument` error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. (). If function `bar` was called with specifically crafted arguments, buffer overflow was possible. This closes . (). We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes . (). Fixes `Data compressed with different methods` in `joinalgorithm='auto'`. Keep LowCardinality as type for left table join key in `joinalgorithm='partial_merge'`. (). Adjust decimals field size in mysql column definition packet. (). Fix bug in table engine `Buffer` which doesn't allow to insert data of new structure into `Buffer` after `ALTER` query. Fixes . (). Fix instance crash when using joinGet with LowCardinality types. This fixes . (). Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. (). Fix MSan report in QueryLog. Uninitialized memory can be used for the field `memory_usage`. (). Fix hang of queries with a lot of subqueries to same table of `MySQL` engine. Previously, if there were more than 16 subqueries to same `MySQL` table in query, it hang forever. (). Fix rare race condition on server startup when system.logs are enabled. (). Fix race condition during MergeTree table rename and background cleanup. (). Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. (). Fixed compression in S3 storage. (). Fix multiple occurrences of column transformers in a select query. (). fixes fix attach mysql database engine throw exception(no query context). (). Report proper error when the second argument of `boundingRatio` aggregate function has a wrong type. (). Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in `ON CLUSTER`. Introduced in . (). Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. (). Fix `Missing columns` errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes . (). Fix bug when `ILIKE` operator stops being case insensitive if `LIKE` with the same pattern was executed. (). Mutation might hang waiting for some non-existent part after `MOVE` or `REPLACE PARTITION` or, in rare cases, after `DETACH` or `DROP PARTITION`. It's fixed. (). Fix 'Database <db> doesn't exist.' in queries with IN and Distributed table when there's no database on initiator. (). Significantly reduce memory usage in AggregatingInOrderTransform/optimizeaggregationin_order. (). Prevent the possibility of error message `Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call`. This fixes . (). Query is finished faster in case of exception. Cancel execution on remote replicas if exception happens. (). Fixed `Element ... is not a constant expression` error when using `JSON` function result in `VALUES`, `LIMIT` or right side of `IN` operator. (). Fix the order of destruction for resources in `ReadFromStorage` step of query plan. It might cause crashes in rare cases. Possibly connected with . (). Proper error handling during insert into MergeTree with S3. (). Fix race condition in AMQP-CPP. (). Fix rare race condition in dictionaries and tables from MySQL.

Fixed too low default value of `maxreplicatedlogstokeep` setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. (). Fix error `Cannot find column` which may happen at insertion into `MATERIALIZED VIEW` in case if query for `MV` containes `ARRAY JOIN`. (). Fix some cases of queries, in which only virtual columns are selected. Previously `Not found column _nothing in block` exception may be thrown. Fixes . (). Control CI builds configuration from the ClickHouse repository. (). Now ClickHouse uses gcc-10 for the release build. Fixes . (). Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with `madvise` to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. (). In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 -> `OFF`, 1 -> `ON` in `option`s default values. - Added some descriptions and links to docs to the options. - Replaced `FUZZER` option (there is another option `ENABLEFUZZING` which also enables same functionality). - Removed `ENABLEGTESTLIBRARY` option as there is `ENABLETESTS`. (). Speed up build a little by removing unused headers. (). Fix build failure in OSX. (). Attempt to speed up build a little. (). Now we use clang-11 to build ClickHouse in CI. (). #14809 fix MaterializeMySQL empty transaction unstable test case found in CI. (). Reformat and cleanup code in all integration test .py files. (). Fixing tests/integration/testdistributedoverliveview/test.py. (). Switch from `clang-tidy-10` to `clang-tidy-11`. (). Convert to python3. This closes . (). Make performance test more stable and representative by splitting test runs and profile runs. (). Maybe fix MSan report in base64 (on servers with AVX-512). This fixes . (). Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes . (). Now all test images use `llvm-symbolizer-11`. (). Splitted huge test `testdictionariesalllayoutsand_sources` into smaller ones. (). Added a script to perform hardware benchmark in a single command. (). Fix CMake options forwarding in fast test script. Fixes error in . (). Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. (). Now we use clang-11 for production ClickHouse build. (). Allow to run AArch64 version of clickhouse-server without configs. This facilitates . (). Fail early in functional tests if server failed to respond. This closes . (). fix bug for build error: . (). fix bug for building querydbgenerator.cpp. (). Allow to build with llvm-11. (). Switch binary builds(Linux, Darwin, AArch64, FreeDSD) to clang-11. (). Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes . (). Make binary a bit smaller (~50 Mb for debug version). (). NO CL ENTRY: 'Bump numpy from 1.19.1 to 1.19.2 in /docs/tools'. (). NO CL ENTRY: 'Modify the minimum version of the Clang compiler'. (). NO CL ENTRY: 'fix a syntax error bug while using copier'. (). NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.9 to 0.4.13 in /docs/tools'. (). NO CL ENTRY: 'Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka"'. (). NO CL ENTRY: 'Revert "Avoid deadlocks in Log/TinyLog"'. (). NO CL ENTRY: 'Bump mkdocs-macros-plugin from 0.4.13 to 0.4.17