Project: ClickHouse (App Definition and Development / Database)
File: v21.2.1.5869-prestable.md

sidebar_position: 1 sidebar_label: 2022 - Fix memory tracking for `OPTIMIZE TABLE`/merges - Account query memory limits and sampling for `OPTIMIZE TABLE`/merges. (). Forbid `lcm`/`gcd` for floats. (). Bitwise functions (`bitAnd`, `bitOr`, etc) are forbidden for floating point arguments. Now you have to do explicit cast to integer. (). add support for zstd long option for better compression of string columns to save space. (). - Added support of mapping LDAP group names, and attribute values in general, to local roles for users from ldap user directories. (). Data type `Nested` now supports arbitrary levels of nesting. Introduced subcolumns of complex types, such as `size0` in `Array`, `null` in `Nullable`, names of `Tuple` elements, which can be read without reading of whole column. (). Add support of tuple argument to `argMin` and `argMax` functions. (). Added `Nullable` support for `FlatDictionary`, `HashedDictionary`, `ComplexKeyHashedDictionary`, `DirectDictionary`, `ComplexKeyDirectDictionary`, `RangeHashedDictionary`. (). Disallow floating point column as partition key related to : #event-4147046255. (). Add function decodeXMLComponent to decode characters for XML. ``` SELECT decodeXMLComponent('Hello,&quot;world&quot;!'); ``` . (). Added PostgreSQL table engine (both select/insert, with support for multidimensional arrays), also as table function. Added PostgreSQL dictionary source. Added PostgreSQL database engine. (). Add `SELECT ALL` syntax. closes . (). Add three functions for map data type: 1. mapContains(map, key) to check weather map.keys include the second parameter key. 2. mapKeys(map) return all the keys in Array format 3. mapValues(map) return all the values in Array format. (). Support MetaKey+Enter hotkey binding in play ui. (). Function formatDateTime support the %Q modification to format date to quarter. ... (). ... (). Add factories' objects names, created during query, into system.query_log. Closes . (). Add `sign` math function. (). Added functions `parseDateTimeBestEffortUSOrZero`, `parseDateTimeBestEffortUSOrNull`. (). ... (). Use a connection pool for S3 connections, controlled by the `s3maxconnections` settings. (). Rewrite `sumIf()` and `sum(if())` function to `countIf()` function when logically equivalent. (). Update libcxx and use unstable ABI to provide better performance. (). Faster parts removal by lowering the number of `stat` syscalls. This returns the optimization that existed while ago. More safe interface of `IDisk`. This closes . (). Speed up aggregate function `sum`. Improvement only visible on synthetic benchmarks and not very practical. (). Support splitting `Filter` step of query plan into `Expression + Filter` pair. Together with `Expression + Expression` merging optimization () it may delay execution for some expressions after `Filter` step. (). Reduce lock contention for multiple layers of the Buffer engine. (). Slightly improve server latency by removing access to configuration on every connection. (). Added support for `WITH ... (). Add optimize_alias_column_prediction (on by default), that will: Respect aliased columns in WHERE during partition pruning and skipping data using secondary indexes Respect aliased columns in WHERE for trivial count queries for optimize_trivial_count Respect aliased columns in GROUP BY/ORDER BY for optimizeaggregationinorder/optimizereadinorder. (). Updated AWS C++ SDK in order to utilize global regions in S3. (). Support insert into table function `cluster`, and for both table functions `remote` and `cluster`, support distributing data across nodes by specify sharding key. Close . (). Support `EXISTS VIEW` syntax. (). Update librdkafka to v1.6.0-RC2. Fixes . (). Allow CTE to be further aliased. Propagate CSE to subqueries in the same level when `enableglobalwith_statement = 1`. This fixes . This fixes https://github.com/ClickHouse/ClickHouse/pull/16575#issuecomment-753416235 . (). Add . (). Added prefix-based S3 endpoint settings. (). Fix issues with RIGHT and FULL JOIN of tables with aggregate function states. In previous versions exception about `cloneResized` method was thrown.

- Check per-block checksum of the distributed batch on the sender before sending (without reading the file twice, the checksums will be verified while reading), this will avoid stuck of the INSERT on the receiver (on truncated .bin file on the sender) - Avoid reading .bin files twice for batched INSERT (it was required to calculate rows/bytes to take squashing into account, now this information included into the header, backward compatible is preserved). (). Add `normalizeQueryKeepNames` and `normalizedQueryHashKeepNames` to normalize queries without masking long names with `?`. This helps better analyze complex query logs. (). Docker image: several improvements for clickhouse-server entrypoint. (). Fixed `PeekableReadBuffer: Memory limit exceed` error when inserting data with huge strings. Fixes . (). Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server images. (). The exception when function `bar` is called with certain NaN argument may be slightly misleading in previous versions. This fixes . (). Allow change `maxservermemory_usage` without restart. This closes . (). Fix wrong alignment of values of `IPv4` data type in Pretty formats. They were aligned to the right, not to the left. This closes . (). Allow docker to be executed with arbitrary uid. (). Add metrics for MergeTree parts (Wide/Compact/InMemory) types. (). Improve MySQL compatibility. (). Add `httpreferer` field to `system.querylog`, `system.processes`, etc. This closes . (). `toIPv6` function parses `IPv4` addresses. (). Support using the new location of `.debug` file. This fixes . (). Enable function length/empty/notEmpty for datatype map, which returns keys number in map. (). Support constant result in function `multiIf`. (). Add an option to disable validation of checksums on reading. Should never be used in production. Please do not expect any benefits in disabling it. It may only be used for experiments and benchmarks. The setting only applicable for tables of MergeTree family. Checksums are always validated for other table engines and when receiving data over network. In my observations there is no performance difference or it is less than 0.5%. (). Dictionary better error message during attribute parsing. (). Fix rare `maxnumberofmergeswithttlin_pool` limit overrun (more merges with TTL can be assigned) for non-replicated MergeTree. (). Insuffiient arguments check in `positionCaseInsensitiveUTF8` function triggered address sanitizer. (). Add separate pool for message brokers (RabbitMQ and Kafka). (). In distributed queries if the setting `asyncsocketfor_remote` is enabled, it was possible to get stack overflow at least in debug build configuration if very deeply nested data type is used in table (e.g. `Array(Array(Array(...more...)))`). This fixes . This change introduces minor backward incompatibility: excessive parenthesis in type definitions no longer supported, example: `Array((UInt8))`. (). Table function `S3` will use global region if the region can't be determined exactly. This closes . (). Clickhouse client query param CTE added test. (). Correctly output infinite arguments for `formatReadableTimeDelta` function. In previous versions, there was implicit conversion to implementation specific integer value. (). `S3` table function now supports `auto` compression mode (autodetect). This closes . (). Set charset to utf8mb4 when interacting with remote MySQL servers. Fixes . (). Add `--reconnect` option to `clickhouse-benchmark`. When this option is specified, it will reconnect before every request. This is needed for testing. (). fix data type convert issue for mysql engine ... (). `SELECT count() FROM table` now can be executed if only one any column can be selected from the `table`. This PR fixes . (). Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes .

Disable constant folding for subqueries on the analysis stage, when the result cannot be calculated. (). Attach partition should reset the mutation. . (). Fix bug when mutation with some escaped text (like `ALTER ... UPDATE e = CAST('foo', 'Enum8(\'foo\' = 1')` serialized incorrectly. Fixes . (). Fix error `Task was not found in task queue` (possible only for remote queries, with `asyncsocketfor_remote = 1`). (). to fix Add a check to avoid exception when long column alias('table.column' style, usually auto-generated by BI tools like Looker) equals to long table name. (). Fix incorrect behavior when `ALTER TABLE ... DROP PART 'part_name'` query removes all deduplication blocks for the whole partition. Fixes . (). Fixed rare crashes when server run out of memory. (). Fixed very rare deadlock at shutdown. (). Fix possible exception `QueryPipeline stream: different number of columns` caused by merging of query plan's `Expression` steps. Fixes . (). Disable `optimizemovefunctionsoutof_any` because optimization is not always correct. This closes . This closes . (). Join tries to materialize const columns, but our code waits for them in other places. (). Fix inserting of `LowCardinality` column to table with `TinyLog` engine. Fixes . (). Fix possible error `Expected single dictionary argument for function` if use function `ignore` with `LowCardinality` argument. Fixes . (). Make sure `groupUniqArray` returns correct type for argument of Enum type. This closes . (). Restrict `MODIFY TTL` queries for `MergeTree` tables created in old syntax. Previously the query succeeded, but actually it had no effect. (). Fixed `There is no checkpoint` error when inserting data through http interface using `Template` or `CustomSeparated` format. Fixes . (). Simplify the implementation of `tupleHammingDistance`. Support for tuples of any equal length. Fixes . (). Fix startup bug when clickhouse was not able to read compression codec from `LowCardinality(Nullable(...))` and throws exception `Attempt to read after EOF`. Fixes . (). Fix bug in merge tree data writer which can lead to marks with bigger size than fixed granularity size. Fixes . (). Fix infinite reading from file in `ORC` format (was introduced in ). Fixes . (). - Split RemoteQueryExecutorReadContext into module part - Fix leaking of pipe fd for `asyncsocketfor_remote`. (). Fix bug when concurrent `ALTER` and `DROP` queries may hang while processing ReplicatedMergeTree table. (). Do not mark file for distributed send as broken on EOF. (). Fix error `Cannot convert column now64() because it is constant but values of constants are different in source and result`. Continuation of . (). Fixed possible wrong result or segfault on aggregation when Materialized View and its target table have different structure. Fixes . (). Fix system.parts state column (LOGICALERROR when querying this column, due to incorrect order). (). Added `cast`, `accurateCast`, `accurateCastOrNull` performance tests. (). - Fix default value in join types with non-zero default (e.g. some Enums). Closes . (). Fix possible buffer overflow in Uber H3 library. See https://github.com/uber/h3/issues/392. This closes . (). Uninitialized memory read was possible in encrypt/decrypt functions if empty string was passed as IV. This closes . (). Fix possible error `Extremes transform was already added to pipeline`. Fixes . (). Fixed very rare bug that might cause mutation to hang after `DROP/DETACH/REPLACE/MOVE PARTITION`. It was partially fixed by for the most cases. (). Mark distributed batch as broken in case of empty data block in one of files. (). Buffer overflow (on memory read) was possible if `addMonth` function was called with specifically crafted arguments. This fixes . This fixes .

Fix wrong deserialization of columns description. It makes INSERT into a table with a column named `\` impossible. (). Fix SIGSEGV with mergetreeminrowsforconcurrentread/mergetreeminbytesforconcurrentread=0/UINT64_MAX. (). Query CREATE DICTIONARY id expression fix. (). `DROP/DETACH TABLE table ON CLUSTER cluster SYNC` query might hang, it's fixed. Fixes . (). Fix use-after-free of the CompressedWriteBuffer in Connection after disconnect. (). Fix wrong result of function `neighbor` for `LowCardinality` argument. Fixes . (). Some functions with big integers may cause segfault. Big integers is experimental feature. This closes . (). Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption. Fixes . (). Fix a segmentation fault in `bitmapAndnot` function. Fixes . (). Fix crash when nested column name was used in `WHERE` or `PREWHERE`. Fixes . (). Fixed stack overflow when using accurate comparison of arithmetic type with string type. (). In previous versions, unusual arguments for function arrayEnumerateUniq may cause crash or infinite loop. This closes . (). The function `greatCircleAngle` returned inaccurate results in previous versions. This closes . (). Fix filtering by UInt8 greater than 127. (). Fix crash when pushing down predicates to union distinct subquery. This fixes . (). Fix argMin/argMax crash when combining with -If. This fixes https://clickhouse-test-reports.s3.yandex.net/19800/7b8589dbde5bc621d1bcfd68a713e4684183f593/fuzzer_ubsan/report.html#fail1. (). Restore Kafka input in FreeBSD builds. (). Add integration tests run with memory sanitizer. (). Add SQLancer test docker image to run check in CI. (). Added tests for defaults in URL and File engine. This closes . (). - Enabling RBAC tests - Tests for SYSTEM privileges - Requirements name changes. (). Query Fuzzer will fuzz newly added tests more extensively. This closes . (). Allow building librdkafka without ssl. (). Avoid UBSan reports in `arrayElement` function, `substring` and `arraySum`. Fixes . Fixes . This closes . (). Fix potential nullptr dereference in table function `VALUES`. (). Allow building ClickHouse with Kafka support on arm64. (). Integrate with for better fuzzing. (). Allow to explicitly enable or disable watchdog via environment variable `CLICKHOUSEWATCHDOGENABLE`. By default it is enabled if server is not attached to terminal. (). Updating TestFlows AES encryption tests to support changes to the encrypt plaintext parameter. (). Made generation of macros.xml easier for integration tests. No more excessive logging from dicttoxml. dicttoxml project is not active for 5+ years. (). Remove --project-directory for docker-compose in integration test. Fix logs formatting from docker container. (). Fixed MemorySanitizer errors in cyrus-sasl and musl. (). Add test for throwing an exception on inserting incorrect data in CollapsingMergeTree. (). Adding retries for docker-compose start, stop and restart in TestFlows tests. (). NO CL ENTRY: 'Remove useless codes'. (). NO CL ENTRY: 'Merging '. (). Add ANTLR tests check (). Add more debug info to PollingQueue exception. (). More checks in merge tree writer wide (). Try to remove ActionsDAG::removeColumn (). Fix rarely flaky test 01459manualwritetoreplicas_quorum (). fix some wrong words in comment (). Add tests from (). Fix manipulators for common string types (). Fix misleading error message while inserting in a table function (). Fix duplicate UUIDs of LiveView on server startup (). (). Avoid redundant exception while dropping part (). More debug for stateless tests writer (). Update test containers (). Support operations with views in ANTLR parser (). Run more tests in SQLancer (). Update Cassandra driver library (). Add metrics for part number in MergeTree in ClickHouse (). Remove useless code (). (). MemoryTracker: Do not ignore server memory limits during blocking by default (). speed up some perf tests (for other machines)

Cover distributed send with different headers (). Add missing type check in dictHas for COMPLEXKEYSSD_CACHE layout (). test for Expandmacrosfor_fetchPartition (). add MySQL Var check (). Tiny changes in DistributedBlockOutputStream (). curl dependency tiny fixes (). Fix MSan error in rocksdb (). Fix MSan report in Kerberos library (). Update Dragonbox (). Make symbolizers available in fuzzer Docker image (). Add redundant test for Yandex banner system (). Make `-- { echo }` hint preserve leading comments (). Disable msan for epoll array in libuv (). Remove tsan supression (). Consolidate the test hint handling (). Support 'keys' and 'values' subcolumns in type Map (). Split TestKeeperStorage and processing thread (). move ctr from private to delete (). Avoid mixing output from parallel test runs (). Fixed race between copy-constructor and addQueryAccessInfo (). Add Sanitizer report issue template (). fix sleep with infinite input (). ISSUES-18684 fix MaterializeMySQL integration test failure (). Fix race condition in TestKeeperHandler on session finish (). Fix several cases, while reading subcolumns (). Reconnect after client errors (). SQLancer binary changed its name (). Better logging in MySQLHandler (). Fix IDisk::open parameters to match posix open (). MacOS fixed build issues (). Add log message with elapsed time while pushing to view (). Avoid UBSan report in aggregate function sum (). Fail fast in incorrect usage of extractAllGroups (). Fix UBSan report in quantileExactWeighted (). Limit max memory usage in fuzz testing (). Check literal types in RewriteSumIfFunctionMatcher (). mirror changes in code and comment (). Fix one more race in TestKeeper (). Continue fix for Block structure mismatch in PipelineExecuting stream (). Integration tests: print stderr of failed subprocess (). Don't dwell on the past in query fuzzer (). Remove outdated suppressions (). Fix build with unbundled poco (). Fix UBSan report in arraySlice and substring (). Fix UBSan report in runningDifference (). Avoid UBSan report in greatCircleDistance (). UBsan report fixes (arraySlice, addMinutes/addHours/addWeeks/addDays, sumWithOverflow(Decimal)) (). Remove complications from FunctionsAES (). Fix insufficient args check in AES functions (). Fix UBSan report in avgWeighted (). remove part of misleading exception message (). Bug fix : support const column processing in mapContains, mapKeys, mapValues functions (). More diagnostics in fuzzer (). Fix 01459manualwritetoreplicas flaky test (). Check for hung queries or server hung in fast test (). DateLUTImpl::addYears(...): suppress UBSan (). Merging (). Update Cassandra submodule with patch by @leshikus (). Remove unused include header (). Fix merge join constants (). Better abstractions in disk interface (). Fix deadlock in testkeeper (). fix special build on clang11 (). Require currentdatabase filter for tests with querylog/querythreadlog (). clickhouse-test tiny cleanup (). Fix crash after merging ActionsDAG. (). Fix flaky test concatnaryconstwithnonconst_segfault (). Run tests in parallel in flaky check (). Fix 01115joinwith_dictionary flakiness (). add empty line after error messages in client (). Update QueryPlan tree optimization traverse. (). Add missing lsof for fasttest docker image (). Make Fuzzer more reliable (). Minor code improvement in JOIN (). Make integration odbc tests idempotent (). Small fixes for fasttest (). LowCardinality UUID fix (). Add log comment when running .sh tests (). Fix UBSan report in "round" (). Add test script for (). Fix dependency on ODBC for Yandex internal build (). Don't run all stateless tests in parallel (). Avoid losing exception messages in logs under high memory pressure (). Try to make test_dir.tar smaller (). style-check tiny fixes (). Fix UBSan report in DateTimeAddInterval (). Fix debug assertion in Hyperscan (). Fix UBSan report in GatherUtils ().