Project: ClickHouse (App Definition and Development / Database)
File: v21.6.1.6891-prestable.md

sidebar_position: 1 sidebar_label: 2022 Add projection support for MergeTree tables. (). Add back `indexHint` function. This is for . This reverts https://github.com/ClickHouse/ClickHouse/pull/9542 . This fixes . (). Add aggregate function sumCount. This function returns a tuple of two fields: sum and count. (). Added less secure IMDS credentials provider for S3 which works under docker correctly. (). - New aggregate function `deltaSumTimestamp` for summing the difference between consecutive rows while maintaining ordering during merge by storing timestamps. (). - LDAP: implemented user DN detection functionality to use when mapping Active Directory groups to ClickHouse roles. (). Introduce a new function: arrayProduct which accept an array as the parameter, and return the product of all the elements in array. Close issue: . (). Add setting `indexes` (boolean, disabled by default) to `EXPLAIN PIPELINE` query. When enabled, shows used indexes, number of filtered parts and granules for every index applied. Supported for `MergeTree` tables. (). Add setting `json` (boolean, 0 by default) for `EXPLAIN PLAN` query. When enabled, query output will be a single `JSON` row. It is recommended to use `TSVRaw` format to avoid unnecessary escaping. (). Added `SYSTEM QUERY RELOAD MODEL`, `SYSTEM QUERY RELOAD MODELS`. Closes . (). Made progress bar for LocalServer and united it for Client and Local. (). Support DDL dictionaries for DatabaseMemory. Closes . Added support for `DETACH DICTIONARY PERMANENTLY`. Added support for `EXCHANGE DICTIONARIES` for Atomic database engine. Added support for moving dictionaries between databases using `RENAME DICTIONARY`. (). Allow globs `{...}`, which act like shards, and failover options with separator `|` for URL table function. Closes . (). If `insertnullas_default` = 1, insert default values instead of NULL in `INSERT ... SELECT` and `INSERT ... SELECT ... UNION ALL ...` queries. Closes . (). Implement table comments. closes . (). Introduce a new function: arrayProduct which accept an array as the parameter, and return the product of all the elements in array. Closes . (). Add postgres-like cast operator (`::`). E.g.: ` (). ... (). Add function splitByRegexp. (). Add `threadname` column in `system.stacktrace`. This closes . (). Enable `compile_expressions` setting by default. When this setting enabled, compositions of simple functions and operators will be compiled to native code with LLVM at runtime. (). ORC input format reading by stripe instead of reading entire table into memory by once which is cost memory when file size is huge. (). Update `re2` library. Performance of regular expressions matching is improved. Also this PR adds compatibility with gcc-11. (). Support Array data type for inserting and selecting data in Arrow, Parquet and ORC formats. (). Add settings `externalstoragemaxreadrows` and `externalstoragemaxreadrows` for MySQL table engine, dictionary source and MaterializeMySQL minor data fetches. (). Retries on HTTP connection drops in S3. (). Fix the case when a progress bar in interactive mode in clickhouse-client that appear in the middle of the data may rewrite some parts of visible data in terminal. This closes . (). Added possibility to restore MergeTree parts to 'detached' directory for DiskS3. (). Skip unavaiable replicas when writing to distributed tables. (). Support LowCardinality nullability with `joinusenulls`, close .

Disable settings `usehedgedrequests` and `asyncsocketfor_remote` because there is an evidence that it may cause issues. (). Fixed `quantile(s)TDigest`. Added special handling of singleton centroids according to tdunning/t-digest 3.2+. Also a bug with over-compression of centroids in implementation of earlier version of the algorithm was fixed. (). Allow user to specify empty string instead of database name for `MySQL` storage. Default database will be used for queries. In previous versions it was working for SELECT queries and not support for INSERT was also added. This closes . This can be useful working with `Sphinx` or other MySQL-compatible foreign databases. (). Disable minbytestousemmap_io by default. (). If user applied a misconfiguration by mistakenly setting `maxdistributedconnections` to value zero, every query to a `Distributed` table will throw exception with a message containing "logical error". But it's really an expected behaviour, not a logical error, so the exception message was slightly incorrect. It also triggered checks in our CI enviroment that ensures that no logical errors ever happen. Instead we will treat `maxdistributedconnections` misconfigured to zero as the minimum possible value (one). (). Keep default timezone on DateTime operations if it was not provided explicitly. For example, if you add one second to a value of `DateTime` type without timezone it will remain `DateTime` without timezone. In previous versions the value of default timezone was placed to the returned data type explicitly so it becomes DateTime('something'). This closes . (). Previously, MySQL 5.7.9 was not supported due to SQL incompatibility. Now leave MySQL parameter verification to the MaterializeMySQL. (). Possibility to change S3 disk settings in runtime via new `SYSTEM RESTART DISK` SQL command. (). Respect lockacquiretimeoutforbackground_operations for OPTIMIZE. (). Make big integers production ready. Add support for `UInt128` data type. Fix known issues with the `Decimal256` data type. Support big integers in dictionaries. Support `gcd`/`lcm` functions for big integers. Support big integers in array search and conditional functions. Support `LowCardinality(UUID)`. Support big integers in `generateRandom` table function and `clickhouse-obfuscator`. Fix error with returning `UUID` from scalar subqueries. This fixes . This fixes . This fixes . This fixes . This fixes . Backward incompatible change: values of `UUID` type cannot be compared with integer. For example, instead of writing `uuid != 0` type `uuid != '00000000-0000-0000-0000-000000000000'`. (). Add `partitionvalue` virtual column to MergeTree table family. It can be used to prune partition in a deterministic way. It's needed to implement partition matcher for mutations. (). Enable `asyncsocketfor_remote` by default. (). When there is some `ReplicatedMergeTree` tables whose zookeeper is expired, it will throw the error below when we select the meta data of some table from `system.tables` with `selectsequentialconsistency` is enabled: `Session expired (Session expired): While executing Tables`. (). Added `region` parameter for S3 storage and disk. (). Allow configuring different log levels for different logging channels. Closes . (). Add `brokendatafiles`/`brokendatacompressedbytes` into `system.distributionqueue`. Add metric for number of files for asynchronous insertion into Distributed tables that has been marked as broken (`BrokenDistributedFilesToInsert`). (). Allow to add specific queue settings via table settng `rabbitmqqueuesettingslist`. (Closes and

Allow user to control all RabbitMQ setup: if table setting `rabbitmqqueue_consume` is set to `1` - RabbitMQ table engine will only connect to specified queue and will not perform any RabbitMQ consumer-side setup like declaring exchange, queues, bindings. (Closes ). Add proper cleanup when RabbitMQ table is dropped - delete queues, which the table has declared and all bound exchanges - if they were created by the table. (). Measure found rate (the percentage for which the value was found) for dictionaries (see `found_rate` in `system.dictionaries`). (). Add hints for Enum names. Closes . (). Add support for HTTP compression (determined by `Content-Encoding` HTTP header) in `http` dictionary source. This fixes . (). Preallocate support for hashed/sparse_hashed dictionaries. (). Support specifying table schema for postgresql dictionary source. Closes . (). Log information about OS name, kernel version and CPU architecture on server startup. (). enable DateTime64 to be a version column in ReplacingMergeTree. (). Add support for `ORDER BY WITH FILL` with `DateTime64`. (). Now `prefercolumnnametoalias = 1` will also favor column names for `group by`, `having` and `order by`. This fixes . (). Do not acquire lock for totalbytes/totalrows for Buffer engine. (). Flush Buffer tables before shutting down tables (within one database), to avoid discarding blocks due to underlying table had been already detached (and `Destination table default.adata01870 doesn't exist. Block of data is discarded` error in the log). (). Preserve dictionaries until storage shutdown (this will avoid possible `external dictionary 'DICT' not found` errors at server shutdown during final Buffer flush). (). Update zstd to v1.5.0. (). Fix crash when memory allocation fails in simdjson. https://github.com/simdjson/simdjson/pull/1567 . Mark as improvement because it's a rare bug. (). This PR fixes a crash on shutdown which happened because of currentConnections() could return zero while some connections were still alive. (). QueryAliasVisitor to prefer alias for ASTWithAlias if subquery was optimized to constant. Fixes . Fixes . (). Fixed `Not found column` error when selecting from `MaterializeMySQL` with condition on key column. Fixes . (). Fixed the behavior when disabling `inputformatwithnamesuse_header ` setting discards all the input with CSVWithNames format. This fixes . (). Add type conversion for optimizeskipunusedshardsrewritein (fixes `use-of-uninitialized-value` with `optimizeskipunusedshardsrewritein`). (). Fixed simple key dictionary from DDL creation if primary key is not first attribute. Fixes . (). Fixed very rare (distributed) race condition between creation and removal of ReplicatedMergeTree tables. It might cause exceptions like `node doesn't exist` on attempt to create replicated table. Fixes . (). Fixed very rare race condition on background cleanup of old blocks. It might cause a block not to be deduplicated if it's too close to the end of deduplication window. (). Fix possible crash in case if `unknown packet` was received form remote query (with `asyncsocketfor_remote` enabled). Maybe fixes . (). Don't relax NOT conditions during partition pruning. This fixes and . (). Fix bug in dict join with join_algorithm = 'auto'. Close . (). Fix possible `Block structure mismatch` error for queries with `UNION` which could possibly happen after filter-push-down optimization. Fixes . (). Fix incompatible constant expression generation during partition pruning based on virtual columns. This fixes https://github.com/ClickHouse/ClickHouse/pull/21401#discussion_r611888913.

`ORDER BY` with `COLLATE` was not working correctly if the column is in primary key (or is a monotonic function of it) and the setting `optimizereadinorder` is not turned off. This closes . Workaround for older versions: turn the setting `optimizereadinorder` off. (). Remove support for `argMin` and `argMax` for single `Tuple` argument. The code was not memory-safe. The feature was added by mistake and it is confusing for people. These functions can be reintroduced under different names later. This fixes and reverts . (). Allow to move more conditions to `PREWHERE` as it was before version 21.1. Insufficient number of moved condtions could lead to worse performance. (). Kafka storage may support parquet format messages. (). Kafka storage may support `arrow` and `arrowstream` format messages. (). Fixed `Cannot unlink file` error on unsuccessful creation of ReplicatedMergeTree table with multidisk configuration. This closes . (). - Bug fix for `deltaSum` aggregate function in counter reset case ... (). Fix bug that does not allow cast from empty array literal, to array with dimensions greater than 1. Closes . (). Fix corner cases in vertical merges with `ReplacingMergeTree`. In rare cases they could lead to fails of merges with exceptions like `Incomplete granules are not allowed while blocks are granules size`. (). When modify column's default value without datatype, and this column is used as ReplacingMergeTree's parameter like column `b` in the below example, then the server will core dump: ``` CREATE TABLE altertest (a Int32, b DateTime) ENGINE = ReplacingMergeTree(b) ORDER BY a; ALTER TABLE altertest MODIFY COLUMN `b` DEFAULT now(); ``` the sever throw error: ``` 2021.04.22 09:48:00.685317 (). Fix `columns` function when multiple joins in select query. Closes . (). Fix bug with `Join` and `WITH TOTALS`, close . (). Fix restart / stop command hanging. Closes . (). Fix misinterpretation of some `LIKE` expressions with escape sequences. (). Fixed server fault when inserting data through HTTP caused an exception. This fixes . (). Added an exception in case of completely the same values in both samples in aggregate function `mannWhitneyUTest`. This fixes . (). Fixed a bug in recovery of staled `ReplicatedMergeTree` replica. Some metadata updates could be ignored by staled replica if `ALTER` query was executed during downtime of the replica. (). Avoid possible "Cannot schedule a task" error (in case some exception had been occurred) on INSERT into Distributed. (). Fix `heapuseafter_free` when reading from hdfs if `Values` format is used. (). Fix crash when `PREWHERE` and row policy filter are both in effect with empty result. (). Fixed remote JDBC bridge timeout connection issue. Closes . (). Fix `CLEAR COLUMN` does not work when it is referenced by materialized view. Close . (). Fix error `Can't initialize pipeline with empty pipe` for queries with `GLOBAL IN/JOIN` and `usehedgedrequests`. Fixes . (). Better handling of URI's in `PocoHTTPClient`. Fixed bug with URLs containing `+` symbol, data with such keys could not be read previously. (). HashedDictionary complex key update field initial load fix. Closes . (). Better handling of HTTP errors in `PocoHTTPClient`. Response bodies of HTTP errors were being ignored earlier.

Fix `distributedgroupbynomerge=2` with `GROUP BY` and aggregate function wrapped into regular function (had been broken in ). Throw exception in case of someone trying to use `distributedgroupbynomerge=2` with window functions. Disable `optimizedistributedgroupbysharding_key` for queries with window functions. (). Fix implementation of connection pool of PostgreSQL engine. Closes . (). Fix keys metrics accounting for CACHE() dictionary with duplicates in the source (leads to `DictCacheKeysRequestedMiss` overflows). (). Fix SIGSEGV for external GROUP BY and overflow row (i.e. queries like `SELECT FROM GROUP BY WITH TOTALS SETTINGS maxbytesbeforeexternalgroupby>0, maxrowstogroupby>0, groupbyoverflowmode='any', totalsmode='beforehaving'`). (). Some `ALTER PARTITION` queries might cause `Part A intersects previous part B` and `Unexpected merged part C intersecting drop range D` errors in replication queue. It's fixed. Fixes . (). Fix crash in MergeJoin, close . (). now64() supports optional timezone argument ... (). Fixed using const `DateTime` value vs `DateTime64` column in WHERE. ... (). Bug: explain pipeline with` select xxx final `shows wrong pipeline: ``` dell123 :) explain pipeline select z from prewheremoveselect_final final;. (). Fix a rare bug that could lead to a partially initialized table that can serve write requests (insert/alter/so on). Now such tables will be in readonly mode. (). Fix race condition which could happen in RBAC under a heavy load. This PR fixes , ,. (). Update nested column with const condition will make server crash. ``` CREATE TABLE testwidenested ( `id` Int, `info.id` Array(Int), `info.name` Array(String), `info.age` Array(Int) ) ENGINE = MergeTree ORDER BY tuple() SETTINGS minbytesforwidepart = 0; set mutations_sync = 1;. (). Fix abnormal server termination due to hdfs becoming not accessible during query execution. Closes . (). Fix wrong typo at StorageMemory, this bug was introduced at , now fixed, Closes . (). Adding Map type tests in TestFlows. (). Testflows tests for DateTime64 Extended Range. (). CMake will be failed with settings as bellow ` -DENABLECASSANDRA=OFF -DENABLEAMQPCPP=ON ` ... (). Add simple tool for benchmarking (). Remove a source of nondeterminism from build. Now builds at different point of time will produce byte-identical binaries. Partially addressed . (). Avoid possible build dependency on locale and filesystem order. This allows reproducible builds. (). Always enable asynchronous-unwind-tables explicitly. It may fix query profiler on AArch64. (). Fix Memory Sanitizer report in GRPC library. This closes . (). Window functions tests in TestFlows. (). Adds support for building on Solaris-derived operating systems. (). Update librdkafka 1.6.0-RC3 to 1.6.1. (). Enabling running of all TestFlows modules in parallel. (). Fixing window functions distributed tests by moving to a deterministic sharding key. (). Add more benchmarks for hash tables, including the Swiss Table from Google (that appeared to be slower than ClickHouse hash map in our specific usage scenario). (). Support building on Illumos. (). Automated backporting now looks at the label 'release' of PRs to consider the release branch. (). Add test cases for arrayElement. related issue: . (). Rename uniqThetaSketch (https://github.com/ClickHouse/ClickHouse/issues/14893) to uniqTheta. (). NO CL ENTRY: 'Revert "Function `arrayFold` for folding over array with accumulator"'. (). NO CL ENTRY: 'Revert " (). NO CL ENTRY: 'Revert "add uniqThetaSketch"'. (). NO CL ENTRY: 'Revert "Fix CI build for gcc-10"'. (). NO CL ENTRY: 'Update syntax.md'.

Apply idleconnnectiontimeout/poll_interval after each query (). Do not silently catch errors for writing to S3 (). Add dockerhub-proxy to runner (). merging sumCount fusion PR (). Minor fixes in ATTACH query (). Merging (). Fix logical error in stress tests (). more stable formatting for negate() (). Use double quote identifier in odbc as default in case of error (). Add -Wundef for gcc builds (). Report an error if jemalloc.background_thread was requested (). Add trace_log into stateless/stress test artifacts (). Zlib use attribute constructor for functable initialization (). Fix integration tests for Hedged requests (). Function toDateTime decimal overflow ubsan fix (). Link keeper-bench to clickhousecommonzookeeper (). Print errors on db creation in clickhouse-test (). fix broken perf test (). Fix bad test 01602maxdistributed_connections (). Fix hdfs reading from files with spaces (). add check that p.pruning works (). Add test for (). Fix flaky test 01666mergetreemaxquery_limit.sh (). Add test for (). Disable clickhouse-odbc-bridge build when ODBC is disabled (). Fix AppleClang build (). Use Atomic database for development environment (). Add test for (). Add test for (). Allow rabbitmq vhost in table settings (). System dictionaries virtual key column (). ISSUES-23310 Try fix MySQL 8.0 address already in use (). Fix error in perf test (). clickhouse-test: print database name on failures (). upload cpu model to perf test db (). Fix function tests flaps (). fix pvs warnings (). ignore empty input chunks generated by joins (). fix window functions for Distributed tables (). add more info to perf test report (). export trace log from stateless tests in flamegraph-friendly format (). add numactl info to perf test run attributes (). Less chance of OOM in stress tests (). Fix inconsistent formatting for tupleElement (for fuzzer) (). Remove unneeded code from CMakeLists (). Somewhat better handling of paths in CMake (incomplete) (). Remove old trash (a little) (). Remove some garbage from RocksDB CMakeLists (). Add warning about gcc usage (). Remove garbage from CMakeLists (2) (). Remove rotten parts of release script (). Remove useless file (). Skip CatBoost tests under MSan (). Fix segfault in TSan on _exit (). Fix bad test 01641memorytrackinginsertoptimize_long (). Remove pbuilder (). Aggregator remove unused code (). Merging (). Move non gtest unit tests to /examples folder (). Kerberized HDFS test is fluky - retries and extra output if failure (). Fix documentation for DETACH ON CLUSTER PERMANENTLY (). Skip integration test for library bridge under MSan (). Fix a few PVS-Studio warnings (). Enable use-after-destruction detection in MSan (). Fix cyrus-sasl msan warning (). A little bit faster merge of aggregating states. (). stable query indexes in perf test (). Minor changes in code (). Fix illiterate error message (). Remove obsolete compilers (). Follow-up for (). Disable testodbcinteraction for MSan build (). fix a typo in query formatting check in fuzzer (). ExternalLoaderRepository fix arcadia (). Ignore cmake-in-clickhouse (). Refactor join step (). clickhouse-test: send TERM to all childs (to avoid hung check triggering) (). Improve testinsertinto_distributed (). Fix CI build for gcc-10 (). Update array-functions.md

Remove unused compilers (fixed for the troublesome "unbundled" build) (). Slightly better hardening for intersecting parts (). Workaround for PVS-Studio (). Add examples folder filter to ya.make.in (). Function default implementation for nulls small optimization (). autodetect arch of gosu in server dockerfile (). Add test for (). clickhouse-test: add missing whitespace before printing database on error (). Improve 00840longconcurrentselectanddropdeadlock (). Flat, Hashed dictionary include update field bytes into bytes_allocated (). XDBCBridgeHelper use global context (). gcc-10 installation no-install-reccomends option fix (). replxx readline compatibility (). Add file paths into logs on failed distributed async sends (). Reduce the amount of logs that StorageMergeTree::selectPartsToMutate outputs in busy systems. (). Add DiskRestartProxy.cpp to ya.make (). Fix some warnings by PVS-Studio (). Add Read/WriteBufferFromFileDecorator.cpp to ya.make (). @CurtizJ convinced me that this test has to be deleted (). bash completion improvements (). Remove obsolete code (). Minor code simplification (implemented TODO) (). Add run options with default config path according to issues/23875 (). Fix testinsertinto_distributed flaps (). Fix bad test about compression codecs (). Cleanup IDatabase.h from extra headers (). Minor simplification (). Drop unnecessary ports where applicable (). Check MIN/MAX attributes in the list of dictionary attributes (). typo: fix a typo in Compression/CodecT64 (). Don't try GLIBC_COMPATIBILITY for i686 Linux (). Function arrayDifference decimal math overflow (). Use 0 over nan for hit_rate in case of 0 queries to the cache dictionary (). Round floats in Aggregator log messages (). support longer query ids in trace log for perf tests (). PVS-Studio fixes, part 6 (). Functional stateless tests fix numbers (). use LowCardinality for AsynchronousMetricLog name column (). Function dictGetOrNull handle empty rows execute (). CompileDAG fix Sip hash (). Fix bad code (). Better error codes in Keeper when no leader alive (). Fix ArenaWithFreeLists test (). Run check_compilerflag earlier (). Performance tests disable compile expressions (). for trivial INSERT SELECT, adjust block size in bytes as well (). Enable thread_local in Arcadia build (). lower two-level aggregation threshold for uniq test to avoid jitter (). Fix empty key projection query analysis (). Generate ya.make for missing UUID.cpp (). bash-completion: complete available formats (). Fix concurrent snapshot read/write (). Also retry database creation in `clickhouse-test` (). Calculate header from ActionsDAG (). Fix arcadia (). TestReadAfterAIO: Use the current path instead of /tmp for temporal files (). Fix a few trailing whitespaces in output (). IFunction refactoring (). Fix add projection to replicated mergetree (). Fix distributed processing when using projection (). fix tiny code style (). fix reinterpretAsFixedString for UUID (). Function move file (). Updated LRUHashMap benchmarks (). Remove temporary files (). Try mute grpc msan (). Update slack link (). Simplier isLocalAddress (). Increase timeout for server restart in integration tests. (). Addition to (). Small code simplification (). Speedup test (). remove useless code (). One more attempt to fix retries in clickhouse-test (). Log exception in Allocator::free (). Fix broken HTML markup on website (). Fix arcadia (). Fixed Arcadia after IFunctionOverloadResolver interface refactoring (). Fix isLocalAddress() (ifa_addr maybe NULL) (). Fix bad test for Enum hints (). - Add uniqThetaSketch to support in ClickHouse. (). - Add uniqThetaSketch to support in ClickHouse. ().