Project: ClickHouse (App Definition and Development / Database)
File: v22.5.1.2079-stable.md

sidebar_position: 1 sidebar_label: 2022 Updated the BoringSSL module to the official FIPS compliant version. This makes ClickHouse FIPS compliant. (). Now, background merges, mutations and `OPTIMIZE` will not increment `SelectedRows` and `SelectedBytes` metrics. They (still) will increment `MergedRows` and `MergedUncompressedBytes` as it was before. (). add implementation of MeiliSearch storage and table function. (). Add support of GROUPING SETS in GROUP BY clause. Follow up after . This implementation supports a parallel processing of grouping sets. (). According to the design mentioned at #issuecomment-1068772646. (). Added `SYSTEM SYNC DATABASE REPLICA` query which allows to sync tables metadata inside Replicated database, because currently synchronisation is asynchronous. (). - Add output format Prometheus, . (). Parse collations in CREATE TABLE, throw exception or ignore. closes . (). Add aliases JSONLines and NDJSON for JSONEachRow. Closes . (). Set partstodelayinsert and partstothrowinsert as query-level settings. If they are defined, they can override table-level settings. (). temporary table can show total rows and total bytes. . (). Added new hash function - wyHash64. (). Window function nth_value was added. (). Add MySQLDump input format. It reads all data from INSERT queries belonging to one table in dump. If there are more than one table, by default it reads data from the first one. (). New single binary based diagnostics tool. (). Description: It is used to count the system table of a request for remote file access, which can help users analyze the causes of performance fluctuations in the scenario of separation of storage and computer. The current system table structure is as follows. When a query reads a segment of a remote file, a record is generated. Read type include READ_FROM_FS_AND_DOWNLOADED_TO_CACHEREAD_FROM_CACHEREAD_FROM_FS_BYPASSING_CACHE*, which used to indicate whether the query accesses the segment from the cache or from a remote file. (). Adds `h3Line`, `h3Distance` and `h3HexRing` functions. (). Related issue - . (). Added system.certificates table. (). Improve performance or ORDER BY, MergeJoin, insertion into MergeTree using JIT compilation of sort columns comparator. (). First commit is to increase the inline threshold. Next commits will improve queries by inlining for those who have shown better performance. This way we will not increase the compile time and binary size and optimize the program. (). Transform OR LIKE chain to multiMatchAny. Will enable once we have more confidence it works. (). Rewrite 'select countDistinct(a) from t' to 'select count(1) from (select a from t groupBy a)'. (). Change structure of `system.asynchronousmetriclog`. It will take about 10 times less space. This closes . The field `eventtimemicroseconds` was removed, because it is useless. (). The default `HashJoin` is not thread safe for inserting right table's rows and run it in a single thread. When the right table is large, the join process is too slow with low cpu utilization. (). Improve performance of reading from storage `File` and table functions `file` in case when path has globs and matched directory contains large number of files. (). Appy parallel parsing for input format `HiveText`, which can speed up HiveText parsing by 2x when reading local file. (). Improves performance of file descriptor cache by narrowing mutex scopes. (). This PR improves the `WATCH` query in WindowView: 1. Reduce the latency of providing query results by calling the `fire_condition` signal. 2. Makes the cancel query operation(ctrl-c) faster, by checking `isCancelled()` more frequently.

Improve performance of `avg`, `sum` aggregate functions if used without GROUP BY expression. (). Improve performance of unary arithmetic functions (`bitCount`, `bitNot`, `abs`, `intExp2`, `intExp10`, `negate`, `roundAge`, `roundDuration`, `roundToExp2`, `sign`) using dynamic dispatch. (). Remind properly if use clickhouse-client --file without preceeding --external. Close . (). Added support for specifying `content_type` in predefined and static HTTP handler config. (). Implement partial GROUP BY key for optimizeaggregationin_order. (). Nullables detection in protobuf using Google wrappers. (). If the required amount of memory is available before the selected query stopped, all waiting queries continue execution. Now we don't stop any query if memory is freed before the moment when the selected query knows about the cancellation. (). Enable memory overcommit by default. (). - Add branch to avoid unnecessary memcpy in readbig. (). Refactor code around schema inference with globs. Try next file from glob only if it makes sense (previously we tried next file in case of any error). Also it fixes . (). Improve schema inference for json objects. (). Add support for force recovery which allows you to reconfigure cluster without quorum. (). We create a local interpreter if we want to execute query on localhost replica. But for when executing query on multiple replicas we rely on the fact that a connection exists so replicas can talk to coordinator. It is now improved and localhost replica can talk to coordinator directly in the same process. (). Show names of erroneous files in case of parsing errors while executing table functions `file`, `s3` and `url`. (). Allowed to increase the number of threads for executing background operations (merges, mutations, moves and fetches) at runtime if they are specified at top level config. (). clickhouse-benchmark can read auth from environment variables. (). Allow names of tuple elements that start from digits. (). Allow file descriptors in table function file if it is run in clickhouse-local. (). Allow to cast columns of type `Object(...)` to `Object(Nullable(...))`. (). Cleanup CSS in Play UI. The pixels are more evenly placed. Better usability for long content in table cells. (). The metrics about time spent reading from s3 now calculated correctly. Close . (). Improve `SYSTEM DROP FILESYSTEM CACHE` query: `<path>` option and `FORCE` option. (). Add `isalldata_sent` column into `system.processes`, and improve internal testing hardening check based on it. (). Now date time conversion functions that generates time before 1970-01-01 00:00:00 with partial hours/minutes timezones will be saturated to zero instead of overflow. This is the continuation of https://github.com/ClickHouse/ClickHouse/pull/29953 which addresses https://github.com/ClickHouse/ClickHouse/pull/29953#discussion_r800550280 . Mark as improvement because it's implementation defined behavior (and very rare case) and we are allowed to break it. (). Allow to cancel query while still keep decent query id in MySQLHandler. (). Properly cancel INSERT queries in `clickhouse-client`/`clickhouse-local`. (). Allow cluster macro in s3Cluster table function. (). Added `userdefinedpath` config setting. (). Allow to execute hash functions with arguments of type `Array(Tuple(..))`. (). Add warning if someone running clickhouse-server with log level "test". The log level "test" was added recently and cannot be used in production due to inevitable, unavoidable, fatal and life-threatening performance degradation. (). Play UI: If there is one row in result and more than a few columns, display the result vertically. Continuation of . (). Add extra diagnostic info (if applicable) when sending exception to other server.

After settings like `backgroundfetchespoolsize` became obsolete and can appear in top level config, but clickhouse throws and exception like `Error updating configuration from '/etc/clickhouse-server/config.xml' config.: Code: 137. DB::Exception: A setting 'backgroundfetchespoolsize' appeared at top level in config /etc/clickhouse-server/config.xml.` This is fixed. (). Finalize write buffers in case of exception to avoid doing it in destructors. Hope it fixes: . (). Play UI: Nullable numbers will be aligned to the right in table cells. This closes . (). Implemented a new mode of handling row policies which can be enabled in the main configuration which enables users without permissive row policies to read rows. (). Fix bug which can lead to forgotten outdated parts in MergeTree table engines family in case of filesystem failures during parts removal. Before fix they will be removed only after first server restart. (). Modify query div in play.html to be extendable beyond 200px height. In case of very long queries it is helpful to extend the textarea element, only today, since the div is fixed height, the extended textarea hides the data div underneath. With this fix, extending the textarea element will push the data div down/up such the extended textarea won't hide it. (). Better read from cache. (). Fix progress indication for `INSERT SELECT` in clickhouse-local for any query and for file progress in client, more correct file progress. (). Disable `logquerythreads` setting by default. It controls the logging of statistics about every thread participating in query execution. After supporting asynchronous reads, the total number of distinct thread ids became too large, and logging into the `querythreadlog` has become too heavy. (). Option `compatibilityignoreautoincrementincreatetable` allows ignoring `AUTO_INCREMENT` keyword in a column declaration to simplify migration from MySQL. (). Added implicit cast for `h3kRing` function second argument to improve usability. Closes . (). Limit the max partitions could be queried for each hive table. Avoid resource overruns. (). Extracts Version ID if present from the URI and adds a request to the AWS HTTP URI. Closes . - - (). Now `clickhouse-keeper` for the `x86_64` architecture is statically linked with and doesn't depend on any system libraries. (). Fail performance comparison on errors in the report. (). Check out the most of build jobs with depth=1. (). Bump minizip-ng to a sane version, or else old git won't be able to address dangling remote ref. . (). Use consistent `force tests` label in CI. (). Limit PowerPC code generation to Power8 for better compatibility. This closes . (). - More robust handling of unknown architectures in CMake. (). Simplify performance test. This will give a chance for us to use it. (). Fix checking for rabbitmq liveness in tests. Fixed incorrect import. (). ClickHouse builds for `PowerPC64LE` architecture are now available in universal installation script `curl https://clickhouse.com/ | sh` and by direct link `https://builds.clickhouse.com/master/powerpc64le/clickhouse`. (). - Make cmake build scripts a bit more robust. (). The ilike() function on FixedString columns could have returned wrong results (i.e. match less than it should). (). Fix implicit cast for optimizeskipunusedshardsrewrite_in. (). Enable `enableglobalwith_statement` for subqueries, close . (). Now WindowView `WATCH EVENTS` query will not be terminated due to the nonempty Chunk created in `WindowViewSource.h:58`. (). Fix "Cannot create column of type Set" for distributed queries with LIMIT BY. (). Fix possible overflow during `OvercommitRatio` comparison. cc @tavplubix.

Update `max_fired_watermark ` after blocks actually* fired, in case delete data that hasn't been fired yet. (). Kafka does not need `group.id` on producer stage. In console log you can find Warning that describe this issue: ``` 2022.05.15 17:59:13.270227 (). fix MySQL database engine to compatible with binary(0) dataType. (). Fix execution of mutations in tables, in which there exist columns of type `Object`. Using subcolumns of type `Object` in `WHERE` expression of `UPDATE` or `DELETE` queries is now allowed yet, as well as manipulating (`DROP`, `MODIFY`) of separate subcolumns. Fixes . (). Fix Nullable(String) to Nullable(Bool/IPv4/IPv6) conversion Closes . (). Fix system.opentelemetryspanlog attribute.values alias to values instead of keys. (). Fix possible deadlock in OvercommitTracker during logging. cc @alesapin @tavplubix Fixes . (). - fix substring function range error length when `offset` and `length` is negative constant and `s` is not constant. (). Accidentally ZSTD support for Arrow was not being built. This fixes . (). Fix ALTER DROP COLUMN of nested column with compact parts (i.e. `ALTER TABLE x DROP COLUMN n`, when there is column `n.d`). (). Fix insertion of complex JSONs with nested arrays to columns of type `Object`. (). Queries with aliases inside special operators returned parsing error (was broken in 22.1). Example: `SELECT substring('test' AS t, 1, 1)`. (). - Fix assertion in JOIN, close . (). Fix dictionary reload for `ClickHouseDictionarySource` if it contains scalar subqueries. (). Fix nullptr dereference in JOIN and COLUMNS matcher. This fixes . This is for https://github.com/ClickHouse/ClickHouse/pull/36417. (). Fix bug in s3Cluster schema inference that let to the fact that not all data was read in the select from s3Cluster. The bug appeared in https://github.com/ClickHouse/ClickHouse/pull/35544. (). Server might fail to start if it cannot resolve hostname of external ClickHouse dictionary. It's fixed. Fixes . (). This code segment can prove bug. ``` int main() { RangeGenerator g{1230, 100}; std::cout << g.totalRanges() << std::endl; int count = 0; while(g.nextRange()) ++count; std::cout << "count:" << count << std::endl; return 0; }. (). Fix clickhouse-benchmark json report results. (). Add missing enum values in system.session_log table. Closes . (). Fix possible exception with unknown packet from server in client. (). Fix usage of executable user defined functions in GROUP BY. Before executable user defined functions cannot be used as expressions in GROUP BY. Closes . (). close . (). Fix hostname sanity checks for Keeper cluster configuration. Add `keeperserver.hostchecks_enabled` config to enable/disable those checks. (). Fix offset update ReadBufferFromEncryptedFile, which could cause undefined behaviour. (). - Fix potential error with literals in `WHERE` for join queries. Close . (). Fix `Missing column` exception which could happen while using `INTERPOLATE` with `ENGINE = MergeTree` table. (). Fix format crash when default expression follow EPHEMERAL not literal. Closes . (). Fix merges of wide parts with type `Object`. (). Fixed parsing of query settings in `CREATE` query when engine is not specified. Fixes https://github.com/ClickHouse/ClickHouse/pull/34187#issuecomment-1103812419. (). Fix possible heap-use-after-free in schema inference. Closes . (). Fix server restart if cache configuration changed. (). In the previous , I found that testing (stateless tests, flaky check (address, actions))* is timeout. Moreover, testing locally can also trigger unstable system deadlocks. This problem still exists when using the latest source code of master. (). Fix server reload on port change (do not wait for current connections from query context). (). Fix vertical merges in wide

Previously an exception `There is no column` can be thrown during merge. (). During the in , I found that the one cache class was initialized twice, it throws a exception. Although the cause of this problem is not clear, there should be code logic of repeatedly loading disk in ClickHouse, so we need to make special judgment for this situation. (). Fix a bug of `groupBitmapAndState`/`groupBitmapOrState`/`groupBitmapXorState` on distributed table. (). Fix timeouts in Hedged requests. Connection hang right after sending remote query could lead to eternal waiting. (). Fix insertion to columns of type `Object` from multiple files, e.g. via table function `file` with globs. (). Fix some issues with async reads from remote filesystem which happened when reading low cardinality. (). Fix creation of tables with `flatten_nested = 0`. Previously unflattened `Nested` columns could be flattened after server restart. (). Fix incorrect cast in cached buffer from remote fs. (). Remove function `groupArraySorted` which has a bug. (). Fix fire in window view with hop window . (). Fix `current_size` count in cache. (). Fix incorrect query result when doing constant aggregation. This fixes . (). Fix bug in clickhouse-keeper which can lead to corrupted compressed log files in case of small load and restarts. (). Fix bugs when using multiple columns in WindowView by adding converting actions to make it possible to call`writeIntoWindowView` with a slightly different schema. (). Fix issue: . (). Fix stuck when dropping source table in WindowView. Closes . (). Fixed logical error on `TRUNCATE` query in `Replicated` database. Fixes . (). Fix sending external tables data in HedgedConnections with maxparallelreplicas != 1. (). Fixed problem with infs in `quantileTDigest`. Fixes . (). Fix LowCardinality->ArrowDictionary invalid output when type of indexes is not UInt8. Closes . (). Fix in-order `GROUP BY` (`optimize_aggregation_in_order=1`) with `Array` (`groupArrayArray`/...) aggregate functions. (). Fixed performance degradation of some INSERT SELECT queries with implicit aggregation. Fixes . (). Fix optimize_aggregation_in_order with prefix GROUP BY and Array aggregate functions. (). NO CL ENTRY: 'Revert "Minor refactor to prefer C++ Standard Algorithms"'. (). NO CL ENTRY: 'Revert "Strict taskstats parser"'. (). NO CL ENTRY: 'Revert "Translate docs/zh/sql-reference/data-types/map.md"'. (). NO CL ENTRY: 'Revert "Update setting.md"'. (). NO CL ENTRY: 'Documentation: Add a missing ESTIMATE* in explain syntax'. (). NO CL ENTRY: ' (). NO CL ENTRY: 'Translate playground.md to Chinese'. (). NO CL ENTRY: 'Revert "Memory overcommit: continue query execution if memory is available"'. (). NO CL ENTRY: 'Revert "Revert "Memory overcommit: continue query execution if memory is available""'. (). NO CL ENTRY: 'Revert "BLAKE3 hash function documentation"'. (). NO CL ENTRY: 'Revert "Remove height restrictions from the query div in play web tool."'. (). Cleanup GROUPING SETS formating (). Minor renames (). clickhouse-test: fix left-queries-check, to fix test log parser (). Regression test for CHECKSUMDOESNTMATCH error because of per-column TTL bug (). Fix performance tests (). Backup for replicated tables and other improvements (). Trying to fix some trash in zero copy replication (). Speed up build a little (). Replace remove-erase idiom by C++20 erase()/erase_if() (). Strict taskstats parser (). Draft changelog for version 22.4 (). Update int-uint.md (). Return back (). Fixed warnings of clang-tidy check "bugprone-branch-clone" (). Clang tidy fixes (). fixed /common/example cow_compositions.cpp clone the inner column when there is no need (). Refactoring dependency for ParserAttachAccessEntity

more specific warning text about low disk space (). Fixed missing enum values for ClientInfo::Interface (). Add passphrase for certificates (). Update version after release (). Followup on password hash salt feature (). Update version_date.tsv after v22.4.2.1-stable (). fix log should print 'from' path (). Add function bin tests for Int/UInt128/UInt256 (). Fix 01161allsystem_tables (). Update PULLREQUESTTEMPLATE.md (). Fixed integer overflow in toStartOfInterval (). Fix issue with broken git ownership (). Fix version autodetection for docker_server.py (). Snyk fixes (). Use just index to split performance tests by group (). Update quantiledeterministic.md (). CodeQL Experiment - Exclude contrib Dependencies (). Small refactoring of Processors and QueryPipeline (). Simplify 01834aliascolumnslazinessfilimonov test (). bash-completion: add completion for sendlogslevel (). client: add a message on reconnect (under warning log level) (). Fix strange whitespace (or I do not know YAML) (). Strict taskstats parser (resubmit) (). Exclude test `02271fixcolumnmatcherandcolumntransformer` from bc check (). Ancient cmake version cleanup (). Cleanup clang-tidy integration. (). ParallelReadBuffer small improvements (). Lambda cancel sync (). Version history benchmarks (). fix typo (). Tiny Mutator code cleanup (). Disble test postgresql replica with asan (). Minor Coverity defects fixes (). Properly wait for queries in 01502longlogtinylogdeadlock_race test (). remove unneeded if statement (). Fix SortingStep::updateOutputStream() (). CodeQL - Run Daily Analysis (). Simplify check_branch, prefetch target branches (). Make backward compatible integration tests runner (). Fix some flaky tests (). Update version_date.tsv after v22.4.3.3-stable (). Fix flaky test (). add missing pandas package (). Fix style issue reported by black formatter (). Fix formatting in drop cache system query (). Fix stress test after 36639 (). Update version_date.tsv after v22.3.4.20-lts (). Update 00170s3cache.sql (). Better version of SeekableReadBufferWithSize (). Better logs for virtual parts (). Check socket is connected in HTTPSession (). fix typo (). Remove excessive logging from S3 (). Tests report (). Use correct nodes in testkeeperznode_time (). Reorganize source files so that base won't depend on Common (). tests: fix 02015asyncinsertsstresslong flakiness (). Add an extra check for RAFT config change (). Update 00170s3cache.sql (). ClickHouseDictionarySource context copy (). Fix build SeekableReadBufferWithSize -> SeekableReadBuffer (). Better error message from NuRaft in case of CRC mismatch (). Fix certs (finishing pr ) (). Another fix for Hung Check (). Deduce `UInt8` type for bools from json instead of `UInt64` (). Add small script for keeper check (). Fix ungrammatical error message (). Fix flaky test `testddlworkernonleader` (). Fix evaluateConstantExpression for subqueries (). Fix clickhouse-test for server without `isalldata_sent` in system.processes (). Init thread pools for clickhouse-local (). Fix performance test (). Better exception messages while socket timeouts (). Improve docs_check to be used in clickhouse-docs (). pull poco #58 (). Tiny cleanup (). Use mutex per worker in ParallelReadBuffer (). Update version_date.tsv after v22.3.5.5-lts (). play.html: add button to transpose table (). Fix style error in DocsCheck (). Set isalldata_sent on exceptions too (). Clang -Tidy Fixes (). Protection from incorrect build (). Avoid recreation of system.asynchronousmetriclog (due to difference in codec) (). Removed forceful drop cache command, fix detached status state (). Log hash table's cache messages with TRACE level (). Cleanup: Remove switches for obsolete GCC version (). Add functions from CREATE FUNCTION to completion (). Validate that function had been passed in CREATE FUNCTION (). (). Disable `mergetreemetadata_cache` by default (). Replace uses of SFINAE by C++20 concepts (). Fix performance test (4)

Remove "preconditions" from performance tests (overengineering, unneeded feature) (). Remove inherited create() method + disallow copying of storages (). tests: disable 02260altercompactpartdropnestedcolumn for <=22.4 (). Fix check black (). Aggregator JIT compilation lock fix (). Add Other Query Time Microseconds Profile Event (). Reproduce and a little bit better fix for LC dict right offset. (). Terminate if S3 buffer is not finalized (). Activated a bunch of LLVM 12/13/14 clang-tidy warnings (). Replace `Timeout` with `Tests not finished` (). Backup improvements (). Integration tests (). Relax cmake check for CFLAGS/CXXFLAGS/LDFLAGS (). Fix insertion of complex JSONs with nested arrays (). Add column to system.filesystemcachelog (). Log into filesystemcachelog when cache not even attempted (). fix typo in comment (). Add some CurrentMetrics for fs cache (). Add some asynchronous metrics for fs cache (). Save system logs in functionsl tests if server crashed (). quick tmp fix for stress test (). Some fixes for replicated merge tree (). Invert .clang-tidy (). Replace makepair()/maketuple() by pair()/tuple() (). Remove log message on client reconnects (reverts ) (). Fix profile events in fs cached buffer (). Benchmark was not loaded properly (). Force keeper snapshot equality on different replicas (). Remove strange code (). Remove nested exception (). Enable clangtidies (). Improve stress tests report a little bit (). Follow up for ConcurrentHashJoin (). Fix formats docs (). Support secure connection in clickhouse-test (). Fix installation_id filter, minimize docker images diff (). Update version_date.tsv after v22.3.6.5-lts (). Update version_date.tsv after v22.4.5.9-stable (). Removing ReplacingWindowColumnTransform in WindowView (). Merging (). Remove CodeQL (). Remove obsolete code (). Try fix flaky test (). fix output error in LabelsCheck (). remove useless code (). Merging . (). Fix benchmark build code (). Add benchmark script for testing clouds (). Fix flaky integration mongodb test (). Print stacks if we cannot terminate server in stress tests (). Remove last mentions of data streams (). Changelog script (). This evening I started using Grammarly. (). update sanity checks warning message text (). Continue fixing . (). just fix a number (). Disable thread fuzzer after server restart (). Option to force crosstoinnerjoinrewrite (). fix wrong argument in proxy resolver of DiskS3 (). tests: fix 01119optimizetrivialinsertselect (due to max_threads randomization) (). Support SELECT query in WindowView (). Fix workflow style check (). Add check for columns size in Block::cloneWithColumns (). Update documentation and defaults for memory overcommit (). Update default remote fs read method in ReadSettings (). Update Exception Message for allowed auth types (). tests/integration: fix possible race for iptables user rules inside containers (). Fix fasttest ccache permissions (). Add const qualifier for few methods from Context (). Reload listenreuseport/listen_backlog from config (). Enable DNS cache for HTTPSClientSession. (). fix async reads from remote fs internal setting not being always turned on (). Backups Improvements 5 (). Add S3Requests metric (). Fix coverity build problem with LD_PRELOAD (). Relax log level for some checks in check thread (). update poco (). Ignore harmful env variables in clickhouse binaries (reexec w/o them) (). Fix wrong comment in IVolume.h (). Cmake cleanup pt2 (). Temporarily fix flaky test `01825typejsoninsertselect.sql` (). Pass needfilter, hasnull_map to joinRightColumns (). Activate more clangtidies (). Fix docker cleaner in workflows (). tidy build fix (). Update run-check.py to match PR template, add comments (). Release without prestable (). Fixed typos ().