Project: ClickHouse (App Definition and Development / Database)
File: v22.12.1.1752-stable.md

sidebar_position: 1 sidebar_label: 2022 Fixed backward incompatibility in (de)serialization of states of `min`, `max`, `any`, `argMin`, `argMax` aggregate functions with `String` argument. The incompatibility was introduced in https://github.com/ClickHouse/ClickHouse/pull/41431 and affects 22.9, 22.10 and 22.11 branches (fixed since 22.9.6, 22.10.4 and 22.11.2 correspondingly). Some minor releases of 22.3, 22.7 and 22.8 branches are also affected: 22.3.13...22.3.14 (fixed since 22.3.15), 22.8.6...22.8.9 (fixed since 22.8.10), 22.7.6 and newer (will not be fixed in 22.7, we recommend to upgrade from 22.7.* to 22.8.10 or newer). This release note does not concern users that have never used affected versions. Incompatible versions append extra `'\0'` to strings when reading states of the aggregate functions mentioned above. For example, if an older version saved state of `anyState('foobar')` to `statecolumn` then incompatible version will print `'foobar\0'` on `anyMerge(statecolumn)`. Also incompatible versions write states of the aggregate functions without trailing `'\0'`. Newer versions (that have the fix) can correctly read data written by all versions including incompatible versions, except one corner case. If an incompatible version saved a state with a string that actually ends with null character, then newer version will trim trailing `'\0'` when reading state of affected aggregate function. For example, if an incompatible version saved state of `anyState('abrac\0dabra\0')` to `statecolumn` then newer versions will print `'abrac\0dabra'` on `anyMerge(statecolumn)`. The issue also affects distributed queries when an incompatible version works in a cluster together with older or newer versions. (). Add "gracehash" joinalgorithm. (). Merging on initiator now uses the same memory bound approach as merging of local aggregation results if `enablememoryboundmergingofaggregationresults` is set. (). Add BSONEachRow input/output format. In this format, ClickHouse formats/parses each row as a separated BSON Document and each column is formatted/parsed as a single BSON field with column name as a key. (). close: . (). Added `multiplyDecimal` and `divideDecimal` functions for decimal operations with fixed precision. (). Added `system.moves` table with list of currently moving parts. (). Keeper feature: add support for embedded Prometheus endpoint. (). Added age function to calculate difference between two dates or dates with time values expressed as number of full units. Close . (). Add settings `maxstreamsformergetreereading` and `allowasynchronousreadfromiopoolformergetree`. Setting `maxstreamsformergetreereading` limits the number of reading streams for MergeTree tables. Setting `allowasynchronousreadfromiopoolformergetree` enables background I/O pool to read from `MergeTree` tables. This may increase performance for I/O bound queries if used together with `maxstreamstomaxthreadsratio` or `maxstreamsformergetreereading`. (). Add the expression of the index on `dataskippingindices` system table. (). New hash function added. Also performance of `xxHash32` and `xxHash64` improved on arm thanks to library update. (). - Temporary data (for external sorting, aggregation, and JOINs) can share storage with the filesystem cache for remote disks and evict it, close . (). Add column `engine_full` to system table `databases` so that users can access whole engine definition of database via system tables. (). Add password complexity rules and checks for creating a new user. (). Add function concatWithSeparator , like concat_ws in spark. (). Added constraints for merge tree settings. (). Support numeric literals with _ as separator. (). Add a new setting `inputformatjsonreadobjectsasstrings` that allows to parse nested JSON objects into Strings in all JSON input formats. This setting is disable by default. (). Optimisation is getting skipped now if `maxsizetopreallocatefor_aggregation` has too small value. Default value of this setting increased to `10^8`. (). Support numeric literals with underscores. closes . (). Add `FROM table SELECT column` syntax.

This PR changes how followed queries delete parts: truncate table, alter table drop part, alter table drop partition. Now these queries make empty parts which cover old parts. This makes truncate query works without exclusive lock which means concurrent reads aren't locked. Also achieved durability in all those queries. If request is succeeded then no resurrected pars appear later. Note that atomicity is achieved only with transaction scope. (). `SET paramx` query no longer requires manual string serialization for the value of the parameter. For example, query `SET parama = ' (). `filesystemAvailable` and related functions support one optional argument with disk name, and change `filesystemFree` to `filesystemUnreserved`. Closes . (). Increased the default value of search_limit to 256, and added LDAP server config option to change that to an arbitrary value. Closes: . (). Add cosine distance for annoy. (). Allow to remove sensitive information from the exception messages also. Resolves . (). Keeper improvement: Add 4lw command `rqld` which can manually assign a node as leader. (). Apply connection timeouts settings for Distributed async INSERT from the query. (). unhex function support FixedString arguments. . (). Priority is given to deleting completely expired Partsrelated . (). Follow-up to https://github.com/ClickHouse/ClickHouse/pull/42484. Mask sensitive information in logs better; mask secret parts in the output of queries `SHOW CREATE TABLE` and `SELECT FROM system.tables`. Also resolves . (). Enable compress marks and primary key. (). resolve issue . Right now async insert doesn't support deduplication, because multiple small inserts will coexist in one part, which corespond multiple `block id`s. This solution is straitfoward: The change involves: 1. mark offsets for every inserts in every chunk 2. calculate multiple `blockid`s when sinker receive a chunk 3. get block number lock by these `blockid`s 3.1. if fails, remove the dup insert(s) and dup `blockid`(s) from block and recalculate `offsets` agian. 3.2. if succeeds, commit `blockid`'s and other items into keeper a. if fails, do 3.1 b. if succeeds, everything succeeds. (). More precise and reactive CPU load indication on client. (). Restrict default access to named collections for user defined in config. It must have explicit `shownamedcollections=1` to be able to see them. (). Support reading of subcolumns of nested types from storage `S3` and table function `s3` with formats `Parquet`, `Arrow` and `ORC`. (). - Systemd integration now correctly notifies systemd that service is really started and is ready to server requests. (). Add table_uuid to system.parts. (). Added client option to display the number of locally processed rows in non-interactive mode (--print-num-processed-rows). (). Show read rows while reading from stdin from client. Closes . (). Keeper improvement: try syncing logs to disk in parallel with replication. (). Show progress bar while reading from s3 table function / engine. (). Progress bar will show both read and written rows. (). Implement `aggregation-in-order` optimization on top of query plan. It is enabled by default (but works only together with `optimizeaggregationinorder`, which is disabled by default). Set `queryplanaggregationin_order = 0` to use previous AST-based version. (). Allow to send profile events with `tracetype = 'ProfileEvent'` to `system.tracelog` on each increment with current stack, profile event name and value of increment. It can be enabled by setting `traceprofileevents` and used to debug performance of queries. (). Keeper improvement: requests are batched more often. The batching can be controlled with the new setting `maxrequestsquickbatchsize`. (). Added possibility to use array as a second parameter for cutURLParameter function. Close .

Implement referential dependencies and use them to create tables in the correct order while restoring from a backup. (). Add a new setting `inputformatmaxbinarystring_size` to limit string size in RowBinary format. (). - Fix some incorrect logic in ast level optimization related. (). Support query like `SHOW FULL TABLES ...`. (). When ClickHouse requests a remote HTTP server, and it returns an error, the numeric HTTP code was not displayed correctly in the exception message. Closes . (). Settings `mergetreeminrowsforconcurrentreadforremotefilesystem/mergetreeminbytesforconcurrentreadforremotefilesystem` did not respect adaptive granularity. Fat rows did not decrease the number of read rows (as it is was done for `mergetreeminrowsforconcurrentread/mergetreeminbytesforconcurrentread`, which could lead to high memory usage. (). Support `optimizeiftransformstringsto_enum` in new analyzer. (). This is to upgrade the new "DeflateQpl" compression codec which has been implemented on previous PR (details: https://github.com/ClickHouse/ClickHouse/pull/39494). This patch improves codec on below aspects: 1. QPL v0.2.0 to QPL v0.3.0 2. Improve CMake file for fixing QPL build issues for QPL v0.3.0 3. Link the QPL library with libaccel-config at build time instead of runtime loading on QPL v0.2.0 (dlopen) 4. Fixed log print issue in CompressionCodecDeflateQpl.cpp. (). Follow-up to https://github.com/ClickHouse/ClickHouse/pull/43834 Fix review issues; dependencies from `Distributed` table engine and from `cluster()` function are also considered now; as well as dependencies of a dictionary defined without host & port specified. (). Fix mutations not making progress when checksums do not match between replicas (e.g. caused by a change in data format on an upgrade). (). fix skipunavailableshards does not work using hdfsCluster table function. (). fix s3 support question mark wildcard. Closes . (). - Fix functions arrayFirstOrNull and arrayLastOrNull or null when array is Nullable. (). - we create a new zk path called "asyncblocks" for replicated tables in . However, for tables created in older versions, this path does not exist and will cause error when doing partition operations. This PR will create this node when initializing replicated tree. - This PR created a flag `asyncinsert_deduplicate` with `false` default value to control whether to use this function. As mentioned in , this function is not yet fully finished. I would turn off it by default. (). Add support for FreeBSD/powerpc64le. (). Bump Testcontainers for Go to v0.15.0. (). ... Enable base64 on s390x > Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/. (). Shutdown will be much faster if do not call clearOldPartsFromFilesystem. Especially this is right for tests with zero-copy due to single thread deletion parts. clearOldPartsFromFilesystem is unnecessary after https://github.com/ClickHouse/ClickHouse/pull/41145. (). Integrate skim into the client/local. (). Allow clickhouse to use openssl as a dynamic library and in-tree for development purposes. (). Closes . (). Bring sha512 sums back to the building step. (). Kill stress tests after 2.5h in case of hanging process. (). Fixed unable to log in (because of failure to create session_log entry) in rare case of messed up setting profiles. ... (). Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting. (). Do not suppress exceptions in web disk. Fix retries for web disk. (). Fixed race condition between inserts and dropping MVs. (). Fixed bug which could lead to deadlock while using asynchronous inserts. (). Additional check on zero uncompressed size is added to `CompressionCodecDelta`. (). An issue with the following exception has been reported while trying to read a Parquet file from S3 into ClickHouse:. (). Fix bad cast from LowCardinality column when using short circuit function execution. Proper fix of https://github.com/ClickHouse/ClickHouse/pull/42937. (). Fixed queries with `SAMPLE BY` with prewhere optimization on tables using `Merge` engine.

Fix `DESCRIBE` for `deltaLake` and `hudi` table functions. (). Check and compare the content of `format_version` file in `MergeTreeData` so tables can be loaded even if the storage policy was changed. (). Fix possible (very unlikely) "No column to rollback" logical error during INSERT into Buffer. (). Fix a bug that allowed FucntionParser to parse an unlimited amount of round brackets into one function if `allowfunctionparameters` is set. (). MaterializeMySQL support ddl drop table t1,t2 and Compatible with most of MySQL drop ddl. (). Fix possible `Cannot create non-empty column with type Nothing` in functions if/multiIf. Closes . (). Fix a bug when row level filter uses default value of column. (). Query with DISTINCT + LIMIT BY + LIMIT can return fewer rows than expected. Fixes . (). Fix sumMap() for Nullable(Decimal()). (). Fix date_diff() for hour/minute on macOS. Close . (). Fix incorrect memory accounting because of merges/mutations. (). Substitute UDFs in `CREATE` query to avoid failures during loading at the startup. Additionally, UDFs can now be used as `DEFAULT` expressions for columns. (). Correctly report errors in queries even when multiple JOINs optimization is taking place. (). Fixed primary key analysis with conditions involving `toString(enum)`. (). - Ensure consistency when copier update status and `attachisdone` in keeper after partition attach is done. (). During recovering of the lost replica there could a situation where we need to atomically swap two table names (use EXCHANGE), but instead previously we tried to use two RENAME queries. Which was obviously failed and moreover failed the whole recovery process of the database replica. (). fix s3Cluster function returns NOTFOUNDCOLUMNINBLOCK error. Closes . (). Optimized number of List requests to ZooKeeper when selecting a part to merge. Previously it could produce thousands of requests in some cases. Fixes . (). Fix posssible logical error 'Array sizes mismatched' while parsing JSON object with arrays with same key names but with different nesting level. Closes . (). Fixed possible exception in case of distributed group by with an alias column among aggregation keys. (). Fix bug which can lead to broken projections if zero-copy replication is enabled and used. (). - Fix using multipart upload for large S3 objects in AWS S3. (). Fixed `ALTER ... RESET SETTING` with `ON CLUSTER`. It could be applied to one replica only. Fixes . (). Fix logical error in right storage join with using. (). Keeper fix: throw if interserver port for Raft is already in use. Fix segfault in Prometheus when Raft server failed to initialize. (). Fix order by positional arg in case unneeded columns pruning. Closes . (). Fix bug with wrong order of keys in Storage Join. (). Fixed exception when subquery contains having but doesn't contain actual aggregation. (). Fix race in s3 multipart upload. This race could cause the error `Part number must be an integer between 1 and 10000, inclusive. (S3_ERROR)` while restoring from a backup. (). Fix undefined behavior in the `quantiles` function, which might lead to uninitialized memory. Found by fuzzer. This closes . (). Prevent dropping nested column if it creates empty part. (). Fix `LOGICAL_ERROR` in case when fetch of part was stopped while fetching projection to the disk with enabled zero-copy replication. (). Fix possible Bad cast from type DB::IAST const to DB::ASTLiteral const*. Closes . (). Prevent `ReadonlyReplica` metric from having negative values. (). Fixed Endian issues in hex string conversion on s390x (which is not supported by ClickHouse). ().

toDateTime64 conversion generates wrong time on z build, add bit_cast swap fix to support toDateTime64 on s390x platform. (). ... s390x support for ip coding functions. (). Fix byte order issue of wide integers for s390x. (). Fixed endian issue in bloom filter serialization for s390x. (). Fixed setting TCP_KEEPIDLE of client connection for s390x. (). Fix endian issue in StringHashTable for s390x. (). NO CL ENTRY: 'Revert "Revert "S3 request per second rate throttling""'. (). NO CL ENTRY: 'Update version after release'. (). NO CL ENTRY: 'Revert "Add table_uuid to system.parts"'. (). NO CL ENTRY: 'Revert "Fix endian issue in integer hex string conversion"'. (). NO CL ENTRY: 'Update replication.md'. (). NO CL ENTRY: 'Revert "Temporary files evict fs cache"'. (). NO CL ENTRY: 'Update html interface doc'. (). NO CL ENTRY: 'Revert "Add function 'age'"'. (). NO CL ENTRY: 'Revert "Builtin skim"'. (). NO CL ENTRY: 'Revert "Add information about written rows in progress indicator"'. (). Build libcxx and libcxxabi from llvm-project (). Allow release only from ready commits (). Add global flags to base/ libraries (). Enable strict typing check in tests/ci (). Add server UUID for disks access checks (read/read-by-offset/write/delete) to avoid possible races (). Do not include libcxx library for C (). Followup fixes for FuseFunctionsPass (). Fix bug in replication queue which can lead to premature mutation finish (). Support `CREATE / ALTER / DROP NAMED COLLECTION` queries under according access types (). Fix race in `IColumn::dumpStructure` (). Sanitize thirdparty libraries for public flags (). stress: increase timeout for server waiting after TERM (). Fix cloning of ASTIdentifier (). Fix race on write in `ReplicatedMergeTree` (). Cancel lambda api url (). Fixed: Typo (). Analyzer small fixes (). Fix: make testreadonly_table more stable (). Make insertRangeFrom() more exception safe (). Analyzer added indexes support (). Allow to "drop tables" from s3_plain disk (so as from web disk) (). Add --max-consecutive-errors for clickhouse-benchmark (). Add (). Suggest users installation troubleshooting (). Update version_date.tsv and changelogs after v22.11.1.1360-stable (). Provide full stacktrace in case of uncaught exception during server startup (). Update SECURITY.md on new stable tags (). Splitting checks in CI more (). Update version_date.tsv and changelogs after v22.8.9.24-lts (). Fix mess with signed sizes in SingleValueDataString (). Add a comment (). Avoid race condition for updating system.distribution_queue values (). Fix flaky 01926orderbydesclimit (). Fix possible heap-use-after-free in local if history file cannot be created (). Fix flaky test (). Fix backward compatibility check (). Fix typo (). Remove noise from logs about NetLink in Docker (). Modify test slightly (). Set run_passes to 1 by default (). Do not reuse jemalloc memory in testglobalovercommit (). Fix createTableSharedID again (). Use smaller buffer for small files (). Merging (). Use all parameters with prefixes from ssm (). Avoid possible DROP hung due to attached web disk (). Improve fuzzy search in clickhouse-client/clickhouse-local (). check ast limits for createparserfuzzer (). Add another test for SingleDataValueString (). Move password reset message from client to server (). Sync everything to persistent storage to avoid writeback affects perf tests (). bump lib for diag (). Temporarily disable `testhivequery` (). Analyzer SumIfToCountIfPass fix (). Analyzer UniqInjectiveFunctionsEliminationPass (). Disable broken 00176bsonparallel_parsing (). Add benchmark for query interpretation with JOINs (). Analyzer table functions untuple fix (). Prepare CI for universal runners preallocated pool (). Iterate list without index-based access (). Remove code that I do not understand (). Add table_uuid to system.parts (resubmit)

Move perf tests for Aarch64 from PRs to master (). Fix flaky 01175distributedddloutputmode_long (). Speedup backup config loading (). Fix (). Do not checkout submodules recursively (). Optimize binary-builder size (). Fix flaky `KeeperMap` integration tests (). Fix data race in `Keeper` snapshot (). Use docker images cache from merged PRs in master and release branches (). Update AsynchronousReadIndirectBufferFromRemoteFS.cpp (). Fix pagination issue in GITHUBJOBID() (). Try fix flaky test 00176bsonparallel_parsing (). Fix log messages in clickhouse-copier (). try to remove clickhouse if already exists (). Fix 43622 (). Fix example of colored prompt in client (). Minor fixes in annoy index documentation (). Terminate lost runners (). Update README.md (). Fix included_elements calculation in AggregateFunctionNullVariadic (). Migrate runnertokenrotation_lambda to zip-package deployment (). Analyzer compound expression crash fix (). Migrate termination lambda to zip-package (). Fix flaky `teststorecleanup` (). Attempt to fix StyleCheck condition (). Rerun PullRequestCI on changed description body (). Yet another fix for AggregateFunctionMinMaxAny (). Add google benchmark to contrib (). Fix EN doc as in (). Detach threads from thread group (). Try making `testkeeperzookeeper_converter` less flaky (). Polish UDF substitution visitor (). Analyzer ConstantNode refactoring (). Update Poco (). Add another BC check suppression (). tests: fix 01676longclickhouseclientautocomplete flakiness (). Use disk operation to serialize and deserialize meta files of StorageFilelog (). Add constexpr (). Do not postpone removal of in-memory tables (). Increase some logging level for keeper client. (). FuseFunctionsPass small fix (). Followup fixes for XML helpers (). Hold ProcessListEntry a bit longer in case of exception from Interpreter (). A little improve performance of PODArray (). Change email for robot-clickhouse to immutable one (). Rerun DocsCheck on edited PR description (). Temporarily disable misc- slow clang-tidy checks (). do not leave tmp part on disk, do not go to the keeper for remove it (). do not read part status just for logging (). Analyzer Context refactoring (). Analyzer CTE resolution fix (). Improve release script (). Use only PRs to our repository in pr_info on push (). Join engine works with analyzer (). Fix reports (). Fix vim settings (and make it compatible with neovim) (). Fix clang tidy errors introduced in (). Fix BACKUP TO S3 for Google Cloud Storage (). Fix tags workflow (). Generate missed changelogs for latest releases (). Fix typo in tests/ci/bugfixvalidatecheck.py (). Remove test logging of signal "EINTR" (). Some cleanup of isDeterministic(InScopeOfQuery)() (). Try to keep runners alive for longer (). Fix relaxed "too many parts" threshold (). Correct CompressionCodecGorilla exception message (). Fix exception message (). Update version_date.tsv and changelogs after v22.8.11.15-lts (). do not hardlink serialization.json in new part (). Fix tracing of profile events (). Slightly better clickhouse disks and remove DiskMemory (). Assign release PRs (). Merging (). add changelogs (). Fix the CACHE_PATH creation for default value (). Fix aspell (). Fix another bug in AggregateFunctionMinMaxAny (). Analyzer aggregate function lambda crash fix (). Fix -Wshorten-64-to-32 on FreeBSD and enable -Werror (). Fix flaky test `02497traceeventsstresslong` (). Minor file renaming (). Fix typo (). Better descriptions of signals (). make calls to be sure that parts are deleted (). Ignore "session expired" errors after BC check (). Fix incorrect assertion (). Close GRPC channels in tests (). Remove misleading message from logs (). Minor clang-tidy fixes in fromUnixTimestamp64() (). Hotfix for "check_status.tsv doesn't exists" in stress tests (). Fix documentation after (). Fix an assertion in transactions (). Add log message ().