Project: ClickHouse (App Definition and Development / Database)
File: v21.4.2.10-prestable.md

sidebar_position: 1 sidebar_label: 2022 Column `keys` in table `system.dictionaries` was replaced to columns `key.names` and `key.types`. Columns `key.names`, `key.types`, `attribute.names`, `attribute.types` from `system.dictionaries` table does not require dictionary to be loaded. (). Fix `cutToFirstSignificantSubdomainCustom()`/`firstSignificantSubdomainCustom()` returning wrong result for 3+ level domains present in custom top-level domain list. For input domains matching these custom top-level domains, the third-level domain was considered to be the first significant one. This is now fixed. This change may introduce incompatibility if the function is used in e.g. the sharding key. (). The `toStartOfIntervalFunction` will align hour intervals to the midnight (in previous versions they were aligned to the start of unix epoch). For example, `toStartOfInterval(x, INTERVAL 11 HOUR)` will split every day into three intervals: 00:00:00..10:59:59, 11:00:00..21:59:59 and 22:00:00..23:59:59. This behaviour is more suited for practical needs. This closes . (). Extended range of `DateTime64` to properly support dates from year 1925 to 2283. Improved support of `DateTime` around zero date (`1970-01-01`). ... (). - Added support of Kerberos authentication for preconfigured users and HTTP requests (GSS-SPNEGO). (). Zero-copy replication for ReplicatedMergeTree over S3 storage. (). Support `dictHas` function for `RangeHashedDictionary`. Fixes . (). Supports implicit key type conversion for JOIN. Closes . (). Allow customizing timeouts for http connections used for replication independently from other http timeouts. (). Added async update in `ComplexKeyCache`, `SSDCache`, `SSDComplexKeyCache` dictionaries. Added support for Nullable type in `Cache`, `ComplexKeyCache`, `SSDCache`, `SSDComplexKeyCache` dictionaries. Added support for multiple attributes fetch with `dictGet`, `dictGetOrDefault` functions. Fixes . (). Added `Grant,` `Revoke` and `System` values of `querykind` column for corresponding queries in `system.querylog` ... (). Added new SQL command ALTER TABLE 'table_name' UNFREEZE (). Added ExecutablePool dictionary source. Close . (). - Add function `isIPAddressInRange` to test if an IPv4 or IPv6 address is contained in a given CIDR network prefix. (). Add `_partition_id` virtual column for `MergeTree` engines. Allow to prune partitions by `partitionid`. Add `partitionID()` function to calculate partition id string. (). Add new column `slowdownscount` to `system.clusters`. When using hedged requests, it shows how many times we switched to another replica because this replica was responding slowly. Also show actual value of `errorscount` in `system.clusters`. (). Add option `--backslash` for clickhouse-format, which can add a backslash at the end of each line of the formatted query. (). Add new optional clause GRANTEES for CREATE/ALTER USER commands:. (). Add `ctime` option to `zookeeper-dump-tree`. It allows to dump node creation time. (). Functions 'dictGet', 'dictHas' use current database name if it is not specified for dictionaries created with DDL. Closes . (). Support `Nullable` type for `PolygonDictionary` attribute. (). Added table function `dictionary`. It works the same way as `Dictionary` engine. Closes . (). Add function `timezoneOf` that returns the timezone name of `DateTime` or `DateTime64` data types. This does not close . Fix inconsistencies in function names: add aliases `timezone` and `timeZone` as well as `toTimezone` and `toTimeZone` and `timezoneOf` and `timeZoneOf`. (). Add `prefercolumnnametoalias` setting to use original column names instead of aliases. it is needed to be more compatible with common databases' aliasing rules. This is for and . (). Support parallel parsing for `CSVWithNames` and `TSVWithNames` formats. This closes . (). Improved performance by replacing `memcpy` to another implementation. This closes . (). Supported parallel formatting in clickhouse-local and everywhere else. (). Optimize performance of queries like `SELECT ... FINAL ...

Now in queries with `FINAL` it's allowed to move to `PREWHERE` columns, which are in sorting key. ... (). Faster `GROUP BY` with small `maxrowstogroupby` and `groupbyoverflow_mode='any'`. (). Avoid unnecessary data copy when using codec `NONE`. Please note that codec `NONE` is mostly useless - it's recommended to always use compression (`LZ4` is by default). Despite the common belief, disabling compression may not improve performance (the opposite effect is possible). The `NONE` codec is useful in some cases: - when data is uncompressable; - for synthetic benchmarks. (). Add cache for files read with `minbytestousemmap_io` setting. It makes significant (2x and more) performance improvement when the value of the setting is small by avoiding frequent mmap/munmap calls and the consequent page faults. Note that mmap IO has major drawbacks that makes it less reliable in production (e.g. hung or SIGBUS on faulty disks; less controllable memory usage). Nevertheless it is good in benchmarks. (). Enable read with mmap IO for file ranges from 64 MiB (the settings `minbytestousemmap_io`). It may lead to moderate performance improvement. (). Introduce a new merge tree setting `minbytestorebalancepartitionoverjbod` which allows assigning new parts to different disks of a JBOD volume in a balanced way. (). Improve performance of aggregation in order of sorting key (with enabled setting `optimizeaggregationin_order`). (). MaterializeMySQL: add minmax skipping index for _version column. (). Do not create empty parts on INSERT when `optimizeoninsert` setting enabled. Fixes . (). - Support more cases to rewrite `CROSS JOIN` to `INNER JOIN`. (). MaterializeMySQL: Attempt to reconnect to MySQL if the connection is lost. (). Improve support of integer keys in data type `Map`. (). Improve clickhouse-format to not throw exception when there are extra spaces or comment after the last query, and throw exception early with readable message when format `ASTInsertQuery` with data . (). Age and Precision in graphite rollup configs should increase from retention to retention. Now it's checked and the wrong config raises an exception. (). Add setting `optimizeskipunusedshardslimit` to limit the number of sharding key values for `optimizeskipunused_shards`. (). Add `lasterrortime`/`lasterrormessage`/`lasterrorstacktrace`/`remote` columns for `system.errors`. (). If PODArray was instantiated with element size that is neither a fraction or a multiple of 16, buffer overflow was possible. No bugs in current releases exist. (). - Propagate query and session settings for distributed DDL queries. Set `distributedddlentryformatversion` to 2 to enable this. - Added `distributedddloutputmode` setting. Supported modes: `none`, `throw` (default), `nullstatusontimeout` and `never_throw`. - Miscellaneous fixes and improvements for `Replicated` database engine. (). Update clusters only if their configurations were updated. (). Support replicas priority for postgres dictionary source. (). Closes . Support non-default table schema for postgres storage/table-function. (). Better formatting for `Array` and `Map` data types in Web UI. (). DiskS3 (experimental feature under development). Fixed bug with the impossibility to move directory if the destination is not empty and cache disk is used. (). Add connection pool for PostgreSQL table/database engine and dictionary source. Should fix . (). Add profile event HedgedRequestsChangeReplica, change read data timeout from sec to ms. (). Support `RANGE OFFSET` frame for floating point types. Implement `lagInFrame`/`leadInFrame` window functions, which are analogous to `lag`/`lead`, but respect the window frame. They are identical when the frame is `between unbounded preceding and unbounded following`. This closes . (). Show path to data directory of `EmbeddedRocksDB` tables in system tables. (). Supported `replicationalterpartitions_sync=1` setting for moving partitions from helping table to destination. Decreased default timeouts. Fixes .

If partition key of a `MergeTree` table does not include `Date` or `DateTime` columns but includes exactly one `DateTime64` column, expose its values in the `mintime` and `maxtime` columns in `system.parts` and `system.partscolumns` tables. Add `mintime` and `maxtime` columns to `system.partscolumns` table (these was inconsistency to the `system.parts` table). This closes . (). Add option strict_increase to windowFunnel function to calculate each event once (resolve ). (). Added case insensitive aliases for `CONNECTION_ID()` and `VERSION()` functions. This fixes . (). Update used version of simdjson to 0.9.1. This fixes . (). Convert `system.errors.stack_trace` from `String` into `Array(UInt64)` (This should decrease overhead for the errors collecting). (). If tuple of NULLs, e.g. `(NULL, NULL)` is on the left hand side of `IN` operator with tuples of non-NULLs on the right hand side, e.g. `SELECT (NULL, NULL) IN ((0, 0), (3, 1))` return 0 instead of throwing an exception about incompatible types. The expression may also appear due to optimization of something like `SELECT (NULL, NULL) = (8, 0) OR (NULL, NULL) = (3, 2) OR (NULL, NULL) = (0, 0) OR (NULL, NULL) = (3, 1)`. This closes . (). Added possibility to migrate existing S3 disk to the schema with backup-restore capabilities. (). Add case-insensitive history search/navigation and subword movement features to clickhouse-client. (). Add `current_database` column to `system.processes` table. It contains the current database of the query. (). Fix MSan report for function `range` with `UInt256` argument (support for large integers is experimental). This closes . (). Fix error `Directory tmpfetchXXX already exists` which could happen after failed fetch part. Delete temporary fetch directory if it already exists. Fixes . (). Better exception message in client in case of exception while server is writing blocks. In previous versions client may get misleading message like `Data compressed with different methods`. (). Fixed open behavior of remote host filter in case when there is `remoteurlallow_hosts` section in configuration but no entries there. :warning: please add a note about potential issue when upgrading - @alexey-milovidov. (). `forcedroptable` flag didn't work for `MATERIALIZED VIEW`, it's fixed. Fixes . (). fix official website documents which introduced cluster secret feature. (). Fix receive and send timeouts and non-blocking read in secure socket. (). Fix Avro format parsing for Kafka. Fixes . (). Fixed race on SSL object inside SecureSocket in Poco. (). Fix that S3 table holds old credentials after config update. (). Fix table function `clusterAllReplicas` returns wrong `shardnum`. close . (). The ```::poll()``` return ```rc == 1 ```, it could be a request or it could be a response. (). In case if query has constant `WHERE` condition, and setting `optimizeskipunused_shards` enabled, all shards may be skipped and query could return incorrect empty result. (). Fix possible error ` Cannot find column` when `optimizeskipunused_shards` is enabled and zero shards are used. (). `std::terminate` was called if there is an error writing data into s3. (). Remove unknown columns from joined table in where for queries to external database engines (MySQL, PostgreSQL). close , close (dup), close (dup). (). Fix fsyncpartdirectory for horizontal merge. (). Fix distributed requests cancellation (for example simple select from multiple shards with limit, i.e. `select from remote('127.{2,3}', system.numbers) limit 100`) with `asyncsocketfor_remote=1`. (). Add type conversion for StorageJoin (previously led to SIGSEGV). (). Start accepting connections after DDLWorker and dictionaries initialization. (). Fix SIGSEGV on not existing attributes from iptrie with accesstokeyfrom_attributes. (). Fix function `arrayElement` with type `Map` for constant integer arguments.

Fix concurrent `OPTIMIZE` and `DROP` for `ReplicatedMergeTree`. (). Fix bug for ReplicatedMerge table engines when `ALTER MODIFY COLUMN` query doesn't change the type of decimal column if its size (32 bit or 64 bit) doesn't change. (). Reverted S3 connection pools. (). Fix adding of parts with already existing in destination table names in query `MOVE PARTITION TO TABLE` with non-replicated `MergeTree` tables. (). Fix scalar subquery index analysis. This fixes , which was introduced in https://github.com/ClickHouse/ClickHouse/pull/18896 . (). Fix possible crashes in aggregate functions with combinator Distinct, while using two-level aggregation. This is a follow-up fix of https://github.com/ClickHouse/ClickHouse/pull/18365 . Can only reproduced in production env. No test case available yet. cc @CurtizJ. (). Better error handling and logging in WriteBufferFromS3. (). Fix incorrect query result (and possible crash) which could happen when `WHERE` or `HAVING` condition is pushed before `GROUP BY`. Fixes . (). Fix deadlock in first catboost model execution. Closes . (). Backported in : Fix bug, which leads to underaggregation of data in case of enabled `optimizeaggregationinorder` and many parts in table. Slightly improve performance of aggregation with enabled `optimizeaggregationinorder`. (). Fix wrong `ORDER BY` results when a query contains window functions, and optimization for reading in primary key order is applied. Fixes . (). Fix reading the HTTP POST request with "multipart/form-data" content type. (). Prevent hedged connections overlaps (`Unknown packet 9 from server` error). (). Reverted that may cause significant increase in memory usage while loading external dictionaries of hashed type. This closes . (). In rare case, merge for `CollapsingMergeTree` may create granule with `index_granularity + 1` rows. Because of this, internal check, added in (affects 21.2 and 21.3), may fail with error `Incomplete granules are not allowed while blocks are granules size`. This error did not allow parts to merge. (). The function `decrypt` was lacking a check for the minimal size of data encrypted in AEAD mode. This closes . (). Docker entrypoint: avoid chown of `.` in case when `LOG_PATH` is empty. Closes . (). Disable `asyncsocketforremote`/`usehedged_requests` for buggy linux kernels. (). Fix waiting for `OPTIMIZE` and `ALTER` queries for `ReplicatedMergeTree` table engines. Now the query will not hang when the table was detached or restarted. (). Fix the background thread pool name. (). Fix error `Invalid number of rows in Chunk` in `JOIN` with `TOTALS` and `arrayJoin`. Closes . (). Fix docker entrypoint in case `http_port` is not in the config. (). Fix uncaught exception in InterserverIOHTTPHandler. (). Use finalize() over next() for nested writers. (). Fix query cancellation with `usehedgedrequests=0` and `asyncsocketfor_remote=1`. (). Fix exception which may happen when `SELECT` has constant `WHERE` condition and source table has columns which names are digits. (). Now clickhouse will not throw `LOGICAL_ERROR` exception when we try to mutate the already covered part. Fixes . (). Backported in : Do not limit HTTP chunk size. Fixes . (). Fixed bug in S3 zero-copy replication for hybrid storage. (). Backported in : Buffer overflow (on read) was possible in `tokenbf_v1` full text index. The excessive bytes are not used but the read operation may lead to crash in rare cases. This closes . (). Add (missing) memory accounting in parallel parsing routines. In previous versions OOM was possible when the resultset contains very large blocks of data. This closes . (). Backported in : Remove socket from epoll before cancelling packet receiver in HedgedConnections to prevent possible race. I hope it fixes .

Backported in : Fix ClickHouseDictionarySource configuration loop. Closes . (). Backported in : Fix bug in partial merge join with `LowCardinality`. Close , close . (). Backported in : Fix deserialization of empty string without newline at end of TSV format. This closes . Possible workaround without version update: set `inputformatnullasdefault` to zero. It was zero in old versions. (). Backported in : Fix UB by unlocking the rwlock of the TinyLog from the same thread. (). Backported in : Avoid UB in Log engines for rwlock unlock due to unlock from another thread. (). Backported in : Try flush write buffer only if it is initialized. Fixes segfault when client closes connection very early . (). Backported in : Fix LOGICAL_ERROR for Log with nested types w/o columns in the SELECT clause. (). Fix macOS shared lib build. (). - Added ALL and NONE privilege tests. - Added ROW POLICY tests. - Cleanup of existing tests. - Tests close faster if something fails. (). Fixing LDAP authentication performance test by removing assertion. (). Updating docker/test/testflows/runner/dockerd-entrypoint.sh to use Yandex dockerhub-proxy. (). Add llvm-12 binaries name to search in cmake scripts. Implicit constants conversions to mute clang warnings. Updated submodules to build with CMake 3.19. Mute recursion in macro expansion in readpassphrase library. Deprecated -fuse-ld changed to --ld-path for clang. (). Updating TestFlows to 1.6.74. (). Add tests for NuKeeper. (). remove decode method with python3. (). Allow to use clang-tidy with release builds by enabling assertions if it is used. (). Introduce 2 arguments for clickhouse-server image Dockerfile: deblocation & singlebinary_location. (). Add `tzdata` to Docker containers because reading ORC formats requires it. This closes . (). Enable status check for SQLancer CI run. (). try fix https://clickhouse-test-reports.s3.yandex.net/22289/c71da4a5c8e655f4bdfaa33b92ab022b97dfdf1a/integrationtests(asan).html#fail1 MySQL is started only once with MaterializeMySQL integration test. (). - Added a way to check memory info for the RBAC testflows tests. (). Fixed compiling on ppc64le and use the correct instruction pointer register on ppc64le. (). Fix ClickHouse's config embedding and cctz's timezone embedding on ppc64le. ... (). Enable building with jemalloc on ppc64le ... (). Fix Fedora\RHEL\CentOS not finding libclang_rt.builtins on ppc64le ... (). Fix CMake error about internal CMake variable CMAKEASMCOMPILE_OBJECT not set on ppc64le ... (). Fix compiling boost on ppc64le ... (). Enable compiling on ppc64le with Clang ... (). Re-enable the S3 (AWS) library on aarch64 ... (). Update tests for hedged requests. (). Don't set the same timeouts in ReadBufferFromPocoSocket/WriteBufferFromPocoSocket in nextImpl because it causes a race. (). NO CL ENTRY: 'Improve the translation of `query_log.md` in Chinese documents'. (). NO CL ENTRY: 'Update gui.md: add SeekTable'. (). NO CL ENTRY: 'Flatten libcpuid PEERDIRs'. (). NO CL ENTRY: 'Revert "quick fix for broken resolution of apt.llvm.org on Yandex infra"'. (). Add soft task timeout for Intergation tests (). Merging geometry functions (). Remove outdated suppressions, part 2 (). Move some CI-related scripts to github (). fix (). Add more tests for quota consumption by the SHOW statement (). Save packed keys for GROUP BY with multiple fixed size keys (). Refine MergeTreeData::loadDataParts to not parse format version file and detached directory (). Persistent nukeeper snapshot storage (). Fix logging for optimizeaggregationinorder=1 (with small maxblock_size) (). Adjust prewherewithrowlevelfilter performance test (). Refactor actions dag (). add query formatting idempotence check to fuzzer (). Fix heap-buffer-overflow in highlighting multi-line comments (). Fix global stop merges in test (). DirectDictionary updated

Avoid processing optimizeskipunused_shards twice (). DOCSUP-7197: Escaped Unicode replaced with symbols (). Pod array left pad not multiple of element crash fix (). ShellCommand waitpid eintr signal fix (). Refactoring of data types serialization (). fix a typo in window functions frame (). Added specialized CacheDictionaryStorage (). (). Enable ipv6 in NuRaft (). add an article about ast-based fuzzer (). Changelog 21.3 (). redefine some errorcode (). Fix ambigous column error in joinsinmemory (). Add test for path as a query parameter in system.zookeeper (). ExecutablePool fix default max execution time setting (). DictionaryStructure fix non unique attribute names (). Fix flaky testreplacepartition (). DOC Fix ORDER BY syntax (). PODArray swap fix (). LibraryDictionarySource fix possible leak (). Run three nodes with Replicated database and NuKeeper in CI (). Fix error message in clickhouse-test (). Set SOCK_CLOEXEC for sockets (hardcoded via poco update) (). Tests fixes (that was found by stress tests) (). Fix log_comment for .sh in clickhouse-test (). Remove useless CMake option (). Fix UBSan report in modulo by constant (). Add more variants for memcpy benchmark (). Do not overlap zookeeper path for ReplicatedMergeTree in stateless .sh tests (). make the fuzzer use sources from the CI (). Add one more variant to memcpy benchmark (). fix incorrect number of rows for Chunks with no columns in PartialSor (). docs(fix): typo (). DDLWorker.cpp: fixed exceeded amount of tries typo (). fix integration MaterializeMySQL test (). more robust error handling in perf test (). test for (). Add bash completion support for clickhouse utils (). LRUCache fix exception unsafe element insertion (). fix fuzzer failure in tupleElement formatting (). Fix possibly dangling reference to Context (). Add stress test for distributed queries (). Fix misleading log in WriteBufferFromS3 (). Add a test for (). Add a test for (). Add a test for (). prevent accidental reinterpret_cast in Field::get<> (). Fix UBSan report in addMonths (). Add a test for (). Fix UBSan report in intDiv (). Cast to enum nullable fix (). Small simplification in ExternalLoader. (). Add test for (). quick fix for broken resolution of apt.llvm.org on Yandex infra (). Darwin cmake disable memcpy benchmark (). Fix UBSan report in TransformDateTime64 (). Fix UBSan report in mapPopulateSeries. (). Fix bar with invalid float value (). remove useless code (). stable formatting for negate() (). adjust perf test thresholds (). Fix sleepinsendtablesstatusms/sleepinsenddata_ms in integration tests (). Update requirements.txt (). Fix some flaky order dependent integration tests. (). Prevent busy waiting in hedged requests when asyncsocketfor_remote=0 (). less flaky functional tests (). test for (). CachedCompressedReadBuffer fix cache usage (). Fix MSan report in `quantileDeterministic` (). Remove harmful default parameters (). Fix build error (). Rename NuKeeper and TestKeeper to Keeper in all places (). Update materialize-mysql.md (). Fix native macOS build for ALL_BUILD (Xcode/AppleClang) (). Add suffixes for dockerfile arguments (). More coarse test for DateLUT (). Remove useless code (). Maybe fix false MSan report in GRPC (). Remove old MSan suppressions (part 3) (). Remove old MSan suppressions (part 4) (). Fix flaky tests test_row_policy and test_quota* (). Try fix flaky rabbitmq test (). Fix UBSan report in mapOp (). Remove recursive submodules from Arrow (). Fix some OOMs in stress tests (). Correctly place debug helpers (). fix error message for invalid window frame start (). Lower scale of a test (). Remove TestFlows (2) (). Support ThetaSketch to do set operations. ().