Project: ClickHouse (App Definition and Development / Database)
File: v23.12.1.1368-stable.md

sidebar_position: 1 sidebar_label: 2023 Fix check for non-deterministic functions in TTL expressions. Previously, you could create a TTL expression with non-deterministic functions in some cases, which could lead to undefined behavior later. This fixes . Disallow TTL expressions that don't depend on any columns of a table by default. It can be allowed back by `SET allowsuspiciousttl_expressions = 1` or `SET compatibility = '23.11'`. Closes . (). Remove function `arrayFold` because it has a bug. This closes . This closes . (). Remove the feature of `is_deleted` row in ReplacingMergeTree and the `CLEANUP` modifier for the OPTIMIZE query. This fixes . This closes . This closes . This closes . This closes . The feature has to be removed because it is not good. We have to remove it as quickly as possible, because there is no other option. (). The MergeTree setting `cleandeletedrows` is deprecated, it has no effect anymore. The `CLEANUP` keyword for `OPTIMIZE` is not allowed by default (unless `allowexperimentalreplacingmergewith_cleanup` is enabled). (). Allow disabling of HEAD request before GET request. (). Add a HTTP endpoint for checking if Keeper is ready to accept traffic. (). Add 'union' mode for schema inference. In this mode the resulting table schema is the union of all files schemas (so schema is inferred from each file). The mode of schema inference is controlled by a setting `schemainferencemode` with 2 possible values - `default` and `union`. Closes . (). Add new setting `inputformatcsvtryinfernumbersfrom_strings` that allows to infer numbers from strings in CSV format. Closes . (). Refreshable materialized views. (). Add more warnings on the number of databases, tables. (). Added a new mutation command `ALTER TABLE <table> APPLY DELETED MASK`, which allows to enforce applying of mask written by lightweight delete and to remove rows marked as deleted from disk. (). Added a new SQL function `sqid` to generate Sqids (https://sqids.org/), example: `SELECT sqid(125, 126)`. (). Dictionary with `HASHEDARRAY` (and `COMPLEXKEYHASHEDARRAY`) layout supports `SHARDS` similarly to `HASHED`. (). Add asynchronous metrics for total primary key bytes and total allocated primary key bytes in memory. (). Table system.droppedtablesparts contains parts of system.dropped_tables tables (dropped but not yet removed tables). (). Add `FORMAT_BYTES` as an alias for `formatReadableSize`. (). Add SHA512_256 function. (). Allow passing optional SESSION_TOKEN to `s3` table function. (). Clause `ORDER BY` now supports specifying `ALL`, meaning that ClickHouse sorts by all columns in the `SELECT` clause. Example: `SELECT col1, col2 FROM tab WHERE (). Added functions for punycode encoding/decoding: `punycodeEncode()` and `punycodeDecode()`. (). This PR reproduces the implementation of `PASTE JOIN`, which allows users to join tables without `ON` clause. Example: ``` SQL SELECT FROM ( SELECT number AS a FROM numbers(2) ) AS t1 PASTE JOIN ( SELECT number AS a FROM numbers(2) ORDER BY a DESC ) AS t2. (). A handler `/binary` opens a visual viewer of symbols inside the ClickHouse binary. (). Made copy between s3 disks using a s3-server-side copy instead of copying through the buffer. Improves `BACKUP/RESTORE` operations and `clickhouse-disks copy` command.

HashJoin respects setting `maxjoinedblocksizerows` and do not produce large blocks for `ALL JOIN`. (). Release memory for aggregation earlier. This may avoid unnecessary external aggregation. (). Improve performance of string serialization. (). Support trivial count optimization for `Merge`-engine tables. (). Optimized aggregation in some cases. (). The `hasAny()` function can now take advantage of the full-text skipping indices. (). Function `if(cond, then, else)` (and its alias `cond ? : then : else`) were optimized to use branch-free evaluation. (). Extract non intersecting parts ranges from MergeTree table during FINAL processing. That way we can avoid additional FINAL logic for this non intersecting parts ranges. In case when amount of duplicate values with same primary key is low, performance will be almost the same as without FINAL. Improve reading performance for MergeTree FINAL when `donotmergeacrosspartitionsselectfinal` setting is set. (). MergeTree automatically derive `donotmergeacrosspartitionsselectfinal` setting if partition key expression contains only columns from primary key expression. (). Speedup MIN and MAX for native types. (). Make inserts into distributed tables handle updated cluster configuration properly. When the list of cluster nodes is dynamically updated, the Directory Monitor of the distribution table cannot sense the new node, and the Directory Monitor must be re-noded to sense it. (). Replace --no-system-tables with loading virtual tables of system database lazily. (). Clickhouse-test print case sn, current time and case name in one test case. (). Do not allow creating replicated table with inconsistent merge params. (). Implement SLRU cache policy for filesystem cache. (). Show uncompressed size in `system.tables`, obtained from data parts' checksums . (). Add `skipunavailableshards` as a setting for `Distributed` tables that is similar to the corresponding query-level setting. Closes . (). Function `substring()` (aliases: `substr`, `mid`) can now be used with `Enum` types. Previously, the first function argument had to be a value of type `String` or `FixedString`. This improves compatibility with 3rd party tools such as Tableau via MySQL interface. (). Better hints when a table doesn't exist. (). Allow to overwrite `maxpartitionsizetodrop` and `maxtablesizetodrop` server settings in query time. (). Add support for read-only flag when connecting to the ZooKeeper server (fixes ). (). Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk). Fix possible issues with `errorcount` from `system.distributionqueue` (in case of `distributeddirectorymonitormaxsleeptimems` >5min). Introduce profile event to track async INSERT failures - `DistributedAsyncInsertionFailures`. (). The limit for the number of connections per endpoint for background fetches was raised from `15` to the value of `backgroundfetchespoolsize` setting. - MergeTree-level setting `replicatedmaxparallelfetchesforhost` became obsolete - MergeTree-level settings `replicatedfetcheshttpconnectiontimeout`, `replicatedfetcheshttpsendtimeout` and `replicatedfetcheshttpreceivetimeout` are moved to the Server-level. - Setting `keepalivetimeout` is added to the list of Server-level settings. (). It is now possible to refer to ALIAS column in index (non-primary-key) definitions (issue ). Example: `CREATE TABLE tab(col UInt32, colalias ALIAS col + 1, INDEX idx (colalias) TYPE minmax) ENGINE = MergeTree ORDER BY col;`. (). Function `format()` now supports arbitrary argument types (instead of only `String` and `FixedString` arguments). This is important to calculate `SELECT format('The {0} to all questions is {1}', 'answer', 42)`. (). Support PostgreSQL generated columns and default column values in

Closes . (). Allow to apply some filesystem cache config settings changes without server restart. (). Handle sigabrt case when getting PostgreSQl table structure with empty array. (). Allows to use the `datetrunc()` function with the first argument not depending on the case of it. Both cases are now supported: `SELECT datetrunc('day', now())` and `SELECT date_trunc('DAY', now())`. (). Expose the total number of errors occurred since last server as a `ClickHouseErrorMetric_ALL` metric. (). Allow nodes in config with fromenv/fromzk and non empty element with replace=1. (). Generate malformed output that cannot be parsed as JSON. (). Consider lightweight deleted rows when selecting parts to merge if enabled. (). Make querying system.filesystem_cache not memory intensive. (). Allow IPv6 to UInt128 conversion and binary arithmetic. (). Support negative positional arguments. Closes . (). Add a setting for `async inserts deduplication cache` -- how long we wait for cache update. Deprecate setting `asyncblockidscacheminupdateinterval_ms`. Now cache is updated only in case of conflicts. (). `sleep()` function now can be cancelled with `KILL QUERY`. (). Slightly better inference of unnamed tupes in JSON formats. (). Refactor UserDefinedSQL classes to make it possible to add SQL UDF storages which are different from ZooKeeper and Disk. (). Forbid `CREATE TABLE ... AS SELECT` queries for Replicated table engines in Replicated database because they are broken. Reference . (). Fix and improve transform query for external database, we should recursively obtain all compatible predicates. (). Support dynamic reloading of filesystem cache size. Closes . (). Fix system.stack_trace for threads with blocked SIGRTMIN. (). Added a new setting `readonly` which can be used to specify a s3 disk is read only. It can be useful to create a table with read only `s3_plain` type disk. (). Support keeper failures in quorum check. (). Add max/peak RSS (`MemoryResidentMax`) into system.asynchronous_metrics. (). Fix system.stacktrace for threads with blocked SIGRTMIN (and also send signal to the threads only if it is not blocked to avoid waiting `storagesystemstacktracepipereadtimeoutms` when it does not make any sense). (). This PR allows users to use s3 links (`https://` and `s3://`) without mentioning region if it's not default. Also find the correct region if the user mentioned the wrong one. ### Documentation entry for user-facing changes. (). `clickhouse-format --obfuscate` will know about Settings, MergeTreeSettings, and time zones and keep their names unchanged. (). Added explicit `finalize()` function in `ZipArchiveWriter`. Simplify too complicated code in `ZipArchiveWriter`. This PR fixes . (). The primary key analysis in MergeTree tables will now be applied to predicates that include the virtual column `partoffset` (optionally with `_part`). This feature can serve as a poor man's secondary index. (). Make caches with the same path use the same cache objects. This behaviour existed before, but was broken in https://github.com/ClickHouse/ClickHouse/pull/48805 (in 23.4). If such caches with the same path have different set of cache settings, an exception will be thrown, that this is not allowed. (). Allow usage of Azure-related table engines/functions on macOS. (). ClickHouse Fast Test now uses Musl instead of GLibc. (). Run ClickBench for every commit. This closes .

Fixed a sorting order breakage in TTL GROUP BY (). fix: split lttb bucket strategy, first bucket and last bucket should only contain single point (). Fix possible deadlock in Template format during sync after error (). Fix early stop while parsing file with skipping lots of errors (). Prevent dictionary's ACL bypass via dictionary() table function (). Fix another case of non-ready set. (). Fix several issues regarding PostgreSQL `array_ndims` usage. (). Fix RWLock inconsistency after write lock timeout (). Fix: don't exclude ephemeral column when building pushing to view chain (). MaterializedPostgreSQL: fix issue , add test for (). Ignore ON CLUSTER clause in grant/revoke queries for management of replicated access entities. (). Fix crash in clickhouse-local (). Materialize block in HashJoin for Type::EMPTY (). Fix possible segfault in PostgreSQLSource (). Fix type correction in HashJoin for nested low cardinality (). Avoid hangs of system.stack_trace by correctly prohibit parallel read from it (). Fix SIGSEGV for aggregation of sparse columns with any() RESPECT NULL (). Fix unary operators parsing (). Fix RWLock inconsistency after write lock timeout (again) (). Table engine MaterializedPostgreSQL fix dependency loading (). Fix retries for disconnected nodes for BACKUP/RESTORE ON CLUSTER (). Fix bug window functions: revert (). Fix result of external aggregation in case of partially materialized projection (). Fix merge in aggregation functions with `Map` combinator (). Disable system.kafka_consumers by default (due to possible live memory leak) (). Fix low-cardinality keys support in MergeJoin (). Create consumers for Kafka tables on fly (but keep them for some period since last used) (). InterpreterCreateQuery sample block fix (). bugfix: addresses_expr ignored for psql named collections (). Fix invalid memory access in BLAKE3 (Rust) (). Resurrect `arrayFold()` (). Normalize function names in CREATE INDEX (). Fix handling of unavailable replicas before first request happened (). Fix literal alias misclassification (). Revert "Fix bug window functions: revert " (). Fix invalid preprocessing on Keeper (). Fix Integer overflow in Poco::UTF32Encoding (). Fix parallel replicas in presence of a scalar subquery with a big integer value (). Fix `accurateCastOrNull` for out-of-range DateTime (). Fix possible PARAMETEROUTOF_BOUND error during subcolumns reading from wide part in MergeTree (). Remove parallel parsing for JSONCompactEachRow (). fix CREATE VIEW hang (). Fix parallel parsing for JSONCompactEachRow (). NO CL ENTRY: 'Revert "Revert "Update Sentry""'. (). NO CL ENTRY: 'Revert "Fix RWLock inconsistency after write lock timeout"'. (). NO CL ENTRY: 'Revert "improve CI with digest for docker, build and test jobs"'. (). NO CL ENTRY: 'Reapply "improve CI with digest for docker, build and test jobs"'. (). NO CL ENTRY: 'Revert "Merge pull request from mkmkme/mkmkme/reload-config"'. (). NO CL ENTRY: 'Revert "Add system.droppedtablesparts table"'. (). NO CL ENTRY: 'Revert "Consider lightweight deleted rows when selecting parts to merge"'. (). NO CL ENTRY: 'Revert "Fix leftover processes/hangs in tests"'. (). NO CL ENTRY: 'Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)"'. (). NO CL ENTRY: 'Revert "Implement punycode encoding/decoding"'. (). Randomize more settings (). Add more tests for `compile_expressions` ().

improve CI with digest for docker, build and test jobs (). Prepare the introduction of more keeper faults (). Analyzer: Fix assert in tryReplaceAndEqualsChainsWithConstant (). Check what will happen if we build ClickHouse with Musl (). support memory soft limit for keeper (). Randomize disabled optimizations in CI (). Don't throw if noop when dropping database replica in batch (). Better JSON -> JSONEachRow fallback without catching exceptions (). Add tests for (). Add profile event for cache lookup in `ThreadPoolRemoteFSReader` (). Remove select() usage (). Parallel replicas: friendly settings (). Fix formatting string prompt error (). Tune CI scale up/down multipliers (). Revert "Revert "Implemented series period detect method using pocketfft lib"" (). Correctly handle errors during opening query in editor in client (). Add a test for (). Add a test for (). Update CHANGELOG.md (). Update version after release (). Update version_date.tsv and changelogs after v23.11.1.2711-stable (). Identify failed jobs in lambda and mark as steps=0 (). Fix flaky test: distinct in order with analyzer (). CHJIT add assembly printer (). Fix parsing virtual hosted S3 URI in clickhouse_backupview script (). Fix docs for `fileCluster` (). Analyzer: Fix logical error in MultiIfToIfPass (). Throw more clear exception (). Fix "logs and exception messages formatting", part 1 (). Fix "logs and exception messages formatting", part 2 (). Fix "logs and exception messages formatting", part 3 (). Fix "logs and exception messages formatting", part 4 (). Remove bad test (1) (). Remove bad test (2) (). ClickHouse Cloud promotion (). Remove bad test (3) (). Remove bad test (4) (). Random changes in random files (). Merge half of (). Analyzer: Fix JOIN ON true with joinusenulls (). Pin alpine version of integration tests helper container (). Add support for system.stack_trace filtering optimizations for analyzer (). test for (). support keeper memory soft limit ratio (). Fix testdictionariesupdateandreload/test.py::testreloadwhile_loading flakiness (). Tune autoscale to scale for single job in the queue (). Tune network memory for dockerhub proxy hosts (). Parallel replicas: announcement response handling improvement (). Fix building Rust with Musl (). Fix flaky testparallelreplicasdistributedreadfromall (). Minor refactoring of toStartOfInterval() (). Don't run test 02919skiplotsofparsing_errors on aarch64 (). More respect to `minnumberof_marks` in `ParallelReplicasReadingCoordinator` (). SerializationString reduce memory usage (). Fix ThreadSanitizer data race in librdkafka (). Rename `system.asyncloader` into `system.asynchronousloader` (). Set replica number to its position in cluster definition (). fix clickhouse-client invocation in 02327capnprotoprotobufemptymessages (). Fix flaky testparallelreplicasoverdistributed (). Revert (). Dumb down `substring()` tests (). Update version_date.tsv and changelogs after v23.11.2.11-stable (). Fix 02906forceoptimizeprojectionname (). ClickBench: slightly better (). Fix 02932killquery_sleep flakiness (). Revert "Replace --no-system-tables with loading virtual tables of system database lazily" (). Fix memory leak in StorageHDFS (). Remove hardcoded clickhouse-client invocations from tests (). Follow up to (). Fix assertion in HashJoin (). More efficient constructor for SerializationEnum (). Fix testunsetskipunavailableshards (). Add argument to fill the gap in cherry-pick (). Delete debug logging in OutputFormatWithUTF8ValidationAdaptor (). Remove heavy rust stable toolchain (). Improvements for 00002logandexceptionmessages_formatting (). Update CHANGELOG.md (). remove cruft from TablesLoader (). Fix `/dashboard` work with passwords (). Remove wrong test

Fix docker image for integration tests (fixes CI) (). Remove C++ templates (normalizeQuery) (). A small fix for dashboard (). Always use `pread` for reading cache segments (). Improve some tests (). Revert "Merge pull request from azat/system.stacktrace-rttgsigqueueinfo" (). Add a test for (). Fix style-check checkout head-ref (). refine error message (). CI for docs only fix (). Replace rust's BLAKE3 with llvm's implementation (). Better trivial count optimization for storage `Merge` (). enhanced docs for `date_trunc()` (). CI: add needschangedfiles flag for pr_info (). more messages in ci (). Test parallel replicas with forceprimarykey setting (). Update 00002logandexceptionmessages_formatting.sql (). Fix rare race in external sort/aggregation with temporary data in cache (). Fix segfault in FuzzJSON engine (). fix freebsd build (). Rename canUseParallelReplicas to canUseTaskBasedParallelReplicas (). Remove fixed tests from analyzertechdebt.txt (). More verbose errors on 00002logandexceptionmessages_formatting (). Make window insert result into constant (). CI: Happy new year (). Follow up for (). always run ast_fuzz and sqllancer (). Add GH status for PR formating (). Small improvement for SystemLogBase (). Bump Azure to v1.6.0 (). Correct values for randomization (). Non post request should be readonly (). Revert "Merge pull request from guoxiaolongzte/clickhouse-test (). fix typo in the test 02479 (). Bump Azure to 1.7.2 (). Fix flaky test `02567andconsistency` (). Fix Tests Bugfix Validate Check (). Fix for nightly job for digest-ci (). Test for parallel replicas with remote() (). Minor cosmetic changes (). Reintroduce OPTIMIZE CLEANUP as no-op (). Add compatibility in the replication protocol for a removed feature (). Flaky 02922analyzeraggregatenothingtype (). Update version_date.tsv and changelogs after v23.11.3.23-stable (). Limited CI on the master for docs only change (). style fix (). Support "do not test" label with ci.py (). Use the single images list for integration tests everywhere (). Disable parallel replicas with IN (subquery) (). Fix clang-tidy (). Run build report check job on build failures, fix (). Fix dashboard legend sorting and rows number (). Remove retryStrategy assignments overwritten in ClientFactory::create() (). Helper improvements (). Pass through exceptions for reading from S3 (). (). Add bytesuncompressed to system.partlog (). Update docker/test/stateful/run.sh (). Update 00165jitaggregate_functions.sql (). Update clickhouse-test (). Profile event 'ParallelReplicasUsedCount' (). Fix flaky test `02719aggregatewithemptystring_key` (). Fix (). Add base backup name to system.backups and system.backup_log tables (). Fix use-after-move (). Looking at strange code (). Fix all Exception with missing arguments (). Fix leftover processes/hangs in tests (). Fix DWARFBlockInputFormat failing on DWARF 5 unit address ranges (). Fix error in archive reader (). Fix DWARFBlockInputFormat using wrong base address sometimes (). Add support for specifying query parameters in the command line in clickhouse-local (). Fix leftover processes/hangs in tests (resubmit) (). Add optimization for AND notEquals chain in logical expression optimizer (). Fix syntax and doc (). Cleanup some known short messages (). Some code refactoring (was an attempt to improve build time, but failed) (). Fix perf test README (). (). Reintroduce compatibility with `is_deleted` on a syntax level (). Avoid throwing ABORTED on normal situations (). Remove mayBenefitFromIndexForIn (). Allow a few retries when committing a part during shutdown (). Revert ().