Project: ClickHouse (App Definition and Development / Database)
File: v22.10.1.1877-stable.md

sidebar_position: 1 sidebar_label: 2022 Rename cache commands: `show caches` -> `show filesystem caches`, `describe cache` -> `describe filesystem cache`. (). Remove support for the `WITH TIMEOUT` section for `LIVE VIEW`. This closes . (). Add Rust code support into ClickHouse with BLAKE3 hash-function library as an example. (). This is the initial implement of Kusto Query Language. (MVP). (). Support limiting of temporary data stored on disk using settings `maxtemporarydataondisksizeforuser`/`maxtemporarydataondisksizeforquery` . (). Support Java integers hashing in `javaHash`. (). This PR is to support the OpenSSL in-house build like the BoringSSL submodule. Build flag i.e. ENABLECHBUNDLE_BORINGSSL is used to choose between BoringSSL and OpenSSL. By default, the BoringSSL in-house build will be used. (). Composable protocol configuration is added. (). Add OpenTelemetry support to ON CLUSTER DDL(require `distributedddlentryformatversion` to be set to 4). (). Add setting `formatjsonobjecteachrowcolumnforobjectname` to write/parse object name as column value in JSONObjectEachRow format. (). adds Morton Coding (ZCurve) encode/decode functions. (). Implement support for different UUID binary formats with support for the two most prevalent ones: the default big-endian and Microsoft's mixed-endian as specified in . (). Added an aggregate function `analysisOfVariance` (`anova`) to perform a statistical test over several groups of normally distributed observations to find out whether all groups have the same mean or not. Original PR . (). Add support for `SET setting_name = DEFAULT`. (). Add `URL` Functions which conform rfc. Functions include: `cutToFirstSignificantSubdomainCustomRFC`, `cutToFirstSignificantSubdomainCustomWithWWWRFC`, `cutToFirstSignificantSubdomainRFC`, `cutToFirstSignificantSubdomainWithWWWRFC`, `domainRFC`, `domainWithoutWWWRFC`, `firstSignificantSubdomainCustomRFC`, `firstSignificantSubdomainRFC`, `portRFC`, `topLevelDomainRFC`. (). Added functions (`randUniform`, `randNormal`, `randLogNormal`, `randExponential`, `randChiSquared`, `randStudentT`, `randFisherF`, `randBernoulli`, `randBinomial`, `randNegativeBinomial`, `randPoisson` ) to generate random values according to the specified distributions. This closes . (). Implement operator precedence element parser to resolve stack overflow issues and make the required stack size smaller. (). DISTINCT in order optimization leverage sorting properties of data streams. This improvement will enable reading in order for DISTINCT if applicable (before it was necessary to provide ORDER BY for columns in DISTINCT). (). ColumnVector: optimize UInt8 index with AVX512VBMI. (). The performance experiments of SSB (Star Schema Benchmark) on the ICX device (Intel Xeon Platinum 8380 CPU, 80 cores, 160 threads) shows that this change could bring a 2.95x* improvement of the geomean of all subcases' QPS. (). Fixed slowness in JSONExtract with LowCardinality(String) tuples. (). Add ldapr capabilities to AArch64 builds. This is supported from Graviton 2+, Azure and GCP instances. Only appeared in clang-15 . (). Improve performance when comparing strings and one argument is empty constant string. (). optimize insertFrom of ColumnAggregateFunction to share Aggregate State in some cases. (). Relax the "Too many parts" threshold. This closes . Now ClickHouse will allow more parts in a partition if the average part size is large enough (at least 10 GiB). This allows to have up to petabytes of data in a single partition of a single table on a single server, which is possible using disk shelves or object storage. (). Make writing to AzureBlobStorage more efficient (respect `maxsinglepartuploadsize` instead of writing a block per each buffer size). Inefficiency mentioned in . (). Make thread ids in the process list and query_log unique to avoid waste.

Added new infrastructure for query analysis and planning under `allowexperimentalanalyzer` setting. (). Support expression `(EXPLAIN SELECT ...)` in a subquery. Queries like `SELECT * FROM (EXPLAIN PIPELINE SELECT col FROM TABLE ORDER BY col)` became valid. (). Currently changing `asyncinsertmaxdatasize` or `asyncinsertbusytimeoutms` in scope of query makes no sense and this leads to bad user experience. E.g. user wants to insert data rarely and he doesn't have an access to server config to tune default settings. (). Embedded Keeper will always start in the background allowing ClickHouse to start without achieving quorum. (). Improvements for reading from remote filesystems, made threadpool size for reads/writes configurable. Closes . (). Made reestablishing a new connection more reactive in case of expiration of the previous one. Previously there was a task which spawns every minute by default and thus a table could be in readonly state for about this time. (). Support all combinators combination in WindowTransform/arratReduce/initializeAggregation/aggregate functions versioning. Previously combinators like `ForEach/Resample/Map` didn't work in these places, using them led to exception like`State function ... inserts results into non-state column`. (). Now projections can be used with zero copy replication. (). - Add function tryDecrypt that returns NULL when decrypt fail (e.g. decrypt with incorrect key) instead of throwing exception. (). Add the `unreserved_space` column to the `system.disks` table to check how much space is not taken by reservations per disk. (). Support s3 authorisation headers from ast arguments. (). Add setting 'allowimplicitnopassword' that forbids creating a user with no password unless 'IDENTIFIED WITH nopassword' is explicitly specified. (). keeper-improvement: add support for uploading snapshots to S3. S3 information can be defined inside `keeperserver.s3snapshot`. (). Add support for MultiRead in Keeper and internal ZooKeeper client. (). add a support for decimal type comparing with floating point literal in IN operator. (). Allow readable size values in cache config. (). Check file path for path traversal attacks in errors logger for input formats. (). ClickHouse could cache stale DNS entries for some period of time (15 seconds by default) until the cache won't be updated asynchronously. During these period ClickHouse can nevertheless try to establish a connection and produce errors. This behaviour is fixed. (). Add interactive history search with fzf-like utility (fzf/sk) for `clickhouse-client`/`clickhouse-local` (note you can use `FZFDEFAULTOPTS`/`SKIMDEFAULTOPTIONS` to additionally configure the behavior). (). For client when connecting to a secure server with invalid certificate only allow to proceed with '--accept-certificate' flag. (). Add function "tryBase58Decode()", similar to the existing function "tryBase64Decode()". (). Improve feedback when replacing partition with different primary key. Fixes . (). Replace back `clickhouse su` command with `sudo -u` in start in order to respect limits in `/etc/security/limits.conf`. (). Fix parallel parsing: segmentator now checks maxblocksize. (). Don't report TABLEISDROPPED exception in order to skip table in case is was just dropped. (). Improve option enableextendedresultsfordatetime_functions to return results of type DateTime64 for functions toStartOfDay, toStartOfHour, toStartOfFifteenMinutes, toStartOfTenMinutes, toStartOfFiveMinutes, toStartOfMinute and timeSlot. (). Improve DateTime type inference for text formats. Now it respect setting `datetimeinput_format` and doesn't try to infer datetimes from numbers as timestamps. Closes Closes . (). Remove confusing warning when inserting with `performttlmoveoninsert`=false.

Allow user to write `countState()` similar to `count(*)`. This closes . (). - Fix rankCorr size overflow. (). Added an option to specify an arbitrary string as an environment name in the Sentry's config for more handy reports. (). Added system table `asynchronousinsertlog `. It contains information about asynchronous inserts (including results of queries in fire-and-forget mode (with `waitforasync_insert=0`)) for better introspection. (). Fix parsing out-of-range Date from CSV:. (). parseDataTimeBestEffort support comma between date and time. Closes . (). Add support for methods lz4, bz2, snappy in 'Accept-Encoding'. (). Various minor fixes for BLAKE3 function. (). Improved stale replica recovery process for `ReplicatedMergeTree`. If lost replica have some parts which absent on a healthy replica, but these parts should appear in future according to replication queue of the healthy replica, then lost replica will keep such parts instead of detaching them. (). Support BACKUP to S3 with as-is path/data structure. (). Add a possibility to use Date32 arguments for datediff function. Fix issue in datediff function when using DateTime64 arguments with start date before Unix epoch and end date after Unix epoch. (). When uploading big parts to minio, 'Complete Multipart Upload' can take a long time. Minio sends heartbeats every 10 seconds (see https://github.com/minio/minio/pull/7198). But clickhouse times out earlier, because the default send/receive timeout is to 5 seconds. (). Add `S3` as a new type of the destination of backups. (). Fix rarely invalid cast of aggregate state types with complex types such as Decimal. This fixes . (). Support skipping cache completely (both download to cache and reading cached data) in case the requested read range exceeds the threshold defined by cache setting `bypasscachethreashold`, requires to be enabled with `enablebypasscachewiththreshold`). (). Merge parts if every part in the range is older than a certain threshold. The threshold can be set by using `minagetoforcemerge_seconds`. This closes . (). Enabled CompiledExpressionCache in clickhouse-local. (). Remove support for the `{database}` macro from the client's prompt. It was displayed incorrectly if the database was unspecified and it was not updated on `USE` statements. This closes . (). - Improve the time to recover lost keeper connections. (). Allow to use Date32 arguments for dateName function. (). Now filters with NULL literals will be used during index analysis. This closes https://github.com/ClickHouse/ClickHouse/pull/41814 . (). - Choose correct aggregation method for LowCardinality with BigInt. (). Fix using subqueries in row policy filters. This PR fixes . (). Added support of WHERE clause generation to AST Fuzzer and possibility to add or remove ORDER BY and WHERE clause. (). Aarch64 binaries now require at least ARMv8.2, released in 2016. Most notably, this enables use of ARM LSE, i.e. native atomic operations. Also, CMake build option "NOARMV81OR_HIGHER" has been added to allow compilation of binaries for older ARMv8.0 hardware, e.g. Raspberry Pi 4. (). After updating runners to 22.04 cgroups stopped to work in privileged mode, here's the issue https://github.com/moby/moby/issues/42275#issuecomment-1115055846. (). Allow building ClickHouse with Musl (small changes after it was already supported but broken). (). - Add the `$CLICKHOUSE_CRONFILE` file checking to avoid running the `sed` command to get the file not found error.

Update cctz to the latest master, update tzdb to 2020e. (). Update tzdata to 2022e to support the new timezone changes. Palestine transitions are now Saturdays at 02:00. Simplify three Ukraine zones into one. Jordan and Syria switch from +02/+03 with DST to year-round +03. (https://data.iana.org/time-zones/tzdb/NEWS). This closes . (). Fix power8 support. (). Several fixes for DiskWeb. (). Fixes issue when docker run will fail if "https_port" is not present in config. (). Mutations were not cancelled properly on server shutdown or `SYSTEM STOP MERGES` query and cancellation might take long time, it's fixed. (). Fix wrong result of queries with `ORDER BY` or `GROUP BY` by columns from prefix of sorting key, wrapped into monotonic functions, with enable "read in order" optimization (settings `optimizereadinorder` and `optimizeaggregationinorder`). (). Fix possible crash in `SELECT` from `Merge` table with enabled `optimizemonotonousfunctionsinorder_by` setting. Fixes . (). Fixed "Part ... intersects part ..." error that might happen in extremely rare cases if replica was restarted just after detaching some part as broken. (). Don't allow to create or alter merge tree tables with virtual column name rowexists, which is reserved for lightweight delete. Fixed . (). Fix a bug that CORS headers are missing in some HTTP responses. (). 22.9 might fail to startup `ReplicatedMergeTree` table if that table was created by 20.3 or older version and was never altered, it's fixed. Fixes . (). When the batch sending fails for some reason, it cannot be automatically recovered, and if it is not processed in time, it will lead to accumulation, and the printed error message will become longer and longer, which will cause the http thread to block. (). Fix compact parts with compressed marks setting. Fixes and . (). Old versions of Replicated database doesn't have a special marker in (). Fix possible exception in fs cache. (). Fix useenvironmentcredentials for s3 table function. (). Fixed "Directory already exists and is not empty" error on detaching broken part that might prevent `ReplicatedMergeTree` table from starting replication. Fixes . (). toDateTime64() now returns the same output with negative integer and float arguments. (). Fix write into AzureBlobStorage. Partially closes . (). Fix the bzip2 decoding issue for specific bzip2 files. (). - Fix SQL function "toLastDayOfMonth()" with setting "enableextendedresultsfordatetimefunctions = 1" at the beginning of the extended range (January 1900). - Fix SQL function "toRelativeWeekNum()" with setting "enableextendedresultsfordatetimefunctions = 1" at the end of extended range (December 2299). - Improve the performance of for SQL functions "toISOYear()", "toFirstDayNumOfISOYearIndex()" and "toYearWeekOfNewyearMode()" by avoiding unnecessary index arithmetics. (). The maximum size of fetches for each table accidentally was set to 8 while the pool size could be bigger. Now the maximum size of fetches for table is equal to the pool size. (). A table might be shut down and a dictionary might be detached before checking if can be dropped without breaking dependencies between table, it's fixed. Fixes . (). Fix bad inefficiency of `remotefilesystemread_method=read` with filesystem cache. Closes . (). Fix possible timeout exception for distributed queries with usehedgedrequests=0. (). Fixed a minor bug inside function `runningDifference` in case of using it with `Date32`

Previously `Date` was used and it may cause some logical errors like `Bad cast from type DB::ColumnVector<int> to DB::ColumnVector<unsigned short>'`. (). Fix reusing of files > 4GB from base backup. (). DISTINCT in order fails with LOGICAL_ERROR if first column in sorting key contains function. (). Fix a bug with projections and the `aggregatefunctionsnullforempty` setting. This bug is very rare and appears only if you enable the `aggregatefunctionsnullforempty` setting in the server's config. This closes . (). - Fix read from buffer with read in order desc. (). Fix a bug which prevents ClickHouse to start when backgroundpoolsize setting is set on default profile but backgroundmergesmutationsconcurrencyratio is not. (). `ALTER UPDATE` of attached part (with columns different from table schema) could create an invalid `columns.txt` metadata on disk. Reading from such part could fail with errors or return invalid data. Fixes . (). Setting `additionaltablefilters` were not applied to `Distributed` storage. Fixes . (). Fix a data race in query finish/cancel. This closes . (). This reverts which introduced a regression in date/time functions. (). Fix assert cast in join on falsy condition, Close . (). Fix buffer overflow in the processing of Decimal data types. This closes . (). `AggregateFunctionQuantile` now correctly works with UInt128 columns. Previously, the quantile state interpreted `UInt128` columns as `Int128` which could have led to incorrect results. (). Fix bad_assert during INSERT into Annoy indexes over non-Float32 columns. (). This closes . (). Fix function `arrayElement` with type `Map` with `Nullable` values and `Nullable` index. (). Fix unexpected table loading error when partition key contains alias function names during server upgrade. (). Fixed SipHash Endian issue for s390x platform. (). Enable lib base64 for ppc64le platform. (). Fixed Endian issue in T64 compression codec on s390x. (). NO CL ENTRY: 'Revert "Disable parallel s3 multipart upload for part moves."'. (). NO CL ENTRY: 'Revert "Attempt to fix abort from parallel parsing"'. (). NO CL ENTRY: 'Revert "Low cardinality cases moved to the function for its corresponding type"'. (). Test for ignore function in PARTITION KEY (). Add fuzzer for table definitions (). Add missing tests for legacy geobase (). Remove obsolete comment from the config.xml (). Resurrect parallel distributed insert select with s3Cluster (). Update runners to a recent version to install on 22.04 (). Refactor wiping sensitive information from logs. (). Better S3 logs (). Fix typos in JSON formats after (). Fix drop for KeeperMap (). increase default maxsuspiciousbroken_parts to 100 (). Release AWS SDK log level + replace one exception (). Fix a destruction order for views ThreadStatus (). Add very explicit logging on disk choice for fetch (). Fix race between ~BackgroundSchedulePool and ~DNSCacheUpdater (). Add changelog for 22.9 (). Update version after release (). Fix error message (). Add test for setting outputformatjsonvalidateutf8 (). Resolve findings from clang-tidy (). Ignore Keeper errors from ReplicatedMergeTreeAttachThread in stress tests (). Collect logs in Stress test using clickhouse-local (). Disable flaky `testmergetreeazureblob_storage` (). Update version_date.tsv and changelogs after v22.9.2.7-stable (). Fix part removal retries (). Try fix azure tests (). Fix test build (). Change logging levels in cache

Revert of "Revert the revert of "ColumnVector: optimize filter with AVX512 VBMI2 compress store" " (). Fix SET query parameters formatting (). Support to run testcases on macOS (). Bump LLVM from 12 to 13 (). ColumnVector: re-enable AVX512VBMI/AVX512VBMI2 optimized filter and index (). Update 02354_annoy.sql (). Fix the typo preventing building latest images (). Make automatic download script choose between ARMv8.0 or ARMv8.2 builds (). Fix tests for docker-ci (). Possible fix for KeeperMap drop (). Fix drop of completely dropped table (). Log git hash during startup (). Revert "ColumnVector: optimize UInt8 index with AVX512VBMI ()" (). Small fix in dashboard (). Keep the most important log in stress tests (). Use copy for some operations instead of hardlinks (). Remove unused variable in registerStorageMergeTree.cpp (). Fix Jepsen (). Increase `requesttimeoutms` for s3 tests in CI (). tests: fix debug symbols (and possible crashes) for backward compatiblity check (). Remove two redundant lines (). Infer Object type only when allowexperimentalobject_type is enabled (). Add default UNION/EXCEPT/INTERSECT to the echo query text (). Consolidate CMake-generated config headers (). Fix 02267fileglobsschemainference.sql flakiness (). Docs: Remove obsolete modelEvaluate() mention (). Better exception message for duplicate column names in schema inference (). Docs: Reference external papers as DOIs (). Make LDAPR a prerequisite for downloading the ARMv8.2 build (). Another sync replicas in testrecoveryreplica (). remove unused code (). Move all queries for MV creation to the end of queue during recovering (). Fix broken testdisksapp_func (). Temporarily disable ThreadFuzzer with TSan (). Enable some disabled S3 tests (). QOL log improvements (). Fix non-deterministic test results (). Earlier throw exception in PullingAsyncPipelineExecutor. (). Fix linker error (). Bump LLVM from 13 to 14 (). Update version_date.tsv and changelogs after v22.3.13.80-lts (). Update version_date.tsv and changelogs after v22.7.6.74-stable (). Update version_date.tsv and changelogs after v22.8.6.71-lts (). Update version_date.tsv and changelogs after v22.9.3.18-stable (). Add a warning message to release.py script, require release type (). Rename maxtempdataondisk -> maxtemporarydataondisk (). Add more checkStackSize calls (). Fix test 02403bighttpchunksize (). More sane behavior of part number thresholds override in query level settings (). Remove useless code (). Refactoring: Uninline some error handling methods (). Fix warning that ENABLE_REPLXX is unused (). Drop leftovers of libexecinfo (). More detailed exception message (). Build against an LLVM version which has clang (). Add log message and lower the retry timeout in MergeTreeRestartingThread (). Update amqp-cpp (). Fix No such key during table drop (). Temporarily disable too aggressive tests (). Fix style check (). Function name normalization fix functions header (). remove unused virtual keyword (). Fix crash in `SummingMergeTree` with `LowCardinality` (). Fix drop of completely dropped table (). Fix assertion in bloom filter index (). Ignore core.autocrlf for tests references (). Fix progress for INSERT SELECT (). Avoid adding extra new line after using fuzzy history search (). Add `at` to runner AMI, bump gh runner version (). Use sendmetadata instead of sendobject_metadata (). Docs: Preparations to remove misc statements page (). Followup for TemporaryDataOnDisk (). Disable 02122joingroupbytimeout for debug (). Update version_date.tsv and changelogs after v22.6.9.11-stable (). JIT compilation migration to LLVM 15 (). Fix build without TSA (). Update

(). Add null pointer checks (). Revert (). Follow up for (). Fix checking parent for old-format parts (). Revert "Resurrect parallel distributed insert select with s3Cluster (). Docs: Add "TABLE" to CHECK/DESCRIBE statements in sidebar (). Add logging during merge tree startup (). Abort instead of `builtin_unreachable` in debug builds (). (). Fix dialect setting description (). Add a test for #658 (). use alias for MergeMutateSelectedEntry share ptr (). Fix LLVM build (). Exclude comments from style-check defined extern (). Update safeExit.cpp (). Disable concurrent parts removal (). Fail fast on empty URL in HDFS (). Add a test for (). Use MultiRead where possible (). Minor cleanups of LLVM integration (). remove useless code (). Early return of corner cases in selectPartsToMutate function (). Refactor the implementation of user-defined functions (). assert unused value in testreplicatedmergetreecompatibility (). Fix Date Interval add/minus over DataTypeDate32 (). Fix log-level in `clickhouse-disks` (). Remove forgotten debug logging (). Fix another trash in zero-copy replication (). go update for diagnostics tool (). Better logging for asynchronous inserts (). Use nfpm packager for archlinux packages (). Bump llvm/clang to 15.0.2 (). Make getResource() independent from the order of the sections (). Smaller threshold for multipart upload part size increase (). Better error message for unsupported delimiters in custom formats (). Fix formatting of `ALTER FREEZE` (). Replace table name in ast fuzzer more often (). Add -15 tools to cmake.tools for GCC build (). Deactivate tasks in ReplicatedMergeTree until startup (). Revert "Revert " (). Update woboq_codebrowser location (). add mdx and jsx to list of doc files (). Remove code browser docs (). Better workaround for emitting .debug_aranges section (). Fix flaky test (). Fix UBSan report in Julian Day functions (). rename filesystemquerycache (). Add convenience typedefs for Date/Date32/DateTime/DateTime64 columns (). Add error "Destination table is myself" to exception list in BC check (). Get current clickhouse version without sending query in BC check (). Fix logical error from welchTTest (). Attempt to fix abort from parallel parsing (). Increase threshold for using physical cores for `max_threads` (). Add a test for (). Add a test for (). Fix aliases (). tests: fix 00705dropcreatemergetree flakiness (). Fix sanitizer reports in integration tests (). Fix `KeeperTCPHandler` data race (). Disable `teststoragenats`, because it's permanently broken (). Better logs in clickhouse-disks (). add libfuzzer and libfuzzernomain to llvm-project build (). Some polishing: replicated merge tree (). Temporarily disable flaky `testreplicatedmergetreehdfszerocopy` (). Adapt internal data structures to 512-bit era (). Fix strange code in date monotonicity (). Clear thread::id when ThreadFromGlobalPool exits. (). ci/stress: fix memory limits overrides for hung check (). tests: avoid model overlap for obfuscator (). Fix possible segfault in expression parser (). Fix incorrect trace log line on dict reload (). Fix flaky 02458datediffdate32 test (). Revert revert 41268 disable s3 parallel write for part moves to disk s3 (). Try to fix data race on zookeeper vs DDLWorker at server shutdown. (). Add a template for installation issues (). Fix typo in cmake code related to fuzzing (). Fix build (). Add .rgignore for test data (). Fix flaky 02457datediffviaunixepoch test ().