Project: ClickHouse (App Definition and Development / Database)
File: v22.11.1.1360-stable.md

sidebar_position: 1 sidebar_label: 2022 JSONExtract family of functions will now attempt to coerce to the request type. (). - Add function `displayName`, closes . (). Added applied row-level policies to `system.query_log`. (). Add Hudi and DeltaLake table engines, read-only, only for tables on S3. (). Add 4LW command `csnp` for manually creating snapshots. Additionally, `lgif` was added to get Raft information for a specific node (e.g. index of last created snapshot, last committed log index). (). Support for keeper request retries during insert into replicated merge trees. Apart from fault tolerance, it aims to provide better user experience, - avoid returning a user an error during insert if keeper is restarted (for example, due to upgrade). (). Add function ascii like in spark: https://spark.apache.org/docs/latest/api/sql/#ascii. (). Add function pmod which return non-negative result based on modulo. (). Published function `formatReadableDecimalSize`. (). Added S3 PUTs and GETs request per second rate throttling. Settings `s3maxgetrps`, `s3maxgetburst`, `s3maxputrps`, `s3maxputburst` are used to configure token bucket throttler. Can be used with both S3 ObjectStorage and S3 table function. Different limits can be configured for different S3 disks or endpoints. (). Add table function hudi and deltaLake. (). Add function factorial, as in Impala or Spark. (). Add function randCanonical, which is similar to rand function in spark or impala. The function generates pseudo random results with independent and identically distributed uniformly distributed values in (). Currently, the only saturable operators are And and Or, and their code paths are affected by this change. (). `match` function can use the index if it's a condition on string prefix. This closes . (). Fixed slowness in JSONExtract with LowCardinality(String) tuples. (). Support parallel parsing for LineAsString input format. This improves performance just slightly. This closes . (). Keeper performance improvement: improve commit performance for cases when many different nodes have uncommitted states. This should help with cases when a follower node can't sync fast enough. (). Parallelized merging of `uniqExact` states for aggregation without a key, i.e. queries like `SELECT uniqExact(number) FROM table`. The improvement becomes noticeable when the number of unique keys approaches 10^6. Also `uniq` performance is slightly optimized. This closes . (). Support type `Object` inside other types, e.g. `Array(JSON)`. (). Remove covered parts for fetched part (to avoid possible replication delay grows). (). ClickHouse Client and ClickHouse Local will show progress by default even in non-interactive mode. If `/dev/tty` is available, the progress will be rendered directly to the terminal, without writing to stderr. It allows to get progress even if stderr is redirected to a file, and the file will not be polluted by terminal escape sequences. The progress can be disabled by `--progress false`. This closes . (). Add, subtract and negate operations are now available on Intervals. In case when the types of Intervals are different they will be transformed into the Tuple of those types. 2. A tuple of intervals can be added to or subtracted from a Date/DateTime field. 3. Added parsing of Intervals with different types, for example: `INTERVAL '1 HOUR 1 MINUTE 1 SECOND'`.

- Add `notLike` to key condition atom map, so condition like `NOT LIKE 'prefix%'` can use primary index. (). Add support for FixedString input to base64 coding functions. (). Add columns `bytesondisk` and `path` to `system.detached_parts`. Closes . (). Improve using structure from insertion table in table functions, now setting `usestructurefrominsertiontableintable_functions` has new possible value - `2` that means that ClickHouse will try to determine if we can use structure from insertion table or not automatically. Closes . (). Added * glob support for recursive directory traversal to filesystem and S3. resolves . (). Mask passwords and secret keys both in `system.query_log` and `/var/log/clickhouse-server/.log` and also in error messages. (). Add a new variable call `limit` in query_info, indicating whether this query is a limit-trivial query. If so, we will adjust the approximate total rows for later estimation. Closes . (). Implement `ATTACH` of `MergeTree` table for `s3plain` disk (plus some fixes for `s3plain`). (). Fix no progress indication on INSERT FROM INFILE. Closes . (). Add `minagetoforcemergeonpartition_only` setting to optimize old parts for the entire partition only. (). Throttling algorithm changed to token bucket. (). Refactor FunctionTokens to enable max tokens returned for related functions(default disabled). (). Added new field allow_readonly in system.table_functions to allow using table functions in readonly mode resolves Implementation: Added a new field allowreadonly to table system.tablefunctions. Updated to use new field allow_readonly to allow using table functions in readonly mode. Testing: Added a test for filesystem tests/queries/0stateless/02473functionsinreadonly_mode.sh Documentation: * Updated the english documentation for Table Functions. (). Allow to use Date32 arguments for formatDateTime and FROM_UNIXTIME functions. (). Update tzdata to 2022f. Mexico will no longer observe DST except near the US border: https://www.timeanddate.com/news/time/mexico-abolishes-dst-2022.html. Chihuahua moves to year-round UTC-6 on 2022-10-30. Fiji no longer observes DST. See https://github.com/google/cctz/pull/235 and https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/1995209. (). Add `FailedAsyncInsertQuery` event metric for async inserts. (). Implement `read-in-order` optimization on top of query plan. It is enabled by default. Set `queryplanreadinorder = 0` to use previous AST-based version. (). Increase the size of upload part exponentially for backup to S3. (). When the merge task is continuously busy and the disk space is insufficient, the completely expired parts cannot be selected and dropped, resulting in insufficient disk space. My idea is that when the entire Part expires, there is no need for additional disk space to guarantee, ensure the normal execution of TTL. (). bugfix ignore Mysql binlog SAVEPOINT event. (). Add support for interactive parameters in INSERT VALUES queries. (). Add generic implementation for arbitrary structured named collections, access type and system.named_collections. (). add oss function and StorageOSS (This is convenient for users). oss is fully compatible with s3. (). Improve error reporting in the collection of OS-related info for the `system.asynchronous_metrics` table. (). The `system.asynchronous_metrics` gets embedded documentation. This documentation is also exported to Prometheus. Fixed an error with the metrics about `cache` disks - they were calculated only for one arbitrary cache disk instead all of them. This closes . (). Modify the `INFORMATION_SCHEMA` tables in a way so that now ClickHouse can connect to itself using the MySQL compatibility protocol. Add columns instead of aliases (related to ). It will improve the compatibility with various MySQL clients. (). Disable `deltaLake` and `hudi` table functions in readonly

(). Updated normaliser to clone the alias ast. resolves Implementation: Updated QueryNormalizer to clone alias ast, when its replaced. Previously just assigning the same leads to exception in LogicalExpressinsOptimizer as it would be the same parent being inserted again. * This bug is not seen with new analyser (allowexperimentalanalyzer), so no changes for it. I added a test for the same. (). Fix race for backup of tables in Lazy databases. (). fix skipunavailableshards does not work using s3Cluster table function. (). Run SQLancer for each pull request and commit to master. is an OpenSource fuzzer that focuses on automatic detection of logical bugs. (). Update to latest zlib-ng. (). use llvm `l64.lld` in macOS suppress ld warnings, close . (). Add support for testing ClickHouse server with Jepsen. By the way, we already have support for testing ClickHouse Keeper with Jepsen. This pull request extends it to Replicated tables. (). Improve bugfix validation check: fix bug with skipping the check, port separate status in CI, run after check labels and style check. Close . (). Wait for all files are in sync before archiving them in integration tests. (). Use https://github.com/matus-chochlik/ctcache for clang-tidy results caching. (). Before the fix, the user-defined config was preserved by RPM in `$file.rpmsave`. The PR fixes it and won't replace the user's files from packages. (). Add a CI step to mark commits as ready for release; soft-forbid launching a release script from branches but master. (). Fix schema inference in s3Cluster and improve in hdfsCluster. (). Fix retries while reading from http table engines / table function. (retrtiable errors could be retries more times than needed, non-retrialble errors resulted in failed assertion in code). (). A segmentation fault related to DNS & c-ares has been reported. The below error ocurred in multiple threads: ``` 2022-09-28 15:41:19.008,2022.09.28 15:41:19.008088 (). Fix `LOGICAL_ERROR` `Arguments of 'plus' have incorrect data types` which may happen in PK analysis (monotonicity check). Fix invalid PK analysis for monotonic binary functions with first constant argument. (). Fix incorrect key analysis when key types cannot be inside Nullable. This fixes . (). Fix typo in setting name that led to bad usage of schema inference cache while using setting `inputformatcsvusebesteffortinschemainference`. Closes . (). Fix create Set with wrong header when data type is LowCardinality. Closes . (). `(U)Int128` and `(U)Int256` values are correctly checked in `PREWHERE`. (). Fix a bug in ParserFunction that could have led to a segmentation fault. (). Fix truncate table does not hold lock correctly. (). Fix possible SIGSEGV for web disks when file does not exist (or `OPTIMIZE TABLE FINAL`, that also can got the same error eventually). (). Fix `authtype` mapping in `system.sessionlog`, by including `SSL_CERTIFICATE` for the enum values. (). Fix stack-use-after-return under ASAN build in ParserCreateUserQuery. (). Fix lowerUTF8()/upperUTF8() in case of symbol was in between 16-byte boundary (very frequent case of you have strings > 16 bytes long). (). Additional bound check was added to lz4 decompression routine to fix misbehaviour in case of malformed input. (). Fix rare possible hung on query cancellation. (). Fix incorrect savedblocksample with multiple disjuncts in hash join, close .

A null pointer will be generated when select if as from three table join , For example, the SQL:. (). Fix memory sanitizer report in ClusterDiscovery, close . (). Fix datetime schema inference in case of empty string. (). Fix rare NOTFOUNDCOLUMNINBLOCK error when projection is possible to use but there is no projection available. This fixes . The bug was introduced in https://github.com/ClickHouse/ClickHouse/pull/25563. (). Fixes for s3_plain disk that will allow to attach Wide parts. (). Fix ATTACH TABLE in PostgreSQL database engine if the table contains DATETIME data type. Closes . (). Fix lambda parsing. Closes . (). Handle (ignore) SAVEPOINT queries in MaterializedMySQL. (). Fix incorrect key analysis when nullable keys appear in the middle of a hyperrectangle. This fixes . (). - Fix several buffer over-reads. (). Fix function if in case of NULL and const Nullable arguments. Closes . (). Fix decimal math overflow in parsing datetime with 'best effort' algorithm. Closes . (). The `indent` field produced by the `git-import` tool was miscalculated. See https://clickhouse.com/docs/en/getting-started/example-datasets/github/. (). Fixed unexpected behaviour of Interval types with subquery and casting. (). Fix logical error in `sumMap/minMap/maxMap` functions executing `TOTALS/ROLLUP/CUBE` on `NULL` values. Close . (). - Fix ubsan in AggregateFunctionMinMaxAny::read with high sizes. (). Fix IS (NOT) NULL operator priority in regard to other operators. (). ... Add support for format ipv6 on s390x. (). NO CL ENTRY: 'Revert "Sonar Cloud Workflow"'. (). NO CL ENTRY: 'Revert " Keeper retries during insert (clean)"'. (). NO CL ENTRY: 'Revert "Revert " Keeper retries during insert (clean)""'. (). NO CL ENTRY: 'Revert "Optimize TTL merge, completely expired parts can be removed in time"'. (). NO CL ENTRY: 'Revert "Randomize keeper fault injection settings in stress tests"'. (). NO CL ENTRY: 'Revert "S3 request per second rate throttling"'. (). Better logging for docs builder (). Save full server log in AST Fuzzer checks (). Build with libcxx(abi) 15 (). Sonar Cloud Workflow (). Invalid type in where for Merge table (logical error) (). Fix frequent memory drift message and clarify things in comments (). Add functions for PowerBI connect (). Try to save `IDataPartStorage` interface (). Remove Ubuntu cruft (). Analyzer change setting into allowexperimentalanalyzer (). Analyzer IQueryTreeNode remove getName method (). Minor fix iotest_nonblock build (). Add tests and doc for some url-related functions (). Update version_date.tsv and changelogs after v22.10.1.1875-stable (). Fix error handling in clickhouse_helper.py (). Fix execution of version_helper.py to use git tweaks (). MergeTree indexes use RPNBuilderTree (). Always run `BuilderReport` and `BuilderSpecialReport` in all CI types (). Support optimizesyntaxfuse_functions for sum/count/avg via analyzer (). Update version after release (). Update version_date.tsv and changelogs after v22.10.1.1877-stable (). OrderByLimitByDuplicateEliminationPass improve performance (). Analyzer improve subqueries representation (). Update version_date.tsv and changelogs after v22.9.4.32-stable (). Update version_date.tsv and changelogs after v22.8.7.34-lts (). Update version_date.tsv and changelogs after v22.7.7.24-stable (). Move SonarCloud Job to nightly (). Update version_date.tsv and changelogs after v22.8.8.3-lts (). Minor fix implicit cast CaresPTRResolver (). Fix build on master (). Update version_date.tsv and changelogs after v22.3.14.18-lts (). Fix anchor links (). Update version_date.tsv and changelogs after v22.3.14.23-lts (). Update README.md (). Slightly better code with projections

Fix some races in MergeTree (). Fix typo in comments (). Fix compilation of LLVM with cmake cache (). Fix link in docs (). Link to proper place in docs (). Fix argument type check in AggregateFunctionAnalysisOfVariance (). Tests/lambda analyzer (). Fix Missing Quotes - Sonar Nightly (). Add exclusions from the Snyk scan (). Fix Missing Env Vars - Sonar Nightly (). Fix typo (). Add timezone to 02458datediffdate32 (). Adjust cancel and rerun workflow names to the actual (). Analyzer subquery in JOIN TREE with aggregation (). Fix getauxval for sanitizer builds (). Update version_date.tsv and changelogs after v22.10.2.11-stable (). Better usability for dashboard.html on changes (). Some fixes for ReplicatedMergeTree (). Validate Query Tree in debug (). changed type name for s3 plain storage (). Cleanup implementation of regexpReplace(All|One) (). Do not show status for Bugfix validate check in non bugfix PRs (). fix(typo): Passible -> Possible (). Pin the cryptography version to not break lambdas (). Fix: bad cast from type DB::ColumnLowCardinality to DB::ColumnString (). Attach thread pool for loading parts to the query (). Fix macOS M1 builds due to sprintf deprecation (). Less use of CH-specific bit_cast() (). Remove some utils (). Fix a bug in CAST function parser (). Fix old bug to remove `refs/head` from ref name (). Add debug information to nightly builds (). Add some guard rails around aggregation memory management (). Add `on: workflow_call` to debug CI (). Analyzer added identifier typo corrections (). Simple fixes for restart replica description (). Cleanup match code (). Fix TSan errors (correctly ignore _exit interception) (). fix bandwidth throttlers initialization order (). Add test for issue (). Analyzer improve ARRAY JOIN with JOIN (). Fix projection part removal with zero-copy replication (). Fix msan warning (). Analyzer AST key condition crash fix (). Better logging for mark range filtering on projection parts (). Fix ub type punning (). Analyzer improve aliases support for table expressions (). Throw not implemented for window frame type 'groups' in analyzer (). Disable clickhouse local and client non-interactive progress by default. (). Make error message after dropping current user more correct. (). More stable test (). Rewrite tests for memory overcommit (). Fix trailing \n from SQLancer status (). Fix `testkeeperfourwordcommand::testcmdstat` (). Enable keeper fault injection for inserts in functional tests (). Analyzer aggregation crash fix (). Analyzer aggregation totals crash fix (). Improve commitstatushelper.py (). Skip hash logging on sanitizer builds (). Analyzer improve JOIN with constants (). Remove POCOCLICKHOUSEPATCH (). Update CompressionCodecDeflateQpl.cpp (). Randomize keeper fault injection settings in stress tests (). Fix for missing columns bug with projections an ALTER UPDATE (). A workaround for LLVM bug, https://github.com/llvm/llvm-project/issues/58633 (). Stop `ConfigReloader` first to avoid data race (). Fix typo (). Miscellaneous changes (). Fix flaky 02449checkdependenciesandtable_shutdown (). Add test to check for all builds (). Don't throw if shared ID already created in `StorageReplicatedMergeTree` (). Fix nullptr dereference in collectScopeValidIdentifiersForTypoCorrection (). Better message in waitzookeeperto_start (). Make testglobalovercommit_tracker non-parallel (). Rename canonicalRand to randCanonical (). check limits for an AST in select parser fuzzer (). Allow autoremoval of old parts if detachnotbyteidenticalparts enabled (). `pmod`: compatibility with Spark, better documentation ().