Project: ClickHouse (App Definition and Development / Database)
File: v23.6.1.1524-stable.md

sidebar_position: 1 sidebar_label: 2023 Delete feature `donotevictindexandmarkfiles` in the fs cache. This feature was only making things worse. (). Remove ALTER support for experimental LIVE VIEW. (). Add setting `session_timezone`, it is used as default timezone for session when not explicitly specified. (). Added overlay database engine and representation of a directory as a database This commit adds 4 databases: 1. DatabaseOverlay: Implements the IDatabase interface. Allow to combine multiple databases, such as FileSystem and Memory. Internally, it stores a vector with other database pointers and proxies requests to them in turn until it is executed successfully. 2. DatabaseFilesystem: allows to read-only interact with files stored on the file system. Internally, it uses TableFunctionFile to implicitly load file when a user requests the table. Result of TableFunctionFile call cached inside to provide quick access. 3. DatabaseS3: allows to read-only interact with s3 storage. It uses TableFunctionS3 to implicitly load table from s3 4. DatabaseHDFS: allows to interact with hdfs storage. It uses TableFunctionHDFS to implicitly load table from hdfs. (). Add a new setting named `usemysqltypesinshow_columns` to alter the `SHOW COLUMNS` SQL statement to display MySQL equivalent types when a client is connected via the MySQL compatibility port. (). Added option `--renamefilesafter_processing <pattern>`. This closes . (). Add `TableFunctionRedis` 3. Add table engine Redis 4. Add `RedisCommon` which contains Redis related tools and types 5. Support `equals` and `in` filter push down into Redis. (). Allow to skip empty files in file/s3/url/hdfs table functions using settings `s3skipemptyfiles`, `hdfsskipemptyfiles`, `enginefileskipemptyfiles`, `engineurlskipemptyfiles`. (). Clickhouse-client can now be called with a connection instead of "--host", "--port", "--user" etc. (). Codec DEFLATEQPL is now controlled via server setting "enabledeflateqplcodec" (default: false) instead of setting "allowexperimentalcodecs". This marks QPL_DEFLATE non-experimental. (). Improve performance with enabled QueryProfiler using thread-local timer_id instead of global object. (). Rewrite CapnProto input/output format to improve its performance. Map column names and CapnProto fields case insensitive, fix reading/writing of nested structure fields. (). Optimize parquet write performance for parallel threads. (). ### Documentation entry for user-facing changes Disable `parallelizeoutputfrom_storages` for processing MATERIALIZED VIEWs and storages with one block only. (). Merge PR https://github.com/ClickHouse/ClickHouse/pull/46558 (Avoid processing already sorted data). Avoid block permutation during sort if the block is already sorted. (). In the earlier PRs (, ), we used to propose an optimization pattern which transforms the predicates with toYear/toYYYYMM into its equivalent but converter-free form. This transformation could bring significant performance impact to some workloads, such as SSB. However, as issue indicated, these two PRs would introduce some issues which may results in incomplete query results, and as a result, they were reverted by . (). Make multiple list requests to ZooKeeper in parallel to speed up reading from system.zookeeper table. (). Speedup initialization of DateTime lookup tables for time zones. This should reduce startup/connect time of clickhouse client especially in debug build as it is rather heavy. (). Allow to cast IPv6 to IPv4 address for CIDR ::ffff:0:0/96 (IPv4-mapped addresses). (). Update MongoDB protocol to support MongoDB 5.1 version and newer. Support for the versions with the old protocol (<3.6) is preserved. Closes , . (). Improved scheduling of merge selecting and cleanup tasks in `ReplicatedMergeTree`. The tasks will not be executed too frequently when there's nothing to merge or cleanup. Added settings `maxmergeselectingsleepms`, `mergeselectingsleepslowdownfactor`, `maxcleanupdelayperiod` and

It should close . (). Support parallel replicas with the analyzer. (). Add setting `inputformatmaxbytestoreadforschemainference` to limit the number of bytes to read in schema inference. Closes . (). Respect setting inputformatas_default in schema inference. (). Make filter push down through cross join. (). Actual lz4 version is used now. (). Allow to skip trailing empty lines in CSV/TSV/CustomSeparated formats via settings `inputformatcsvskiptrailingemptylines`, `inputformattsvskiptrailingemptylines` and `inputformatcustomskiptrailingemptylines` (disabled by default). Closes . (). Functions "toDateOrDefault|OrNull()" and "accuateCast" now correctly parse numeric arguments. (). Currently, the csv input format can not parse the csv file with whitespace or \t field delimiter, and these delimiters is supported in spark. (). Settings `numberofmutationstodelay` and `numberofmutationstothrow` are enabled by default now with values 500 and 1000 respectively. (). Keeper improvement: add feature flags for Keeper API. Each feature flag can be disabled or enabled by defining it under `keeperserver.featureflags` config. E.g. to enable `CheckNotExists` request, `keeperserver.featureflags.checknotexists` should be set to `1` on Keeper. (). The dashboard correctly shows missing values. This closes . (). CGroups metrics related to CPU are replaced with one metric, `CGroupMaxCPU` for better usability. The `Normalized` CPU usage metrics will be normalized to CGroups limits instead of the total number of CPUs when they are set. This closes . (). Relax the thresholds for "too many parts" to be more modern. Return the backpressure during long-running insert queries. (). Added the possibility to use date and time arguments in syslog timestamp format in functions parseDateTimeBestEffort() and parseDateTime64BestEffort*(). (). Suggest using `APPEND` or `TRUNCATE` for `INTO OUTFILE` when file exists. (). Add embedded keeper-client to standalone keeper binary. (). Command line parameter "--password" in clickhouse-client can now be specified only once. (). Fix data lakes slowness because of synchronous head requests. (Related to Iceberg/Deltalake/Hudi being slow with a lot of files). (). Use `hashofall_files` from `system.parts` to check identity of parts during on-cluster backups. (). The system table zookeeperconnection connectedtime identifies the time when the connection is established (standard format), and sessionuptimeelapsed_seconds is added, which labels the duration of the established connection session (in seconds). (). Show halves of checksums in `system.parts`, `system.projection_parts` and in error messages in the correct order. (). Do not replicate `ALTER PARTITION` queries and mutations through `Replicated` database if it has only one shard and the underlying table is `ReplicatedMergeTree`. (). Improve the progress bar for file/s3/hdfs/url table functions by using chunk size from source data and using incremental total size counting in each thread. Fix the progress bar for Cluster functions. This closes . (). Add totalbytesto_read to Progress packet in TCP protocol for better Progress bar. (). Better checking of data parts on disks with filesystem cache. (). Disable cache setting `donotevictindexandmarkfiles` (Was enabled in `23.5`). (). Fix sometimes not correct currentelementsnum in fs cache. (). Add random sleep before merges/mutations execution to split load more evenly between replicas in case of zero-copy replication. (). The function `transform` as well as `CASE` with value matching started to support all data types. This closes . This closes . This closes . This closes . This closes . (). We have found a bug in LLVM that makes the usage of `compile_expressions` setting unsafe. It is disabled by default. (). Issue reports a core in `grace_hash`

We finally reproduce the exception on local, and found that the issue is related to the failure of creating temporary file. Somehow this is triggered in https://github.com/ClickHouse/ClickHouse/pull/49816 https://github.com/ClickHouse/ClickHouse/pull/49483. (). Update contrib/re2 to 2023-06-02. (). ClickHouse server will print the list of changed settings on fatal errors. This closes . (). In https://github.com/ClickHouse/ClickHouse/pull/51143 the fasstests failed, but the status wasn't created because of the chown `file not found`. This addresses it. Decrease the default values for `http-max-field-value-size` and `httpmaxfieldnamesize` to 128K. (). Update Ubuntu version in docker containers. (). Allow building ClickHouse with clang-17. (). check is considered stable as bugs that were triggered by it are fixed. Now failures of SQLancer check will be reported as failed check status. (). Making our CI even better. (). Split huge `RUN` in Dockerfile into smaller conditional. Install the necessary tools on demand in the same `RUN` layer, and remove them after that. Upgrade the OS only once at the beginning. Use a modern way to check the signed repository. Downgrade the base repo to ubuntu:20.04 to address the issues on older docker versions. Upgrade golang version to address golang vulnerabilities. (). This a follow-up for , the cleanup was lost during refactoring. (). Report loading status for executable dictionaries correctly (). Proper mutation of skip indices and projections (). Cleanup moving parts (). Fix backward compatibility for IP types hashing in aggregate functions (). Fix Log family table return wrong rows count after truncate (). Fix bug in `uniqExact` parallel merging (). Revert recent grace hash join changes (). Query Cache: Try to fix bad cast from ColumnConst to ColumnVector<char8_t> (). Do not read all the columns from right GLOBAL JOIN table. (). Avoid storing logs in Keeper containing unknown operation (). SummingMergeTree support for DateTime64 (). Add compat setting for non-const timezones (). Fix type of LDAP server params hash in cache entry (). Fallback to parsing big integer from String instead of exception in Parquet format (). Fix checking the lock file too often while writing a backup (). Do not apply projection if read-in-order was enabled. (). Fix race azure blob storage iterator (). Fix erroneous `sort_description` propagation in `CreatingSets` (). Fix iceberg V2 optional metadata parsing (). MaterializedMySQL: Keep parentheses for empty table overrides (). Fix crash in BackupCoordinationStageSync::setError() (). Fix subtly broken copy-on-write of ColumnLowCardinality dictionary (). Generate safe IVs (). Fix ineffective query cache for SELECTs with subqueries (). Fix Set index with constant nullable comparison. (). Fix a crash in s3 and s3Cluster functions (). Fix core dump when compile expression (). Fix use-after-free in StorageURL when switching URLs (). Updated check for parameterized view (). Fix multiple writing of same file to backup (). Fix fuzzer failure in ActionsDAG (). Remove garbage from function `transform` (). Fix MSan report in lowerUTF8/upperUTF8 (). fs cache: fix a bit incorrect use_count after (). Fix segfault in MathUnary (). Fix logical assert in `tupleElement()` with default values (). fs cache: remove file from opened file cache immediately when evicting file (). Deprecate delete-on-destroy.txt (). Attempt to increase the general runners' survival rate (). Refactor subqueries for IN (). Test plan optimization analyzer (). Implement endianness-independent serialization for quantileTiming (). require `finalize()` call before d-tor for all writes buffers (). Implement big-endian support for the deterministic reservoir sampler (). Fix compilation error on big-endian platforms

Attach gdb in stateless tests (). JIT infrastructure refactoring (). Analyzer: Do not apply Query Tree optimizations on shards (). Increase max array size in group bitmap (). Misc Annoy index improvements (). Fix reading negative decimals in avro format (). Unify priorities for connection pools (). Prostpone check of outdated parts (). Unify priorities: `IExecutableTask`s (). Disable grace_hash join in stress tests (). ReverseTransform small improvement (). Support OPTIMIZE for temporary tables (). Refactor reading from object storages (). Fix data race in log message of cached buffer (). Add new keywords into projections documentation (). Fix build for aarch64 (temporary disable azure) (). Update version after release (). Update version_date.tsv and changelogs after v23.5.1.3174-stable (). Update CHANGELOG.md (). Update version_date.tsv and changelogs after v23.2.7.32-stable (). Desctructing --> Destructing (). Don't mark a part as broken on `Poco::TimeoutException` (). Rename azureblobstorage to azureBlobStorage (). Fix ParallelReadBuffer seek (). (). Add tests for function "transform" (). Update version_date.tsv and changelogs after v23.5.2.7-stable (). Updated changelog with azureBlobStorage table function & engine entry (). Update easytaskssorted_ru.md (). Document x86 / ARM prerequisites for Docker image (). MaterializedMySQL: Add testnamedcollections (). Update version_date.tsv and changelogs after v22.8.18.31-lts (). Update version_date.tsv and changelogs after v23.3.3.52-lts (). Update version_date.tsv and changelogs after v23.4.3.48-stable (). MaterializedMySQL: Add additional test case to insertwithmodifybinlogchecksum (). Update broken tests list (). Fix LOGICAL_ERROR in snowflakeToDateTime() (). Tests with parallel replicas are no more "always green" (). Slightly more information in error message about cached disk (). do not call finalize after exception (). Update Annoy docs (). A bit safer UserDefinedSQLFunctionVisitor (). Update contribe/orc in .gitmodules (). MaterializedMySQL: Add missing DROP DATABASE for tests (). Fix 'Illegal column timezone' in stress tests (). Fix tests sanity checks and avoid dropping system.query_log table (). Fix tests for throttling by allowing more margin of error for trottling event (). 01746converttypewithdefault: Temporarily disable flaky test (). Fix the statless tests image for old commits (). Fix logic in `AsynchronousBoundedReadBuffer::seek` (). Uncomment flaky test (01746converttypewithdefault) (). Fix keeper-client help message (). fix build issue on clang 15 (). Docs: Fix embedded video link (). Change submodule capnproto to it's fork in ClickHouse (). Attempt to make 01281groupbylimitmemory_tracking not flaky (). Fix flaky 02561nullasdefaultmore_formats (). Fix flaky testseekableformats (). Follow-up to (). Fix a versions' tweak for tagged commits, improve version_helper (). Sqlancer has changed master to main (). Do not spam sqlancer build log (). Refactor IColumn::forEachSubcolumn to make it slightly harder to implement incorrectly (). MaterializedMySQL: Rename materializewithddl.py -> materializedwithddl (). Improve woboq browser report (). Fix for partnamesmutex used after destruction (). Fix ColumnConst::forEachSubcolumn missing from previous PR (). Fix the test 02783parsedatetimebesteffortsyslog flakiness (). Compatibility with clang-17 (). Make more parallel get requests to ZooKeeper in system.zookeeper (). Fix 02703maxlocalwritebandwidth flakiness (). Update version_date.tsv and changelogs after v23.5.3.24-stable (). Update version_date.tsv and changelogs after v23.4.4.16-stable (). Update version_date.tsv and changelogs after v23.3.4.17-lts (). Update version_date.tsv and changelogs after v22.8.19.10-lts (). Fix typo (). Slightly better diagnostics (). Small fix in `MergeTreePrefetchedReadPool` (). Don't report table function accesses to system.errors (). Fix SQLancer branch name (). Revert "Added ability to implicitly use file/hdfs/s3 table functions in clickhouse-local" (). More profile events for fs cache (). Unforget to pass callback to readBigAt() in ParallelReadBuffer (). Update README.md (). Update exception message

Split long test 02149schemainferenceformatswith_schema into several tests to avoid timeout in debug (). Avoid initializing DateLUT from emptyArray function registration (). Suppress check for covered parts in ZooKeeper (). One more profile event for fs cache (). Typo: passowrdsha256hex --> passwordsha256hex (). Introduce settings enum field with auto-generated values list (). Drop session if we fail to get Keeper API version (). Revert "Fix a crash in s3 and s3Cluster functions" (). fix flaky `AsyncLoader` destructor (). Docs: little cleanup of configuration-files.md (). Fix a stupid bug on Replicated database recovery (). FileCache: tryReserve() slight improvement (). Ugly hotfix for "terminate on uncaught exception" in WriteBufferFromOStream (). Avoid too many calls to Poco::Logger::get (). Update version_date.tsv and changelogs after v23.3.5.9-lts (). Better reporting of broken parts (). Update ext-dict-functions.md (). Disable table structure check for secondary queries from Replicated db (). Define Thrift version for parquet and use correct arrow version (). Restore Azure build on ARM (). Query Cache: Un-comment settings in server cfg (). Require more checks (). Fix metadata loading test (). Scratch the strange Python code (). Add a test for (). Add a test for (). Add a test for (). Fix long test `functionsbadarguments` (). Unify merge predicate (). Fix using locks in ProcessList (). Add a test for (). Fix performance tests due to warnings from jemalloc about Per-CPU arena disabled (). Fix "mergetruncatelong" test (). Increase timeout of Fast Test (). Fix bad tests for DNS (). Attempt to fix the `relaxtoomany_parts` test (). Fix MySQL test in Debug mode (). Fix bad test `01018Distributedshardnum` (). Fix "logical error" in addressToLineWithInlines (). Fix test 01280ttlwheregroupby (). Attempt to fix `testsslcert_authentication` (). Revert "Merge pull request from ZhiguoZh/20230607-toyear-fix" (). Two tests are twice longer in average with Analyzer and sometimes failing (). Fix 00899longattachmemorylimit (). Fix test 01293optimizefinal_force (). Fix test 02481parquetlistmonotonicallyincreasing_offsets (). Fix test 02497traceeventsstresslong (). Fix broken labeling for `manual approve` (). Fix parts lifetime in `MergeTreeTransaction` (). Fix flaky test testskipempty_files (). fix flacky test testprofileevents_s3 (). Update README.md (). Replace try/catch logic in hasTokenOrNull() by something more lightweight (). Add retries to `tlsv1_3` tests (). Update exception message (). fs cache: add check for intersecting ranges (). Slightly better code around packets for parallel replicas (). Update system_warnings test (). Many fixes (). Fix test 01605adaptivegranularityblockborders (). Try fix flaky 02497storagefilereaderselection (). Try making Keeper in `DatabaseReplicated` tests more stable (). Convert 02003memorylimitinclient from expect to sh test (to fix flakiness) (). Fix testdiskoverwebserver (). Delay shutdown of system and temporary databases (). Fix memory leakage in CompressionCodecDeflateQpl (). Increase retries in testmultipledisks/test.py::teststartstop_moves (). Fix race in BoundedReadBuffer (). Fix flaky unit test (). Fix flaky test `testhostregexpmultipleptr_records` (). Add a comment (). Make `testsslcertauthentication` similar to `testtlvs1_3` (). Fix duplicate storage set logical error. (). Update teststoragepostgresql/test.py::testconcurrentqueries (). Fix FATAL: query context is not detached from thread group (). Update version_date.tsv and changelogs after v23.3.6.7-lts (). Decoupled commits from for backports (). Try to fix deadlock in ZooKeeper client (). Retry chroot creation in ZK before stateless tests (). use timeout instead trap in 01443mergetruncate_long.sh (). Update version_date.tsv and changelogs after v23.5.4.25-stable (). Fix MergeTreeMarksLoader segfaulting if marks file is longer than expected (). Update version_date.tsv and changelogs after v23.4.5.22-stable (). Update version_date.tsv and changelogs after v23.3.7.5-lts (). Update parts.md ().