Project: ClickHouse (App Definition and Development / Database)
File: v22.3.1.1262-prestable.md

sidebar_position: 1 sidebar_label: 2022 Improvement the toDatetime function overflows. When the date string is very large, it will be converted to 1970. (). Make arrayCompact behave as other higher-order functions: perform compaction not of lambda function results but on original array. If you using nontrivial lambda functions in arrayCompact you may restore old behaviour by wrapping arrayCompact arguments into arrayMap. Closes . (). New data type `Object(<schema_format>)`, which supports storing of semi-structured data (for now JSON only). Data is written to such types as string. Then all paths are extracted according to format of semi-structured data and written as separate columns in most optimal types, that can store all their values. Those columns can be queried by names that match paths in source data. E.g `data.key1.key2` or with cast operator `data.key1.key2::Int64`. (). Support authentication of users connected via SSL by their X.509 certificate. (). related to issue: . Add three functions for map data type: 1. mapReplace(map1, map2) - replaces values for keys in map1 with the values of the corresponding keys in map2; adds keys from map2 that don't exist in map1. 2. mapFilter 3. mapMap mapFilter and mapMap are higher order functions , accept two arguments, first argument is a lambda function with k, v pair , the second argument is a map type column. (). Add local cache for disk s3. Closes . (). - Add startsWith & endsWith function for arrays, closes . (). Implement DateTime64 transform from and to arrow column, which closes and closes . (). Add cpu/mem metric for clickhouse-local. Close . (). Support schema inference for inserting into table functions file/hdfs/s3/url. (). A new settings called <allowplaintextpassword><allownopassword> is added in server configuration which on/off insecure AUTHTYPE plaintext-password and nopassword. By default the property is set to true which means authType Plaintextpassword & NOpassword is allowed. (). Add new table function `hive`, usage as follow ``` hive('<hive metastore url>', '<hive database>', '<hive table name>', '<columns definition>', '<partition columns>') ``` for example ``` SELECT FROM hive('thrift://hivetest:9083', 'test', 'demo', '`id` Nullable(String), `score` Nullable(Int32), `day` Nullable(String)', 'day') ```. (). - When use clickhouse-client logining, If user and password is not specified in command line or configuration file, get them from `CLICKHOUSEUSER` and `CLICKHOUSEPASSWORD` environment variables. Close . (). Added datetimeinputformat = 'besteffort_us'. Closes . (). Changed the Play UI to select a theme by the following priority: 'theme' GET parameter 'theme' in localStorage According to OS preference (didn't work before). (). ``` sql ) explain ast graph = 1 select from system.parts;. (). Add `databasereplicatedallowonlyreplicated_engine` setting. When enabled, it only allowed to create `Replicated` tables in `Replicated` database. (). Calling std::distance on large list will decrease performance, use a version in Node to substitute with more memory 80MB/1000w Nodes. The logic is: Every node is list has a version with type sizet, setting in insert or insertOrReplace method using the class member variable currentversion. The version is only increase at enableSnapshot method. When doing a snapshot, call snapshotSizeWithVersion to get snapshot size and

When traversing the list, if node version is less then or equal to snapshotuptoversion, then protects it from deleting if node version is bigger than snapshotupto_version, we can do anything to it. (). Compaction of log store in Nuraft need acquire an inner lock which also used in normal commit process, so we delete useless logs in `compact` method of Changelog class in a background thread. See details on: https://github.com/ClickHouse-Extras/NuRaft/blob/1707a7572aa66ec5d0a2dbe2bf5effa3352e6b2d/src/handle_commit.cxx#L560. (). Don't hold the latest snapshot in memory, instead, reading the snapshot if needed, sequence reading is fast to 200+MBps even on HDD using mmap system call. Writing snapshot data directly to disk using compression method without holding original data and compressed data in memory. (). MergeTree improve insert performance replacing std::stable_sort with pdqsort. (). Improve the performance of the `ANY` aggregation function by acting over batches. (). Improve performance of `detectCharset `, `detectLanguageUnknown ` functions. Improve performance of `DirectDictionary` if dictionary source is `ClickHouse`. Improve performance of processing queries with large `IN` section. (). Less lock on connection using atomic stat. Notice that it is an approximate stat. (). Make the znode ctime and mtime consistent between servers. (). Hold time lock while assigning tasks to clear old temporary directories in StorageMergeTree. (). When large files were written with `s3` table function or table engine, the content type on the files was mistakenly set to `application/xml` due to a bug in the AWS SDK. This closes . (). Improve schema inference with globs in FIle/S3/HDFS/URL engines. Try to use the next path for schema inference in case of error. (). - Improve the opentelemetry span logs for INSERT operation on distributed table. (). MaterializedMySQL support materializedmysqltables_list(a comma-separated list of mysql database tables, which will be replicated by MaterializedMySQL database engine. Default value: empty list means whole tables will be replicated) settings, mentioned at . (). This PR changes restrictive row policies a bit to make them an easier alternative to permissive policies in easy cases. If for a particular table only restrictive policies exist (without permissive policies) users will be able to see some rows. Also `SHOW CREATE ROW POLICY` will always show `AS permissive` or `AS restrictive` in row policy's definition. (). Add `encodeURLComponent`, 'encodeURLFormComponent' function. Closes . (). Now you can read `system.zookeeper` table without restrictions on path or using `like` expression. This reads can generate quite heavy load for zookeeper so to enable this ability you have to enable setting `allowunrestrictedreadsfromkeeper`. (). Some refactoring and improvement over async and remote buffer related stuff. Separated in each commit. (). ExecutableUserDefinedFunctions allow to specify argument names. This is necessary for formats where argument name is part of serialization, like `Native`, `JSONEachRow`. Closes . (). Extract schema only once on table creation and prevent reading from local files/external sources to extract schema on each server startup. (). Do not reset logging that configured via --log-file/--errorlog-file in case of empty logger.log/logger.errorlog. (). Support `remote()`/`cluster()` for `paralleldistributedinsert_select=2`. (). Add name hints for data skipping indices. Closes . (). Now `ALTER TABLE DROP COLUMN columnX` queries for `MergeTree` table engines will work instantly when `columnX` is `ALIAS` column. Fixes .

In previous versions the progress bar in clickhouse-client can jump forward near 50% for no reason. This closes . (). Fix reading only columns which user asked for. Closes . (). Implement MemoryStatisticsOS for FreeBSD. (). Allow to open empty sqlite db file if it does not exist. Closes . (). Allow LowCardinality strings for ngrambfv1/tokenbfv1 indexes. Closes . (). Ignore per-column `TTL` in `CREATE TABLE AS` if new table engine does not support it (i.e. if the engine is not of `MergeTree` family). (). Use connection pool for hive metastore client. (). Currently, if the user changes the settings of the system tables there will be tons of logs and ClickHouse will rename the tables every minute. This fixes . (). remove unnecessary columns for reading parquet/orc files. (). For random access readbuffer in hive, the first time to read the readbuffer would use the original readbuffer instead of local file. When we read a parquet/orc format file, the readbuffer seeks to the end of the file, which will be blocked until the local file finishes download, and make the whold process slow. (). Add more sanity checks for keeper configuration: now mixing of localhost and non-local servers is not allowed, also add checks for same value of internal raft port and keeper client port. (). Functions `dictGetHierarchy`, `dictIsIn`, `dictGetChildren`, `dictGetDescendants` support implicit key cast and constant arguments. Closes . (). Avoid division by zero in Query Profiler if Linux kernel has a bug. Closes . (). Avoid possible `MEMORYLIMITEXCEEDED` during `INSERT` into `Buffer` with `AggregateFunction`. (). Support `view()` for `paralleldistributedinsert_select`. (). Add setting to lower column case when reading parquet/ORC file. (). Do not retry non-rertiable errors. Closes . (). Added diskname to system.partlog. (). Currently, ClickHouse validates hosts defined under <remoteurlallow_hosts> for URL and Remote Table functions. This PR extends the RemoteHostFilter to Mysql and PostgreSQL table functions. (). Sometimes it is not enough for us to distinguish queries hierachy only by isinitialquery in system.querylog and system.processes. So distributeddepth is needed. (). Support test mode for clickhouse-local. (). Return const for function getMacro if not in distributed query. Close . (). Reload `remoteurlallow_hosts` after config update. (). Ignore obsolete grants in ATTACH GRANT statements. This PR fixes . (). When the inner readbuffer's buffer size is too small, NEEDMOREINPUT in `HadoopSnappyDecoder` will run multi times (>=3)for one compressed block. This makes the input data be copied into the wrong place in `HadoopSnappyDecoder::buffer`. (). Randomize some settings in functional tests. This closes . (). NA. (). Debian package clickhouse-test.deb removed completely. CI use tests from repository and standalone testing via deb package is no longer supported. (). Set timeout 40 minutes for fast tests. (). Drop PVS test from CI. (). Limit DWARF version for debug info by 4 max, because our internal stack symbolizer cannot parse DWARF version 5. This makes sense if you compile ClickHouse with clang-15. (). Improve CI scripts arguments. (). Use @robot-clickhouse as an author and committer for PRs like https://github.com/ClickHouse/ClickHouse/pull/34685. (). Separate smaller clickhouse-keeper build. (). Clion has the following problems "The breakpoint will not currently be

No executable code is associated with this line". (). Add an ability to build stripped binaries with cmake. (). Fix distributed subquery maxquerysize limitation inconsistency. (). Fix incorrect trivial count result when part movement feature is used . (). Stop to select part for mutate when the other replica has already updated the /log for ReplatedMergeTree engine. (). Fix `allowexperimentalprojectionoptimization` with `enableglobalwithstatement` (before it may lead to `Stack size too large` error in case of multiple expressions in `WITH` clause, and also it executes scalar subqueries again and again, so not it will be more optimal). (). Fix serialization/printing for system queries `RELOAD MODEL`, `RELOAD FUNCTION`, `RESTART DISK` when used `ON CLUSTER`. Closes . (). Fix ENOENT with fsyncpartdirectory and Vertical merge. (). Fix bug for h3 funcs containing const columns which cause queries to fail. (). Fix possible failures in S2 functions when queries contain const columns. (). Fix bugs for multiple columns group by in WindowView. (). Support DDLs like CREATE USER to be executed on cross replicated cluster. (). Fix asynchronous inserts to table functions. Fixes . (). Fix possible "Part directory doesn't exist" during `INSERT`. (). Fix postgres datetime64 conversion. Closes . (). Avoid busy polling in keeper while searching for changelog files to delete. (). Unexpected result when use `in` in `where` in hive query. (). Fix wrong schema inference for unquoted dates in CSV. Closes . (). Fix possible rare error `Cannot push block to port which already has data`. Avoid pushing to port with data inside `DelayedSource`. (). Fix possible segfault in filelog. Closes . (). Fix unexpected result when use -state type aggregate function in window frame. (). Fix possible exception `Reading for MergeTree family tables must be done with last position boundary`. Closes . (). Fix reading from `system.asynchronous_inserts` table if there exists asynchronous insert into table function. (). Fix missing alias after function is optimized to subcolumn when setting `optimizefunctionsto_subcolumns` is enabled. Closes . (). Avoid possible deadlock on server shutdown. (). Fixed the "updatelag" external dictionary configuration option being unusable with the error message ``Unexpected key `updatelag` in dictionary source configuration``. (). fix issue: . (). Fix `optimizeskipunusedshardsrewrite_in` for signed columns and negative values. (). Fixed the incorrect translation YAML config to XML. (). Fix partition pruning error when non-monotonic function is used with IN operator. This fixes . (). Fix materialised postrgesql adding new table to replication (ATTACH TABLE) after manually removing (DETACH TABLE). Closes . Closes . Closes . (). Fix materialised postgres `table overrides` for partition by, etc. Closes . (). Schema inference didn't work properly on case of `INSERT INTO FUNCTION s3(...) FROM ...`, it tried to read schema from s3 file instead of from select query. (). - Fix `replaceRegexpAll`, close . (). Fix error in query with `WITH TOTALS` in case if `HAVING` returned empty result. This fixes . (). Fix reading port from config, close . (). Make function `cast(value, 'IPv4')`, `cast(value, 'IPv6')` behave same as `toIPv4`, `toIPv6`

Changed behavior of incorrect IP address passed into functions `toIPv4`,` toIPv6`, now if invalid IP address passes into this functions exception will be raised, before this function return default value. Added functions `IPv4StringToNumOrDefault`, `IPv4StringToNumOrNull`, `IPv6StringToNumOrDefault`, `IPv6StringOrNull` `toIPv4OrDefault`, `toIPv4OrNull`, `toIPv6OrDefault`, `toIPv6OrNull`. Functions `IPv4StringToNumOrDefault `, `toIPv4OrDefault `, `toIPv6OrDefault ` should be used if previous logic relied on `IPv4StringToNum`, `toIPv4`, `toIPv6` returning default value for invalid address. Added setting `castipv4ipv6defaultonconversionerror`, if this setting enabled, then IP address conversion functions will behave as before. Closes . Closes . Closes . (). Wait for IDiskRemote thread pool properly. (). Fix `CHECK TABLE` query in case when sparse columns are enabled in table. (). Fix possible Abort while using Brotli compression with a small `maxreadbuffer_size` setting value. The bug was found in https://github.com/ClickHouse/ClickHouse/pull/35047. (). Fix possible segfault in JSONEachRow schema inference. (). Fix possible `Assertion 'position() != workingbuffer.end()' failed` while using lzma compression with small `maxreadbuffersize` setting value. The bug was found in https://github.com/ClickHouse/ClickHouse/pull/35047. (). Fix possible segfault while using lz4 compression with a small maxreadbuffer_size setting value. The bug was found in https://github.com/ClickHouse/ClickHouse/pull/35047. (). Fix possible `Assertion 'position() != workingbuffer.end()' failed` while using bzip2 compression with small `maxreadbuffersize` setting value. The bug was found in https://github.com/ClickHouse/ClickHouse/pull/35047. (). - Fix partial merge join duplicate rows bug, close . (). Fix segfault in Postgres database when getting create table query if database was created using named collections. Closes . (). Fix bug in S3 zero-copy replication which can lead to errors like `Found parts with the same min block and with the same max block as the missing part` after concurrent fetch/drop table. (). NO CL ENTRY: ' (). NO CL ENTRY: 'Revert "Allow restrictive row policies without permissive"'. (). NO CL ENTRY: 'Revert "Remove "bugs" that do not exist anymore"'. (). NO CL ENTRY: 'Revert "Change timezone in Docker"'. (). NO CL ENTRY: 'Revert "Fix 00900longparquet_load"'. (). (). Remove useless setting experimentalquerydeduplicationsendallpartuuids (). Apply joinusenulls on types before join (). Adjust 01681argminmaxif_fix (). Check overflow in addSeconds/Minues/etc functions (). Update version after release (). Fix `positionUTF8` on aarch64 (). Update version_date.tsv after v22.2.2.1-stable (). Fix ZooKepper paths in zerocopyschema_converter.py (). Changelog for version 22.2 (). Follow up to PR (). Always update ProfileEvents (even on exceptions) (). Function encodeURLComponent minor fixes (). CMake llvm add status message if not used (). Adding noexcept for move constructor SharedContextHolder (). Add new team members to company page (). Fix flaky 00502custompartitioning_local (). Workaround for a bug in NuRaft library (). Fix possible memory_tracker use-after-free (for async s3 writes) for merges/mutations (). Add taiyang-li to trusted contributors (). rename name in ci check (). Performance tests fix H3 (). Enable async writes to S3. (). Add a patch release for stable/lts packages to release.py (). Print only total profile events for --profile-events-delay-ms=-1 (). Fix `parallelreadingfrom_replicas` with `clickhouse-bechmark` (). Submodules are mandatory (). Changelog for version 22.2 (). Sort additionalhtmlurls in test report in CI (). Fix undefined pthreadmutexlock/unlock for glibc 2.34+/DISABLEHERMETICBUILD (). jemalloc: fix includes order to avoid overlaps (). Fix using host/port from config for client (). Update links for 22.2 and 22.3 release webinars (). Add opensee blog post (). Try remove trash from StorageFileLog

Fix generating system.build_options (). Comment output header for team keys (). Fix untuple condition in IN function (). Improve certificate reloader (). Update DiskS3.cpp (). Clang tidy improvements (). Update version_date.tsv after v22.2.3.5-stable (). Add check_prerequisites to release.py (). Fix filelog storage data paths (). Update version_date.tsv after v22.1.4.30-stable (). Fix alignment for aligned_alloc() to fix build for glibc2.35+ (). Stop processing multiqueries in clickhouse-client/local on SIGINT (). Fix some code comments style (). Adjust couple of tests for aarch64 (). Add benchmarks (). Catch exception in ~WriteBufferFromS3 (). Disable memory checking test with thread sanitizer (). Print number of available logical cores in configuration (). Fix possible "nmalloc >= ndalloc" jemalloc assertion (). Add some benchmarks (). Fix typo (). Fix signal-unsafe TSan report in client (). Fix non-MergeTree engines for system._log (). Fix teststoragepostgresql::testconcurrentqueries (). Fix race between INSERT asyncinsert=1 and system.asynchronousinserts (). Add option to clickhouse-test to skip aarch64 build (). clickhouse obfuscator aarch64 fix (). Fix tutorial link (). ExecutableUDF function deterministic in scope of query fix (). fasttest: add timings for "Test time" column in the results table (). Always write preprocessed config in XML format (). Fix read old records from logs (). Fix trim function (). Skip 01086odbcroundtrip for aarch, disable force_tests (). Remove useless define (). Take flush_time into account for scheduling background flush of the Buffer (). Try to fix failed tests (). Do not hide exceptions during mutations (). Put downloaded artifacts to a temorary path (). Get rid of duplicate query planing. (). Temporary supress move partition long for storage S3 (). Fix inconsistency in DiskLocal (). Try to build llvm for Aarch64 (). fix build fail with gcc (). Fix hardcoded page size (). Delete nodes with attributes `remove` or `replace` if they didn't merge (). Upgrade icu to icu-release-70-1 (). Merging (). fix typos (). Adjust maxmemoryusage in external_aggregation.sql (). MaterializedMySQL protocol example fix (). Remove useless files (). Remove useless file (). Add Segment script to <head> of docs pages (). Cache fix (). Use C++14 aliases for some type traits (). Remove redundant configs for TestFlows (). Correct some integration tests (). Change timezone in Docker (). Change timezone example in server config (). Adjust timezone in performance tests (). Remove "bugs" that do not exist anymore (). Change timezone in stateful tests. (). Moved submodules from ClickHouse-Extras to ClickHouse (). Remove "bugs" that do not exist anymore (). Update CachedReadBufferFromRemoteFS.cpp (). Add a test for (). Enable clang tidy for header files (). Fix possible 01506buffertablealterblockstructure2 flakiness (). Mark build action as failed if it was retried and there was no succeeded builds (). Fix flaky test (). Fix logical error in remote fs cache (). Fix code style and other minor corrections after implementing allownopassword. (). Add remote fs cache optimization (). Remove utils/github-hook/hook.py (). Fix test (). Fix 00284externalaggregation.sql (). Fix 00900longparquet_load (). Maybe fix use-after-free inside S3 upload thread (). Modified python packages (). Fix disappeared host and port from client help message (). Changelog 22.3 (). Replace a few uses of enable_if for SFINAE by concepts ().