Project: ClickHouse (App Definition and Development / Database)
File: v20.6.1.4066-prestable.md

`clickhouse-local` now uses an unique temporary data directory by default, not the current directory as before. If needed, the data directory can be explicitly specified with the `-- --path` option. (). Add `Alter table drop replica replica_name` support. This fixes . (). Added new in-memory format of parts in `MergeTree`-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholds `minrowsforcompactpart` and `minbytesforcompactpart`. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by setting `inmemorypartsenablewal`. (). Add -Distinct combinator for aggregate functions. (). Support table engine mongo(host:port, database, collection, user, password). (). Add storage RabbitMQ. (). Opt-in settings to send crash reports to the ClickHouse core team via . (). Add ORCBlockOutputFormat. (). `maxthreadpool_size` config for changing the maximum number of Threads in Global Thread Pool. (). Initial implementation of `EXPLAIN` query. Syntax: `EXPLAIN SELECT ...`. This fixes . (). Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. (). Avro UUID input/output support. (). Added read-only MongoDB table engine. Allows reading flat (primitive types, not nested) fields. (). Add setting to fields not found in Avro schema. (). add function `parseDateTimeBestEffortUS`. (). #4006 Support ALTER TABLE ... (). Add a function initializedAggregation to initialize an aggregation based on a single value. (). Support RIGHT and FULL JOIN with `set joinalgorithm=partialmerge`. Only ALL strictness is supported (ANY, SEMI, ANTI, ASOF are not). (). Implementation of PostgreSQL-like ILIKE operator for . (). Allow Nullable types as keys in MergeTree tables. . (). Allow to use directio and mmapio for secondary indices if the settings `minbytestousedirectio` or `minbytestousemmapio` are configured. (). Fix estimation of the number of marks while reading from MergeTree. This is needed to correctly handle the settings `mergetreemaxrowstousecache`, `mergetreemaxbytestousecache`, `mergetreeminrowsforconcurrentread`, `mergetreeminbytesforconcurrentread`, `mergetreeminrowsforseek`, `mergetreeminbytesforseek`. Now settings `minbytestousemmap_io` also applied to read index and compact parts in MergeTree table engines family. (). Fix " Index not used for IN operator with literals", performance regression introduced around v19.3. (). Remove injective functions inside `uniq()` if `set optimizeinjectivefunctionsinsideuniq=1`. (). Add order by optimisation that rewrites `ORDER BY x, f(x)` with `ORDER by x` if `set optimizeredundantfunctionsinorder_by = 1`. (). Lower memory usage for some operations up to 2 times. (). Add number of errors to ignore while choosing replicas (`distributedreplicaerror_ignore`). (). Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key. (). - Add `optimizeskipunusedshardsnesting` (allows control nesting level for shards skipping optimization) - Add `forceskipoptimizeshardsnesting` (allows control nesting level for checking was shards skipped or not) - Deprecate `forceoptimizeskipunusedshardsnonested` (`forceskipoptimizeshardsnesting` should be used instead) - Disable `optimizeskipunusedshards` if shardingkey has non-deterministic func (i.e. `rand()`, note that this does not changes anything for INSERT side). (). Multiversion metadata for storages without structure locks. (). Slightly relax the validation of ODBC connection

If the hostname or username contains only word characters along with `.` and `-`, don't put it into curly braces. It is needed, because some ODBC drivers (e.g. PostgreSQL) don't understand when hostname is enclosed in curly braces. (). Support parse UUID without separator(separators are always removed in most implementations, this is helpful for users to write data). (). Support `SIGNED` and `UNSIGNED` modifiers of standard integer types (`BIGINT`, `INT`, ...) for compatibility with MySQL. (). Allow to use `sumWithOverflow` as `SimpleAggregateFunction`. Closes . (). Add FixedString support in Hashing functions. (). Rewrite code for `optimizearithmeticoperationsinaggregate_functions` optimisation. (). Improve path concatenation and fix double slashed paths using std::filesystem::path instead of std::string in `DatabaseOrdinary.cpp`. (). Deprecate the old regular style and use the new globalVariable method . (). related to . (). `system.tables` now considers column capacities for Memory and Buffer table engines, which is better approximation for resident memory size. (). Add CPU frequencies to system.asynchronous_metrics. Make the metric collection period configurable. (). Allow to perform "metadata-only" ALTER of partition key such as adding more elements to Enum data type. This fixes . (). Add replica priority for load_balancing (for manual prioritization of the load balancing). (). Support MySQL engine reading Enums type . (). Implemented single part uploads for DiskS3. (). Moved useless S3 logging to TRACE level. (). Improves `REVOKE` command: now it requires grant/admin option for only access which will be revoked. For example, to execute `REVOKE ALL ON .* FROM user1` now it doesn't require to have full access rights granted with grant option. Added command `REVOKE ALL FROM user1` - it revokes all granted roles from `user1`. (). Add 'type' column in system.disks. (). Added support for `%g` (two digit ISO year) and `%G` (four digit ISO year) substitutions in `formatDateTime` function. (). Add `KILL QUERY . (). Support MySQL 'SELECT DATABASE()' 2. Add MySQL replacement query integration test. (). This setting allows to chose charset for printing grids (either utf8 or ascii). (). Write the detail exception message to the client instead of 'MySQL server has gone away'. (). lifetimerows/lifetimebytes for Buffer engine. (). Use correct default secure port for clickhouse-benchmark with `--secure` argument. This fixes . (). Fix unexpected behaviour of queries like `SELECT , xyz.*` which were success while an error expected. (). Fix wrong result for `if()` with NULLs in condition. (). Fix memory accounting via HTTP interface (can be significant with `waitendof_query=1`). (). Fix rare crash caused by using `Nullable` column in prewhere condition. Continuation of . (). Fix potential floating point exception when parsing DateTime64. This fixes . (). Fixed bug with no moves when changing storage policy from default one. (). Fix rare crash caused by using Nullable column in prewhere condition. Continuation of . (). Keep aliases for substitutions in query (parametrized queries). This fixes . (). Fix unitialized memory in partitions serialization.

Use the correct current database for checking access rights after statement `USE database`. (). Fixed `Query parameter was not set` in `Values` format. Fixes . (). Fix race condition in extractAllGroups functions. (). Make mmap IO work again (experimental). Continuation of . (). Fix wrong setting name in log message at server startup. (). Partial revokes work correctly in complex cases as well, for example. (). Fix potential floating point exception. This closes . (). Avoid returning wrong number of geohashes in function `geoHashesInBox` due to accumulation of floating point error. This fixes . (). Fix potential array size overflow in generateRandom that may lead to crash. This fixes . (). Fix calculation of access rights when allow_ddl=0. (). When adding floating point number of intervals to date/datetime, the result may be calculated incorrectly. This fixes . (). A query with function `neighbor` as the only returned expression may return empty result if the function is called with offset `-9223372036854775808`. This fixes . (). Do not mention in changelog, because the bug did not come to release. Fix potential crash when doing ORDER BY multiple columns with specified COLLATE on one of the column when this column is constant. This fixes . The bug was introduced in in version 20.5. (). Fix wrong result and potential crash when invoking function `if` with arguments of type `FixedString` with different sizes. This fixes . (). Fix calculation of access rights when allowintrospectionfunctions=0. (). Fix crash in JOIN with LowCardinality type with `joinalgorithm=partialmerge`. (). Fix incorrect comparison of tuples with `Nullable` columns. Fixes . (). Fix constraints check if constraint is a constant expression. This fixes . (). Make `topK` aggregate function return Enum for Enum types. This fixes . (). Parse tables metadata in parallel when loading database. This fixes slow server startup when there are large number of tables. (). Fix error `Cannot capture column` for higher-order functions with `Tuple(LowCardinality)` argument. Fixes . (). Fix error `Expected single dictionary argument for function` for function `defaultValueOfArgumentType` with `LowCardinality` type. Fixes . (). Fix possible crash while using wrong type for `PREWHERE`. Fixes , . (). Fix SIGSEGV in StorageKafka on DROP TABLE. (). Fix unnecessary limiting the number of threads for selects from `VIEW`. Fixes . (). Fix empty `resultrows` and `resultbytes` metrics in `system.quey_log` for selects. Fixes . (). Fix segfault with `-StateResample` combinators. (). Fix performance for selects with `UNION` caused by wrong limit for the total number of threads. Fixes . (). Format `Parquet` now properly works with `LowCardinality` and `LowCardinality(Nullable)` types. Fixes , . (). Fix handling dependency of table with ENGINE=Dictionary on dictionary. This fixes . This fixes . (). Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.querylog, metriclog, etc) or to engine=Buffer underlying table. (). Fix bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes . (). Normalize "pid" file

In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes . (). Fix potential infinite loop in `greatCircleDistance`, `geoDistance`. This fixes . (). Fix potential overflow in integer division. This fixes . (). Fix bad code in redundant ORDER BY optimization. The bug was introduced in . (). Fix transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes . (). Fix wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes . This closes . (). Don't split the dictionary source's table name into schema and table name itself if ODBC connection doesn't support schema. (). Fixed the behaviour when `SummingMergeTree` engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes . (). Fix dictGet arguments check during GROUP BY injective functions elimination. (). Cap max_memory_usage limits to the process resident memory. (). Fixed logical functions for UInt8 values when they are not equal to 0 or 1. (). Fixed the behaviour when during multiple sequential inserts in `StorageFile` header for some special types was written more than once. This fixed . (). Fixed behaviour on reaching redirect limit in request to S3 storage. (). Not for changelog. Cherry-pick after . (). Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes . This fixes . (). The function `arrayFill` worked incorrectly for empty arrays that may lead to crash. This fixes . (). Show error after TrieDictionary failed to load. (). Fix typo in setting name. (). Some threads might randomly hang for a few seconds during DNS cache updating. It's fixed. (). Fix TTL after renaming column, on which depends TTL expression. (). Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like `_table` in `Merge` tables) or by "index" columns in system tables such as filtering by database name when querying from `system.tables`, and this expression returns `Nullable` type. This fixes . (). Fix order of columns in `WITH FILL` modifier. Previously order of columns of `ORDER BY` statement wasn't respected. (). When using codec `Delta` or `DoubleDelta` with non fixed width types, exception with code `LOGICALERROR` was returned instead of exception with code `BADARGUMENTS` (we ensure that exceptions with code logical error never happen). This fixes . (). Fix very rare race condition in ReplicatedMergeTreeQueue. (). Fix error message and exit codes for `ALTER RENAME COLUMN` queries, when `RENAME` is not allowed. Fixes and .

Fix TOTALS/ROLLUP/CUBE for aggregate functions with `-State` and `Nullable` arguments. This fixes . (). Allow to CLEAR column even if there are depending DEFAULT expressions. This fixes . (). Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes . (). Additional check for arguments of bloom filter index. This fixes . (). Reverts change introduced in to resolve . (). Fixed possible segfault if StorageMerge. Closes . (). Fix overflow when very large LIMIT or OFFSET is specified. This fixes . This fixes . (). Fix UBSan report in base64 if tests were run on server with AVX-512. This fixes . Author: @qoega. (). Add simple GitHub hook script for the serverless environment. (). Send logs to client on fatal errors if possible. This will make test results more readable. (). Allow ClickHouse to run on Android. (). Clean up unused header files from `Obfuscator.cpp` and `DatabaseAtomic.cpp`. (). Fix some typos in code. (). Scripts for automated scheduled backporting based on PR labels. (). Add new type of tests based on Testflows framework. (). Install `ca-certificates` before the first `apt-get update` in Dockerfile. (). Daily check by security analysis tool that looks for . (). Regular check by security analysis tool that looks for in `clickhouse-server` Docker image. Also confirms that `Dockerfile` is buildable. Runs daily on `master` and on pull-requests to `Dockerfile`. (). Add `UNBUNDLED` flag to `system.build_options` table. Move skip lists for `clickhouse-test` to clickhouse repo. (). Implement AST-based query fuzzing mode for clickhouse-client. See for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and `00746sqlfuzzy.pl`. (). Perform an upgrade of system packages in the `clickhouse-server` docker image. (). Added a showcase of the minimal Docker image without using any Linux distribution. (). Minor CMake fixes for UNBUNDLED build. (). Missed `<atomic>` is required for `std::atomic<>`. (). Fix warnings from CodeQL. `CodeQL` is another static analyzer that we will use along with `clang-tidy` and `PVS-Studio` that we use already. (). Record additional detail on Dockerfile scan reports. (). Place common docker compose files to integration docker container. (). Remove verbosity from the binary builds. (). Remove strange file creation during build in `orc`. (). Now functional and stress tests will be able to run with old version of `clickhouse-test` script. (). Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. (). Added S3 HTTPS integration test. (). Fix UBSan report in HDFS library. This closes . (). Update word break characters to match readline default - all non-alphanumeric characters. ... (). Not for changelog. (). ... (). NO CL ENTRY: 'Bump certifi from 2020.4.5.2 to 2020.6.20 in /docs/tools/translate'. (). NO CL ENTRY: 'Merging '. (). NO CL ENTRY: 'Revert " (). NO CL ENTRY: 'Bump idna from 2.9 to 2.10 in /docs/tools'. (). NO CL ENTRY: 'Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools'. ().