Project: Tarantool (App Definition and Development / Database)
File: 3.0.0-alpha1.md

Date: 2023-07-05 Tag: 3.0.0-alpha1 3.0.0-alpha1 is the version of the 3.0 release series. This release introduces 42 new features and resolves 70 bugs since the 2.11 version. There can be bugs in less common areas. If you find any, feel free to on GitHub. Notable changes are: New configuration capabilities (gh-8724) Tarantool 3.x is backward compatible with Tarantool 2.x in the binary data layout, client-server protocol, and replication protocol. Please using the `box.schema.upgrade()` procedure to unlock all the new features of the 3.x series. [Breaking change] Disabled DDL operations with an old system schema. Now you have to run `box.schema.upgrade()` before you can execute any DDL operations (gh-7149). The `schema.maxid` field in the `box.space._schema` function is now deprecated. Now the `space:max()` function is used instead of `schema.max_id` (gh-5997). Added the new function `box.malloc.info()` for reporting memory usage of Tarantool internal data structures allocated with `malloc()`. The function is available only on Linux (gh-7311). Now `index_object.parts` contains the following methods, similar to the `keydef` Lua module: `extractkey()`, `compare()`, `comparewithkey()`, `merge()` (gh-7356). Introduced the `box.watch_once()` function to get the value currently associated with a notification key on the local instance without subscribing to future changes (gh-6493). Introduced the new `IPROTOWATCHONCE` request to get the value currently associated with a notification key on a remote instance without subscribing to future changes (gh-6493). Introduced the `conn:watch_once()` net.box connection method to get the value currently associated with a notification key on a remote instance without subscribing to future changes. The new method is implemented using the `IPROTOWATCHONCE` request type (gh-6493). Introduced the default field values in the space format (gh-8157). Added memtx read view statistics to `box.stat.memtx()` (gh-8501). Added a new `bootstrap_leader` configuration option to specify the node from which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` to `'config'` and set `bootstrap_leader` value to either the URI or UUID of the desired bootstrap leader. For example: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'localhost:3301', replication = { 'localhost:3301', 'localhost:3302', }, listen = '3302', } ``` (gh-7999). You may now control which node new replicas choose as a bootstrap leader without touching node config. To do so, set `box.cfg.bootstrap_strategy` to `'supervised'`, and the nodes will only bootstrap off the node on which you called `box.ctl.makebootstrapleader()` last. This works on an empty replica set bootstrap as well: start the admin console before configuring the nodes. Then configure the nodes: ```lua box.cfg{ bootstrap_strategy = 'supervised', replication = ..., listen = ..., } ``` Finally, call `box.ctl.makebootstrapleader()` through the admin console on the node you want to promote. All the nodes will bootstrap off that node (gh-8509). A new option `box.cfg.cluster_name` allows assigning a human-readable name to the entire cluster. It has to match in all instances and is displayed in `box.info.cluster.name` (gh-5029). A new option `box.cfg.replicaset_name` allows assigning a human-readable name to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value must be the same across all instances of one replicaset. The replicaset name is displayed in `box.info.replicaset.name` (gh-5029). A new option `box.cfg.instance_name` allows assigning a human-readable name to the instance. It works the same as `box.cfg.instance_uuid`. Its value must be unique in the replicaset. The instance name is displayed in

Names of other replicas in the same replicaset are visible in `box.info.replication[id].name` (gh-5029). Instance at rebootstrap can change its UUID while keeping its numeric ID if it has the same non-empty instance name (gh-5029). Added the ability to set the `bootstrap_leader` configuration option to the instance name of the desired bootstrap leader: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'leader-name', replication = { ... }, ... } ``` (gh-7999, gh-8539). Implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option (gh-7932). LuaJIT now can handle exceptions on traces (gh-7745). Embedded the `tarantool/metrics` module for metrics aggregation and export (gh-7725). Metrics can now be configured using the `box.cfg()` function (gh-7725). [Breaking change]* Added the new `varbinary` type to Lua. An object of this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so it can be used for storing binary blobs in the database. This also works the other way round: data fields stored as `MP_BIN` are now decoded in Lua as varbinary objects, not as plain strings, as they used to be. Since the latter may cause compatibility issues, the new compat option `binarydatadecoding` was introduced to revert the built-in decoder to the old behavior (gh-1629). Introduced the `SHOW CREATE TABLE` statement (gh-8098). Non-system spaces in a snapshot file are now sorted by their IDs. As before, all non-system spaces are stored after system spaces (gh-7954). [Breaking change]* Dropped support for `IPROTO_CALL16` from `net.box` (i.e., the `call16` option and the `call16` method for `net.box` connections) (gh-8636). Added support for square brackets in procedure resolution for Lua calls (gh-8604). Added support for accepting IPROTO requests with a space or index name instead of an identifier (gh-8146). Added a new `box.info` parameter `hostname` (gh-8605). Added the `box.cfg.memtxsortthreads` parameter that specifies the number of threads used to sort indexes keys on loading a memtx database. OpenMP is not used to sort keys anymore (gh-3389). Changed the output of the `fselect` index method from a table of strings to a multi-line string and dropped the `print` and `use_nbsp` options. With the `yamlprettymultiline` compat option enabled by default, multi-line strings now look good in the console, so there's no need to return a table of strings to prettify the `fselect` output anymore. Allowed multiple updates to the same tuple field in a single call (gh-8226). Introduced `box.tuple.format` that enables format definition for tuples created via `box.tuple.new` (standalone tuples) (gh-4693). [Breaking change]* Disabled argument list syntax of `box.tuple.new` (this was needed for gh-4693). It is possible to switch to the old behavior using the compatibility option `boxtuplenew_vararg`. Implemented a way to set a table as box.cfg{} options value via environment variables (gh-8051). [Behavior change]* Disabled starting the Lua REPL by default when running Tarantool. Now, Tarantool yields the message that shows the command usage. To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via `stdin`, use dash (`-`) as the script name. For more information see a help message by running `tarantool -h` (gh-8613). Introduced new command-line options `--name` and `--config`

The following compatibility options' defaults were switched to new behavior: `yamlprettymultiline` `sqlseqscan_default` `jsonescapeforward_slash` `fiberchannelclose_mode` `fiberslicedefault` `boxcfgreplicationsynctimeout` More information on the new behavior can be found on the page. Now multiline commands can be used by setting continuation marker using the `\set continuation` command. Now LuaJIT can be fuzzed using grammar-based fuzzer (gh-4823). Added a grammar-based SQL fuzzer (gh-4826). The `zstd` version was updated to pre-1.5.5 (gh-8391). Added `tt` weak dependency. The `tt` command line utility combines `tarantoolctl` and `cartridge-cli` functionality into one tool. The change is a part of pushing `tt` to the public (tarantool/tt#286). Tarantool does not depend on libgomp anymore (gh-7689). Hardening against memory corruption attacks is now enabled by default on FreeBSD (gh-7536). Hardening against memory corruption attacks is now enabled by default on AArch64 systems (gh-7536). Added building static `deb` and `rpm` packages. Dropped building distro-specific `deb` and `rpm` Tarantool packages. Dropped building Tarantool packages with GC32. Now only GC64 packages are shipped. Fixed various bugs related to unsafe (i.e., from an unknown source) decoding and validating of MsgPack extensions (ghs-73). Fixed a crash in the `net.box` client when a remote space had a field type that the client version did not support (gh-4632). Fixed a bug when a space creation failed with a duplication error. The issue occurred if the explicit and implicit space IDs were mixed. Now the actual maximal space `id` is used to generate a new one (gh-8036). Relaxed the tuple format requirements on tuples passed as the page starting position to `index:tuple_pos()` or to the `after` option of `index:select`. Now, Tarantool validates only the key parts of the index being used and all primary indexes (gh-8511). Fixed a crash that could happen when preparing a crash report on macOS (gh-8445). Fixed an integer overflow issue in `net.box` (ghs-121). An `IPROTO_EVENT` packet now has the same sync number as the last corresponding `IPROTO_WATCH` request (gh-8393). Fixed a bug because of which a dirty (not committed to WAL) DDL record could be written to a snapshot and cause a recovery failure (gh-8530). [Breaking change]* The key `box.space._schema['cluster']` is renamed to `'replicasetuuid'`. That is not expected to be breaking because `schema` is an internal system space, but the key was visible in public and documented (gh-5029). [Breaking change]* The table `box.info.cluster` is renamed to `box.info.replicaset`. The behaviour can be reverted using the `compat` option `boxinfocluster_meaning` (https://tarantool.io/compat/boxinfocluster_meaning) (gh-5029). Fixed a bug when a tuple could be inserted even if it violates a `unique` constraint of a functional index that has a nullable part (gh-8587). Fixed a bug when Tarantool failed to decode a request containing an unknown IPROTO key. The bug resulted in broken connectivity between Tarantool 2.10 and 2.11 (gh-8745). Fixed the node writing an empty `00000000000000000000.xlog` file regardless of the actual vclock when interrupted during the initial `box.cfg()` call (gh-8704). Fixed a bug causing the `ERCURSORNO_TRANSACTION` failure for transactions on synchronous spaces when the `oncommit/onrollback` triggers are set (gh-8505). Fixed a bug causing the effective session and user are not propagated to `box.oncommit` and `box.onrollback` trigger callbacks when transaction is synchronous (gh-8742). Eliminated implicit conversion of unprintable utf-8 strings to binary blobs when encoded in YAML. Now unprintable characters are encoded as escaped utf-8 code points, for example, `\x80` or `\u200B`

Fixed a bug when MVCC sometimes lost gap record (gh-8326). Fixed a bug when MVCC rollback of prepared statement could break internal invariants (gh-8648). Now MVCC engine automatically aborts a transaction if it reads changes of a prepared transaction and this transaction is aborted (gh-8654). Disabled the backtrace collection feature on the AArch64 Linux platform (gh-8572). Fixed a crash that could happen when Tarantool is started in the (gh-6128). Fixed the collection of fiber backtraces on the M1/M2 macOS platform (gh-8074). Fixed a bug that caused writing incorrect values into the `stream_id` field of xlog headers (gh-8783). Fixed decoding of escape sequences for single-byte character codes from YAML. Before the fix, single-byte character codes between `0x80` and `0xff` would be erroneously converted to two-byte UTF-8 code points, for example, `\x80` would be decoded as `\uC280` (gh-8782). Fixed the ability for the `memtx_memory` parameter to specify a memory size less than required for the stable operation of the engine (gh-7389). Fixed a heap-use-after-free bug in the transaction manager, which could occur when performing a DDL operation concurrently with a transaction on the same space (gh-8781). Vinyl space and index directories are now created on demand. They are removed as soon as they become empty (gh-8441). Fixed a bug related to `box.info.replication[...].upstream` being stuck in the "connecting" state for several minutes after a replica DNS record change (gh-7294). Fixed an issue when the nodes synchronizing with a hung leader reported the leader as alive. This behavior led to the delay of the new elections (gh-7515). Fixed a bug that occurred on applier failure: a node could start an election without having a quorum to do this (gh-8433). Fixed a bug that allowed to change replicaset UUID via `_schema` space. Fixed a bug when new instances could try to register via an anon instance which previously failed to apply `box.cfg{replication_anon = false}`. Fixed a possible failure to promote the desired node by `box.ctl.promote()` on a cluster with nodes configured with `election_mode = "candidate"` (gh-8497). Fixed nodes configured with `election_mode = 'candidate'` spuriously detecting a split-vote when another candidate should win with exactly a quorum of votes for it (gh-8698). Fixed a crash when using transactions with the linearizable isolation level during a replica reconnect (gh-7991). Fixed a possible crash on bootstrap with `box.cfg.bootstrap_strategy = 'auto'` when some of the bootstrapping nodes were stopped (gh-8757). Backported patches from the vanilla LuaJIT trunk (gh-8069). The following issues were fixed as part of this activity: Fixed successful `math.min/math.max` call with no args (gh-6163). Fixed inconsistencies in `math.min/math.max` calls with a NaN arg (gh-6163). Fixed `pcall()` call without arguments on arm64. Fixed assembling of IR_{AHUV}LOAD specialized to boolean for aarch64. Fixed constant rematerialization on arm64. Fixed `emit_rma()` for the x64/GC64 mode for non-`mov` instructions. Limited Lua C library path with the default `PATH_MAX` value of 4096 bytes. The JIT engine was disabled by default on macOS platforms to improve the user experience. If necessary, you can enable it with `jit.on` (gh-8252). Backported patches from the vanilla LuaJIT trunk (gh-8516). The following issues were fixed as part of this activity: Fixed `IRLREF` assembling for the GC64 mode on x8664. Fixed canonicalization of +-0.0 keys for `IR_NEWREF`. Fixed result truncation for

on x86_64 platforms. Fixed saved bytecode encapsulated in ELF objects. Fixed `lua_yield()` invocation inside C hooks. Fixed memory chunk allocation beyond the memory limit. Fixed TNEW load forwarding with instable types. Fixed use-def analysis for `BCVARG`, `BCFUNCV`. The `box.stat.memtx` function is now callable, it returns all memtx statistics. The `box.stat.memtx.tx()` function is now equivalent to the `box.stat.memtx().tx` function (gh-8448). Exported the missing `AUTH_TYPE` key to `box.iproto.key` (gh-8574). Stripped the unwanted `IPROTO` prefix from the `POSITION`, `AFTERPOSITION`, and `AFTER_TUPLE` keys exported to `box.iproto.key` (gh-8577). Fixed the xlog reader Lua module to show unknown row header fields. Before this change the xlog reader silently skipped them. The default permission mode for `fio.open()` was changed for newly created files to 0666 (before umask) (gh-7981). Fixed a bug where a response body cannot be decoded (gh-8363). Fixed the `Transfer-Encoding: chunked` setting being enabled even if the `Content-Length` header exists for stream requests (gh-8744). The `ANY` field type now supports collation in SQL (gh-8070). Now the `ARRAY`, `MAP`, and `INTERVAL` values cannot be used as arguments in the `ORDER BY` clause (gh-6668). Fixed incorrect conversion of an integer greater than `INT64_MAX` or less than `0` to a decimal number during SQL arithmetic operations (gh-8460). Fixed an assertion when selecting tuples with incomplete internal format (gh-8418). Fixed integer overflow issues in built-in functions (ghs-119). Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause (ghs-119). Fixed a possible assertion or segmentation fault when optimizing `INSERT INTO ... SELECT FROM` (gh-8661). Fixed an integer overflow issue and added check for the `printf()` failure due to too large size (ghs-122). Fixed a bug where `box.cfg.force_recovery` doesn't work when there are no user spaces in a snapshot (gh-7974). Fixed a bug when large numbers were encoded incorrectly by `msgpackffi`. It could lead to wrong select results with large number keys (gh-6119). Fixed a crash on an unknown option of the function (gh-8463). The `takesrawargs` option is now handled correctly during downgrading (gh-8457). Fixed several memory leaks on tuple encoding failures (gh-7939). Fixed the ability to drop `vinyldeferred_delete` system space (gh-5279). Fixed a bug (bad error message) in pagination related to the validation of the `after` position option of the `:select` and `:pairs` methods of space and index objects (gh-8716). Fixed a bug when it was not possible to access spaces named `_stream` and `streamspacecache` and indexes named `space` using the `net.box` stream objects (gh-8598). Fixed the memory leaks caused by the multi-statement transaction errors in the space index building and the space format checking operations (gh-8773). [Breaking change]* The `cordslabcache` symbol was removed from the public API export (gh-7124). Fixed the privilege check when using spaces with functional indexes and constraints (gh-7873). Fixed `console.local_print()` failing on non-string arguments, which led to some rare errors. For example, when connecting via tarantoolctl to cartridged tarantool with incorrect credentials, a cdata error was passed through the `local_print()`, which failed to interpret it (gh-8374). Fixed a bug with buffer overflow in `tnt_strptime` (gh-8502). Fixed an error in `datetime.set` when `timestamp` is passed along with `nsec`, `usec`, or `msec` (gh-8583). Fixed errors when the string representation of a datetime object had a negative nanosecond part (gh-8570). Enabled compiler optimizations for static build dependencies, which were erroneously disabled in version 2.10.3 (gh-8606).