Project: Tarantool (App Definition and Development / Database)
File: 3.0.0-alpha2.md

Date: 2023-08-23 Tag: 3.0.0-alpha2 3.0.0-alpha2 is the version of the 3.0 release series. This release introduces 74 new features and resolves 87 bugs since the 2.11.0 version. There can be bugs in less common areas. If you find any, feel free to on GitHub. Notable changes are: Added initial support of configuring vshard's storage and router roles in the declarative configuration (gh-9007). Fixes and improvements of the configuration process based on early adopters feedback. New config sections: metrics, feedback, vinyl, flightrec (EE), security (EE) (gh-8861). Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data layout, client-server protocol, and replication protocol. Please using the `box.schema.upgrade()` procedure to unlock all the new features of the 2.x series. [Breaking change] Disabled DDL operations with an old system schema. Now you have to run `box.schema.upgrade()` before you can execute any DDL operations (gh-7149). The `schema.maxid` field in the `box.space._schema` function is now deprecated. Now the `space:max()` function is used instead of `schema.max_id` (gh-5997). Added the new function `box.malloc.info()` for reporting memory usage of Tarantool internal data structures allocated with `malloc()`. The function is available only on Linux (gh-7311). Now `index_object.parts` contains the following methods, similar to the `keydef` Lua module: `extractkey()`, `compare()`, `comparewithkey()`, `merge()` (gh-7356). Introduced the `box.watch_once()` function to get the value currently associated with a notification key on the local instance without subscribing to future changes (gh-6493). Introduced the new `IPROTOWATCHONCE` request to get the value currently associated with a notification key on a remote instance without subscribing to future changes (gh-6493). Introduced the `conn:watch_once()` net.box connection method to get the value currently associated with a notification key on a remote instance without subscribing to future changes. The new method is implemented using the `IPROTOWATCHONCE` request type (gh-6493). Introduced the default field values in the space format (gh-8157). Added support for a functional variant of the default field values (gh-8609). Added memtx read view statistics to `box.stat.memtx()` (gh-8501). Added a new `bootstrap_leader` configuration option to specify the node from which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` to `'config'` and set `bootstrap_leader` value to either the URI or UUID of the desired bootstrap leader. For example: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'localhost:3301', replication = { 'localhost:3301', 'localhost:3302', }, listen = '3302', } ``` (gh-7999). You may now control which node new replicas choose as a bootstrap leader without touching node config. To do so, set `box.cfg.bootstrap_strategy` to `'supervised'`, and the nodes will only bootstrap off the node on which you called `box.ctl.makebootstrapleader()` last. This works on an empty replica set bootstrap as well: start the admin console before configuring the nodes. Then configure the nodes: ```lua box.cfg{ bootstrap_strategy = 'supervised', replication = ..., listen = ..., } ``` Finally, call `box.ctl.makebootstrapleader()` through the admin console on the node you want to promote. All the nodes will bootstrap off that node (gh-8509). A new option `box.cfg.cluster_name` allows assigning a human-readable name to the entire cluster. It has to match in all instances and is displayed in `box.info.cluster.name` (gh-5029). A new option `box.cfg.replicaset_name` allows assigning a human-readable name to the

It works the same as `box.cfg.replicaset_uuid`. Its value must be the same across all instances of one replicaset. The replicaset name is displayed in `box.info.replicaset.name` (gh-5029). A new option `box.cfg.instance_name` allows assigning a human-readable name to the instance. It works the same as `box.cfg.instance_uuid`. Its value must be unique in the replicaset. The instance name is displayed in `box.info.name`. Names of other replicas in the same replicaset are visible in `box.info.replication[id].name` (gh-5029). Instance at rebootstrap can change its UUID while keeping its numeric ID if it has the same non-empty instance name (gh-5029). Added the ability to set the `bootstrap_leader` configuration option to the instance name of the desired bootstrap leader: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'leader-name', replication = { ... }, ... } ``` (gh-7999, gh-8539). Implemented correct recovery of mixed transactions. To do this, set `box.cfg.force_recovery` to `true`. If you need to revert to the old behavior, don't set the `force_recovery` option (gh-7932). LuaJIT now can handle exceptions on traces (gh-7745). Embedded the `tarantool/metrics` module for metrics aggregation and export (gh-7725). Metrics can now be configured using the `box.cfg()` function (gh-7725). [Breaking change]* Added the new `varbinary` type to Lua. An object of this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so it can be used for storing binary blobs in the database. This also works the other way round: data fields stored as `MP_BIN` are now decoded in Lua as varbinary objects, not as plain strings, as they used to be. Since the latter may cause compatibility issues, the new compat option `binarydatadecoding` was introduced to revert the built-in decoder to the old behavior (gh-1629). Added the `:isnew()` and `:isold()` helpers to `tarantool.compat` options to simplify effective value checks (gh-8807). Introduced the new `net.box` module function `from_fd` for creating a new connection from a socket file descriptor number (gh-8984). Introduced new option `inherit_fds` for `popen.new`. The option takes an array of file descriptor numbers that should be left open in the child process (gh-8926). Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and `socket:detach` (gh-8927). Introduced the `SHOW CREATE TABLE` statement (gh-8098). Non-system spaces in a snapshot file are now sorted by their IDs. As before, all non-system spaces are stored after system spaces (gh-7954). [Breaking change]* Dropped support for `IPROTO_CALL16` from `net.box` (i.e., the `call16` option and the `call16` method for `net.box` connections) (gh-8636). Added support for square brackets in procedure resolution for Lua calls (gh-8604). Added support for accepting IPROTO requests with a space or index name instead of an identifier (gh-8146). Added a new `box.info` parameter `hostname` (gh-8605). Added the `box.cfg.memtxsortthreads` parameter that specifies the number of threads used to sort indexes keys on loading a memtx database. OpenMP is not used to sort keys anymore (gh-3389). Changed the output of the `fselect` index method from a table of strings to a multi-line string and dropped the `print` and `use_nbsp` options. With the `yamlprettymultiline` compat option enabled by default, multi-line strings now look good in the console, so there's no need to return a table of strings to prettify the `fselect` output

Allowed multiple updates to the same tuple field in a single call (gh-8226). Introduced `box.tuple.format` that enables format definition for tuples created via `box.tuple.new` (standalone tuples) (gh-4693). [Breaking change]* Disabled argument list syntax of `box.tuple.new` (this was needed for gh-4693). It is possible to switch to the old behavior using the compatibility option `boxtuplenew_vararg`. Allowed truncation of temporary and local spaces in the read-only mode (gh-5616). Sped up `index.select` and `index.pairs` with the `after` option by up to 30% in a synthetic test by eliminating an extra buffer allocation. The function `box.session.push` is now deprecated. Consider using `box.broadcast` instead (gh-8802). Introduced the new function `box.session.new` for creating a new IPROTO session from a socket file descriptor number (gh-8801). Introduced the new `luaeval` and `luacall` object types for `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval` allows the user to execute an arbitrary Lua expression with the `IPROTOEVAL` request. Granting the `'execute'` privilege on `luacall` allows the user to execute any global user-defined Lua function with the `IPROTO_CALL` request (gh-8803). [Breaking change]* Introduced the new `sql` object type for `box.schema.user.grant`. Now only users with the `'execute'` privilege granted on `sql` or `universe` can execute SQL expressions with the `IPROTOEXECUTE` or `IPROTOPREPARE` requests. To revert to the old behavior (no SQL access checks), use the `sql_priv` compat option (gh-8803). Implemented a way to set a table as box.cfg{} options value via environment variables (gh-8051). [Behavior change]* Disabled starting the Lua REPL by default when running Tarantool. Now, Tarantool yields the message that shows the command usage. To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via `stdin`, use dash (`-`) as the script name. For more information see a help message by running `tarantool -h` (gh-8613). Introduced new command-line options `--name` and `--config` (gh-8613). The following compatibility options' defaults were switched to new behavior: `yamlprettymultiline` `sqlseqscan_default` `jsonescapeforward_slash` `fiberchannelclose_mode` `fiberslicedefault` `boxcfgreplicationsynctimeout` More information on the new behavior can be found on the page. Initial version of the declarative server and cluster configuration (gh-8724). All vinyl options are now supported (gh-8861). All feedback options are now supported (gh-8861). The `memtx.sort_threads` option is now supported (gh-8861). The `bootstrap_leader` option is now supported (gh-8861). All flight recorder options are now supported (gh-8861). All security hardening `box.cfg{}` options are now supported by config (gh-8861). Added a low priority environment configuration source, which looks into `TT*DEFAULT` variables. It is useful to declare defaults (gh-8862). It is now possible to access configuration from the application script using the `config:get()` method (gh-8862). All metrics configuration options from `box.cfg{}` are now supported in the YAML config (gh-8861). Added `--help-env-list` CLI option (gh-8862). Introduced a non-public API for extending the declarative config functionality in Tarantool Community Edition (gh-8862). Improved the credentials applier: now it supports two-way synchronization of roles and privileges for both users and roles (gh-8861). Removed the sha1 and sha256 hash entries from the config credentials schema as not planned for development (gh-8967). Implemented a full password support in the

schema, including a password setting, updating and removal for the `chap-sha1` auth type (supported by both Tarantool Community Edition and Tarantool Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition where it is available) (gh-8967). A group name from a topology defined by the cluster configuration doesn't persisted anymore as `box.cfg.cluster_name` (gh-8862). The default directory and file paths are changed to `var/run/<...>`, `var/log/<...>`, `var/lib/<...>` and so on (gh-8862). The directory that contains the configuration file is now added into the Lua modules search paths (gh-8862). Most of vshard options are now added in the config (gh-9007). Added initial support of vshard (gh-9007). Now multiline commands can be used by setting the continuation marker using the `\set continuation` command. Introduced readline support to tarantool debugger (gh-7738). Now LuaJIT can be fuzzed using grammar-based fuzzer (gh-4823). Added a grammar-based SQL fuzzer (gh-4826). The `zstd` version was updated to pre-1.5.5 (gh-8391). Added `tt` weak dependency. The `tt` command line utility combines `tarantoolctl` and `cartridge-cli` functionality into one tool. The change is a part of pushing `tt` to the public (tarantool/tt#286). Tarantool does not depend on libgomp anymore (gh-7689). Hardening against memory corruption attacks is now enabled by default on FreeBSD (gh-7536). Hardening against memory corruption attacks is now enabled by default on AArch64 systems (gh-7536). Added building static `deb` and `rpm` packages. Dropped building distro-specific `deb` and `rpm` Tarantool packages. Dropped building Tarantool packages with GC32. Now only GC64 packages are shipped. Exported all LuaJIT public functions from Tarantool binary (gh-3680). Added the CMake option `FIBERSTACKSIZE` to set the default fiber stack size. Fixed various bugs related to unsafe (i.e., from an unknown source) decoding and validating of MsgPack extensions (ghs-73). Fixed a crash in the `net.box` client when a remote space had a field type that the client version did not support (gh-4632). Fixed a bug when a space creation failed with a duplication error. The issue occurred if the explicit and implicit space IDs were mixed. Now the actual maximal space `id` is used to generate a new one (gh-8036). Relaxed the tuple format requirements on tuples passed as the page starting position to `index:tuple_pos()` or to the `after` option of `index:select`. Now, Tarantool validates only the key parts of the index being used and all primary indexes (gh-8511). Fixed a crash that could happen when preparing a crash report on macOS (gh-8445). Fixed an integer overflow issue in `net.box` (ghs-121). An `IPROTO_EVENT` packet now has the same sync number as the last corresponding `IPROTO_WATCH` request (gh-8393). Fixed a bug because of which a dirty (not committed to WAL) DDL record could be written to a snapshot and cause a recovery failure (gh-8530). [Breaking change]* The key `box.space._schema['cluster']` is renamed to `'replicasetuuid'`. That is not expected to be breaking because `schema` is an internal system space, but the key was visible in public and documented (gh-5029). [Breaking change]* The table `box.info.cluster` is renamed to `box.info.replicaset`. The behavior can be reverted using the `compat` option `boxinfocluster_meaning` (https://tarantool.io/compat/boxinfocluster_meaning) (gh-5029). Fixed a bug when a tuple could be inserted even if it violates a `unique` constraint of a functional index that has a nullable part

Fixed a bug when Tarantool failed to decode a request containing an unknown IPROTO key. The bug resulted in broken connectivity between Tarantool 2.10 and 2.11 (gh-8745). Fixed the node writing an empty `00000000000000000000.xlog` file regardless of the actual vclock when interrupted during the initial `box.cfg()` call (gh-8704). Fixed a bug causing the `ERCURSORNO_TRANSACTION` failure for transactions on synchronous spaces when the `oncommit/onrollback` triggers are set (gh-8505). Fixed a bug causing the effective session and user are not propagated to `box.oncommit` and `box.onrollback` trigger callbacks when transaction is synchronous (gh-8742). Eliminated implicit conversion of unprintable utf-8 strings to binary blobs when encoded in YAML. Now unprintable characters are encoded as escaped utf-8 code points, for example, `\x80` or `\u200B` (gh-8756). Fixed a bug when MVCC sometimes lost gap record (gh-8326). Fixed a bug when MVCC rollback of prepared statement could break internal invariants (gh-8648). Now MVCC engine automatically aborts a transaction if it reads changes of a prepared transaction and this transaction is aborted (gh-8654). Disabled the backtrace collection feature on the AArch64 Linux platform (gh-8572). Fixed a crash that could happen when Tarantool is started in the (gh-6128). Fixed the collection of fiber backtraces on the M1/M2 macOS platform (gh-8074). Fixed a bug that caused writing incorrect values into the `stream_id` field of xlog headers (gh-8783). Fixed decoding of escape sequences for single-byte character codes from YAML. Before the fix, single-byte character codes between `0x80` and `0xff` would be erroneously converted to two-byte UTF-8 code points, for example, `\x80` would be decoded as `\uC280` (gh-8782). Fixed a bug when a space that is referenced by a foreign key could not be truncated even if the referring space was empty (gh-8946). Now foreign keys from non-temporary to temporary and from non-local to local spaces are prohibited since they can potentially break foreign key consistency (gh-8936). Fixed a crash when a collation used by a space was deleted (gh-4544). Fixed a crash that could happen when Tarantool is compiled by `clang` version 15 and above with enabled AddressSanitizer (tarantool/tarantool-qa#321). Fixed a bug because of which it was impossible to set the `hint` option to `true` for TREE indexes (gh-8937). Now `indexobject.parts.excludenull` always contains `false` rather than `null` when it is actually `false` (gh-8649). Fixed the ability for the `memtx_memory` parameter to specify a memory size less than required for the stable operation of the engine (gh-7389). Fixed a heap-use-after-free bug in the transaction manager, which could occur when performing a DDL operation concurrently with a transaction on the same space (gh-8781). Vinyl space and index directories are now created on demand. They are removed as soon as they become empty (gh-8441). Fixed a heap-use-after-free bug in the Vinyl read iterator caused by a race between a disk read and a memory dump task. The bug could lead to a crash or an invalid query result (gh-8852). Fixed a bug related to `box.info.replication[...].upstream` being stuck in the "connecting" state for several minutes after a replica DNS record change (gh-7294). Fixed an issue when the nodes synchronizing with a hung leader reported the leader as alive. This behavior led to the delay of the new elections

Fixed a bug that occurred on applier failure: a node could start an election without having a quorum to do this (gh-8433). Fixed a bug that allowed to change replicaset UUID via `_schema` space. Fixed a bug when new instances could try to register via an anon instance which previously failed to apply `box.cfg{replication_anon = false}`. Fixed a possible failure to promote the desired node by `box.ctl.promote()` on a cluster with nodes configured with `election_mode = "candidate"` (gh-8497). Fixed nodes configured with `election_mode = 'candidate'` spuriously detecting a split-vote when another candidate should win with exactly a quorum of votes for it (gh-8698). Fixed a crash when using transactions with the linearizable isolation level during a replica reconnect (gh-7991). Fixed a possible crash on bootstrap with `box.cfg.bootstrap_strategy = 'auto'` when some of the bootstrapping nodes were stopped (gh-8757). Backported patches from the vanilla LuaJIT trunk (gh-8069). The following issues were fixed as part of this activity: Fixed successful `math.min/math.max` call with no args (gh-6163). Fixed inconsistencies in `math.min/math.max` calls with a NaN arg (gh-6163). Fixed `pcall()` call without arguments on arm64. Fixed assembling of IR_{AHUV}LOAD specialized to boolean for aarch64. Fixed constant rematerialization on arm64. Fixed `emit_rma()` for the x64/GC64 mode for non-`mov` instructions. Limited Lua C library path with the default `PATH_MAX` value of 4096 bytes. The JIT engine was disabled by default on macOS platforms to improve the user experience. If necessary, you can enable it with `jit.on` (gh-8252). Backported patches from the vanilla LuaJIT trunk (gh-8516). The following issues were fixed as part of this activity: Fixed `IRLREF` assembling for the GC64 mode on x8664. Fixed canonicalization of +-0.0 keys for `IR_NEWREF`. Fixed result truncation for `bit.rol` on x86_64 platforms. Fixed saved bytecode encapsulated in ELF objects. Fixed `lua_yield()` invocation inside C hooks. Fixed memory chunk allocation beyond the memory limit. Fixed TNEW load forwarding with instable types. Fixed use-def analysis for `BCVARG`, `BCFUNCV`. Backported patches from the vanilla LuaJIT trunk (gh-8825). The following issues were fixed as part of this activity: Fixed `BC_UCLO` insertion for returns. Fixed recording of `BC_VARG` with unused vararg values. Initialization instructions on trace are now emitted only for the first member of a union. The `box.stat.memtx` function is now callable, it returns all memtx statistics. The `box.stat.memtx.tx()` function is now equivalent to the `box.stat.memtx().tx` function (gh-8448). Exported the missing `AUTH_TYPE` key to `box.iproto.key` (gh-8574). Stripped the unwanted `IPROTO` prefix from the `POSITION`, `AFTERPOSITION`, and `AFTER_TUPLE` keys exported to `box.iproto.key` (gh-8577). Fixed the xlog reader Lua module to show unknown row header fields. Before this change the xlog reader silently skipped them. The default permission mode for `fio.open()` was changed for newly created files to 0666 (before umask) (gh-7981). Fixed a bug where a response body cannot be decoded (gh-8363). Fixed the `Transfer-Encoding: chunked` setting being enabled even if the `Content-Length` header exists for stream requests (gh-8744). Fixed a heap-use-after-free bug in the function creating a tuple format Lua object for `net.box` (gh-8889). The `ANY` field type now supports collation in SQL (gh-8070). Now the `ARRAY`, `MAP`, and `INTERVAL` values cannot be used as arguments in the `ORDER BY` clause

Fixed incorrect conversion of an integer greater than `INT64_MAX` or less than `0` to a decimal number during SQL arithmetic operations (gh-8460). Fixed an assertion when selecting tuples with incomplete internal format (gh-8418). Fixed integer overflow issues in built-in functions (ghs-119). Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause (ghs-119). Fixed a possible assertion or segmentation fault when optimizing `INSERT INTO ... SELECT FROM` (gh-8661). Fixed an integer overflow issue and added a check for the `printf()` failure due to too large size (ghs-122). Fixed a bug where `box.cfg.force_recovery` doesn't work when there are no user spaces in a snapshot (gh-7974). Fixed a bug when large numbers were encoded incorrectly by `msgpackffi`. It could lead to wrong select results with large number keys (gh-6119). Fixed a crash on an unknown option of the function (gh-8463). The `takesrawargs` option is now handled correctly during downgrading (gh-8457). Fixed several memory leaks on tuple encoding failures (gh-7939). Fixed the ability to drop `vinyldeferred_delete` system space (gh-5279). Fixed a bug (bad error message) in pagination related to the validation of the `after` position option of the `:select` and `:pairs` methods of space and index objects (gh-8716). Fixed a bug when it was not possible to access spaces named `_stream` and `streamspacecache` and indexes named `space` using the `net.box` stream objects (gh-8598). Fixed the memory leaks caused by the multi-statement transaction errors in the space index building and the space format checking operations (gh-8773). [Breaking change]* The `cordslabcache` symbol was removed from the public API export (gh-7124). Fixed the privilege check when using spaces with functional indexes and constraints (gh-7873). Fixed the inability to insert an integral number into a double-formatted field (gh-7483). Fixed a bug in the box console implementation because of which the `language` parameter was shared between connected clients (gh-8817). Fixed the invalid memory access in a corner case of a specialized comparison function (gh-8899). Fixed a crash when `box.iproto.override` was called with an unconfigured box. Now, an error is raised instead (gh-8975). Fixed an error message if the cluster configuration was not provided or the instance was not found in the cluster configuration during reload (gh-8862). Support parent directories creation for options that accept a directory or a file (gh-8862). Create parent directories for `console.socket` and `log.file` (gh-8862). Create the `process.work_dir` directory (gh-8862). Consider all the paths as relative to `process.work_dir` when creating necessary directories (gh-8862). Fixed `console.local_print()` failing on non-string arguments, which led to some rare errors. For example, when connecting via tarantoolctl to cartridged tarantool with incorrect credentials, a cdata error was passed through the `local_print()`, which failed to interpret it (gh-8374). Fixed a bug with buffer overflow in `tnt_strptime` (gh-8502). Fixed an error in `datetime.set` when `timestamp` is passed along with `nsec`, `usec`, or `msec` (gh-8583). Fixed errors when the string representation of a datetime object had a negative nanosecond part (gh-8570). Fixed a bug raising a false positive error when creating new intervals with range boundary values (gh-8878). Fixed decoding datetime intervals with fields larger than possible int32 values (gh-8887). Enabled compiler optimizations for static build dependencies, which were erroneously disabled in version 2.10.3 (gh-8606).