Project: Tarantool (App Definition and Development / Database)
File: 3.0.0-alpha3.md

Date: 2023-09-12 Tag: 3.0.0-alpha3 3.0.0-alpha3 is the version of the 3.0 release series. This release introduces 71 new features and resolves 23 bugs since the 2.11.1 version. There can be bugs in less common areas. If you find any, feel free to on GitHub. Notable changes are: Added initial support of configuring vshard's storage and router roles in the declarative configuration (gh-9007). Fixes and improvements of the configuration process based on early adopters feedback. New config sections: metrics, feedback, vinyl, flightrec (EE), security (EE) (gh-8861). Added initial support of roles in the declarative configuration. Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data layout, client-server protocol, and replication protocol. Please using the `box.schema.upgrade()` procedure to unlock all the new features of the 2.x series. [Breaking change] Disabled DDL operations with an old system schema. Now you have to run `box.schema.upgrade()` before you can execute any DDL operations (gh-7149). Now `index_object.parts` contains the following methods, similar to the `keydef` Lua module: `extractkey()`, `compare()`, `comparewithkey()`, `merge()` (gh-7356). Introduced the `box.watch_once()` function to get the value currently associated with a notification key on the local instance without subscribing to future changes (gh-6493). Introduced the new `IPROTOWATCHONCE` request to get the value currently associated with a notification key on a remote instance without subscribing to future changes (gh-6493). Introduced the `conn:watch_once()` net.box connection method to get the value currently associated with a notification key on a remote instance without subscribing to future changes. The new method is implemented using the `IPROTOWATCHONCE` request type (gh-6493). Introduced the default field values in the space format (gh-8157). Added support for functional variant of the default field values (gh-8609). Added memtx read view statistics to `box.stat.memtx()` (gh-8501). Introduced the `memory.tuple` statistic for `box.stat.vinyl()` that shows the total size of memory occupied by all tuples allocated by the Vinyl engine (gh-8485). Added a new `bootstrap_leader` configuration option to specify the node from which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` to `'config'` and set `bootstrap_leader` value to either the URI or UUID of the desired bootstrap leader. For example: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'localhost:3301', replication = { 'localhost:3301', 'localhost:3302', }, listen = '3302', } ``` (gh-7999). You may now control which node new replicas choose as a bootstrap leader without touching node config. To do so, set `box.cfg.bootstrap_strategy` to `'supervised'`, and the nodes will only bootstrap off the node on which you called `box.ctl.makebootstrapleader()` last. This works on an empty replica set bootstrap as well: start the admin console before configuring the nodes. Then configure the nodes: ```lua box.cfg{ bootstrap_strategy = 'supervised', replication = ..., listen = ..., } ``` Finally, call `box.ctl.makebootstrapleader()` through the admin console on the node you want to promote. All the nodes will bootstrap off that node (gh-8509). A new option `box.cfg.cluster_name` allows assigning a human-readable name to the entire cluster. It has to match in all instances and is displayed in `box.info.cluster.name` (gh-5029). A new option `box.cfg.replicaset_name` allows assigning a human-readable name to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value must be the same across all instances of one replicaset. The replicaset name is displayed in `box.info.replicaset.name` (gh-5029). A new option `box.cfg.instance_name` allows assigning a human-readable name to the instance. It works the same as `box.cfg.instance_uuid`. Its value must be unique in the replicaset. The instance name is displayed in

Names of other replicas in the same replicaset are visible in `box.info.replication[id].name` (gh-5029). Instance at rebootstrap can change its UUID while keeping its numeric ID if it has the same non-empty instance name (gh-5029). Added the ability to set the `bootstrap_leader` configuration option to the instance name of the desired bootstrap leader: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'leader-name', replication = { ... }, ... } ``` (gh-7999, gh-8539). LuaJIT now can handle exceptions on traces (gh-7745). [Breaking change]* Added the new `varbinary` type to Lua. An object of this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so it can be used for storing binary blobs in the database. This also works the other way round: data fields stored as `MP_BIN` are now decoded in Lua as varbinary objects, not as plain strings, as they used to be. Since the latter may cause compatibility issues, the new compat option `binarydatadecoding` was introduced to revert the built-in decoder to the old behavior (gh-1629). Introduced the new Lua module `trigger`. This module enables managing and calling triggers stored in trigger registry (gh-8656). Introduced the new `net.box` module function `from_fd` for creating a new connection from a socket file descriptor number (gh-8984). Introduced new option `inherit_fds` for `popen.new`. The option takes an array of file descriptor numbers that should be left open in the child process (gh-8926). Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and `socket:detach` (gh-8927). Introduced the `SHOW CREATE TABLE` statement (gh-8098). Non-system spaces in a snapshot file are now sorted by their IDs. As before, all non-system spaces are stored after system spaces (gh-7954). [Breaking change]* Dropped support for `IPROTO_CALL16` from `net.box` (i.e., the `call16` option and the `call16` method for `net.box` connections) (gh-8636). Added support for square brackets in procedure resolution for Lua calls (gh-8604). Added support for accepting IPROTO requests with a space or index name instead of an identifier (gh-8146). Added a new `box.info` parameter `hostname` (gh-8605). Added the `box.cfg.memtxsortthreads` parameter that specifies the number of threads used to sort indexes keys on loading a memtx database. OpenMP is not used to sort keys anymore (gh-3389). Changed the output of the `fselect` index method from a table of strings to a multi-line string and dropped the `print` and `use_nbsp` options. With the `yamlprettymultiline` compat option enabled by default, multi-line strings now look good in the console, so there's no need to return a table of strings to prettify the `fselect` output anymore. Allowed multiple updates to the same tuple field in a single call (gh-8226). Introduced `box.tuple.format` that enables format definition for tuples created via `box.tuple.new` (standalone tuples) (gh-4693). [Breaking change]* Disabled argument list syntax of `box.tuple.new` (this was needed for gh-4693). It is possible to switch to the old behavior using the compatibility option `boxtuplenew_vararg`. Sped up `index.select` and `index.pairs` with the `after` option by up to 30% in a synthetic test by eliminating an extra buffer allocation. The function `box.session.push` is now deprecated. Consider using `box.broadcast` instead (gh-8802). Introduced the new function `box.session.new` for creating a new IPROTO session from a socket file descriptor number (gh-8801). Introduced the new `luaeval` and `luacall` object types for `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval` allows the user to execute an arbitrary Lua expression with the `IPROTOEVAL`

Granting the `'execute'` privilege on `luacall` allows the user to execute any global user-defined Lua function with the `IPROTO_CALL` request (gh-8803). [Breaking change]* Introduced the new `sql` object type for `box.schema.user.grant`. Now only users with the `'execute'` privilege granted on `sql` or `universe` can execute SQL expressions with the `IPROTOEXECUTE` or `IPROTOPREPARE` requests. To revert to the old behavior (no SQL access checks), use the `sql_priv` compat option (gh-8803). The maximum length of `box.cfg{}` string parameters is now 512 instead of 256. Added a new flag `tarantool.build.asan` that shows whether build flag `ENABLE_ASAN` is set. Added the `leader_name` field to `box.info.election` (gh-8931). [Behavior change]* Disabled starting the Lua REPL by default when running Tarantool. Now, Tarantool yields the message that shows the command usage. To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via `stdin`, use dash (`-`) as the script name. For more information see a help message by running `tarantool -h` (gh-8613). Introduced new command-line options `--name` and `--config` (gh-8613). The following compatibility options' defaults were switched to new behavior: `yamlprettymultiline` `sqlseqscan_default` `jsonescapeforward_slash` `fiberchannelclose_mode` `fiberslicedefault` `boxcfgreplicationsynctimeout` More information on the new behavior can be found on the page. Initial version of the declarative server and cluster configuration (gh-8724). All vinyl options are now supported (gh-8861). All feedback options are now supported (gh-8861). The `memtx.sort_threads` option is now supported (gh-8861). The `bootstrap_leader` option is now supported (gh-8861). All flight recorder options are now supported (gh-8861). All security hardening `box.cfg{}` options are now supported by config (gh-8861). Added a low priority environment configuration source, which looks into `TT*DEFAULT` variables. It is useful to declare defaults (gh-8862). It is now possible to access configuration from the application script using the `config:get()` method (gh-8862). All metrics configuration options from `box.cfg{}` are now supported in the YAML config (gh-8861). Added `--help-env-list` CLI option (gh-8862). Introduced a non-public API for extending the declarative config functionality in Tarantool Community Edition (gh-8862). Improved the credentials applier: now it supports two-way synchronization of roles and privileges for both users and roles (gh-8861). Removed the sha1 and sha256 hash entries from the config credentials schema as not planned for development (gh-8967). Implemented a full password support in the `config.credentials` schema, including a password setting, updating and removal for the `chap-sha1` auth type (supported by both Tarantool Community Edition and Tarantool Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition where it is available) (gh-8967). A group name from a topology defined by the cluster configuration doesn't persisted anymore as `box.cfg.cluster_name` (gh-8862). The default directory and file paths are changed to `var/run/<...>`, `var/log/<...>`, `var/lib/<...>` and so on (gh-8862). The directory that contains the configuration file is now added into the Lua modules search paths (gh-8862). Most of vshard options are now added in the config (gh-9007). Added initial support of vshard (gh-9007). All audit options are now supported (gh-8861). Non-default privileges are now revoked from default users and roles when they are removed from the config (gh-8967). Introduced the initial support for roles - programs that run when a configuration is loaded or reloaded (gh-9078). Now multiline commands can be used by setting continuation marker using the `\set continuation` command. Introduced readline support to tarantool debugger (gh-7738). Introduced space type: a new space definition field. Introduced the fully temporary space type. It is the same as data-temporary but also has temporary metadata. Temporary spaces can now be created in read_only mode, they disappear after server restart and don't exist on replicas

Added a grammar-based SQL fuzzer (gh-4826). Tarantool does not depend on libgomp anymore (gh-7689). Added building static `deb` and `rpm` packages. Dropped building distro-specific `deb` and `rpm` Tarantool packages. Dropped building Tarantool packages with GC32. Now only GC64 packages are shipped. Exported all LuaJIT public functions from Tarantool binary (gh-3680). [Breaking change]* The key `box.space._schema['cluster']` is renamed to `'replicasetuuid'`. That is not expected to be breaking because `schema` is an internal system space, but the key was visible in public and documented (gh-5029). [Breaking change]* The table `box.info.cluster` is renamed to `box.info.replicaset`. The behaviour can be reverted using the `compat` option (gh-5029). Eliminated implicit conversion of unprintable utf-8 strings to binary blobs when encoded in YAML. Now unprintable characters are encoded as escaped utf-8 code points, for example, `\x80` or `\u200B` (gh-8756). Fixed decoding of escape sequences for single-byte character codes from YAML. Before the fix, single-byte character codes between `0x80` and `0xff` would be erroneously converted to two-byte UTF-8 code points, for example, `\x80` would be decoded as `\uC280` (gh-8782). Now `indexobject.parts.excludenull` always contains `false` rather than `null` when it is actually `false` (gh-8649). Fixed a crash that could happen when inserting an index definition with an empty parts list directly into `box.space._index` (gh-8688). Fixed the memory leak on unpacking an invalid MsgPack error extension (gh-8921). Fixed a use-after-free bug in fiber recycling code (gh-9020). Fixed a use-after-free bug in iproto server code (gh-9037). Fixed a heap-buffer-overflow bug in fiber creation code (gh-9026). An error is no longer raised on an attempt to print a dead fiber (gh-4265). Fixed a possible out-of-bound write in debug build on unpacking a MsgPack error extension (gh-9098). Fixed a bug that allowed to change replicaset UUID via `_schema` space. Fixed a bug when new instances could try to register via an anon instance which previously failed to apply `box.cfg{replication_anon = false}`. Fixed a bug when `box.ctl.demote()` with `box.cfg{election_mode = 'off'}` and an owned synchro queue could simply not do anything (gh-6860). Backported patches from the vanilla LuaJIT trunk (gh-8825). The following issues were fixed as part of this activity: Fixed `BC_UCLO` insertion for returns. Fixed recording of `BC_VARG` with unused vararg values. Initialization instructions on trace are now emitted only for the first member of a union. Prevent integer overflow while parsing long strings. Fixed various `^` operator and `math.pow()` function inconsistencies. Fixed parsing with predicting `next()` and `pairs()`. Fixed binary number literal parsing. Parsing of binary number with a zero fractional part raises error too now. Fixed load forwarding optimization applied after table rehashing. Fixed recording of the `BC_TSETM`. The default permission mode for `fio.open()` was changed for newly created files to 0666 (before umask) (gh-7981). Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause (ghs-119). Fixed the ability to drop `vinyldeferred_delete` system space (gh-5279). Fixed a bug when it was not possible to access spaces named `_stream` and `streamspacecache` and indexes named `space` using the `net.box` stream objects (gh-8598). [Breaking change]* The `cordslabcache` symbol was removed from the public API export (gh-7124). Fixed an error message if the cluster configuration was not provided or the instance was not found in the cluster configuration during reload (gh-8862). Support parent directories creation for options that accept a directory or a file (gh-8862). Create parent directories for `console.socket` and `log.file` (gh-8862). Create the `process.work_dir` directory (gh-8862). Consider all the paths as relative to `process.work_dir` when creating necessary directories (gh-8862).