Project: Tarantool (App Definition and Development / Database)
File: 3.0.0-beta1.md

Date: 2023-11-20 Tag: 3.0.0-beta1 3.0.0-beta1 is the version of the 3.0 release series. This release introduces 95 new features and resolves 51 bugs since the 2.11 version. There can be bugs in less common areas. If you find any, feel free to on GitHub. Notable changes are: New configuration capabilities (gh-8724) Fixes and improvements of the configuration process based on early adopters feedback. Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data layout, client-server protocol, and replication protocol. Please using the `box.schema.upgrade()` procedure to unlock all the new features of the 2.x series. [Breaking change] Disabled DDL operations with an old system schema. Now you have to run `box.schema.upgrade()` before you can execute any DDL operations (gh-7149). Now `index_object.parts` contains the following methods, similar to the `keydef` Lua module: `extractkey()`, `compare()`, `comparewithkey()`, `merge()` (gh-7356). Introduced the `box.watch_once()` function to get the value currently associated with a notification key on the local instance without subscribing to future changes (gh-6493). Introduced the new `IPROTOWATCHONCE` request to get the value currently associated with a notification key on a remote instance without subscribing to future changes (gh-6493). Introduced the `conn:watch_once()` net.box connection method to get the value currently associated with a notification key on a remote instance without subscribing to future changes. The new method is implemented using the `IPROTOWATCHONCE` request type (gh-6493). Introduced the default field values in the space format (gh-8157). Added support for functional variant of the default field values (gh-8609). `box.schema` can now be accessed before `box.cfg()`. Introduced new methods `tuple:info()` and `space:stat()` with the detailed information on memory consumed by data (gh-6762). Added memtx read view statistics to `box.stat.memtx()` (gh-8501). Introduced the `memory.tuple` statistic for `box.stat.vinyl()` that shows the total size of memory occupied by all tuples allocated by the Vinyl engine (gh-8485). Added a new `bootstrap_leader` configuration option to specify the node from which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` to `'config'` and set `bootstrap_leader` value to either the URI or UUID of the desired bootstrap leader. For example: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'localhost:3301', replication = { 'localhost:3301', 'localhost:3302', }, listen = '3302', } ``` (gh-7999). You may now control which node new replicas choose as a bootstrap leader without touching node config. To do so, set `box.cfg.bootstrap_strategy` to `'supervised'`, and the nodes will only bootstrap off the node on which you called `box.ctl.makebootstrapleader()` last. This works on an empty replica set bootstrap as well: start the admin console before configuring the nodes. Then configure the nodes: ```lua box.cfg{ bootstrap_strategy = 'supervised', replication = ..., listen = ..., } ``` Finally, call `box.ctl.makebootstrapleader()` through the admin console on the node you want to promote. All the nodes will bootstrap off that node (gh-8509). A new option `box.cfg.cluster_name` allows assigning a human-readable name to the entire cluster. It has to match in all instances and is displayed in `box.info.cluster.name` (gh-5029). A new option `box.cfg.replicaset_name` allows assigning a human-readable name to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value must be the same across all instances of one replicaset. The replicaset name is displayed in `box.info.replicaset.name` (gh-5029). A new option `box.cfg.instance_name` allows assigning a human-readable name to the instance. It works the same as `box.cfg.instance_uuid`. Its value must be unique in the replicaset. The instance name is displayed in `box.info.name`. Names of other replicas in the same replicaset are visible in `box.info.replication[id].name`

Instance at rebootstrap can change its UUID while keeping its numeric ID if it has the same non-empty instance name (gh-5029). Added the ability to set the `bootstrap_leader` configuration option to the instance name of the desired bootstrap leader: ```lua box.cfg{ bootstrap_strategy = 'config', bootstrap_leader = 'leader-name', replication = { ... }, ... } ``` (gh-7999, gh-8539). LuaJIT now can handle exceptions on traces (gh-7745). [Breaking change]* Added the new `varbinary` type to Lua. An object of this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so it can be used for storing binary blobs in the database. This also works the other way round: data fields stored as `MP_BIN` are now decoded in Lua as varbinary objects, not as plain strings, as they used to be. Since the latter may cause compatibility issues, the new compat option `binarydatadecoding` was introduced to revert the built-in decoder to the old behavior (gh-1629). Introduced the new Lua module `trigger`. This module enables managing and calling triggers stored in trigger registry (gh-8656). Triggers from `space_object`, `box.session`, and `box.ctl` were moved to the trigger registry (gh-8657). Introduced a new event `'tarantool.trigger.on_change'` in the trigger registry. It is called when any event in the trigger registry is modified (gh-8664). Introduced helpers for encoding and decoding IPROTO packets in Lua (gh-8054). Improved error reporting for `msgpack.decode`. Now, an error raised by `mgpack.decode` has a detailed error message and the offset in the input data. If `msgpack.decode` failed to unpack a MsgPack extension, it also includes the error cause pointing to the error in the extension data (gh-7986). Introduced the new `net.box` module function `from_fd` for creating a new connection from a socket file descriptor number (gh-8984). Introduced new option `inherit_fds` for `popen.new`. The option takes an array of file descriptor numbers that should be left open in the child process (gh-8926). Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and `socket:detach` (gh-8927). Introduced the `xlog.meta()` method for reading a meta block from an xlog file. Introduced the `SHOW CREATE TABLE` statement (gh-8098). [Breaking change]* The `DROP CONSTRAINT` statement has been improved (gh-9112). [Breaking change]* Names in SQL are now case sensitive. To support backward compatibility, a second lookup using a name normalized using the old rules is added (gh-4467). `SHOW CREATE TABLE` no longer supports the DEFAULT clause (gh-8793). `SQL_EXPR` functions can now be set as a default value (gh-8793). A literal set as the default value can no longer have a `+` or `-` sign unless the literal is numeric (gh-8793). [Breaking change]* SQL now uses the `box` mechanism for default values (gh-8793). Added the new `--force-recovery` CLI option (gh-8876). Non-system spaces in a snapshot file are now sorted by their IDs. As before, all non-system spaces are stored after system spaces (gh-7954). [Breaking change]* Dropped support for `IPROTO_CALL16` from `net.box` (i.e., the `call16` option and the `call16` method for `net.box` connections) (gh-8636). Added support for square brackets in procedure resolution for Lua calls (gh-8604). Added support for accepting IPROTO requests with a space or index name instead of an identifier (gh-8146). Added a new `box.info` parameter `hostname` (gh-8605). Added the `box.cfg.memtxsortthreads` parameter that specifies the number of threads used to sort indexes keys on loading a memtx database. OpenMP is not used to sort keys anymore (gh-3389). Changed the output of the `fselect` index method from a table of strings to a multiline string and dropped the `print` and `use_nbsp`

With the `yamlprettymultiline` compat option enabled by default, multiline strings now look good in the console, so there's no need to return a table of strings to prettify the `fselect` output anymore. Allowed multiple updates to the same tuple field in a single call (gh-8226). Introduced `box.tuple.format` that enables format definition for tuples created via `box.tuple.new` (standalone tuples) (gh-4693). [Breaking change]* Disabled argument list syntax of `box.tuple.new` (this was needed for gh-4693). It is possible to switch to the old behavior using the compatibility option `boxtuplenew_vararg`. Sped up `index.select` and `index.pairs` with the `after` option by up to 30% in a synthetic test by eliminating an extra buffer allocation. The function `box.session.push` is now deprecated. Consider using `box.broadcast` instead (gh-8802). Introduced the new function `box.session.new` for creating a new IPROTO session from a socket file descriptor number (gh-8801). Introduced the new `luaeval` and `luacall` object types for `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval` allows the user to execute an arbitrary Lua expression with the `IPROTOEVAL` request. Granting the `'execute'` privilege on `luacall` allows the user to execute any global user-defined Lua function with the `IPROTO_CALL` request (gh-8803). [Breaking change]* Introduced the new `sql` object type for `box.schema.user.grant`. Now only users with the `'execute'` privilege granted on `sql` or `universe` can execute SQL expressions with the `IPROTOEXECUTE` or `IPROTOPREPARE` requests. To revert to the old behavior (no SQL access checks), use the `sql_priv` compat option (gh-8803). The maximum length of `box.cfg{}` string parameters is now 512 instead of 256. Added a new flag `tarantool.build.asan` that shows whether build flag `ENABLE_ASAN` is set. Added the `leader_name` field to `box.info.election` (gh-8931). Now it is possible to specify the sort order for each part of an index (gh-5529). Added support for sending tuple formats in IPROTO responses. Added a `boxtupleextension` backward compatibility option to disable sending tuple formats in responses to IPROTOCALL and IPROTOEVAL requests (gh-8146). Added support for sending tuple formats in IPROTOCALL and IPROTOEVAL request arguments. `boxtupleextension` backward compatibility option can be used to disable receiving tuple formats in IPROTOCALL and IPROTOEVAL request arguments (gh-8633). [Behavior change]* Disabled starting the Lua REPL by default when running Tarantool. Now, Tarantool yields the message that shows the command usage. To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via `stdin`, use dash (`-`) as the script name. For more information see a help message by running `tarantool -h` (gh-8613). Introduced new command-line options `--name` and `--config` (gh-8613). The following compatibility options' defaults were switched to new behavior: `yamlprettymultiline` `sqlseqscan_default` `jsonescapeforward_slash` `fiberchannelclose_mode` `fiberslicedefault` `boxcfgreplicationsynctimeout` `cfunciproto_multireturn` More information on the new behavior can be found on the page. Initial version of the declarative server and cluster configuration (gh-8724). All vinyl options are now supported (gh-8861). All feedback options are now supported (gh-8861). The `memtx.sort_threads` option is now supported (gh-8861). The `bootstrap_leader` option is now supported (gh-8861). All flight recorder options are now supported (gh-8861). All security hardening `box.cfg{}` options are now supported by config (gh-8861). Added a low priority environment configuration source, which looks into `TT*DEFAULT` variables. It is useful to declare defaults (gh-8862). It is now possible to access configuration from the application script using the `config:get()` method (gh-8862). All metrics configuration options from `box.cfg{}` are now supported in the YAML config (gh-8861). Added `--help-env-list` CLI option (gh-8862). Introduced a non-public API for extending the declarative config functionality in Tarantool Community Edition

Improved the credentials applier: now it supports two-way synchronization of roles and privileges for both users and roles (gh-8861). Removed the sha1 and sha256 hash entries from the config credentials schema as not planned for development (gh-8967). Implemented a full password support in the `config.credentials` schema, including a password setting, updating and removal for the `chap-sha1` auth type (supported by both Tarantool Community Edition and Tarantool Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition where it is available) (gh-8967). A group name from a topology defined by the cluster configuration doesn't persisted anymore as `box.cfg.cluster_name` (gh-8862). The default directory and file paths are changed to `var/run/<...>`, `var/log/<...>`, `var/lib/<...>` and so on (gh-8862). The directory that contains the configuration file is now added into the Lua modules search paths (gh-8862). Most of vshard options are now added in the config (gh-9007). Added initial support of vshard (gh-9007). All audit options are now supported (gh-8861). Non-default privileges are now revoked from default users and roles when they are removed from the config (gh-8967). Introduced the initial support for roles - programs that run when a configuration is loaded or reloaded (gh-9078). On read-only instances, Tarantool now synchronizes credentials in the background when switching to read-write mode instead of skipping the synchronization (gh-8967). Now the permissions for space, function, and sequence can be granted via the config. If the object has not been created yet, the grant is postponed and executed after the creation of the object (gh-8967). Now the brand-new `luaeval`, `luacall`, and `sql` object types are supported by credentials (gh-8967). Now a password hash (and salt) will be regenerated for users managed in the configuration file if `security.auth_type` differs from a user's `auth_type` (gh-8967). Introduced validation for replicasetname/uuid and instancename/uuid mismatches before the recovery process when Tarantool is configured via a YAML file or etcd. If Tarantool is configured via a YAML file or etcd, then instance and replica set names are automatically set when possible (gh-8978). Added support for the `{{ replicasetname }}` and `{{ groupname }}` templates in addition to the existing `{{ instance_name }}` (gh-8862). Now multiline commands can be used by setting continuation marker using the `\set continuation` command. Introduced readline support to tarantool debugger (gh-7738). Exported `fibersetnamen`, `fibername`, `fiberid`, `fibercsw` and `fiber_find` into the public C API and usable via FFI as well. Make `fibersetjoinable`, `fibersetctx` and `fibergetctx` treat the NULL argument as the current fiber. Introduced space type: a new space definition field. Introduced the fully temporary space type. It is the same as data-temporary but also has temporary metadata. Temporary spaces can now be created in read-only mode, they disappear after server restart and don't exist on replicas (gh-8323). Added a grammar-based SQL fuzzer (gh-4826). Tarantool does not depend on libgomp anymore (gh-7689). Added building static `deb` and `rpm` packages. Dropped building distro-specific `deb` and `rpm` Tarantool packages. Dropped building Tarantool packages with GC32. Now only GC64 packages are shipped. Exported all LuaJIT public functions from Tarantool binary (gh-3680). Updated libcurl to version 8.3.0. Updated luarocks version to 3.9.2 (gh-6597). [Breaking change]* The key `box.space._schema['cluster']` is renamed to `'replicasetuuid'`. That is not expected to be breaking because `schema` is an internal system space, but the key was visible in public and documented (gh-5029). [Breaking change]* The table `box.info.cluster` is renamed to `box.info.replicaset`. The behaviour can be reverted using the `compat` option `boxinfocluster_meaning` (https://tarantool.io/compat/boxinfocluster_meaning) (gh-5029). Eliminated implicit conversion of unprintable utf-8 strings to binary blobs when encoded in

Now unprintable characters are encoded as escaped utf-8 code points, for example, `\x80` or `\u200B` (gh-8756). Fixed decoding of escape sequences for single-byte character codes from YAML. Before the fix, single-byte character codes between `0x80` and `0xff` would be erroneously converted to two-byte UTF-8 code points, for example, `\x80` would be decoded as `\uC280` (gh-8782). Now `indexobject.parts.excludenull` always contains `false` rather than `null` when it is actually `false` (gh-8649). Fixed a crash that could happen when inserting an index definition with an empty parts list directly into `box.space._index` (gh-8688). Fixed the memory leak on unpacking an invalid MsgPack error extension (gh-8921). Fixed a use-after-free bug in fiber recycling code (gh-9020). Fixed a use-after-free bug in iproto server code (gh-9037). Fixed a heap-buffer-overflow bug in fiber creation code (gh-9026). An error is no longer raised on an attempt to print a dead fiber (gh-4265). Fixed a possible out-of-bound write in debug build on unpacking a MsgPack error extension (gh-9098). Fixed a bug that caused a replication error after calling `box.schema.downgrade` (gh-9049). Fixed a possible crash when unpacking an invalid MsgPack error extension (gh-9136). Fixed a wrong assertion in index comparators when comparing decimals with floats greater than `1e38`. The error was present only in the debug build. Despite the failing assertion, the behavior after the assertion was correct (gh-8472). Now the "Type Ctrl-C to exit Tarantool" message is displayed when a local console is exited, but background fibers are running (gh-7017). Fixed a bug that could result in the incorrect `space:bsize()` when altering a primary index concurrently with DML operations (gh-9247). Fixed a crash that happened while printing the stack trace on a stack overflow bug (gh-9222). Fixed the behavior of `log.cfg{modules = ...}`. Now, instead of merging the new log modules configuration with the old one, it completely overwrites the current configuration, which is consistent with `box.cfg{log_modules = ...}` (gh-7962). Fixed a bug in the network buffer that could result in a crash when there are a lot of pending requests (gh-9218) Fixed a potential data corruption when the number of tuples in a hash index is more than 2147483648 (gh-3594). Decremented the max space id (`box.schema.SPACE_MAX`). Now, the max space id equals 2147483646. The limit was decremented because the old value is used as an error indicator in the box C API. It's still possible to revert to the old behavior with the compatibility module option `boxspacemax` (gh-9118). The `tarantool -v` output and `tarantool.build.flags` string now include build type-specific and LTO flags if any (gh-8022). Fixed a possible inconsistent state entering if fibers are joined incorrectly. Now the `fibersetjoinable` function panics if the fiber is dead or joined already. The `fiberjoin` and `fiberjoin_timeout` functions now panic on a double join if it is possible to detect it (gh-7562). Fixed a bug that allowed to change replicaset UUID via `_schema` space. Fixed a bug when new instances could try to register via an anon instance which previously failed to apply `box.cfg{replication_anon = false}`. Fixed a bug when `box.ctl.demote()` with `box.cfg{election_mode = 'off'}` and an owned synchro queue could simply not do anything (gh-6860). Fixed a false-positive split-brain in a replica set on the first promotion after an upgrade from versions before 2.10.1 (gh-8996). Fixed replicas writing corrupted xlogs when appending data to a local space from an `onreplace` or `beforereplace` trigger on a global replicated space. Such xlogs were unrecoverable and caused other nodes to break replication with the replica (gh-8746, gh-8958). Backported patches from the vanilla LuaJIT trunk

The following issues were fixed as part of this activity: Fixed `BC_UCLO` insertion for returns. Fixed recording of `BC_VARG` with unused vararg values. Initialization instructions on trace are now emitted only for the first member of a union. Prevent integer overflow while parsing long strings. Fixed various `^` operator and `math.pow()` function inconsistencies. Fixed parsing with predicting `next()` and `pairs()`. Fixed binary number literal parsing. Parsing of binary number with a zero fractional part raises error too now. Fixed load forwarding optimization applied after table rehashing. Fixed recording of the `BC_TSETM`. Fixed the panic routine when `mprotect` fails to change flags for mcode area. Fixed frame for on-trace OOM handling. Fixed handling of instable types in TNEW/TDUP load forwarding. Handled table unsinking in the presence of `IRFLTABNOMM`. Fixed a bug when an error could be raised on the non-currently executed coroutine (gh-6323). Backported patches from the vanilla LuaJIT trunk (gh-9145). The following issues were fixed as part of this activity: Fixed error handling after return from a child coroutine. The default permission mode for `fio.open()` was changed for newly created files to 0666 (before umask) (gh-7981). Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause (ghs-119). Fixed a memory leak in a parser for a column's default rule (gh-9159). Fixed assertion in a debug build when a collation was added after an index with more than one field (gh-9229). Fixed a bug that in some cases would not assign collations to an index created during `CREATE TABLE` (gh-9229). A space no longer loses a default value after `ALTER TABLE ADD COLUMN` (gh-8793). Fixed the ability to drop `vinyldeferred_delete` system space (gh-5279). Fixed a bug when it was not possible to access spaces named `_stream` and `streamspacecache` and indexes named `space` using the `net.box` stream objects (gh-8598). [Breaking change]* The `cordslabcache` symbol was removed from the public API export (gh-7124). All DDL functions from the `box.schema` module are now wrapped into a transaction to avoid database inconsistency on failed operations (gh-4348). Fixed crashes if `box.info.memory()`, `box.info.gc()`, `box.info.vinyl()`, and `box.info.sql()` are called before `box.cfg{}` (gh-9173). [Breaking change]* Added a `cfunciproto_multireturn` option to the `compat` module. The new behavior drops an additional array that wraps multiple results returned via iproto (gh-4799). Fixed a bug that allows downgrading from a schema version more recent than a Tarantool version (gh-9182). Fixed a bug when `on_shutdown` triggers weren't run if `os.exit()` was called from `-e` command-line option (gh-9266). [Breaking change]* `box.schema.user.grant()` now raises an error on an attempt to grant the `execute` privilege on a space. Historically, this action was allowed although it had no effect. It's still possible to revert to the old behavior with the new compatibility option `boxspaceexecute_priv` (gh-9277). Added a check that disables setting `box.cfg` and `log.cfg` options to infinite numbers (NaN, Inf). Setting a `box.cfg` or `log.cfg` option to an infinite number could result in a crash or invalid behavior (gh-4962). Fixed an error message if the cluster configuration was not provided or the instance was not found in the cluster configuration during reload (gh-8862). Support parent directories creation for options that accept a directory or a file (gh-8862). Create parent directories for `console.socket` and `log.file` (gh-8862). Create the `process.work_dir` directory (gh-8862). Consider all the paths as relative to `process.work_dir` when creating necessary directories (gh-8862). Added the validation of instance, replica set, and group names in the configuration, `--name` CLI option, and `TTINSTANCENAME` environment variable (gh-8862). Fixed a streaming connection stuck if etcd is stopped unexpectedly (gh-9086).