Project: Tarantool (App Definition and Development / Database)
File: 4182-persistent-lua-functions.md

Status*: In progress Start date*: 10-05-2019 Authors*: Kirill Shcherbatov @kshcherbatov kshcherbatov@tarantool.org, Vladimir Davydov @locker vdavydov.dev@gmail.com, Konstantin Osipov @kostja kostja@tarantool.org Issues*: , Persistent Lua functions in Tarantool are Lua functions that are part of the schema and are thus immediately available after restart. Today Lua functions defined in Tarantool are a part of the runtime environment and must be defined again after program restart. We need to introduce a machinery to persist them (make them a part of the database state). This is a useful feature itself and moreover it is a dependency for functional indexes: all parts of a functional index definition must be persisted in database schema. Let's store function definition in _func and automatically load/define function on bootstrap (as well as replicate it). We need to extend _func:format with new fields: name: ``id``\ type: ``unsigned``\ Unique routine id name: ``owner``\ type: ``unsigned``\ The object owner name: ``name``\ type: ``string``\ collation: ``unicode_ci``\ Routine name name: ``stuid``\ type: ``unsigned``\ This makes Tarantool treat the functions caller as the functions creator. name: ``language``\ type: ``string``\ default: ``LUA``\ Procedure language - `LUA`, `C`, `SQL`(reserved) name: ``body``\ type: ``string``\ default: ``''``\ Function language-dependent body name: ``routine_type``\ type: ``string``\ default: ``FUNCTION``\ Type of registered object: `FUNCTION` or `PROCEDURE`; The procedure is a function that never returns result. name: ``param_list``\ type: ``map``\ is_nullable: ``true``\ default: ``{}``\ An array of maps describing arguments e.g. {name = `a`, type = `string`}. `{}`(undefined) by default. This field will be usefull when static field types validation is performed for function arguments. name: ``returns``\ type: ``string``\ default: ``any``\ A field_type-compatible string describing the returned value type. name: ``aggregate``\ type: ``string``\ default: ``NONE``\ Whether this routine is an SQL aggregate function: `NONE` or `GROUP`. name: ``sqldataaccess``\ type: ``string``\ default: ``NONE``\ Returns one of the following values:\ `NONE` = Function does not contain SQL.\ `CONTAINS` = Function possibly contains SQL.\ `READS` = Function possibly reads SQL data.\ `MODIFIES` = Function possibly modifies SQL data. name: ``is_deterministic``\ type: ``boolean``\ default: ``false``\ Whether the routine is deterministic (can produce only one result for a given list of parameters) or not. name: ``is_sandboxed``\ type: ``boolean``\ default: ``false``\ Whether the Lua routine should be executed in isolated environment with limited number of available modules. This option is compatible only with Lua function. name: ``isnullcall``\ type: ``boolean``\ default: ``true``\ Indicates whether the routine will be called if any one of its arguments is NULL. This option is compatible only with SQL

name: ``exports``\ type: ``array``\ default: ``{'Lua'}``\ An array of `enum('LUA', 'SQL')` strings - the Tarantool's frontends where new function must be also available. name: ``opts``\ type: ``map``\ default: ``{}``\ Options map (reserved). name: ``comment``\ type: ``string``\ default: ``''`` Comment associated with the routine. name: ``created``\ type: ``string``\ Date and time the routine was created (format `0000-00-00 00:00:00`). name: ``last_altered``\ type: ``string``\ Date and time the routine was modified (format `0000-00-00 00:00:00`). Initially `last_altered == created`. The updated interface to create a function in Tarantool is: ``` body_string = [[function(a, b) return a + b end]] box.schema.func.create('funcname', <ifnotexists = boolean [false]>, <setuid = boolean [false]>, <language = enum('LUA', 'C') ['LUA']>, <body = string [''], <is_deterministic = boolean [false]>, <is_sandboxed = boolean [false]>, <returns = string ['any']> <param_list = array [{}]>, <comment = string ['']>' }) ``` Persistent Lua function object must be compiled and expored into Lua on insertion into the system table (in contrast with a C function, which .so file is loaded on first use). This is necessary to export a name referencing an entry in the 'persistent' table and to safely construct a sandbox, as well as verify function body. The function body still may be invalid, or undefined, but there's nothing we can do about it. This impacts functional indexes, which need to evaluate the function to fill the index: such index may still not be constructible even if its function is defined. The function creation process could also be unsafe. To create a new function object, we must evaluate an expression given by a user. This expression may be invalid, for example `body = 'fiber.yield()'` instead of `body = 'function() return fiber.yield() end`. Tarantool thus will load such function definitions using a Lua sandbox. Persistent Lua functions mustn't refer to runtime environment objects like global variables or Lua modules: using these in a function may lead to a yield, event loop stalls, or break completely if such module or object is not available when a function is invoked. The function itself may leave traces on the global environment, which is also undesirable. To address these issues, Tarantool will for now only support functions with `is_sandboxed = true` property. For such functions it will create a unique sandbox via `setfenv` for each function object when loading

The sandbox will provide access to the following functions and modules: ``` assert, error, type, unpack, pairs, ipairs, pcall, xpcall, print, next, select, string, table, tonumber, math, utf8 ``` All Lua sandboxed functions are expected to be stateless. The reworked persistent Lua functions use the same security model as was implemented for functions not having a body. All access checks are performed on each function call. When ``setuid`` field is defined, the id of the user is a set-definer-uid one. Currently Tarantool has a ``box.internal.sqlcreatefunction`` mechanism to make Lua functions callable from SQL statements. ``` sqlcreatefunction("funcname", "type", funclua_object, funcargnum, is_deterministic); ``` That internally calls ``` int sqlcreatefunction_v2(sql db, const char zFunc, enum field_type returns, int nArg, int flags, void *p, void (xSFunc) (sql_context , int, sql_value ), void (xStep) (sql_context , int, sql_value ), void (xFinal) (sql_context ), void (xDestroy) (void )); ``` With prepared context ``` struct luasqlfunc_info { int func_ref; } funcinfo = {.funcref = luaLref(L, LUAREGISTRYINDEX);}; sqlcreatefunctionv2(db, normalizedname, type, funcargnum, isdeterministic ? SQLDETERMINISTIC : 0, funcinfo, luasql_call, NULL, NULL, luasqldestroy); ``` A persistent Lua function has everything what ``sqlcreatefunction_v2`` needs: ``func_lua.base.def.name``, ``funclua.luaref``, ``funclua.base.def.isdeterministic`` ``funclua.base.def.paramcount`` (it is the size of the ``param_list`` array) Therefore the internal ``box.internal.sqlcreatefunction`` endpoint becomes redundant and will be deleted. To control which Lua-functions are exported into SQL frontend, we would use `_func.exports` array. Only a function that has `'SQL'` string in this array is exported in SQL. The SQL subsystem has an own function hash and a function definition object FuncDef. It is inconsistent with Tarantool function definition cache and it is not scallable, so it will be reworked. SQL subsystem will use Tarantool function hash. SQL defines some names for builtins. They are: ``` TRIM, TYPEOF, PRINTF, UNICODE, CHAR, HEX, VERSION, QUOTE, REPLACE, SUBSTR, GROUP_CONCAT, JULIANDAY, DATE, TIME, DATETIME, STRFTIME, CURRENTTIME, CURRENTTIMESTAMP, CURRENT_DATE, LENGTH, POSITION, ROUND, UPPER, LOWER, IFNULL, RANDOM, CEIL, CEILING, CHARACTER_LENGTH, CHARLENGTH, FLOOR, MOD, OCTETLENGTH, ROW_COUNT, COUNT, LIKE, ABS, EXP, LN, POWER, SQRT, SUM, TOTAL, AVG, RANDOMBLOB, NULLIF, ZEROBLOB, MIN, MAX, COALESCE, EVERY, EXISTS, EXTRACT, SOME, GREATER, LESSER, SOUNDEX ``` (the functions are currently not defined as Tarantool UDFs but may appear in future) To avoid name clash, we will reserve these names by adding entries for them in `func` system space. `func.name` index collation will change to use unicode_ci. All built-ins will be added to the bootstrap snapshot. We also reserve service SQL method names ``` sqlstatget, sqlstatpush, sqlstat_init ``` required to work with SQL analyze statistics. SQL may call all functions having 'SQL' in their exports array. A new function `LUA` will be added to evaluate an arbitrary Lua string in an SQL request. ``` box.execute('SELECT lua(\'return box.cfg.memtx_memory\')') ```