Durable functions in database systems

A database system and a computer implemented method for managing functions in the database system is provided. The method, implemented using instructions that may be stored in the database system, involves obtaining first data representing a set of one or more operations to be performed on data in the database system. Second data, representing a function definition, is derived from the first data and the function definition includes the set of one or more operations and a set of state variables. A function is generated according to the second data. Generating the function includes storing a set of one or more values for respective ones of the state variables. The set of values are stored in an isolated computing environment in the database system and the function is configured to operate on the values in the isolated computing environment.

BACKGROUND OF THE INVENTION

Field of the Invention

The present application relates to database management systems, and more specifically, methods and systems for implementing functions in relational database management systems.

Description of the Related Technology

As technologies advance, the amount of information stored in electronic form, the desire for the real-time, or pseudo real-time, ability to search, organize, and/or manipulate such information is ever increasing. Database management systems, sometimes also referred to as database systems, databases, datastores, and data warehouses, organize data in a form that facilitates efficient search, retrieval, and/or manipulation of selected information. Typical database management systems allow a user to submit a “query” or call one or more functions in a query language for searching, organizing, retrieving, and/or manipulating information that satisfies particular conditions.

Certain databases may be transactional, which is to say their primary purpose is to record transactions. Such transactions may be thought of as one or more operations performed on data, which are recorded in logs. A log may comprise a continuous stream of log records, each of which corresponds to a transaction. This may allow transactions to be replayed or undone following an event such as a system crash. Certain databases may additionally, or alternatively, be analytical, which is to say their purpose is to execute queries and generate analytics on data stored in the database.

Demands on database systems may vary and to handle increased demand, database systems may be scaled up (and down) by increasing (and decreasing) the resources of an existing server by increasing (or decreasing) the memory or upgrading the CPUs. Scale-out database systems increase the capacity by adding new nodes, for example in the form of new machines, to the database system.

User defined workloads may be deployed to operate on data stored in a database. Implementing these workloads in a database system presents challenges when trying to allocate resources, managing the interaction between workloads, and maintaining data consistency and durability across the database system. It would be desirable to provide a system which is capable of flexibly and efficiently handling complex workloads which operate on data stored in a database for a plurality of users.

SUMMARY

According to a first aspect of the present disclosure there is provided a computer-implemented method for managing functions in a database system, the method comprising: obtaining first data representing a set of one or more operations to be performed on data in a database managed by a database system; deriving second data from the first data, the second data representing a function definition for implementation in the database system, wherein the function definition includes the set of operations and a set of one or more state variables; and generating a function according to the second data, wherein generating the function comprises storing a set of one or more values for respective ones of the state variables, wherein the set of values are stored in an isolated computing environment in the database system, and the function is configured to operate on the set of values in the isolated computing environment.

The computer-implemented method of the present disclosure aims to provide a method which enables functions to be implemented in a database system which can be readily paused, upgraded, and redeployed. In particular, providing a respective set of values in an isolated computing environment which are modifiable by the function allows the state and operating status of the function to be durably maintained. Other functions, or users, may be able to query these values to determine the state and operating status of the function, thereby allowing further functions or workloads to operate with and instruct the function in a way which maintains consistency and durability of the function and its associated values. These functions can be paused, for example, by first determining whether there are any active operations being implemented by the function, and subsequently issuing an instruction to cease performing operations. The function can also be redeployed in a different database system, or on a different set of computing resources, for example, by copying the function from one set of computing resources to another, and by transferring the set of values from the isolated computing environment to a different isolated computing environment on the different set of computing resources. Providing functions in this way enables these functions to be paused, updated, and redeployed effectively, allowing the database system to more easily manage computational resources, adhere to relevant jurisdictional data protection laws, and react quickly to user requests, such as updates and/or modifications to these functions. These functions may be durable in the sense that each respective set of values is durably maintained in an isolated environment, the values are operated on with durable transactability, and the functions can be redistributed and a redeployed within the database system in way which maintains consistency and the characteristics of the functions.

According to a second aspect of the present disclosure there is provided a database system for implementing one or more functions which are configured to operate on data stored in the database system, the database system comprising at least one processor; and storage comprising computer executable instructions which, when executed by the at least one processor, cause the at least one processor to: obtain first data representing a set of one or more operations to be performed on data in a database managed by a database system; derive second data from the first data, the second data representing a function definition for implementation in the database system, wherein the function definition includes the set of operations and a set of one or more state variables; and generate a function according to the second data, wherein generating the function comprises storing a set of one or more values for respective ones of the state variables, wherein the set of values are stored in an isolated computing environment in the database system, the function being configured to operate on the set of values in the isolated computing environment.

According to a third aspect of the present disclosure there is provided a non-transitory computer-readable storage medium comprising computer executable instructions which, when executed by at least one processor, cause the at least one processor to: obtain first data representing a set of one or more operations to be performed on data in a database managed by a database system; derive second data from the first data, the second data representing a function definition for implementation in the database system, wherein the function definition includes the set of operations and a set of one or more state variables; and generate a function according to the second data, wherein generating the function comprises storing a set of one or more values for respective ones of the state variables, wherein the set of values are stored in an isolated computing environment in the database system, the function being configured to operate on the set of values in the isolated computing environment.

DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS

Details of systems and methods according to examples will become apparent from the following description, with reference to the Figures. This description includes numerous specific details of certain examples for the purpose of explanation. Reference in the specification to “an example” or similar language means that a particular feature, structure, or characteristics described in connection with the example is included in at least that one example, but not necessarily other examples. It should be further noted that certain examples are described schematically with certain features omitted and/or necessarily simplified for ease of explanation and understanding of the concepts underlying the examples.

A database may generally be considered as an organized collection of data stored electronically in a computer system. The computer systems which store and manage databases and the data stored therein are commonly referred to as database management systems, or database systems. As well as storing databases, database systems may also be configured to perform certain management functions such as replicating the databases to provide resiliency to crashes and performing operations to record new data in the database and/or to modify data already stored in the database. Some database management systems may also provide an interface to enable users to implement workloads which are configured to perform operations on data stored in the database.

Workloads are typically computer programs or applications which are configured to perform one or more operations. In some cases, workloads may be simple, for example, a workload may be a very specific program for performing one or more operations, such as reading, writing, or otherwise modifying data in a database system. Alternatively, workloads can be complex programs which are configured to implement many functions, each function being configured to perform one or more low level operations on data stored in the database. For example, a simple workload may be a program which is configured to go through a database and modify a single data value type for a plurality of entries in the database. An example of a more complex workload may be a program written to manage and perform analysis on data for a large number of clients, or users, whose data is stored in a database. A complex workload such as this may include functions for monitoring the status of each client whose data is stored in the database, functions which are configured to run analysis on client's data in the database, and functions which are configured to manage the implementation of other functions in the database system to ensure concurrency, consistency, and efficient distribution of computing resources in the database system.

In general, workloads which are configured to perform operations on data stored in a database may be implemented on computing resources which are separate to the database system and may use an interface provided by the database system to access data stored in the database. Alternatively, these workloads may be deployed within these database systems. As technologies advance, the amount of data stored in databases and the complexity of workloads operating on this data is ever increasing.

Complex workloads, such as workloads which perform machine learning and artificial intelligence operations, can be long running and may compete for computing resources with other workloads. Additionally, these workloads may need to be retrained, and redeployed, either within the same database system or to other database systems. The long running nature of these workloads present challenges when attempting to pause, redeploy, and modify these workloads. Long running workloads are generally very computationally expensive to start from scratch, can lose valuable data generated during the operation of the workload when being paused or canceled, and/or can fail to adequately process real-time or pseudo real-time events that the database system is configured to analyze and transact. Pseudo real-time events may generally be events which logically appear to occur in real-time in a computer system although, may be related to events which, temporally, occur asynchronously.

Certain examples described herein aim to provide a database system and methods for managing functions in a database system which enable functions to be more easily paused, redeployed, updated, and monitored. The functions described herein are provided with the ability to control and modify respective values which correspond to state variables defined in the function. These values are isolated in the database system such that they are protected from being modified by other functions. Isolating these values simplifies the management of these functions by making it easier to identify the state of the function and to copy or move the function between computing resources in the database system.

FIG.1shows a database system100comprising a set of one or more processors102, storage104, and one or more communications modules106. The processor(s)102, storage104, and communications module(s)106are connected over an interface108, for example, a bus. The processor(s)102includes any suitable combination of processing circuitry which is configurable to perform the functions described herein with reference toFIGS.2to12. For example, the processor(s)102may include central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), fixed function processing circuitry, any other suitable processing circuitry, and/or any combination of these.

The storage104may include a combination of volatile storage, or memory, and non-volatile storage, which may be referred to as disk storage. The volatile storage may include a combination of random access memory (RAM), static random-access memory (SRAM), and/or types of dynamic random-access memory (DRAM). While the volatile storage has been described in relation to the storage104in some cases, part or all of the volatile storage may be directly attached to or integrated with the processor(s)102.

Non-volatile storage, while sometimes being referred to as “a disk” or “disk storage”, may include any suitable combination of non-volatile storage types. For example, the non-volatile storage may include hard disk drives, solid state drives, flash memory, magnetic tape storage, or any combination of these storage types. In some cases, the non-volatile storage may include hierarchical storage, in which different storage types are used to store data according to a respective age and/or frequency of access of the data. For example, solid state drives may be used to store data which is frequently accessed, or which is likely to be accessed frequently. While hard disk drives, or tape storage, which have a higher read/write latency than solid state drives or flash memory may be used to store data which is older or less likely to be accessed frequently.

The storage104is used to store computer executable instructions110, and a database112. The computer-executable instructions110, when executed by the processor(s)102may implement one or more programs. These programs may include general database management programs for maintaining data in the database112, performing backup and replication processes, providing an interface to the database112, and so forth. The computer-executable instructions110may also include instructions for implementing user defined workloads. The database112included in the storage104may include a collection of structured data in the form of a combination of row store format and column store format data, as well as collections of unstructured data such as binary large objects.

The communications module(s)106may include any suitable combination of network enabled communications modules and/or user interfaces. For example, the communications module(s)106may include network interfaces to enable the database system100to communicate with one or more further computer systems over a network such as a local area network, a wide area network such as the internet, and so forth. The communications module(s)106may alternatively, or additionally, include user interfaces such as input and output devices such as a screens, keyboards, touch screens and so forth.

While the database system100is shown inFIG.1as being a single device, it will be appreciated that the database system100may comprise a collection of separate computing systems. For example, the database system100may be implemented on one or more collocated servers, or cluster computers. Alternatively, the database system100may be a distributed database system, being implemented as a plurality of computer, or servers, located remotely from one another and communicatively coupled over a network.

The computer-executable instructions110include a set of instructions114which, when executed by the processor(s)102cause the database system100to perform a computer-implemented method200for managing functions in a database system100shown inFIG.2and which will be described further below in relation toFIGS.2to12.

Referring toFIGS.2and3, method200comprises obtaining202first data300representing a set of one or more operations302to be performed on data in a database112managed by a database system100. Obtaining202the first data300may include receiving the first data300over the one or more communications modules106. For example, the first data300may be received over a network from a user operating a terminal or external computing device which is configured to interface with the database system100. A user may write a program in a high-level computer programming language, such as Python, or Java, C++, and so forth, and upload this program to the database system100via the communications module(s)106.

Alternatively, the first data300may alternatively be distributed by a centralized management system which is configured to distribute first data300to one or more database systems100. For example, the first data300may represent a set of one or more operations302which are to be implemented in a variety of different database systems which are provided and/or managed by a centralized database management service.

In other examples, obtaining202the first data300may include reading the first data300from storage104, such as non-volatile storage, in the database system100. Where the database system100comprises a user interface, a developer environment may be provided in the database system100allowing users to write programs, including writing a program represented by the first data300, directly in the database system100. This first data300may be written into storage104and obtained202from the storage when the method200is implemented.

Second data304representing a function definition306for implementation in the database system100is derived204from the first data300. The function definition306represents information which is used to create specific instances of the function in the database system100. Instances of the function definition306are generally referred to as functions. The function definition306includes the set of operations302and a set of one or more state variables308.

As described above, the set of operations302may include low level operations to be performed to data in the database, such as reading, writing, and/or otherwise modifying the data. These operations302may also include more sophisticated operations which each involve reading multiple data values from the database, operating on and/or with these data values and storing results in the database.

The set of one or more state variables308represent variables which are used to define each instance of the function generated from the function definition306and may represent the data on which the operations in the function are configured to operate. The set of state variables308in the function definition306may be implemented as part of a schema which defines the set of state variables308and a data structure within which the state variables308are implemented and mapped.

By way of illustration, one implementation of a function definition306may be where a company uses the function definition306to establish a system for managing client's data and lifecycle in the database system100. According to this example, the set of one or more state variables308may represent data associated with an individual, such as age, height, location, a subscription type, and so forth. The set of operations302in this case could represent operations for example to modify these data, to make predictions regarding the individual based on these data, or to cause certain operations or events to be performed in response to certain conditions being represented by the individual's data.

Deriving the second data304may in some cases include processing the first data300and/or converting the data into a format which is suitable for implementation in the database system100. As described above, the first data300may represent a program written by a developer, the program including the set of operations302. The developer may write this program in a language of their choosing, for example, in a high-level computing language such as Python, C++, Java, and the like. Where the database system100is configured to implement such languages, deriving204the second data304may be a simple operation to copy, move, and or compile the first data300such that it can be implemented in the database system100. In other examples, deriving204the second data304may be a more complicated process involving identifying the set of operations302, any associated variables, and generating a function definition306which is suitable for implementing in the database system100, for example written in a language and/or using data structures which are suitable for the database system100.

A function310is generated206according to the second data304; this may involve storing a set of one or more values312for respective ones of the state variables308. The set of values312may be data values on which the one or more operations302may be performed. For example, the set of values312may be user data relating to users of the database system100, or their clients.

InFIG.3the set of values312are each represented with a subscript numeral which corresponds to a subscript numeral of the respective variable308. Generating206the function310may be triggered by a request to generate a new function310. In the request to generate206a new function310, an initial set of values312, each relating to a respective state variable308, may be provided. Taking the example provided above in which the function definition306defines a function310which stores, manages, and controls an individual's data and lifecycle, the request to generate a function310may be triggered when e.g. a new customer signs up to the services provided by the company. The customer would provide an initial set of values including their age, height, location, and a selected subscription type. A function310, associated with this customer, may then be generated206and these values312are stored.

In preferred arrangements the set of values312are stored in an isolated computing environment314in the database system100and the function310is configured to operate on the set of values312in the isolated computing environment314. This prevents the set of values312from being modified by functions which are not authorized to do so and/or on the request of an unauthorized user. By restricting the ability to operate on and/or modify the set of values312to the function310, the durability and consistency of the set of values312can be maintained.

Further, by simplifying the number of functions which are able to modify the set of values312, it becomes easier to move or modify the set of values312and the function310to a different set of computing resources in the database system100while ensuring that they are not actively being operated on, which might otherwise cause errors or data corruption.

The function310may be configured to operate on the set of values312in a way which maintains characteristics such as atomicity, consistency, isolation, and durability (ACID) so that the integrity and transactability of the set of values312are maintained. For example, operations may be performed by the function in units of compute referred to as transactions, which are atomic in the database system100. A transaction may involve one or more low level operations. When performing a transaction, the individual operations included in the transaction may be performed logically, in memory, and only persisted to disk when the full transaction has been completed, making the operations permanent in the database system100. Persisting the transaction to disk, and performing adequate backup procedures, may ensure durability in the database system100. Once a transaction is completed it is said to be committed in the database system100. As the set of values312are only operated on by transactions performed by the function310, the transactions are isolated, meaning that the set of values312may not be subjected to two or more operations at the same time. Isolation and consistency may also be supported by configuring the function310to perform operations asynchronously, which will be described further below with reference toFIG.7.

The set of values312may be configured to allow one or more further functions in the database system100to query the set of values312such that these further functions are able to read the values312but cannot perform operations on or modify the set of values312. In this way, it is possible for other functions in the database system100to monitor the set of values312associated with the function310while also ensuring that the permission to modify these values312is restricted thereby simplifying the management of the set of values312in the database system100and enabling the system to maintain the ACID characteristics described above.

The set of values312may be stored in a relational data table, referred to as a state table316, which is able to be queried using appropriate computing instructions. The data structure and mapping of the state table316used to store the set of values312may be defined in the function definition306as the state variables308. Where the set of values312are stored in a state table316, the values can be queried by suitably authorized functions and/or systems, for example, using structured query language (SQL).

The isolated computing environment314may be implemented in a number of ways. In a first example, the isolated computing environment314is implemented using a private table, e.g. the state table316may be a private table. The function310may be configured with authorization to modify the set of values312stored in the private table. For example, the private table may require a key, or other type of identification, to be presented before allowing values stored therein to be processed or modified. The function310may be configured with this key or other type of identification. Where the isolated computing environment314is implemented using a private table, the table may be stored in a database comprising a plurality of private tables each associated with a respective function.

In a second example, the isolated computing environment314is implemented using a virtual machine. The function310may be configured to interface with the virtual machine and/or may be implemented using the same virtual machine on which the values312are stored. In other examples, the set of values312may be logically isolated in the database system such that only the function310has information, such as a schema, needed to locate the set of values312in order to operate on and/or modify them.

Storing the set of values312in an isolated computing environment314and configuring the function310to operate on the set of values312in the isolated computing environment may involve providing the function310with control over the set of values312. This control may be exclusive such that the function310has sole authority to modify the set of values312in the isolated computing environment314.

By storing a set of values312in an isolated computing environment314and configuring the function310to operate on the set of values312, control over the set of values312may be simplified. Providing a system in which only the function310, or otherwise authorized functions, which is associated with the set of values312is able to operate on them makes it possible to determine the current state of the function and its associated values quickly and consistently. This is because it is neither necessary to query a set of data values which are distributed in tables throughout the database, nor necessary to confirm that any other functions which have access to the set of data values312are actively operating on these values312.

Simplifying the access and control over the set of values312which are relevant to the function310in this way makes it possible to quickly pause, modify, redeploy, and/or redistribute the function310within the database system100without having to manage other functions which may otherwise be concurrently operating on the set of values312.

The function310shown inFIG.3includes the set of operations302as defined in the function definition306. However, it will be appreciated that including the set of operations302may involve storing a reference to the set of operations302in the function310. When performing one of the operations302the function310may call the operation from the function definition306and provide values and/or references to values when calling the operation.

At least one of the set of operations302may be configured to update one or more of the set of values312stored in the isolated computing environment314. In this way the function310provides a way to update the set of values312. Other functions which need to modify the set of one or more values312may do so using an operation provided in the function310. Users, or other functions which are separate to the function310may send a request to the function310, in order to update the set of values312.

FIG.4shows an example of the database system100in which the method200has been performed and the database system100comprises a plurality of functions310A to310D each of which is configured to operate on a respective set of values stored in respective isolated computing environments314A to314D. As described above, these isolated computing environments314A to314D may be private tables in a database, isolated physical computing resources in the database, and/or isolated virtual machines in the database system100. Each of the functions310A to310D may include a reference to the function definition306from which they are generated. In this way, changes to the function definition306may be correlated to the functions310A to310D. Further, it becomes possible to readily identify the set of operations302and the state variables308which are associated with the functions310A to310D. Each of the functions310A to310D may comprise a respective unique identification number, or ID, allowing the function310A to310D to be referenced, called, or queried by other functions operating in the database system100.

The method200may involve storing an execution log502, as shown in the example ofFIG.5, comprising data indicative of operations302having been performed by the function310. The function310may be configured to write data to the execution log502that is indicative of operations which are performed by the function310, for example in the form of log records. Each log record may store data representing a specific operation, or unit of compute. By storing records of the operations302performed by the function310it becomes possible to monitor past and ongoing operations performed by the function310and to identify whether operations have been successfully performed by the function310, and/or whether the operation requires attention from an administrator. An administrator in this context may be a user who is actively monitoring the database system100and/or an automated function configured to identify errors, or stalled operations and to act accordingly to mitigate and/or correct these occurrences in the database system100.

In some examples, the execution log502used to store data indicative of operations performed by the function310may be unique to the function310. That is to say that the execution log502may only record operations performed by the function310and may not include data indicative of operations performed by other functions in the database system100. Alternatively, the execution log502may be used to store data indicative of operations performed by a plurality of functions310A to310D which are implemented in the database system100. In this case, the execution log502may be used to store data indicative of operations performed by all functions310A to310D in the database system100or to store data indicative of operations performed by a subset of functions310A to310D implemented in the database system100. The subset of functions310A to310D may include functions of the same type and/or functions310A to310D associated with the same respective function definition306.

Operations302defined in the function310may be event-driven, which is to say that operations302may be called by the function310in response to events. To this end, each operation may be associated with one or more events, such that following an event a respective operation can be identified to be performed.FIG.6shows an example in which the method200involves configuring the function310to obtain an indication of an event602and, in response to the indication602, perform at least one associated operation. Each operation may be associated with a respective event ID which may correspond to an event ID included in the indication602.

On completion, the operation generates an output. The output may include data used to modify604one or more of the set of values312and/or data which is provided to one or more further functions. For example, the output of an operation performed by the function310may be an event which causes a further function, not shown inFIG.6, to perform a further operation.

The operation performed by the function310in response to the indication602may be recorded606in the execution log502by writing data indicative of the operation in the execution log502. The data written to the execution log502may include any one or more of an indication of the operation which is performed, an indication of whether the operation is successfully performed, an indication of an output of the operation, and/or an indication of the event associated with the operation. Writing data to the execution log502representing the successful or unsuccessful performance of the operation makes it possible to easily identify whether an operation has completed, is ongoing, is waiting for an input, and/or is competing for resources in the database system100and is unable to be completed.

Data indicative of one or more events which occur in the database system100may also be stored in the execution log502. For example, when an indication602of an event is obtained by a function the execution log502may be updated with data indicative of this event. By storing data indicative of operations performed by the function310and data indicative of the events which occur in the database system100it becomes possible to identify where an operation, which is expected to be performed in response to a given event, is not successfully called and/or does not successfully complete. This can be used to identify where a function310is struggling to compete for computational resources to implement a given operation. As well as records of operations and events, the execution log502may be used to store further metadata associated with the function310for diagnostic and analytical purposes. General signal traffic and communications sent to and from the operation may also be stored in the execution log502which can be reviewed by an administrator to tune or otherwise reconfigure the database system100.

The aforementioned events may include any one or more of an input from a user, an instruction from one or more further functions operating in the database system100, an output from one or more further functions operating in the database system100, or an output from another operation performed by the function310.

Inputs from a user may include a user providing an instruction, over a user interface and/or the communications module(s)106, to the function310to perform one or more operations of the set of operations302. The database system100may convert the input from the user into an indication of an event602, for example, including a relevant event ID which can be used to identify which of the operations302to perform in response to the event and, if applicable, input data to be used when performing one of the set of operations302.

Some functions310may include operations302which are dependent on calling one or more further operations either included in the same function310or in other functions in the database system100. For example, where the function310relates to managing the data and lifecycle of a particular individual, one of the operations of the set of operations302may involve generating a prediction of one or more values associated with the individual at some future point in time. In this case, generating a prediction may require one or more further functions operating in the database system100to provide an output which is used in generating the prediction for the individual associated with the function310. The function310may send an indication of an event, representing a request for a specific output, to one or more further functions in the database system100which in turn is used by these one or more further functions to identify and perform a respective operation. On completion an output from this operation is returned to the function310allowing it to complete its prediction.

As described above, the output of an operation from within the function310or from a further function in the database system100can serve as an event causing one or more further operations to be performed by the function310. In this way, operations302and/or functions310may be nested in the database system100meaning that operations302may wait for the output of one or more further operations to be completed before completing. This allows complex operations, which might otherwise be implemented as multi-threaded programs to instead be implemented in a linear fashion using a single thread in the database system100, as will now be explained.

The function310, and other functions310A to310D generated according to the second data304, may be configured to perform operations asynchronously, as shown inFIG.7. This means functions can readily be paused, updated, and redeployed. In the example illustrated inFIG.7there are several active operations702,704, and706but only one operation is processed at any one time. For example, several events may occur in the database system100causing the function310to start performing respective operations702,704, and706. In some instances, none of the respective operations702,704, and706may have priority over the other operations702,704, and706and so the function310may process the operations702,704, and706on a first come first serve basis, wherein the operation702corresponding to the first indication of an event which is received by the function310is performed first. The other operations704,706may be paused while the first operation702is being performed. In some instances, an operation704will not be able to be completed until an input from another operation is received, or until certain data is made available to the operation704. Where this is the case, the operation704may be paused until the input is received. While the operation704is paused, the function310may continue with performing another one of the active operations706. Using single-threaded processes in the database system100in this manner simplifies the implementation of functions310in the database system100and enables the database system100to cancel or pause operations when redeploying or upgrading the function310while mitigating any potential errors which would otherwise affect the consistency and/or durability of the set of values312.

Alternatively, performing operations302asynchronously may include only having one operation active at any one time and queuing further operations which are to be activated on completion of a present operation.

Referring toFIG.8, function310may be implemented using a stack-based virtual machine800, which is to say a virtual machine which processes data using an interaction which involves using push and pull operations to move short-lived temporary values to and from a push down stack according to a last in first out data structure. In this case the method200may involve configuring the function310to create a stack802in response to obtaining the indication of the event602, performing the operation702using the stack802, and, in response to the operation702generating an output, deleting the stack802. In this way, each stack802,804,806may correspond to a respective operation702,704,706performed by the function310.

Using stacks802,804,806to implement each operation702,704,706means that where an operation704is paused, the respective stack804maintains a current status of the operation, allowing it to be easily resumed from where the operation704left off. Stacks802,804,806may also be implemented using contiguous blocks of memory, thereby simplifying the read and write procedure when processing data using a stack. As such, monitoring, pausing, and redeploying functions310which are actively performing operations702to706is simplified as the progress of operations702to706can be easily identified and copied from the stacks802to806.

In some examples, the function310may be implemented using a linear memory model. A linear memory model is a memory addressing technique in which memory is organized in a single contiguous address space. This single contiguous address space may be a physical storage space or may be implemented using virtualized memory. Using a linear memory model to implement a function310means that while processing operations in the function310it becomes possible to access memory locations directly as well as linearly. Using a linear memory model also enables the database system100to easily identify and collect memory used to implement the function310, which is useful when migrating and updating the function310as will be described below in relation toFIGS.9to12.

Turning toFIG.9, the method200may comprise obtaining a request902to update the function definition306. The request902may include third data904representing an update to be performed to the function definition306. The update that is to be performed to the function definition306may include modifying the set of operations302in the function definition306, such as adding a new operation, deleting an operation, and/or otherwise modifying one of the set of operations302. Alternatively, or additionally, the update may include modifying the set of state variables308, such as by adding a new state variable VarN+1, deleting a state variable, and/or otherwise modifying the state variable or a data structure, or schema, defining a relation between the set of state variables308.

The request902to update the function definition306may be provided by a user. For example, a user may update a program, from which the function definition306is derived, and may upload the updated program to the database system100to cause the database system100to modify the function definition306accordingly. Alternatively, the request to update902the function definition306may be automatically generated in the database system100.

The second data304is then updated906based on the third data904. That is to say that the second data304may be modified according to the update represented by the third data including, for example, modifying the set of state variables308, the set of one or more operations302, and/or modifying metadata associated with the function definition306. An updated function910is then generated908according to the updated second data304. When generating the updated function910, the updated function910may be provisioned with a reference to the set of values312that are associated with the original function310. The updated function910may then be configured to operate on the set of values312stored in the isolated computing environment314.

In this way, the function310may be updated by creating a new updated function910and transferring control of the set of values312to this updated function910. This allows the function310to be updated seamlessly while mitigating the amount of down time which would otherwise occur if the function310had to be taken offline before being updated. After the updated function910is generated and configured to operate on the set of values312, the original function310may be instructed to shut down and transfer control of the set of values312over to the updated function910. The updated function910may be provided with an ID which is associated with an ID of the original function310and/or which is derived from the ID of the original function310such that a relation to the original function310can be maintained in metadata.

Where the updated function910is associated with a different ID to the original function310, generating the updated function910may also involve updating metadata comprising an ID for the original function310by replacing this ID with an ID associated with the updated function910.

As described above in relation toFIG.3, the set of values312may be stored in a state table316. Where the set of values312are stored in a state table316, and the update includes modifying the state variables308associated with the function definition306, generating the updated function910may involve modifying the state table316. The state table316may be modified to account for modifications made to the state variables308in the function definition306. As described above, updating the state variables308may include adding state variables, removing state variables, and/or otherwise modifying the set of state variables308. Modifying the state table316may involve extending the state table316, to account for additional values corresponding to additional state variables308in the updated function definition306, deleting portions of the state table316to account for the removal of state variables in the updated function definition306, and/or otherwise modifying the data structure or mapping of the state table316to address the changes made to the state variables308.

FIG.10shows an example of a procedure for updating the function310by generating an updated function910similar to the process described in relation toFIG.9. In the example shown inFIG.10, generating the updated function910includes generating an updated state table1016according to the modified state variables308. In the example shown inFIG.10the updated state table1016which is generated accounts for the additional state variable VarN+1. The updated function910may be configured to transfer the set of values312from the original state table316to the updated state table1016, which now additionally includes value ValN+1. In the example shown inFIG.10the state table1016is stored in a new isolated computing environment1014corresponding to the updated function910.

As described above in relation toFIG.8, functions310,910in the database system may be implemented using stack-based virtual machines. If the function310is to be updated while operations702to706are being performed by the function310, the function310may be configured to handle this in one of a plurality of ways. In the example shown inFIG.11, if an operation702is being performed when the updated function910is being configured to operate on the set of values312, the function310may be configured to pause the operation702. Data in the stack802, controlled by the function310and being used to perform the operation702, may be transferred to a new stack1102controlled by the updated function910. The new stack1102controlled by the updated function910may then be used to resume the operation702.

Alternatively, when the updated function910is configured to operate on the set of values312, the function310may be configured to monitor the operation702. In this case completing the configuring of the updated function910to operate on the set of values312may be dependent on an output from the operation702being identified. In this way, control over the set of values312may only be completely handed over to the updated function910after the operations being performed by the function310are completed. Waiting for operations702to be completed before handing over to the updated function910may simplify the procedure for updating the function310and thereby mitigate the risk of errors occurring, and ensuring durability and consistency of the functions310,910and the set of values312across the database system100.

The function310in the database system100may be configured to handle some running operations702differently to other operations704when the updated function910is being configured to operate on the set of values312in the isolated computing environment314. For example, the function310may include rules for determining whether a particular operation702should be paused, and the respective data transferred to a new stack1102, whether the particular operation702should be left to complete, and/or whether this operation702should be canceled before completing the configuration of the updated function910to operate on the set of values312.

In some examples, the database system100and/or the function310may be configured to process ongoing operations702to706which are being performed during the update procedure and determine whether each of the ongoing operations702to706should be completed, canceled, or transferred to the updated function910.

As described briefly above in relation toFIG.1, the database system100may be implemented using a cluster computer in which computing resources can be scaled up and scaled down to accommodate for demand. In such a device, it is possible to logically separate portions of computing resources from other computing resources and assign these resources to specific functions.

FIG.12shows an example in which the function310is implemented on a first set of computing resources1202in the database system100. For example, the function310may be implemented using a specific set of processors and/or storage in the database system100. The set of computing resources1202may be a set of physical resources which are reserved for implementing the function310. Alternatively, the set of resources1202may be virtualized to logically appear as a contiguous set of resources. In some examples, the set of values312and the function310are implemented on a set of computing resources1202which are located in the same jurisdiction i.e. within the same country or in a location governed under the same laws. In jurisdictions where there are data protection laws which govern the movement and storage of data relating to citizens, implementing the function310and the set of values312on computing resources1202in the same jurisdiction provides adherence to these laws.

The method200may comprise obtaining a request to migrate the function310from the first set of computing resources1202to a second, different, set of computing resources1204.

The request to migrate the function310to the second of computing resources1204may be triggered by any of a number of conditions, such as the first set of computing resources1202having insufficient capacity to implement the function310effectively. This may occur where the first set of computing resources1202is being used to implement a plurality of such functions and the function310is unsuccessfully competing for resources with these other functions. This may also occur where one or more of the set of operations302to be implemented by the function310requires a larger amount of processing power or storage than is available in the first set of computing resources1202. The need to migrate the function310to the second set of computing resources1204may be identified using the execution log502in which there will be record of the function310struggling to perform operations, either by having a large number of incomplete operations, errors, or high latency times for completing operations recorded in the execution log502. In other examples, the need to migrate the function310may be determined based on a benchmark comparison of the performance of the function310against one or more performance thresholds relating to, for example, latency, error rate, power consumption, and so forth.

Other conditions which may trigger the request to migrate the function310to the second set of resources1202are where one or more of the set of operations302to be performed by the function310requires data that is stored in the second set of resources1204. Implementing a function310on the same set of computing resources1204where relevant data is stored can reduce latency time when the function310is performing operations. Where one of the set of operations302implements a high volume of reads and/or writes to data, reducing the latency of each read and/or write operation even by a small amount can yield large gains in overall efficiency.

In some examples, there may be legal restrictions on the movement of data between jurisdictions. Some jurisdictions have data protection laws which set out restrictions on the movement of data relating to citizens outside of these jurisdictions. Civil contracts may also be in place between companies and/or individuals which restrict what can and cannot be done with an individual's data, including the movement of this data. In some examples, the first set of computing resources1202are at a first geographic location and the second set of computing resources120are at a second geographic location that is different to the first geographic location. The geographic location in this case may relate to a separate physical computing device located near to each other, for example in the same data warehouse, in separate physical computing devices located in different datacenters or warehouses, or in different countries or jurisdictions.

In response to the request to migrate the function310to the second set of resources1204, a new function1206may be generated according to the second data304, the new function1206being implemented on the second set of computing resources1204. The new function1206may be provisioned with a reference to the set of values312. This reference may be a memory or storage address in which the set of values312are located in the first set of computing resources1202, or, where the set of values312are stored in a state table316, the reference may be a reference to the state table316and/or information needed to search the state table316to identify or modify the set of values312stored therein. In other examples, such as where the isolated computing environment314is implemented as a virtual machine, the reference may be a virtual machine ID.

The method200may comprise configuring the new function1206to operate on the set of values312in the isolated computing environment314. This may involve providing the new function1206with a key and/or otherwise authorizing the second function1206to operate in the isolated environment314. In the example shown inFIG.12, the method200may also comprise generating a new state table1208in the second set of computing resources1204and the new function1206may migrate the set of values312from the original state table316in the first set of resources1202to the new state table1208in the second set of resources1204. In examples where the new function1206migrates the set of values312to the second set of resources1204, an isolated computing environment1210may be established in the second set of computing resources1204to store the set of values312. A new execution log1212may also be implemented in the second set of resources1204to record operations performed by the new function1206. Alternatively, the new function1206may use the same execution log502as the original function310. While execution logs502and1212are shown inFIG.12as being included in each of the first1202and second1204sets of computing resources respectively, it is to be appreciated that in other examples, the execution log(s)502,1212may be stored separately to the first1202and second1204sets of computing resources. For instance, the execution log502may be stored in a set of computing resources which are separate to the first or second sets of computing resources and may be used to record data indicating events and operations relating a plurality of different functions, including functions310and1206.

FIG.13shows a non-transitory computer-readable storage medium1302comprising computer-executable instructions1304A to1304C which, when executed by one or more processors1306, cause the one or more processors1306to perform a method200. A first instruction block1304A, when executed, causes the processor(s)1306to obtain first data300representing a set of one or more operations302to be performed on data in a database managed by a database system100. A second instruction block1304B, when executed, causes the processor(s)1306to derive second data304from the first data300, the second data304representing a function definition, not shown inFIG.12for simplicity, for implementation in the database system100. The function definition includes the set of operations and a set of one or more state variables. A third instruction block1304C, when executed, causes the processor(s)1306to generate a function310according to the second data304, wherein generating the function310comprises storing a set of one or more values312for respective ones of the state variables. The set of values312are stored in an isolated computing environment314in the database system100, the function310being configured to operate on the set of values in the isolated computing environment314.