Patent Description:
<CIT> describes system for building and validating an application (including e.g., various software versions and revisions, programming languages, code segments, among other examples) without any scripting required by a system user. In one embodiment, an SDLC system is configured to construct a build and test environment, by automatically analyzing a submitted project. The build environment is configured to assemble existing user code, for example, to generate an application to test. <CIT> describes a method for deploying source code from a version control system to at least one of a web server and an application server is provided. <NPL>, describes decorators in Python.

Many large-scale data analytics systems are designed to efficiently run large-scale data transformation jobs. Such large-scale data analytics systems apply transformations to one or more input datasets to generate one or more output datasets. Such data analytics systems include multiple subsystems that are tightly coupled, making it difficult to add new transformation features, as changes to support such features need to be made across many subsystems. For example, expanding existing workflows to support a new coding language will require making changes to many tightly coupled subsystems. Furthermore, transformations may operate on data that is subject to security restrictions. Developers may wish to author transformations in any of several programming languages. Auditors may wish to track the origin of columns in transformation output and yet with current systems it is difficult to know which transformation or elements thereof contributed to particular output columns in an output dataset. Thus, there is a need for a distributed data analytics system that allows for efficient development and deployment of new features to one or more subsystems without having to make changes to all subsystems.

In a first aspect, there is provided a computer-implemented method as set forth in claim <NUM>. In a second aspect, there is provided a non-transitory computer-readable medium as set forth in claim <NUM>. In a third aspect, there is provided a system as set forth in claim <NUM>.

The example embodiment(s) of the present invention are illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:.

While each of the figures illustrates a particular embodiment for purposes of illustrating a clear example, other embodiments may omit, add to, reorder, and/or modify any of the elements shown in the figures.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the example embodiment(s) of the present invention. It will be apparent, however, that the example embodiment(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the example embodiment(s).

Techniques for data transformation in a distributed system of applications and data repositories are described. The subsystems for the overall framework are distributed, thereby allowing for customization to require only isolated changes to one or more subsystems, instead of to all subsystems. In some embodiments, a source code repository is used to receive and digitally store source code. A build subsystem can retrieve source code from the source code repository and build it, using one or more criteria. By building the source code, the build subsystem can generate an artifact, which is executable code, such as a JAR or SQL file. Likewise, by building the source code, the build subsystem can generate one or more job specifications for executing the executable code. In some embodiments, the artifact and job specification may be used to launch an application server in a cluster. The application server can then receive data transformation instructions and execute the data transformation instructions.

The present system provides an extensible framework that can be easily customized with new features. New features can be implemented in a single subsystem or a handful of subsystems without requiring updates to all subsystems of the overall data transformation system. Examples of new features include support for new coding languages. The system provides common security, testing, and configuration mechanisms that work with multiple different source programming languages. The system creates and stores metadata useful in tracking dataset provenance for columns of data in datasets that result from transforms. Improved security for the execution of transformation code is provided so that interactions with secure datasets do not compromise the security policies of that data.

In some embodiments, a data transformation may be specified via a function in a Python coding language. Python is desirable because existing Python libraries provide rich support for data transformation functionality. The Python function itself provides a sequence of instructions for a dataset transformation. A Python decorator that corresponds to the function may be used to specify one or more metaprogramming dependencies, such as dependent datasets, necessary for the execution of the function. Thus, the decorator may be used to retrieve dependent datasets that are necessary for the execution of the function when a command is received to execute a data transformation using the function.

A dataset may be defined as a collection of data. A dataset may include multiple versions that describe the content of the dataset at different points in time. Thus, by looking at a different version of the dataset, it is possible to understand the state of the data of that dataset at a particular point in time. A dataset may include one or more transactions that modify the state of the data in the dataset at different versions. For example, a transaction may add new data files to a dataset, overwrite existing data files in a dataset, or delete data files in a dataset.

For example, <FIG> illustrates dataset versioning according to one example. On Day One, an initial version of dataset A is created in the context of transaction TX1 resulting in data file F1. For example, data file F1 may contain web access log entries for the past six months. Also on Day One, an initial version of dataset B is created in the context of transaction TX2 resulting in data file F2. For example, data file F2 may contain rows corresponding to users of an online web service and associating user name identifiers with network addresses from which the users access the web service. Also on Day One, a driver program P1 is executed in the context of transaction TX3 that performs a join based on network address between dataset A, consisting of the initial version of dataset A, and dataset B, consisting of the initial version of dataset B. This execution results in an initial version of dataset C and data file F3 containing the results of the join operation executed in the context of transaction TX3.

On Day Two, the previous day's (i.e., Day One's) web access log entries are added to dataset A in the context of transaction TX4 thereby producing data file F4. In this example, data file F4 contains only the previous day's (i.e., Day One's) web access log entries. Also on Day Two, the driver program P1 is executed again in the context of transaction TX5. In this example, the join performed in the context of transaction TX5 is between the web access log entries in data file F4 and the entries in data file F2. This execution results in a second version of dataset C and data file F5 containing the results of the join operation executed in the context of transaction TX5.

Similarly, on Day Three, the previous day's (i.e., Day Two's) web access log entries are added to dataset A in the context of transaction TX6 and resulting in data file F6. In this example, data file F6 contains only the previous day's (i.e., Day Two's) web access log entries. Also on Day Two, the driver program P1 is executed again in the context of transaction TX7. In this example, the join performed in the context of transaction TX7 is between the web access log entries in data file F6 and the entries in data file F2. This execution results in a third version of dataset C and data file F7 containing the results of the join operation executed in the context of transaction TX7. As a result, there are three versions of dataset A corresponding to transactions TX1, TX4, and TX6 and data files F1, F4, and F6. There is one version of dataset B corresponding to transaction TX2 and data file F2. And there are three versions of dataset C corresponding to transactions TX3, TX5, and TX7 and data files F3, F5, and F7.

A data transformation job is a unit of work that can be executed to perform the computation of a new dataset or a new version of an existing dataset. In some embodiments, a data transformation job computes one or more output datasets from the data of a set of input datasets. Thus, a data transformation job may take one or more input datasets, perform data transformation processing on those input datasets, and generate one or more output datasets. The output datasets may be a new version of an existing dataset or a new dataset altogether.

<FIG> illustrates an example block diagram of a data analytics system <NUM> in which the techniques described herein may be practiced, according to some embodiments. In the example of <FIG>, data analytics system <NUM> is a system configured or programmed to perform data transformations. The various components of data analytics system <NUM> are implemented, at least partially, by hardware at one or more computing devices, such as one or more hardware processors executing stored program instructions stored in one or more memories for performing the functions that are described herein. In other words, all functions described herein are intended to indicate operations that are performed using programming a special-purpose computer or general-purpose computer, in various embodiments. Data analytics system <NUM> illustrates only one of many possible arrangements of components configured to executing the programming described herein. Other arrangements may include fewer or different components, and the division of work between the components may vary depending on the arrangement.

Data analytics system <NUM> includes a code repository <NUM>. Code repository <NUM> is programmed or configured to serve as an archive for storing, managing, and accessing source code. Thus, code repository <NUM> can be used for digitally storing source code for a computer program. For example, in some embodiments, code repository <NUM> may be programmed or configured to allow for the checking in, checking out, committing, merging, branching, forking, or other management of source code files. Source code files can be in any programming language, including, but not limited to Java, Structured Query Language (SQL), Python, Scala, etc. Code repository <NUM> may be programmed or configured to provide version control for source code files. In some embodiments, code repository <NUM> may be accessible via a web interface and/or a command line interface. In some embodiments, code repository <NUM> may be implemented as a Git repository.

In some embodiments, code repository <NUM> may be communicatively coupled to client device <NUM> and/or automated build system <NUM>. Thus, a user can use client device <NUM> to commit source code to code repository <NUM>.

Data analytics system <NUM> may further include an automated build system <NUM>. Automated build system <NUM> is programmed or configured to retrieve source code from code repository <NUM>. Automated build system <NUM> is further programmed or configured to compile and build that source code into one or more artifacts and/or one or more job specifications. In some embodiments, automated build system <NUM> may implemented as a Gradle build system.

In some embodiments, automated build system <NUM> may be configured via one or more build tasks. A build task is a sequence of instructions that provides criteria for building source code retrieved from code repository <NUM>. A build task may include various criteria, including, but not limited to, a schedule for building source code, an identification of what source code should be retrieved and built, one or more libraries or plugins to use in the build process or include in an artifact, an identification of a location or path of an artifact repository <NUM> and/or build service <NUM> that the automated build system <NUM> should send the built code to. In some embodiments, different build tasks may be used for different types of source code, such as source code of different programming languages.

In some embodiments, automated build system <NUM> may be accessible via a user interface, such as an application, a web interface, or a command line interface.

The output of automated build system <NUM> retrieving, compiling, and building source code files from code repository <NUM> may include one or more artifacts and/or one or more job specifications.

An artifact is a set of executable code, such as a JAR file, a SQL file, an executable file (. EXE), library, plugin, or any other form of executable code.

A job specification is a set of instructions that specifies how to construct a data transformation job. For example, a job specification may include instructions for detailing necessary dataset dependencies for a data transformation job. These dataset dependencies that are included in the job specification may include a unique identifier that identifies dataset dependencies, a path to a dataset that is necessary for the performance of a data transformation job, a data transformation job dependency that indicates a prior job that needs to be completed before the present job can be processed, and/or a subset of data from a dataset, such as a particular version of a dataset to use in the data transformation job. A job specification may further include instructions that indicate a job type that specifies the type of transform worker <NUM> to use to run a data transformation job. For example, a job type may specify a particular programming language and/or a particular transform worker <NUM> capable of handling that programming language. A job specification may further include various user-defined configuration settings for running a data transformation job, such as when to execute the data transformation job, where to output the output dataset for the data transformation job, the permission settings for the user that is attempting to execute the data transformation job, etc..

Automated build system <NUM> may be communicatively coupled to artifact repository <NUM> and/or build service <NUM>. In some embodiments, automated build system <NUM> may send any artifacts that it builds to artifact repository <NUM>. In some embodiments, automated build system <NUM> may send any job specifications it builds to build service <NUM>.

Data analytics system <NUM> may further include artifact repository <NUM>. Artifact repository <NUM> is a data storage system that is programmed or configured to serve as an archive for storing, managing, and accessing artifacts. In some embodiments, artifact repository <NUM> is a blobstore.

Artifact repository <NUM> may be programmed or configured to provide version control for artifacts. In some embodiments, artifact repository <NUM> may be accessible via a web interface and/or a command line interface. In some embodiments, artifact repository <NUM> may be communicatively coupled to transform worker <NUM>.

In some embodiments, artifact repository <NUM> may be programmed or configured to periodically retrieve artifacts from automated build system <NUM> according to a schedule or configuration setting.

Storing artifacts in artifact repository <NUM> provides various security advantages. For example, downstream systems do not have access to the underlying source code and only have access to fully built artifacts. This can help to ensure that downstream systems, and user accounts that have access to downstream systems, cannot access or modify the raw source code files in code repository <NUM>. Furthermore, by storing an artifact in artifact repository <NUM>, the artifact will be fully built and ready to use for subsequent build dataset commands that require the artifact, without having to rebuild the artifact, assuming no changes to the underlying source code are made. Therefore, data transformation jobs can be efficiently processed using existing artifacts in artifact repository <NUM>, without having to rebuild the same artifact over and over between data transformation jobs.

Data analytics system <NUM> may further include build service <NUM> that is programmed or configured to manage a data transformation job. Build service <NUM> may be implemented as an executable program, process or thread that is defined by stored program instructions in the system <NUM>. Build service <NUM> may be communicatively coupled to automated build system <NUM> and receive a job specification from automated build system <NUM>.

Build service <NUM> may further be communicatively coupled to client device <NUM>. For example, client device <NUM> may send various commands to build service <NUM> via an interface, such as a web interface, command line interface, or application programming interface (API). In some embodiments, client device <NUM> may be configured to send a build dataset command to a build service <NUM>. A build dataset command is a sequence of data transformation instructions to initiate a data transformation job to either build a new dataset or build a new version of an existing dataset. In some embodiments, the data transformation instructions specify a process or parameters for a build transformation job.

In some embodiments, build service <NUM> does not actually perform any data transformation job itself, instead, it invokes a registered transform worker <NUM> to perform the data transformation job. In some embodiments, build service <NUM> is communicatively coupled to one or more transform workers <NUM>. Although depicted in <FIG> as a single transform worker <NUM>, in some embodiments, multiple transform workers <NUM> may exist. In some embodiments, different transform workers <NUM> may exist for different job types. For example, a first transform worker <NUM> may be used for data transformation jobs that use SQL, and a second transform worker <NUM> may be used for data transformation jobs that use another programming language, such as Java.

In some embodiments, once build service <NUM> receives a build dataset command and job specification, build service <NUM> identifies the appropriate transform worker <NUM> to perform the data transformation job, based on the job specification. For example, build service <NUM> may store mapping data that maps a job type of a job specification to a particular transform worker <NUM>. Build service <NUM> then sends a build dataset command and the job specification to the identified transform worker <NUM> for execution. In some embodiments, build service <NUM> may communicate with transform worker <NUM> via a Hypertext Transfer Protocol (HTTP) and/or JavaScript Object Notation (JSON) API.

In some embodiments, build service <NUM> may receive job status information from transform worker <NUM> about the status of a data transformation job. In some embodiments, status information may include one or more of the following:.

In some embodiments, build service <NUM> may make status information available to a client device <NUM> to allow a user to view the status of a data transformation job for which a build dataset command was sent.

Transform worker <NUM> is programmed or configured to invoke a data transformation job. Transform worker <NUM> may be implemented as an executable program, process or thread that is defined by stored program instructions in the system <NUM>. In some embodiments, multiple transform workers <NUM> may exist. For example, different transform workers <NUM> may be used to invoke data transformation jobs having different job types.

A transform worker <NUM> invokes a data transformation job by launching an application module <NUM>, via module launcher <NUM>, and executing the data transformation job via that application module <NUM>.

In an embodiment, transform worker <NUM> sends a request or command to module launcher <NUM> to launch an application module for the data transformation job. The request from transform worker <NUM> may include the job specification and/or the coordinates for one or more necessary artifacts for the data transformation job. The coordinates may identify a particular artifact or location of an artifact in artifact repository <NUM>. Module launcher may then launch an application module <NUM>.

Once the application module <NUM> is launched by module launcher <NUM>, transform worker <NUM> can then send a build dataset command directly to application module <NUM> in order to cause application module <NUM> to retrieve the necessary artifacts from the artifact repository and execute the data transformation job. For example, in some embodiments, either the launch module command and/or the build dataset command may include coordinates that identify one or more artifacts in artifact repository <NUM>. Thus, upon receiving a job specification and a build dataset command from build service <NUM>, transform worker <NUM> causes module launcher <NUM> to retrieve the necessary artifacts to build the dataset from artifact repository <NUM> for the execution of the data transformation job. In some embodiments, the necessary artifacts or artifact coordinates may be specified in the job specification. Transform worker <NUM> then sends a request or command to module launcher <NUM> to launch an application module for the data transformation job. The request from transform worker <NUM> may include the job specification and/or the artifact coordinates. The application module <NUM>, once launched, will retrieve the necessary artifacts from artifact repository <NUM> based on the coordinates received from transform worker <NUM>.

Application module <NUM> can send status information regarding the execution of the data transformation job to transform worker <NUM>, which can, in turn, relay that status information to build service <NUM>.

Module launcher <NUM> is an application server that is programmed or configured to receive a launch module command from transform worker <NUM>. In response to receiving a launch module command from transform worker <NUM>, module launcher <NUM> initializes an application module <NUM> in cluster <NUM>. Module launcher <NUM> is not involved in the actual data transformation job processing, but rather, merely provides a workflow for initializing and starting up an application module <NUM> that can handle data transformation job processing.

Upon initializing application module <NUM>, module launcher <NUM> can be programmed or configured to send a response to transform worker <NUM> that includes a link to application module <NUM>. In some embodiments, the link may be implemented as a Representational State Transfer (REST) API. Thus, the module launcher <NUM> allows the transform worker <NUM> to communicate directly to the application module <NUM>, via the link.

In some embodiments, the module launcher <NUM> may return status information to transform worker <NUM> regarding the status of the initialization of application module <NUM>. This status information may be related, via build service <NUM> to client device <NUM>.

Application module <NUM> is an application server that is programmed or configured to receive data transformation instructions and execute a data transformation job. In some embodiments, application module <NUM> may be an HTTP server. Application module <NUM> can be initialized based on a job specification and one or more artifacts stored in artifact repository <NUM>. Application module <NUM> is communicatively coupled to transform worker <NUM> via a link, such as a REST API. In some embodiments, application module <NUM> may receive a build dataset command that includes data transformation instructions from a transform worker <NUM>. Upon receiving a build dataset command from transform worker <NUM>, application module <NUM> can execute a data transformation job as specified in the build dataset command. In some embodiments, application module <NUM> may retrieve one or more necessary artifacts from artifact repository <NUM> based on artifact coordinates received from transform worker <NUM>. The one or more artifacts may be used to execute the data transformation job as specified in the build dataset command. Thus, application module <NUM> handles the actual processing of the data transformation job. In some embodiments, application module <NUM> may return status information to transform worker <NUM> before, during, and/or after the execution of a data transformation job regarding the status of the execution of the data transformation instructions.

In some embodiments, application module <NUM> may be initialized in a cluster <NUM> with other application modules <NUM> (not depicted). Thus, a cluster of application modules <NUM> may exist for different types of data transformation jobs. Cluster <NUM> may be used for caching shared resources across multiple application modules <NUM>, such as in-memory copies of datasets or cached copies of artifacts. An application module <NUM> may run indefinitely, thus providing availability for processing particular types of data transformation jobs as required by the transform worker <NUM>.

In some embodiments, a different application module <NUM> may be initialized in cluster <NUM> for each supported programming language, such as Java, Python, SQL, etc. This allows different application modules <NUM> to be responsible for different programming languages. In some embodiments, transform worker <NUM> stores mapping data that maps a particular one of the application modules <NUM> to a particular job type, and/or mapping data that maps a particular job type to a particular application module <NUM>, such as by programming language. This allows for the rapid development and deployment of new functionality to support a new programming language on the fly by initializing a new application module <NUM> for the new programming language.

Data analytics system <NUM> may include a client device <NUM>. Client device <NUM> may be implemented across one or more computing devices, such as a mobile computing device, tablet computer, desktop computer, or laptop computer, none of which is intended as generic, since it is loaded with instructions in a new ordered combination as otherwise disclosed herein to implement the functions and algorithms of this disclosure. Client device <NUM> may be accessible to a user and may provide a user interface for a user to access code repository <NUM> and/or build service <NUM> of data analytics system <NUM>. The user interface may be provided via an application, a web service, a command line interface, or some other graphical display. Client device <NUM> is coupled to code repository <NUM> and/or build service <NUM>. For example, client device <NUM> may be communicatively coupled to code repository <NUM> and/or build service <NUM> via a network, such as the Internet or a private network.

Client device <NUM> may be programmed or configured to commit, merge, branch, fork, check in, check out or perform other interactions with code repository <NUM>. Client device <NUM> may be programmed or configured to send a build dataset command to build service <NUM> that includes one or more dataset transformation instructions. Client device <NUM> may be programmed or configured to receive information from build service <NUM>, such as status information on data transformation jobs, and display that information via a user interface to allow a user to view the status of a data transformation job.

The present system provides an extensible framework that can be easily customized with new features. New features can be implemented in a single subsystem or a small subset of subsystems of data analytics system <NUM> without requiring updates to all subsystems. Examples of new features are described below, but other features may be easily added to data analytics system <NUM> with minimal changes to the overall architecture.

One risk of a data analytics system is that a user that has permission to author or modify source code that is used in a data transformation job and execute that source code can potentially access sensitive data on the computing device(s) where the data transformation job is executed. Such sensitive data can be data that needs to be secure from malicious users or data that, if modified, may affect data integrity as a whole for the data analytics system. The present system may provide improved security by segregating security permissions for committing source code to code repository <NUM> from the security permissions for executing a build dataset command on build service <NUM>. Thus, some users may have access to committing source code that will be used in a data transformation job, but those users may be denied access from actually performing a data transformation job via a build dataset command. By segregating these permissions away from the users that are committing source code, a second set of users can oversee and verify the committed source code before actually executing a data transformation job via a build dataset command. Such a feature can be added at the code repository <NUM> and build service <NUM> subsystems without a need for global changes across more subsystems.

Datasets can include multiple versions of information over multiple transactions, and the dataset versions can be generated based on multiple dataset inputs. Over time, it can be challenging to identify the provenance of the original source of where data in a dataset originated. The present system provides for easy extensibility to add dataset provenance tracking to datasets.

In some embodiments, certain data in a dataset may be sensitive information. For example, if a dataset contains data files that specify social security numbers, the social security number data in the dataset may be sensitive. Whenever a dataset with such sensitive information is processed using a dataset transformation job, the data related to such sensitive information may need to be obfuscated to preserve the sensitive information.

In some embodiments, a shrinkwrap library can be included as part of build task for the inclusion in libraries and/or job specifications. The shrinkwrap library is a customized library that can be used for the secure obfuscation of sensitive data, for example, with a hashing function. In a traditional data analytics system, such a shrinkwrap library functionality would need to be included in every source code commit for any dataset transformation job. However, in the present system, such a shrinkwrap library can be easily added to all data transformation jobs. The shrinkwrap library can be added to the automated build system <NUM> and referenced via a set of build task criteria. Then, when the automated build system <NUM> builds code from a code repository <NUM> using build task criteria that specifies the shrinkwrap library, the shrinkwrap library functionality can be included in the outputted artifacts and job specifications automatically. This is an example of how a single change at a single subsystem can provide robust features for all data transformation jobs handled by data analytics system <NUM>.

The present system makes it easy to add support for new programming languages that are to be used in data transformation jobs. A new programming language can be added to the existing data analytics system <NUM> by defining a new build task that can take the source code of the new programming language and generate artifacts based on source code in the new programming language, as well as updating a configuration setting of the transform worker <NUM> that maps the new programming language to a new application module <NUM>.

Caching datasets across data transformation jobs can be accomplished with the present system. In some embodiments, cluster <NUM> may include functionality for caching datasets that can be shared across multiple application modules <NUM>.

If multiple jobs depend on the same artifact, there will be an incurred cost to resolve them every time when executing a data transformation job. In some embodiments, artifacts may be distributed to a cache shared across a cluster <NUM>. Then, application modules <NUM> can reference artifacts in the cache of cluster <NUM> instead of referring to the artifacts by coordinates in the artifact repository <NUM>. This will improve uptime, as only those application modules <NUM> in the same cluster will be competing for the artifacts. Those application modules <NUM> in a separate cluster will have their own cache of artifacts.

During compilation of source code, various validation can be performed of the source code to ensure that the source code conforms to certain requirements.

In the case of a data transformation job that consists of multiple stages of computation, the computations can be inlined. Inlining comprises running a sequence of instructions without persisting intermediate computation results to disk. Inlining can be challenging to achieve when computations are performed across languages, where the computed output from one computation cannot be referenced in the language of another computation.

In an embodiment, a Python-based implementation of data analytics system <NUM> may be used. In a Python-based implementation, transform worker <NUM> may be implemented as a Python server. Likewise, code submitted to code repository <NUM> may be coded in a Python coding language.

Code may include one or more Python functions which define computational logic for a data transformation job. Thus, by using a Python coding language, the code may be implemented to take advantage of Python syntax, features, and data structures. For example, the code may be implemented to use for-loops, recursion, and similar types of features native to Python. This allows for fully-featured implementation of data transformation jobs using Python coding language features.

Code may further include Python decorators. A Python decorator is an annotation syntax in a Python coding language that can be used for implementing higher-order functions. Specifically, a decorator is a higher-order function that takes a lower-order function as an input and extends the lower-order function without explicitly modifying it. Python decorators can thus be used to define metaprogramming dependencies for functions of computational logic. For example, a lower-order Python function may define a data transformation job that outputs an output dataset and Python decorators can be used to define one or more expected input datasets for the lower-order Python function. The decorators can define, for example, the expected location of the input dataset(s), the expected type of the input dataset(s), the expected content of the input dataset(s), or the expected characteristic(s) of the input datasets. Thus, a Python decorator that corresponds to a function may be utilized to allow users to define, directly via the code, pipeline dependencies for data transformation jobs.

A Python-based implementation of data analytics system <NUM> provides various benefits. First, Python is a fully featured programming language. Thus, code for data transformation jobs may utilize the existing syntax, features, and data structures of the Python programming language to implement complex data transformation jobs. For example, for-loops, conditional programming, dynamic imports, reflection, and recursion may be implemented in the data transformation jobs. This allows for the implementation of complex data transformation jobs.

Second, Python is widely utilized as a programming language for data science analysis. Thus, by providing a Python-based implementation of code for computational logic in the present data analytics system <NUM>, a user may utilize various Python-specific libraries and tools for data sciences as part of the computational logic of data transformation jobs. Examples of such Python-specific libraries and tools include, but are not limited to: Pandas, NumPy, SciPy, and IPython Notebook.

Third, a Python-based implementation of data analytics system <NUM> allows users to specify metaprogramming dependencies of input and output datasets via Python decorators. In an embodiment, transform worker <NUM> can analyze an artifact built from Python code or the underlying code itself to determine metaprogramming dependencies for one or more data transformation jobs. In an embodiment, transform worker <NUM> is programmed or configured to analyze the Python decorators in the Python code or artifact to identify dataset dependencies for data transformation jobs specified by functions without having to execute the computational logic of the functions themselves. Thus, the transform worker <NUM> can determine metaprogramming dependencies of the code or artifact. In an embodiment, transform worker <NUM> can generate a dependency file based on this analysis. This dependency file therefore defines the dataset dependencies for one or more data transformation jobs in a set of code. This dependency file can be used to define a pipeline of data transformations across multiple datasets.

In an embodiment, transform worker <NUM> is further programmed or configured to use the Python decorators to retrieve one or more dependent datasets necessary for the computation of a data transformation job.

In an embodiment, a Java Virtual Machine (JVM)-based implementation of data analytics system <NUM> may be used. In a JVM-based implementation, transform worker <NUM> may be implemented using a JVM. Code submitted to code repository <NUM> may be coded in a programming language that is compatible with a JVM, such as Java or Scala. Although Java and Scala are two examples of JVM-compatible languages, any language that is compatible with a JVM may be used.

Using Java or Scala for defining data transformation jobs via code in a JVM-based implementation of data analytics system <NUM> provides various benefits. First, Java and Scala are each strongly-typed programming languages, which, in some cases, can make programming data transformation jobs via code easier. Second, Java and Scala are each object-oriented programming languages, thus, implementing data transformation jobs via Java or Scala allows for code reuse, object encapsulation and abstraction, and polymorphism.

In an embodiment, a JVM-based implementation of data analytics system <NUM> allows a transform worker <NUM> to determine whether to recompute a dataset if byte code changes to a data transformation job are detected. For example, assume that code checked into code repository <NUM> includes computational logic for a data transformation job that is used for computing a first dataset. The code may be built into an artifact stored into artifact repository <NUM>. In an embodiment, when a transform worker <NUM> retrieves the artifact from artifact repository <NUM>, the transform worker <NUM> is programmed or configured to determine whether the byte code of data transformation job specified in the artifact has changed from a prior version of the data transformation job. Transform worker <NUM> is programmed or configured to compare the byte code of the current version of the data transformation job to the byte code of the prior version of the data transformation job. If the transform worker <NUM> detects that there have been no byte code changes, the transform worker <NUM> will not recompute the first dataset. However, if the transform worker <NUM> detects that there have been byte code changes to the data transformation job, the transform worker <NUM> will recompute the first dataset as the computational logic for the data transformation job has been changed. Thus, transform worker <NUM> is programmed or configured to dynamically determine whether recomputation of datasets is necessary based on byte code changes to a data transformation job.

<FIG> illustrates a flow diagram of an example process <NUM> for performing a build of source code files, according to some embodiments. <FIG>, and each other flow diagram in the drawing figures or process description in the text, is intended to disclose an algorithm or functional description that may be used as a basis of writing computer programs to implement the functions that are described herein, and which cause a computer to operate in the new manner that is disclosed herein. Further, <FIG> and each other flow diagram or process description are provided to communicate such an algorithm at the same level of detail that is normally used, by persons of skill in the art to which this disclosure is directed, to communicate among themselves about plans, designs, specifications and algorithms for other computer programs of a similar level of complexity.

The process <NUM> may start at step <NUM>. In step <NUM>, code repository <NUM> is programmed or configured to receive one or more source code files and commit those source code files. The source code files may be committed in response to a request from client device <NUM>. In some embodiments, committing the source code files may include merging, branching, forking, checking out, and/or checking in source code files. Once the source code files are committed, the process <NUM> may then proceed to step <NUM>.

In step <NUM>, automated build system <NUM> is programmed or configured to receive the source code files from code repository <NUM>. In some embodiments, the source code files are pushed by the code repository <NUM> to the automated build system <NUM> upon being committed. In some embodiments, automated build system <NUM> may retrieve the source code files from code repository <NUM> based on a build task that specifies one or more criteria for retrieving the source code files, such as a build schedule. The build task criteria may include an identification of what source code files to build, how frequently to retrieve the source code files, and where to send the output of the build. Once automated build system <NUM> receives the source code files from code repository <NUM>, the process <NUM> may proceed to step <NUM>.

In step <NUM>, the automated build system <NUM> is programmed or configured to use a build task to build the source code into one or more artifacts and/or one or more job specifications. The build task may specific various criteria for building the source code files, such as when the source code files should be built, what libraries and virtual machine to use for building the source code, configuration settings for the build process, and where the output of the build process should be sent. The build process of the automated build system <NUM> may generate one or more artifacts, which are executable code, such as JARs,. EXE files, SQL, etc. The build process of the automated build system <NUM> may generate one or more job specifications that are sequences of instructions for how to construct a data transformation job. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the process <NUM> may store the one or more artifacts generated in step <NUM> in an artifact repository <NUM>. Artifact repository <NUM> is a repository for storing executable code. In some embodiments, artifact repository <NUM> may be implemented as a blobstore. In some embodiments, artifact repository <NUM> may include coordinates for the location of artifacts stored in the artifact repository <NUM> to allow transform workers <NUM> to easily retrieve necessary artifacts.

In step <NUM>, the automated build system <NUM> is programmed or configured to send the job specification generated in step <NUM> to build service <NUM>. Build service <NUM> will later serve as an entry point for the invoking of data transformation jobs, via build dataset commands, and the job specification will be used as criteria for the execution of the data transformation job. The process <NUM> may then end.

<FIG> illustrates a flow diagram of an example process <NUM> for executing a data transformation job. The process <NUM> may start at step <NUM>. At step <NUM>, the build service <NUM> is programmed or configured to receive a job specification and a build dataset command. The job specification is received from automated build system <NUM> and includes a set of instructions that specifies how to construct a data transformation job. For example, a job specification may include instructions for detailing necessary dataset dependencies for a data transformation job or additional user-defined configuration settings for a data transformation job. The build dataset command may be received from client device <NUM>. A build dataset command may be any request for the execution of a data transformation job. For example, the build dataset command may be received via a user interface on client device <NUM>, such as an application, web interface, or command line interface. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the process <NUM> determines the appropriate transform worker <NUM> to handle the data transformation job. For example, in some embodiments, the build service <NUM> may determine a particular transform worker <NUM> to handle the data transformation job based on the job type specified in the job specification. In another embodiment, build service <NUM> may determine a particular transform worker <NUM> based on the programming language identified in the job specification, as a specific transform worker <NUM> may be dedicated to a handling jobs of a particular programming language. Once the particular transform worker <NUM> is identified, the build service <NUM> may send the job specification and/or a build dataset command to the transform worker <NUM>. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may determine whether an existing application module <NUM> is available that can handle the data transformation job or not. For example, in some embodiments, transform worker <NUM> may maintain a catalogue of available application modules <NUM> in cluster <NUM>. The catalogue may be maintained by adding new application modules <NUM> to the catalogue when they are initialized, and removing application modules <NUM> from the catalogue when they shut down or become unavailable. The catalogue may further contain mapping information that maps particular job types to particular application modules <NUM>. Thus, the transform worker <NUM> can maintain state information on the availability of application modules <NUM>, as well as the types of data transformation jobs that those application modules <NUM> can handle. If transform worker <NUM> determines that there is no application module <NUM> that is available that can handle the build dataset command, the process <NUM> may proceed to step <NUM>. If transform worker <NUM> determines that an application module <NUM> is available that can handle the build dataset command, the process <NUM> may proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may send a launch module command to module launcher <NUM>. A launch module command may be any request for the initialization of an application module <NUM> that can be used for handling a data transformation job. In some embodiments, the launch module command may include the job specification or a subset of the job specification. In some embodiments, the launch module command may include the coordinates of the artifacts from the artifact repository <NUM>. In some embodiments, the launch module command may include one or more server settings that are configured by the transform worker, such as settings of when to launch the application module <NUM>, the location of where to launch the application module <NUM>, how long the application module <NUM> should be available, security settings for the application module <NUM>, or any other configuration related to the lifecycle of the application module <NUM>. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the module launcher <NUM> initializes the application module <NUM> in cluster <NUM>, based on the launch module command received form the transform worker <NUM>. In some embodiments, the module launcher may return status information regarding the initialization of application module <NUM> to the transform worker <NUM>. In some embodiments, once application module <NUM> has successfully completed launching, the module launcher <NUM> can send a link to transform worker <NUM> that the transform worker <NUM> can use to directly communicate with application module <NUM>, such as a REST API URL. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may send a build dataset command to application module. A build dataset command may be any request for the execution of a data transformation job. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the application module <NUM> receives the build dataset command from transform worker <NUM>, and, in response, retrieves one or more artifacts from the repository and executes the data transformation job. In some embodiments, application module <NUM> may retrieve one or more artifacts from the artifact repository <NUM> that are necessary for the data transformation job. For example, in some embodiments, the necessary artifacts may be identified in the job specification via coordinates. In an embodiment, transform worker <NUM> may include the artifact coordinates as part of either the launch module command of step <NUM> or the build dataset command of step <NUM>. Application module <NUM> may then use those coordinates provided by transform worker <NUM> to retrieve the one or more artifacts from artifact repository <NUM>. The execution of the data transformation job may include the taking one or more input datasets and generating one or more output datasets or a new version of one or more existing datasets. In some embodiments, the execution of the data transformation job may be performed using the artifacts retrieved from the artifact repository <NUM> and/or the job specification. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the application module <NUM> may send status information of the data transformation job to the transform worker <NUM>. Status information may indicate that the data transformation job was completed, failed, aborted, or any other information regarding the interim processing of the data transformation job. This information can be used by the transform worker <NUM> to notify build service <NUM> and/or client device <NUM> regarding the status of the data transformation job. The process <NUM> may then end.

<FIG> illustrates a flow diagram of an example process <NUM> for executing a data transformation job in a Python-based implementation, according to some embodiments. Process <NUM> assumes that code in Python coding language has already been committed to code repository <NUM>, built into an artifact, and stored in artifact repository <NUM>, as per the process <NUM> described earlier. The process <NUM> may start at step <NUM>. At step <NUM>, the build service <NUM> is programmed or configured to receive a job specification and a build dataset command. The job specification is received from automated build system <NUM> and includes a set of instructions that specifies how to construct a data transformation job. For example, a job specification may include instructions for detailing necessary dataset dependencies for a data transformation job or additional user-defined configuration settings for a data transformation job. The build dataset command may be received from client device <NUM>. A build dataset command may be any request for the execution of a data transformation job. For example, the build dataset command may be received via a user interface on client device <NUM>, such as an application, web interface, or command line interface. In an embodiment, the build dataset command may specific one or more Python functions for execution of a data transformation job. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the process <NUM> determines the appropriate transform worker <NUM> to handle the data transformation job. In this particular example, the transform worker <NUM> may be a Python-based server. Once the particular transform worker <NUM> is identified, the build service <NUM> may send the job specification and/or a build dataset command to the transform worker <NUM>. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may send a launch module command to module launcher <NUM>. A launch module command may be any request for the initialization of an application module <NUM> that can be used for handling a data transformation job. In some embodiments, the launch module command may include the job specification. In some embodiments, the launch module command may include the artifacts retrieved form the artifact repository <NUM> or the coordinates of the artifacts from the artifact repository <NUM>. In some embodiments, the launch module command may include one or more server settings that are configured by the transform worker, such as settings of when to launch the application module <NUM>, the location of where to launch the application module <NUM>, how long the application module <NUM> should be available, security settings for the application module <NUM>, or any other configuration related to the lifecycle of the application module <NUM>. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the module launcher <NUM> initializes the application module <NUM> in cluster <NUM>, based on the launch module command received form the transform worker <NUM>. In on embodiment, the module launcher may return status information regarding the initialization of application module <NUM> to the transform worker <NUM>. In some embodiments, once application module <NUM> has successfully completed launching, the module launcher <NUM> can send a link to transform worker <NUM> that the transform worker <NUM> can use to directly communication with application module <NUM>, such as a REST API link. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may analyze one or more Python decorators in the code or in the artifact to identify one or more dataset dependencies for the execution of the data transformation job. Dataset dependencies may include one or more expected input datasets, their locations such as relative file path, their expected contents and their expected dataset types. In an embodiment, transform worker <NUM> may generate a dependency file that identifies the one or more dataset dependencies for the data transformation job. In an embodiment, transform worker <NUM> may further retrieve the dependent datasets. Thus, transform worker <NUM> is programmed or configured to determine what dataset dependencies exist and retrieve the necessary datasets for the execution of the data transformation job. The process <NUM> may then proceed to step <NUM>.

In step <NUM>, the transform worker <NUM> may send a build dataset command to application module. A build dataset command may be any request for the execution of a data transformation job. In an embodiment, the build dataset command may be packaged with the dataset dependencies, the dependency file, and/or the input datasets retrieved in step <NUM>. The process <NUM> may then proceed to step <NUM>.

Referring now to <FIG>, it is a block diagram that illustrates a computing device <NUM> in which the example embodiment(s) of the present invention may be embodied. Computing device <NUM> and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the example embodiment(s). Other computing devices suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

Computing device <NUM> may include a bus <NUM> or other communication mechanism for addressing main memory <NUM> and for transferring data between and among the various components of device <NUM>.

Computing device <NUM> may also include one or more hardware processors <NUM> coupled with bus <NUM> for processing information. A hardware processor <NUM> may be a general purpose microprocessor, a system on a chip (SoC), or other processor.

Main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, also may be coupled to bus <NUM> for storing information and software instructions to be executed by processor(s) <NUM>. Main memory <NUM> also may be used for storing temporary variables or other intermediate information during execution of software instructions to be executed by processor(s) <NUM>.

Software instructions, when stored in storage media accessible to processor(s) <NUM>, render computing device <NUM> into a special-purpose computing device that is customized to perform the operations specified in the software instructions. The terms "software", "software instructions", "computer program", "computer-executable instructions", and "processor-executable instructions" are to be broadly construed to cover any machine-readable information, whether or not human-readable, for instructing a computing device to perform specific operations, and including, but not limited to, application software, desktop applications, scripts, binaries, operating systems, device drivers, boot loaders, shells, utilities, system software, JAVASCRIPT, web pages, web applications, plugins, embedded software, microcode, compilers, debuggers, interpreters, virtual machines, linkers, and text editors.

Computing device <NUM> also may include read only memory (ROM) <NUM> or other static storage device coupled to bus <NUM> for storing static information and software instructions for processor(s) <NUM>.

One or more mass storage devices <NUM> may be coupled to bus <NUM> for persistently storing information and software instructions on fixed or removable media, such as magnetic, optical, solid-state, magnetic-optical, flash memory, or any other available mass storage technology. The mass storage may be shared on a network, or it may be dedicated mass storage. Typically, at least one of the mass storage devices <NUM> (e.g., the main hard disk for the device) stores a body of program and data for directing operation of the computing device, including an operating system, user application programs, driver and other support files, as well as other data files of all sorts.

Computing device <NUM> may be coupled via bus <NUM> to display <NUM>, such as a liquid crystal display (LCD) or other electronic visual display, for displaying information to a computer user. In some configurations, a touch sensitive surface incorporating touch detection technology (e.g., resistive, capacitive, etc.) may be overlaid on display <NUM> to form a touch sensitive display for communicating touch gesture (e.g., finger or stylus) input to processor(s) <NUM>.

An input device <NUM>, including alphanumeric and other keys, may be coupled to bus <NUM> for communicating information and command selections to processor <NUM>. In addition to or instead of alphanumeric and other keys, input device <NUM> may include one or more physical buttons or switches such as, for example, a power (on/off) button, a "home" button, volume control buttons, or the like.

Another type of user input device may be a cursor control <NUM>, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <NUM> and for controlling cursor movement on display <NUM>.

While in some configurations, such as the configuration depicted in <FIG>, one or more of display <NUM>, input device <NUM>, and cursor control <NUM> are external components (i.e., peripheral devices) of computing device <NUM>, some or all of display <NUM>, input device <NUM>, and cursor control <NUM> are integrated as part of the form factor of computing device <NUM> in other configurations.

Functions of the disclosed systems, methods, and modules may be performed by computing device <NUM> in response to processor(s) <NUM> executing one or more programs of software instructions contained in main memory <NUM>. Such software instructions may be read into main memory <NUM> from another storage medium, such as storage device(s) <NUM>. Execution of the software instructions contained in main memory <NUM> cause processor(s) <NUM> to perform the functions of the example embodiment(s).

While functions and operations of the example embodiment(s) may be implemented entirely with software instructions, hard-wired or programmable circuitry of computing device <NUM> (e.g., an ASIC, a FPGA, or the like) may be used in other embodiments in place of or in combination with software instructions to perform the functions, according to the requirements of the particular implementation at hand.

The term "storage media" as used herein refers to any non-transitory media that store data and/or software instructions that cause a computing device to operate in a specific fashion. Non-volatile media includes, for example, non-volatile random access memory (NVRAM), flash memory, optical disks, magnetic disks, or solid-state drives, such as storage device <NUM>. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, flash memory, any other memory chip or cartridge.

Various forms of media may be involved in carrying one or more sequences of one or more software instructions to processor(s) <NUM> for execution. For example, the software instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the software instructions into its dynamic memory and send the software instructions over a telephone line using a modem. A modem local to computing device <NUM> can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. Bus <NUM> carries the data to main memory <NUM>, from which processor(s) <NUM> retrieves and executes the software instructions. The software instructions received by main memory <NUM> may optionally be stored on storage device(s) <NUM> either before or after execution by processor(s) <NUM>.

Computing device <NUM> also may include one or more communication interface(s) <NUM> coupled to bus <NUM>. A communication interface <NUM> provides a two-way data communication coupling to a wired or wireless network link <NUM> that is connected to a local network <NUM> (e.g., Ethernet network, Wireless Local Area Network, cellular phone network, Bluetooth wireless network, or the like). Communication interface <NUM> sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information. For example, communication interface <NUM> may be a wired network interface card, a wireless network interface card with an integrated radio antenna, or a modem (e.g., ISDN, DSL, or cable modem).

Network link(s) <NUM> typically provide data communication through one or more networks to other data devices. For example, a network link <NUM> may provide a connection through a local network <NUM> to a host computer <NUM> or to data equipment operated by an Internet Service Provider (ISP) <NUM>. Local network(s) <NUM> and Internet <NUM> use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link(s) <NUM> and through communication interface(s) <NUM>, which carry the digital data to and from computing device <NUM>, are example forms of transmission media.

Computing device <NUM> can send messages and receive data, including program code, through the network(s), network link(s) <NUM> and communication interface(s) <NUM>. In the Internet example, a server <NUM> might transmit a requested code for an application program through Internet <NUM>, ISP <NUM>, local network(s) <NUM> and communication interface(s) <NUM>.

<FIG> is a block diagram of a software system <NUM> that may be employed for controlling the operation of computing device <NUM>. Software system <NUM> and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

Software system <NUM> is provided for directing the operation of computing device <NUM>. Software system <NUM>, which may be stored in system memory (RAM) <NUM> and on fixed storage (e.g., hard disk or flash memory) <NUM>, includes a kernel or operating system (OS) <NUM>.

The OS <NUM> manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as 602A, 602B, 602C. 602N, may be "loaded" (e.g., transferred from fixed storage <NUM> into memory <NUM>) for execution by the system <NUM>. The applications or other software intended for use on device <NUM> may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, an app store, or other online service).

Software system <NUM> includes a graphical user interface (GUI) <NUM>, for receiving user commands and data in a graphical (e.g., "point-and-click" or "touch gesture") fashion. These inputs, in turn, may be acted upon by the system <NUM> in accordance with instructions from operating system <NUM> and/or application(s) <NUM>. The GUI <NUM> also serves to display the results of operation from the OS <NUM> and application(s) <NUM>, whereupon the user may supply additional inputs or terminate the session (e.g., log off).

OS <NUM> can execute directly on the bare hardware <NUM> (e.g., processor(s) <NUM>) of device <NUM>. Alternatively, a hypervisor or virtual machine monitor (VMM) <NUM> may be interposed between the bare hardware <NUM> and the OS <NUM>. In this configuration, VMM <NUM> acts as a software "cushion" or virtualization layer between the OS <NUM> and the bare hardware <NUM> of the device <NUM>.

VMM <NUM> instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine comprises a "guest" operating system, such as OS <NUM>, and one or more applications, such as application(s) <NUM>, designed to execute on the guest operating system. The VMM <NUM> presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.

In some instances, the VMM <NUM> may allow a guest operating system to run as if it is running on the bare hardware <NUM> of device <NUM> directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware <NUM> directly may also execute on VMM <NUM> without modification or reconfiguration. In other words, VMM <NUM> may provide full hardware and CPU virtualization to a guest operating system in some instances.

In other instances, a guest operating system may be specially designed or configured to execute on VMM <NUM> for efficiency. In these instances, the guest operating system is "aware" that it executes on a virtual machine monitor. In other words, VMM <NUM> may provide para-virtualization to a guest operating system in some instances.

The above-described computer hardware and software is presented for purpose of illustrating the underlying computer components that may be employed for implementing the example embodiment(s). The example embodiment(s), however, are not necessarily limited to any particular computing environment or computing device configuration. Instead, the example embodiment(s) may be implemented in any type of system architecture or processing environment that one skilled in the art, in light of this disclosure, would understand as capable of supporting the features and functions of the example embodiment(s) presented herein.

Although some of the figures described in the foregoing specification include flow diagrams with steps that are shown in an order, the steps may be performed in any order, and are not limited to the order shown in those flowcharts. Additionally, some steps may be optional, may be performed multiple times, and/or may be performed by different components. All steps, operations and functions of a flow diagram that are described herein are intended to indicate operations that are performed using programming in a special-purpose computer or general-purpose computer, in various embodiments. In other words, each flow diagram in this disclosure, in combination with the related text herein, is a guide, plan or specification of all or part of an algorithm for programming a computer to execute the functions that are described. The level of skill in the field associated with this disclosure is known to be high, and therefore the flow diagrams and related text in this disclosure have been prepared to convey information at a level of sufficiency and detail that is normally expected in the field when skilled persons communicate among themselves with respect to programs, algorithms and their implementation.

Claim 1:
A computer-implemented method, comprising:
receiving source code from a computer data storage device or input from a computer input device, the source code comprising:
a lower-order function defining a data transformation job that comprises a sequence of computation instructions for a dataset transformation that generates an output dataset; and
a decorator as a higher order function that corresponds to the lower-order function and identifies one or more dependent datasets that are necessary to execute the lower-order function and one or more expected input datasets for the lower-order function;
digitally (<NUM>) storing the received source code at a first code repository (<NUM>);
building (<NUM>) the source code into executable code comprising one or more machine executable computer program files; and
digitally storing (<NUM>) the executable code in a second repository (<NUM>);
receiving (<NUM>), from a client device (<NUM>), a data transformation command that identifies the data transformation job using the lower-order function; and
in response to receiving the data transformation command:
retrieving (<NUM>) the executable code associated with the lower-order function and the decorator from the second repository (<NUM>);
based on the one or more expected input datasets for the lower-order function, analysing (<NUM>) the decorator in the source code or the executable code to identify and retrieve the one or more dependent datasets and the one or more expected input datasets for the lower-order function; and
executing (<NUM>) the data transformation job, using the one or more expected input datasets and executable code associated with the lower-order function, to generate a particular output dataset.