Certain aspects of the present disclosure provide techniques for multitenancy pipeline orchestration. According to certain embodiments, a user stores user code in a code repository and defines user code permissions needed to execute the code in an ETL system, that are stored in a databased on a processing platform. The user code is imported to the ETL environment, and a plugin obtains the user code permissions from the processing platforms and generates a DAG having an operational context that includes the user code and user code permissions, that govern access permissions during execution on an ETL engine. When execution of the DAG ceases, the user code and user code permissions are removed from the ETL system, according to certain embodiments.

INTRODUCTION

Aspects of the present disclosure relate to multitenant environments, and more particularly to determining operational permissions for a tenant in a multitenant extract, transform, load (ETL) environment.

Extract, transform, and load (ETL) processes schedule and orchestrate access to data sources by multiple user code bases. User code bases may be represented as directed acyclic graphs (DAGs) which visually diagrams the operations of an ETL instance. For a given DAG, user code represents a node in a graph, input arrows into the node represent dependencies of the code associated with that node, and the results generated by the user code are carried forward (e.g., as an input into another node). The DAG representing user code may thus be a graph that is directed and acyclic, as the output of the DAG may not serve as input into the same DAG, and each vertex between nodes in the DAG may have a directionality (e.g., indicating that the output of a node is an input or dependency of another node). Multiple DAGs may be executed in an instance of an ETL process, with the instance of the ETL process providing functionality of scheduling and orchestration of their execution in access to one or more data sources, also called data warehouses, data lakes, and the like. Further, the instance of the ETL process manages resources in a computing environment, spinning up additional compute resources as needed for operation of the multiple DAGs. To manage resources for the execution of DAGs in the computing environment, the instance of the ETL process has a permission role in the computing environment for obtaining and releasing computing resources.

Within the instance of the ETL process, a default identity and access management (IAM) role generally governs operation of the various DAGs, enabling the DAGs to access compute resources (e.g., memory resources, processing resources, etc.) as well as other DAGs. However, because there are generally a limited number of default IAM modes that provide access context across a universe of DAGs, managing multiple DAGs becomes challenging. For example, a default IAM permissions role that allows for access to all DAGs in the instance of the ETL process can result in unintended access to compute resources, as one DAG can access the resources of another DAG. In response, ETL administrators may spin up multiple instances of the ETL process, but this increases complexity as a DAG in one instance of the ETL process may require access to a DAG in another instance of the ETL process, even though the compute resources for one ETL process may be isolated from the compute resources for another ETL process (e.g., inaccessible to other ETL processes). Moreover, acquiring additional CPU, memory, network, load balancers, and other resources for provisioning multiple instances of the ETL process increases cost and operational complexity with each additional instance of the ETL process.

Thus, what is needed are systems and methods for providing individual internal IAM role permissions for individual DAGs in an ETL instance.

BRIEF SUMMARY

Certain embodiments provide a method for executing code in a multitenant extract, transform, and load (ETL) environment. The method generally includes receiving user code at the ETL environment, the ETL environment being configured with an ETL permissions role. User code is executed in the ETL environment. Executing the user code in the ETL environment generally includes receiving a user code permissions role, generating a context based on the user code permissions role, and executing the user code in the ETL environment based on the context.

Certain embodiments provide for a system for provisioning a multitenant extract, transform, and load (ETL) environment, which includes a memory comprising computer-readable instructions, and a processor configured to execute the computer-readable instructions. The computer-readable instructions cause the system to generate a reference to user code associated with a user account, create a pipeline comprising a user code permission associated with the user code, provide the pipeline and the reference to user code, responsive to a first request from an ETL environment, and receive request for validation of user code permissions associated with the user code, responsive to a validation request from the ETL environment.

DETAILED DESCRIPTION

Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums for multitenancy pipeline orchestration tools for ETL systems. According to certain embodiments, when a user creates code in a code repository, the user creates a set of permissions needed for the user code to run in an ETL system. The user code, also referred to as a directed acyclic graph (DAG) herein is stored in a code repository, and permissions are stored in a database.

Within an instance of an ETL process (also referred to as an “ETL instance”), a plugin is provided to obtain and manage the IAM role permissions of individual DAGs. A user logs into the ETL instance and imports code from the code repository, invoking the plugin to retrieve the permissions for the DAG, that are embedded in the DAG at execution time. The embedded permissions generally provide an operational context for the DAG during execution, preventing access and operations not provided in the permissions either to ETL resources or to other DAGs operating in the ETL instance.

By providing discrete permissions for individual DAGs, the ETL instance becomes a multitenant environment in which multiple DAGs can be executed, with each DAG being governed by its associated permissions. In this context, a multitenant environment is a computing environment in which each operating instance of user code, or DAG, has unique operational permissions governing access to other DAGs or compute resources available in the ETL instance. The techniques discussed herein may, thus, allow DAGs for an organization requiring access to each other and/or common data and compute resources to be executed in a single multitenant ETL instance. By operating multiple DAGs according to principles of systems and methods disclosed herein, CPU time, memory, power, network bandwidth, and other resources may be saved by operating these DAGs in a single ETL multitenant instance, instead of multiple ETL instances in which default IAM roles provide access context for the DAGs executing within each of the multiple ETL instances. Further, security issues in ETL instances using default IAM roles may be remedied by providing a multitenant ETL for multiple DAGs, with each DAG having defined operational permissions preventing access to resources (e.g., compute resources and other DAGs) that are not included in the permissions of the DAG. Moreover, by operating multiple DAGs in a single instance, security issues caused by the need to have multiple ETL instances, and networks connecting the multiple instances, needed for a DAG requiring access to DAG in a different instance.

Example System for Multitenancy Extract, Transform, Load (ETL) Pipeline Orchestration

FIG.1depicts a system100for multitenancy extract, transform, load (ETL) pipeline orchestration, according to certain embodiments. As discussed herein, an ETL pipeline generally refers to a process by which user code defines a process by which data is extracted from a data source, processed, and loaded into a destination system (e.g., represented by another artifact of user code). ETL pipelines within system100may be defined, for example, for multiple artifacts of user code, and each artifact of user code104may be associated with a set of permissions to be used in executing the user code defining or otherwise associated with an ETL pipeline. These ETL pipelines, with the associated sets of permissions, may allow for an ETL system to become a multitenant environment so that a reduced number of ETL process instances are needed to execute different user code artifacts. The permissions may allow user code artifacts for ETL pipelines, for example, to interact with other user code artifacts while blocking the user code artifacts from interacting with other artifacts for which permission has not been granted. Thus, multiple user code artifacts may execute within a single ETL process while maintaining data security within the ETL process and minimizing the amount of compute resources dedicated to executing user code in an ETL environment.

A user develops user code104and stores the user code in a code repository108. Generally, the user code104may be a code artifact that defines a process for extracting data from a data repository, processing the data (e.g., transforming the extracted data from one format to another format), and loading the processed data into another process. Associated with the user code104, the user creates user code permissions112required to execute the user code in the pipeline component116of a processing platform120, and the user code permissions112are stored in a database124. Generally, the user code permissions112associated with the user code104may define the permissions that user code104has with respect to other user code artifacts executing within ETL system134. By way of example and not limitation using AIVIAZON™ AWS™ computing environments, user code permissions112may be a permission to access a computing resource such as a particular EC2™ instance, LAMBDA™ instance, S3™ instance, SAGEMAKER™ instance, or other computing instance or environment, or other user code embodied in a DAG discussed below. According to certain embodiments, user code permissions112includes a user ID of the creator and a unique identifier, that may be created based on the user code104file name or other aspect of the user code104. The unique identifier may be a hash of the file name or other aspect of the user code104. Further, upon creation of user code permissions112, an authorization component128may provide the unique identifier in the form of a user credential related to a user account132of the user.

System100further includes an ETL system134. According to certain embodiments, ETL system134may be one of AIRFLOW™, AIRBYTE™, FIVETRAN™, STITCH™, MATILLION™, SINGER™, or other ETL systems. DAG138includes user code104, which in some embodiments is imported from code repository108, and user code permissions112, provided by a plugin142, that make up a context140.

ETL system134, by way of example, may be utilized by computing environments having separate transaction database systems and data warehouses. ETL systems such as ETL system134extract, transform, and load data from a variety of sources (e.g., transaction databases, data warehouses) and update relevant data sources, based on user code embodied in DAGs. In this context, an ETL system schedules operation of user code, and orchestrates execution of user code based on dependencies in user code upon other user code, data sources, and transactions occurring in other systems. Conventionally, ETL system134has an ETL permissions role that provides the ETL system134with permissions in the context of a computing environment to allocate and deallocate computing resources, by way of example using AWS™, these resources may include and are not limited to EC2™, S3™, LAMBDA™, SAGEMAKER™, and other available resources. ETL system134includes default permissions136that may in some embodiments provide an operational context for one or more DAGs. User code permissions112incorporated into a DAG as described herein replace, or operate in addition to, the default DAG permissions.

When user code104is uploaded to DAG138, plugin142makes an application program interface (API) call to the processing platform120for user authentication credentials130related to user code104. Based on the user authentication credentials130and file name, user code permissions are provided to the plugin142. The user code permissions are integrated into the DAG138as the context140, for execution on the ETL engine146. According to certain embodiments, context validation component150validates that the user code permissions112are the correct permissions for use with user code104embodied in the DAG138by verifying the unique identifier, validating the user authentication credentials130with an API call to the authorization component128, that validates the user authentication credentials130with the user account132. According to certain embodiments, context validating component150may use the unique identifier associated with the user code permissions112and related user ID to validate the user code permissions112against user code. Validating the user code permissions112prevents user code from a different user from utilizing the permissions associated with user code104, preventing potential security issues that may arise from these different users interacting with user code104and the data associated with or processed by user code104.

Example Process

FIG.2depicts a process200, according to certain embodiments. At block204a pipeline is created at the pipeline component. The pipeline created at the pipeline component generally implements user code permissions, such as user code permission112illustrated inFIG.1, for a user executing code within an ETL system134. As discussed, user code permission112is generally developed by a user and is associated with user code104, and user code permission112generally defines permissions for user code104(e.g., with respect to systems or other user code with which the user code104is allowed to interact and with which the user code104is prohibited from interacting) when embodied in DAG138in the ETL system134.

When the user code permission112are provided as part of DAG138, ETL system134can apply the user code permissions112provided as part of DAG138instead of a default set of permissions defined for the ETL system134. That is, the DAG138is limited in access and execution (e.g., interaction with other systems and/or user code) by the user code permissions112. By limiting the permissions of the DAG138implementing user code104, DAG138may be isolated from DAGs associated with unrelated user code (e.g., user code developed by other users of ETL system134) and thus may not access resources to which the user code104ordinarily would not have rights to access and should not have rights to access. Further, by providing multiple DAGs138with unique user code permission, ETL system134may operate as a multitenant environment in which each DAG is limited in execution by its code permissions, enabling multiple DAGs to be securely, independently scheduled, and orchestrated within the ETL engine146.

At block208, the user code permissions, and related user ID, and in some embodiments, user credentials, and a unique identifier, are stored in database124.

At block212, user code104is provided to the instance of ETL system134. Plugin142is invoked to retrieve user code permissions112to incorporate the user code104into DAG138. At block216, the user code104is authenticated with the authorization component128based on the user authentication credentials130of the user account132.

At block220, user code permissions112are validated by the context validation component150. Using the user code permissions112and associated user ID, and in some embodiments user authentication credentials130, from database124, user code permissions112is validated against user authentication credentials130of the user account132. Upon validation, at block224authentication credentials are generated for the user code permissions112in the plugin142, that are provided to the ETL system134, enabling integration of the user code104and user code permissions112to form DAG138. At block228, the authentication credentials are provided to the ETL system134. At block232, the DAG138, including user code104and user code permissions112, are provided to the ETL engine146by the ETL system134for execution. As discussed, because the ETL engine146is provisioned with the DAG138including user code104and user permissions112, the ETL engine146can execute the DAG to perform an ETL process while maintaining the isolation of the ETL process for user code104from unrelated ETL processes and/or other computing resources.

Example Timing Diagram for Executing User Code in an ETL System

FIG.3depicts a timing diagram300, according to certain embodiments. At arrow308, a user304creates user code permissions112needed by user code104to cause the ETL system134to execute the DAG138based on the user code104and user code permissions112as a tenant of a multitenant environment. Generally, the user code permissions112generally define systems and user code artifacts with which user code104is permitted to interact. For example, the permissions may indicate that user code104is allowed to interact with other user code or other systems owned by the user associated with user code104, but may not be allowed to interact with user code or other systems owned by some other user. In another example, the permissions may indicate that user code104is allowed to interact with user code or systems owned by the user associated with user code104or other users in a defined list of users. The permissions may indicate user code and systems that user code104is allowed to interact with, and it may be understood, when ETL system134generates a DAG for the user code104, that the user code104is blocked from interacting with user code or systems not identified (explicitly or implicitly) in the associated user code permissions112.

At arrow312, the user code permissions112are stored in the database124, in addition to the user ID, a unique name associated with the user code permissions, and in some embodiments, user authentication credentials130associated with the user ID.

At arrow316, DAG138is defined in the ETL system. In this context, the DAG138is defined by the user code104and user code permissions112. Generally, DAG138implements a process defined by the user code104for extracting data from one or more data sources, performing one or more transformations on the data, and outputting the transformed data for use by one or more other DAGs or other systems.

At arrow320, the plugin142validates the user code permissions as being associated with the user code104by context validation component150, as described above.

At arrow324, one of the user ID, permissions code unique identifier, or user credential are provided to the processing platform120for validation against the database124record associated with user code104embodied in the DAG138. At arrow328, an affirmative validation message is received from the processing platform120based on validating the provided information against the database124record associated with user code104, indicating permission to proceed. Generally, permission to proceed may be received if the provided information matches information included in user code permissions112(e.g., information provided about a user to processing platform is also included in user code permissions112). When permission to proceed is received from ETL system134, the DAG is executed in the ETL engine146of the ETL system134. If an affirmative validation message is not received from the processing platform120, execution of the DAG138does not proceed, as the user may not have rights or permission to execute the DAG138embodying user code104(e.g., because the user code104is owned by a different user and this different user has not granted permissions.

At arrow332, the DAG138is executed by the ETL engine. When execution is complete, the DAG138, in addition to the associated user code104and user code permissions112, are deleted from the ETL system134.

At arrow334, status of the execution, or deletion, of the DAG138is transmitted to the processing platform120by the plugin142of the ETL system134, and at arrow336, the processing platform120transmits the status to the user304.

Example Methods for Executing User Code in a Multitenant ETL Environment

FIG.4depicts an example method400for executing user code in a multitenant extract, transform, load (ETL) environment, according to certain embodiments.

At block404, the ETL system134receives the user code104. According to certain embodiments, the ETL system134is configured with default permissions136, and executing the user code in the ETL system134.

At block408, the executing includes receiving user code permissions112from the processing platform120. According to certain embodiments, user code permissions are generated based on a code file associated with the user code, the code file having a code file name. According to certain embodiments, the user code comprises a user code name, and the method further comprises validating the context at least in part by receiving an indication that the user code name is the same as the code file name.

At block412, the ETL system134generates, via plugin142, a context140based on the user code permissions112. According to certain embodiments, generating the context further comprises receiving user authentication credentials. According to certain embodiments, receiving the user credentials comprises receiving the user authentication credentials from a database row identified with a hash based on the user authentication credentials and the code file name

At block416ETL system134executes the user code104in the ETL system134based on the context140. According to certain embodiments, executing the user code in the ETL system comprises generating a connection with the ETL system and providing three user code and validated context to the ETL system.

According to certain embodiments, the method further comprises generating a user code authentication token based on the user authentication credentials, the user code authentication token comprising an expiration time.

Note thatFIG.4is just one example of a method, and other methods including fewer, additional, or alternative steps are possible consistent with this disclosure.

Example Methods for Provisioning a Multitenant ETL Environment

FIG.5depicts an example method500for provisioning a multitenant ETL environment, according to certain embodiments. At block504, the processing platform120generates a reference to user code associated with a user account.

At block508, the processing platform120creates a user code permission associated with the user code.

At block512, the processing platform120provides the user code permission and the reference to user code, responsive to a first request from an ETL system134.

At block516, the processing platform120receives request for validation of user code permissions associated with the user code, responsive to a validation request from the ETL system134.

According to certain embodiments, the processing platform120receives an authentication request from the ETL system134, generates user authentication credentials associated with the user account, and provides the user authentication credentials to the ETL system, responsive to the authentication request. According to certain embodiments, a context140is generated at the ETL system134based on the user code and user code permissions. According to certain embodiments, the user code is executed at the ETL system134based on the context140.

According to certain embodiments, the processing platform120generates a unique name for the user code permission, and stores the user code permission such that the user code permission is indexed based on the unique name.

According to certain embodiments, the processing platform120receives a validation request for a second user code name based on the unique name, compares the second user code name to the unique name, and validates the second user code name responsive to the second user code name being different than the unique name.

Example Processing System for Provisioning and Executing User Code in a Multitenant ETL Environment

FIG.6depicts an example processing system600, according to certain embodiments that may perform methods described herein, such as the method for orchestrating a multitenant ETL environment described with respect toFIGS.3-5.

Processing system600includes a central processing unit (CPU)602connected to a data bus616. CPU602is configured to process computer-executable instructions, e.g., stored in memory608, and to cause the processing system600to perform methods described herein, for example, with respect toFIGS.3-5. CPU602is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, and other forms of processing architecture capable of executing computer-executable instructions. Memory608is included to be representative of one or more memory devices such as volatile memories, that may be a RAM, cache, or other short-term memory that may be implemented in hardware or emulated in software, one or more non-volatile memories such as a hard drive, solid state drive, or other long term memory that may be implemented in hardware or emulated in software, or a combination of volatile and non-volatile memories. Moreover, one or more memory devices that makeup memory608may be located remotely from processing system600and accessed via a network.

Processing system600further includes input/output (I/O) device(s)612and interfaces604, which allows processing system600to interface with input/output devices612, such as, for example, keyboards, displays, mouse devices, pen input, and other devices that allow for interaction with processing system600. Note that processing system600may connect with external I/O devices through physical and wireless connections (e.g., an external display device).

Processing system600further includes a network interface606, which provides processing system600with access to external network614and thereby external computing devices.

Processing system600further includes memory608, which in this example includes a receiving component618, an executing component620, and a generating component622, for performing operations described inFIGS.3-5. Memory608further includes ETL system data628, user code data630, default permissions data632, user code permissions image data634, and context data636, that may be used in performing operations described inFIGS.3-5.

Note that while shown as a single memory608inFIG.6for simplicity, the various aspects stored in memory608may be stored in different physical memories, including memories remote from processing system600, but all accessible by CPU602via internal data connections such as bus616.

Note thatFIG.6is just one example of a processing system, and other processing systems including fewer, additional, or alternative steps are possible consistent with this disclosure.

Example Clauses

Clause 1: A method for a multitenant extract, transform, and load (ETL) environment system (ETL), comprising: receiving user code at the ETL system, the ETL system being configured with a default permissions; executing the user code in the ETL system, including: receiving user code permissions; generating a context based on the user code permissions; and executing the user code in the ETL system based on the context.

Clause 2: A method, wherein generating the context further comprises receiving user authentication credentials for performing a method in accordance with Clause 1.

Clause 3: A method, wherein the user code permissions are generated based on a code file associated with the user code, the code file having a code file name, for performing a method in accordance with any one of Clauses 1-2.

Clause 4: A method, wherein: the user code comprises a user code name, and the method further comprises validating the context at least in part by receiving an indication that the user code name is the same as the code file name, for performing a method in accordance with any one of Clauses 1-3.

Clause 5: A method, further comprising generating a user code authentication token based on the user authentication credentials, the user code authentication token comprising an expiration time, for performing a method in accordance with any one of Clauses 1-4.

Clause 6: A method, wherein executing the user code in the ETL system comprises: generating a connection with the ETL system; and providing the user code and validated context to the ETL system, for performing a method in accordance with any one of Clauses 1-5.

Clause 7: A method, wherein receiving user authentication credentials comprises receiving the user authentication credentials from a database row identified with a hash based on the user authentication credentials and the code file name, for performing a method in accordance with any one of Clauses 1-6.

Clause 8: A system for provisioning a multitenant extract, transform, and load (ETL) system, comprising: a memory comprising computer-readable instructions; a processor configured to execute the computer-readable instructions that cause the system to: generate a reference to user code associated with a user account; create a user code permission associated with the user code; provide the user code permission and the reference to user code, responsive to a first request from an ETL system; and receive request for validation of user code permissions associated with the user code, responsive to a validation request from the ETL system.

Clause 9: A system, wherein the computer-readable instructions further cause the processor to: receive an authentication request from the ETL system; generate user authentication credentials associated with the user account; and provide the user authentication credentials to the ETL system, responsive to the authentication request, to cause the system to perform a method in accordance with Clause 8.

Clause 10: A system, wherein the computer-readable instructions further cause the system to generate the context further comprises receiving user authentication credentials, to cause the system to perform a method in accordance with any one of Clauses 8-9.

Clause 11: A system, wherein the user code is executed at the ETL system based on the context, to cause the system to perform a method in accordance with any one of Clauses 8-10.

Clause 12: A system, wherein the computer-readable instructions further cause the processor to: generate a unique name for the user code permission; and store the user code permission such that the user code permission is indexed based on the unique name, to perform a method in accordance with any one of Clauses 8-11.

Clause 13: A system, wherein the computer-readable instructions further cause the processor to: receive a validation request for a second user code name based on the unique name; compare the second user code name to the unique name; and validate the second user code name responsive to the second user code name being different than the unique name, to perform a method in accordance with any one of Clauses 8-12.

Additional Considerations