Dataflow Pipeline Deployment System and Method

A computer-based method for managing a dataflow pipeline across a plurality of environments, includes the steps of: receiving, for a first environment, a check-in comprising source code, a task definition, a configuration, and a flow definition; receiving a definition of an application secret and an associated credential; storing the application secret and credential in a dataflow pipeline deployer data store; creating a data flow pipeline package, further comprising the steps of: merging the source code, the configuration, and the secrets; and defining a pipeline graph in the first environment pipeline flow registry.

FIELD OF THE INVENTION

The present invention relates to continuous integration and continuous delivery (CI/CD) software development tools, and more particularly, is related to in-service software updates.

BACKGROUND OF THE INVENTION

Availability of computer programs is essential to many businesses. A business may lose valuable time during software updates for mission critical programs. For example, large, distributed software applications may include multiple processes that may each have their own development schedules, which can lead to frequent outages while the various processes are updated. Further, dependencies across components may be complex, for example, if updating a first process requires additional updates to upstream and/or downstream processes. Typically, updating of any one component in a multi-component system results in downtime for the entire system.

For example, a developer implements a change by making a working copy (“branch”) of the current code base, updating the code in a snapshot of the code base at one moment in time. As other developers submit changed code to the source code repository, this snapshot ceases to reflect the (live) repository code. As the existing code base changes, new code may also be added, as well as new libraries, and other resources that create dependencies, and potential conflicts. The longer development continues on a branch without merging back to the mainline, the greater the risk of multiple integration conflicts and failures when the developer branch is eventually merged back. When developers submit code to the repository, they must first update their code to reflect the changes in the repository since they took their copy. The more changes the repository contains, the more work developers must do before submitting their own changes.

Therefore, there is a need in the industry to address the abovementioned shortcomings.

SUMMARY OF THE INVENTION

The present system and method provides a dataflow pipeline deployment system and method. A computer based software development system is configured to deploy a plurality of package environments, comprising: a processor and storage device configured to store non-transient instructions that when implemented by the processor comprise: a dataflow pipeline deployer (110); a Git repository (120) configured to store a flow definition and an environment specific parameter; a secret store (125) configured to store an API key/secret and data pipeline credentials; and for each environment of the plurality of environments: a pipeline flow registry (140,170) configured to store a static definition (151,181) of an environment specific production dataflow pipeline (152,182); a dataflow pipeline container (150,180) configured to contain the production dataflow pipeline and a store (154,184) of environment specific secrets and parameters.

A computer-based method for managing a dataflow pipeline across a plurality of environments, is also provided, comprising the steps of: receiving, for a first environment, a check-in comprising source code, a task definition, a configuration, and a flow definition; receiving a definition of an application secret and an associated credential; storing the application secret and credential in a dataflow pipeline deployer data store; creating a data flow pipeline package, further comprising the steps of: merging the source code, the configuration, and the secrets; and defining a pipeline graph in the first environment pipeline flow registry.

Other systems, methods and features of the present invention will be or become apparent to one having ordinary skill in the art upon examining the following drawings and detailed description. It is intended that all such additional systems, methods, and features be included in this description, be within the scope of the present invention and protected by the accompanying claims.

DETAILED DESCRIPTION

The following definitions are useful for interpreting terms applied to features of the embodiments disclosed herein and are meant only to define elements within the disclosure.

As used within this disclosure, “continuous integration” (CI) is a software engineering term referring to the practice of regularly merging working copies of all developers into a shared mainline, for example, several times a day. CI is often intertwined with continuous delivery (CD) or continuous deployment in a CI/CD pipeline. “Continuous delivery” ensures the software checked in on the mainline is always in a state that can be deployed to users and “continuous deployment” makes the deployment process fully automated.

A CI/CD pipeline automates the software delivery process. The pipeline builds code, runs tests (CI), and safely deploys a new version of the application (CD). Automated pipelines remove manual errors, provide standardized feedback loops to developers, and enable fast product iterations.

As used within this disclosure, a “pipeline” (or “Git pipeline,” for instance an Apache Beam Pipeline) refers to an extensible set of tools for modeling build, testing and deploying code. Pipelines are a top-level component of continuous integration, delivery, and deployment. Pipelines include jobs and stages. Pipeline jobs define what to do, for example, compile or test code. Pipeline stages, define when to run the jobs. For example, a pipeline may have four stages, executed in the following order:A build stage, with a job called compileA test stage, with two jobs called test1 and test2A staging stage, with a job called deploy-to-stageA production stage, with a job called deploy-to-prod

Jobs are executed by runners. Multiple jobs in the same stage are executed in parallel if there are enough concurrent runners. If all jobs in a stage succeed, the pipeline moves on to the next stage. If any job in a stage fails, the next stage is typically not executed, and the pipeline ends early.

As used within this disclosure, “Git” refers to an open-source software used for tracking project changes and revisions across different development teams. Git saves different versions of projects in a folder known as a Git repository (“Git repo”). A Git repository tracks and saves the history of all changes made to the files in a Git project. The Git repository saves this data in a directory called .git, also known as the repository folder. Git uses a version control system to track all changes made to the project (including source code) and saves them in the repository.

As used within this disclosure, a “dataflow” refers to a template configured to allow development teams to share pipelines with team members and across their organization. A dataflow may implement one or more data processing tasks.

As used within this disclosure a “secret” generally refers to secure/sensitive data to be accessed used while processing a pipeline. A secret generally requires some sort of token (e.g., username and password, decryption key, etc.) for access. Such a token may be provided via a resolving entity.

As used within this disclosure, an “environment” refers to one of several spaces where related software systems are maintained. For example, a development environment where software is initially developed, a Quality Assurance (QA) environment where the software is tested in the context of a system before deployment, and a production (prod) environment where the released system operates.

Embodiments of the present invention are directed towards a dataflow pipeline deployer for real-time modification of software while the software is being run. The system determines what modules of the software are being utilized in the current running of the application and will be utilized in the immediate future, based on the process that is running and associated processes that need to run for the process to be complete. The software only modifies the modules that are not being used currently or will need to be used in the near future. Such modifications may include selective updating of software modules.

The dataflow pipeline deployer manages source code in a Git repo, secrets in secret stores for each of a plurality of environments, a sequencer and specifics for each environment, and a pipeline registry which has a static definition of the pipeline.

FIG.1is a block diagram of an exemplary first embodiment of a system100for dataflow management. A dataflow pipeline manager110manages pipelines in multiple environments, for example, a production environment130, a QA environment160, and a stage environment (not shown). The dataflow pipeline deployer110accesses a Git repository120of source code and a secret store125. The Git repository120includes flow definitions and environment specific parameters. The secret store125may include environment specific API application programming interface (API) keys and/or secrets, as well as environment specific application data pipeline credentials.

In the production environment130, a production pipeline container150contains a production pipeline152and a store of secrets and parameters154. The production pipeline152includes a plurality of components (shown as rectangles with solid lines) and flows between the components (shown as solid arrows). A production pipeline flow registry140has a static definition151of the production dataflow pipeline152. The dashed boxes represent static definitions of the components of the production dataflow pipeline152.

The QA environment160has a similar structure to the production environment. In the QA environment160, a QA pipeline container180contains a QA pipeline182and a store of secrets and parameters184. The QA pipeline182includes a plurality of components (shown as rectangles with solid lines) and flows between the components (shown as solid arrows). A QA pipeline flow registry170has a static definition181of the QA dataflow pipeline182. The dashed boxes represent static definitions of the components of the QA dataflow pipeline182.

The dataflow pipeline manager110performs several functions, including (but not limited to):Managing versioning of data flow definitions in a Git repository120.Instantiating and deploying runtime data flow definitions in each environment, for example, the production environment130and the QA environment160.Checking out runtime data flow definitions to the pipeline container150.Managing secrets per environment.Deploying new versions of secrets and parameters only affecting a subset of the flows, andPromoting the package to production.

For example, under the first embodiment, a component first version v1of the production pipeline152in the production pipeline container150is to be updated by a component second version v2in the production pipeline flow registry140. A third version v3of the component is in the QA pipeline flow registry170. Once the third version v3of the component is validated in the QA environment160, the third version v3may be promoted to the production environment150, along with any associated secrets and parameters from the QA dataflow pipeline container180.

It is desirable for the update to only replace components in the production environment130that are affected by the change, thereby only stopping associated portions of the production pipeline152while leaving other components operations. Likewise, if the update only involves changing a secret or a parameter, only portions of the production pipeline152affected by the change of secret/parameter are stopped. Here, the deployer110combines the source code, the secrets/parameters, and the flow definitions.

Under the present embodiments, the definition of a pipeline differs from a traditional Git pipeline. A traditional Git pipeline store source code and run-time definitions. Under the first embodiment the Git pipeline stores source code, dataflow, and secrets as parameters. In a traditional Git pipeline versions are promoted from environment to environment and eventually published. Under the dataflow pipeline embodiments, other components are orchestrated along with source code promotion.

For example, a component to be paused may receive input data from an upstream queue, and provide output data to a downstream queue. In order to pause the component, the upstream queue must similarly be paused while the component is updating, so that data from the upstream queue is not lost during the update.

During the update/promotions, only specifically identified components are paused, refreshed, and then un-paused.

FIG.2is a flowchart200of an exemplary embodiment for a computer based method for managing a dataflow pipeline across a plurality of environments. It should be noted that any process descriptions or blocks in flowcharts should be understood as representing modules, segments, portions of code, or steps that include one or more instructions for implementing specific logical functions in the process, and alternative implementations are included within the scope of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

FIG.2is directed to initial deployment of source code into a first environment, for example, a QA environment. The dataflow pipeline deployer receives a check-in, for example, from a development engineer, as shown by block210. The check-in may include one or more of source code, a task definition, a configuration, and a flow definition. The dataflow pipeline deployer receives a definition of an application secret and an associated credential, for example, from an administrator, as shown by block220. The dataflow pipeline manager stores the application secret and credential in a dataflow pipeline deployer data store, for example, the secret store125(FIG.1), as shown by block230. The dataflow pipeline manager creates a data flow pipeline package, as shown by block240. The Creating the data flow pipeline package may include, for example, merging the source code, the configuration, and the secrets, and defining a pipeline graph in the first environment pipeline flow registry.

FIG.3Ais a flowchart300directed to deployment of updated source code into the first environment. The dataflow pipeline deployer receives an updated version including at least one of an updated flow definition, an updated task definition, an updated configuration, and an updated source code, as shown by block310. The dataflow pipeline deployer identifies a change in the updated version, as shown by block320. The dataflow pipeline deployer identifies a task affected by the identified change, as shown by block330.

After the updated version is verified in the first environment, the dataflow pipeline deployer promotes the updated version to a second environment, for example, a staging environment or a production environment, as shown by the flowchart301ofFIG.3B. The updated version is promoted from the first environment to the second environment, for example, a staging or production environment, as shown by block340. A task flow associated with the identified task in the second environment is paused, as shown by block350. The updated version is deployed in the second environment, as shown by block360. Task flows in the second environment not associated with the identified task in the second environment are not paused.

FIG.4Ais a flowchart400directed to deployment of an updated secret and/or configuration into the first environment. The dataflow pipeline deployer receives an updated version including an updated configuration and/or an updated secret, as shown by block410. The dataflow pipeline deployer identifies a change in the updated version, as shown by block420. The dataflow pipeline deployer identifies a task affected by the identified change, as shown by block430.

After the updated version is verified in the first environment, the dataflow pipeline deployer promotes the updated version to a second environment, for example, a staging environment or a production environment, as shown by the flowchart401ofFIG.4B. The updated version is promoted from the first environment to the second environment, for example, a staging or production environment, as shown by block440. A task flow associated with the identified task in the second environment is paused, as shown by block450. The updated version is deployed in the second environment, as shown by block460. Task flows in the second environment not associated with the identified task in the second environment are not paused.

As shown byFIG.6, under a second embodiment600of the present invention, a single pipeline flow registry is shared across two or more environments.

The present system for executing the functionality described in detail above may be a computer, an example of which is shown in the schematic diagram ofFIG.5. The system500contains a processor502, a storage device504, a memory506having software508stored therein that defines the abovementioned functionality, input, and output (I/O) devices510(or peripherals), and a local bus, or local interface512allowing for communication within the system500. The local interface512can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface512may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface512may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.

The processor502is a hardware device for executing software, particularly that stored in the memory506. The processor502can be any custom made or commercially available single core or multi-core processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the present system500, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or generally any device for executing software instructions.

The memory506can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.). Moreover, the memory506may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory506can have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor502.

The software508defines functionality performed by the system500, in accordance with the present invention. The software508in the memory506may include one or more separate programs, each of which contains an ordered listing of executable instructions for implementing logical functions of the system500, as described below. The memory506may contain an operating system (O/S)520. The operating system essentially controls the execution of programs within the system500and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.

The I/O devices510may include input devices, for example but not limited to, a keyboard, mouse, scanner, microphone, etc. Furthermore, the I/O devices510may also include output devices, for example but not limited to, a printer, display, etc. Finally, the I/O devices510may further include devices that communicate via both inputs and outputs, for instance but not limited to, a modulator/demodulator (modem; for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, or other device.

When the system500is in operation, the processor502is configured to execute the software508stored within the memory506, to communicate data to and from the memory506, and to generally control operations of the system500pursuant to the software508, as explained above.

When the functionality of the system500is in operation, the processor502is configured to execute the software508stored within the memory506, to communicate data to and from the memory506, and to generally control operations of the system500pursuant to the software508. The operating system520is read by the processor502, perhaps buffered within the processor502, and then executed.

When the system500is implemented in software508, it should be noted that instructions for implementing the system500can be stored on any computer-readable medium for use by or in connection with any computer-related device, system, or method. Such a computer-readable medium may, in some embodiments, correspond to either or both the memory506or the storage device504. In the context of this document, a computer-readable medium is an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer-related device, system, or method. Instructions for implementing the system can be embodied in any computer-readable medium for use by or in connection with the processor or other such instruction execution system, apparatus, or device. Although the processor502has been mentioned by way of example, such instruction execution system, apparatus, or device may, in some embodiments, be any computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “computer-readable medium” can be any means that can store, communicate, propagate, or transport the program for use by or in connection with the processor or other such instruction execution system, apparatus, or device.

In an alternative embodiment, where the system500is implemented in hardware, the system500can be implemented with any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon data signals, an application specific integrated circuit (ASIC) having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.