Package: adhesive
Adhesive is a micro BPMN runner written in Python.You can easily model complex logic in BPMN, and Adhesive will execute it
for you taking care of parallelism, joining, etc. using the standard
BPMN notation.Since it’s small, it can easily be embedded in containers, or replace
complex scripts.InstallationpipinstalladhesiveGetting StartedSimple BuildsTo create a basic build you just create a file in your project named_adhesive.py. In it you then declare some tasks. For example:importadhesive@adhesive.task("Checkout Code")defcheckout_code(context):adhesive.scm.checkout(context.workspace)@adhesive.task("Run Build")defrun_build(context):context.workspace.run("mvn clean install")adhesive.build()Since no process was defined, adhesive takes the defined tasks, stitches
them in order, and has a process defined as<start>→Checkout Code→Run Build→<end>.To run it simply calladhesivein the terminal:adhesiveThis is the equivalent of Jenkins stages. But we can do better:Programmatic BuildsIn order to use the full programmatic functionalities that adhesive
offers, you are able to stitch your BPM process manually. You have sub
processes, branching and looping available:importadhesiveimportuuid@adhesive.task("Run in parallel item{loop.value}")defcontext_to_run(context):ifnotcontext.data.executions:context.data.executions=set()context.data.executions.add(str(uuid.uuid4()))data=adhesive.process_start()\.branch_start()\.sub_process_start()\.task("Run in parallel",loop="items")\.sub_process_end()\.branch_end()\.branch_start()\.sub_process_start()\.task("Run in parallel item{loop.value}",loop="items")\.sub_process_end()\.branch_end()\.process_end()\.build(initial_data={"items":[1,2,3,4,5]})assertlen(data.executions)==10Here you see the full BPMN power starting to unfold. We create a process
that branches out, creates sub processes (sub processes can be looped as
a single unit). Loops are creating execution tokens that also run in
parallel in the same pool.Note that you can passinitial_datainto the process, and you can
also get thecontext.datafrom the last execution token.BPMN ProcessLast but not least, adhesive reads BPMN files, and builds the process
graph from them. This is particularly good if the process is complex and
has a lot of dependencies:Thebuild of adhesiveis modeled as aBPMN processitself, so we
load it from the file directly using:adhesive.build_bpmn("adhesive-self.bpmn")importadhesive@adhesive.task("Read Parameters")defread_parameters(context)->None:context.data.run_mypy=Falsecontext.data.test_integration=True@adhesive.task(re=r"^Ensure Tooling:\s+(.+)$")defgbs_ensure_tooling(context,tool_name)->None:ge_tooling.ensure_tooling(context,tool_name)# ...adhesive.build_bpmn("adhesive-self.bpmn")As you see steps are parametrizable, and use the data from the task name
into the step definition.Defining BPMN TasksFor example here, we define an implementation of tasks using regex
matching, and extracting values:@adhesive.task(re=r"^Ensure Tooling:\s+(.+)$")defgbs_ensure_tooling(context,tool_name)->None:# ...Or a user task (interactive form):@adhesive.usertask('Publish to PyPI?')defpublish_to_pypi_confirm(context,ui):ui.add_checkbox_group("publish",title="Publish",values=(("nexus","Publish to Nexus"),("pypitest","Publish to PyPI Test"),("pypi","Publish to PyPI"),),value=("pypitest","pypi"))Don’t forget, the@adhesive.taskand@adhesive.usertaskare just
defining mappings for implementations of the task names available in the
process. Only theadhesive.build()creates a linear process out of
the declaration of the tasks.As you notice, there’s always a first parameter namedcontext. Thecontextparameter contains the following information:task- the Task in the graph that’s currently matched against
this execution.task_name- The resolved name, with the variables interpolated.
Matching is attemptedafterthe name is resolved.data- Data that the current execution token contains. This data
is always cloned across executions, and `set`s and `dict`s are
automatically merged if multiple execution tokens are merged. So you
have a modifiable copy of the data that you’re allowed to change, and
is propagated into the following execution tokens.loop- if the current task is in a loop, the entry contains itsindex, thekeyandvalueof the items that are iterating,
and theexpressionthat was evaluated. Note that loop execution
happens in parallel since these are simple execution tokens.lane- the current lane where the tasks belongs. Implicitly it’sdefault.workspace- a way to interact with a system, and execute
commands, create files, etc.adhesiveruns all the tasks on a parallel process pool for better
performance. This happens automatically.The tasks perform the actual work for the build. But in order to have
that, we need to be able to execute commands, and create files. For that
we have theworkspace.Start Event MessagesAdhesive supports also start events with messages in the process. Each
message start event, is being processed in its own thread andyieldresults:@adhesive.message('Generate Event')defmessage_generate_event(context):foriinrange(10):yieldi@adhesive.task('Process Event')defprocess_event(context):print(f"event data:{context.data.event}")Each yield generates a new event that fires up the connected tasks. The
data yielded is present in theeventattribute in the token, for the
following tasks.Callback MessagesThe other option to push messages into a process is to use callback
messages:@adhesive.message_callback('REST: /rest/process-resource')defmessage_rest_rest_process_resource(context,callback):@app.route("/rest/resource/create")defcreate_resource():callback(Dict({"type":"CREATE"}))return"Create event fired"Using this we’re able to hook into other systems that have their own
loop, such as in this case the Flask server, and push messages using thecallback. This approach has also the advantage of not creating new
threads for each message endpoint.ConnectionsTasks are linked using connections. In some cases, connections can have
conditions. Conditions are expressions that when evaluated toTruewill allow the token to pass the connection. In the connection there is
access to thetask,task_name,data,loop,laneandcontext, as well as the variables defined in thecontext.data.So if in a task there is defined a data field such as:@adhesive.task('prepare data')defprepare_data(context):context.data.navigation_direction="forward"Thenavigation_directioncan be validated in the condition with any
of the following:context.data.navigation_direction == "forward"data.navigation_direction == "forward"navigation_direction == "forward"WorkspaceWorkspaces are just a way of interacting with a system, running
commands, and writing/reading files. Currently there’s support for:the local systemdocker containerskubernetesremote SSH connectionsWhen startingadhesiveallocates a default workspace folder in the
configured temp location (implicitly/tmp/adhesive). TheWorkspaceAPI is an API that allows you to run commands, and create
files, taking care of redirecting outputs, and even escaping the
commands to be able to easily run them inside docker containers.The workspace is available from the cotext directly from thecontext, by callingcontext.workspace.For example callingcontext.workspace.run(…​)will run the command
on the host where adhesive is running:@adhesive.task("Run Maven")defbuild_project(context)->None:context.workspace.run("mvn clean install")If we’re interested in the program output we simply do arunwith acapture_stdoutthat returns the output as a string:@adhesive.task("Test")defgbs_test_linux(context)->None:content=context.workspace.run("echo yay",capture_stdout=True)assertcontent=="yay"or we can use the simplified call withrun_outputthat guarantees astras result, unlike theOptional[str]forrun:@adhesive.task("Test")defgbs_test_linux(context)->None:content=context.workspace.run_output("echo yay")assertcontent=="yay"Theruncommands implicitly use/bin/sh, but a custom shell can
be specified by passing theshellargument:content=context.workspace.run_output("echo yay",shell="/bin/bash")Docker WorkspaceTo create a docker workspace that runs inside a container with the
tooling you just need to:fromadhesive.workspaceimportdockerThen to spin up a container that has the current folder mounted in,
where you’re able to execute commandsinsidethe container. You just
need to:@adhesive.task("Test")defgbs_test_linux(context)->None:image_name='some-custom-python'withdocker.inside(context.workspace,image_name)asw:w.run("python -m pytest -n 4")This creates a container using our current context workspace, where we
simply execute what we want, using therun()method. After thewithstatement the container will be teared down automatically.SSH WorkspaceIn order to have ssh, make sure you installedadhesivewith SSH
support:pipinstall-Uadhesive[ssh]To have a SSH Workspace, it’s again the same approach:fromadhesive.workspaceimportsshThen to connect to a host, you can just use thessh.insidethe same
way like in the docker sample:@adhesive.task("Run over SSH")defrun_over_ssh(context)->None:withssh.inside(context.workspace,"192.168.0.51",username="raptor",key_fileaname="/home/raptor/.ssh/id_rsa")ass:s.run("python -m pytest -n 4")The parameters are being passed to paramiko, that’s the implementation
beneath theSshWorkspace.Kubernetes WorkspaceTo run things in pods, it’s the same approach:fromadhesive.workspaceimportkubeThen we can create a workspace to run things in kubernetes pods. The
workspace, as well as the API, will use thekubectlcommand
internally.@adhesive.task("Run things in the pod")defrun_in_the_pod(context)->None:withkube.inside(context.workspace,pod_name="nginx-container")aspod:pod.run("ps x")# This runs in the podKubernetes APIAdhesive also packs a kubernetes api, that’s available on theadhesive.kubeapi:fromadhesive.kubeapiimportKubeApiTo use it, we need to create an instance against a workspace.@adhesive.gateway('Determine action')defdetermine_action(context):kubeapi=KubeApi(context.workspace,namespace=context.data.target_namespace)Let’s create a namespace:kubeapi.create(kind="ns",name=context.data.target_namespace)Or let’s create a service using thekubectl applyapproach:kubeapi.apply(f"""
    apiVersion: v1
    kind: Service
    metadata:
        name: nginx-http
        labels:
            app:{context.data.target_namespace}spec:
        type: ClusterIP
        ports:
        - port: 80
          protocol: TCP
          name: http
        selector:
          app:{context.data.target_namespace}""")Or let’s get some pods:pod_definitions=kubeapi.getall(kind="pod",filter=f"execution_id={context.execution_id}",namespace=context.data.target_namespace)These returns objects that allow navigating properties as regular python
attributes:new_pods=dict()forpodinpod_definitions:ifnotpod.metadata.name:raiseException(f"Wrong definition{pod}")new_pods[pod.metadata.name]=pod.status.phaseYou can also navigate properties that are not existing yet, for example
to wait for the status of a pod to appear:@adhesive.task('Wait For Pod Creation{loop.key}')defwait_for_pod_creation_loop_value_(context):kubeapi=KubeApi(context.workspace,namespace=context.data.target_namespace)pod_name=context.loop.keypod_status=context.loop.valuewhilepod_status!='Running':time.sleep(5)pod=kubeapi.get(kind="pod",name=pod_name)pod_status=pod.status.phaseTo get the actual data from the wrappers that the adhesive API creates,
you can simply call the_rawproperty.Workspace APIHere’s the full API for it:classWorkspace(ABC):"""
    A workspace is a place where work can be done. That means a writable
    folder is being allocated, that might be cleaned up at the end of the
    execution.
    """@abstractmethoddefwrite_file(self,file_name:str,content:str)->None:pass@abstractmethoddefrun(self,command:str,capture_stdout:bool=False)->Union[str,None]:"""
        Run a new command in the current workspace.

        :param capture_stdout:
        :param command:
        :return:
        """pass@abstractmethoddefrm(self,path:Optional[str]=None)->None:"""
        Recursively remove the file or folder given as path. If no path is sent,
        the whole workspace will be cleared.

        :param path:
        :return:
        """pass@abstractmethoddefmkdir(self,path:str=None)->None:"""
        Create a folder, including all its needed parents.

        :param path:
        :return:
        """pass@abstractmethoddefcopy_to_agent(self,from_path:str,to_path:str)->None:"""
        Copy the files to the agent from the current disk.
        :param from_path:
        :param to_path:
        :return:
        """pass@abstractmethoddefcopy_from_agent(self,from_path:str,to_path:str)->None:"""
        Copy the files from the agent to the current disk.
        :param from_path:
        :param to_path:
        :return:
        """pass@contextmanagerdeftemp_folder(self):"""
        Create a temporary folder in the current `pwd` that will be deleted
        when the `with` block ends.

        :return:
        """pass@contextmanagerdefchdir(self,target_folder:str):"""
        Temporarily change a folder, that will go back to the original `pwd`
        when the `with` block ends. To change the folder for the workspace
        permanently, simply assing the `pwd`.
        :param target_folder:
        :return:
        """passUser TasksIn order to create user interactions, you have user tasks. These define
form elements that are populated in thecontext.data, and available
in subsequent tasks.When a user task is encountered in the process flow, the user is
prompted to fill in the parameters. Note that the other started tasks
continue running, proceeding forward with the build.Thenameused in the method call defines the name of the variable
that’s in thecontext.data.For example in here we define a checkbox group that allows us to pick
where to publish the package:@adhesive.usertask("Read User Data")defread_user_data(context,ui)->None:ui.add_input_text("user",title="Login",value="root")ui.add_input_password("password",title="Password")ui.add_checkbox_group("roles",title="Roles",value=["cyborg"],values=["admin","cyborg","anonymous"])ui.add_radio_group("disabled",# title is optionalvalues=["yes","no"],value="no")ui.add_combobox("machine",title="Machine",values=(("any","<any>"),("win","Windows"),("lin","Linux")))This will prompt the user with this form:This data is also available for edge conditions, so in the BPMN modeler
we can define a condition such as"pypi" in context.data.roles, or
sincedatais also available in the edge scope:"pypi" in data.roles.The other option is simply reading what the user has selected in a
following task:@adhesive.task("Register User")defpublish_items(context):forroleincontext.data.roles:# ...User tasks support the following API, available on theuiparameter,
the parameter after the context:classUiBuilderApi(ABC):defadd_input_text(self,name:str,title:Optional[str]=None,value:str='')->None:defadd_input_password(self,name:str,title:Optional[str]=None,value:str='')->None:defadd_combobox(self,name:str,title:Optional[str]=None,value:Optional[str]=None,values:Optional[Iterable[Union[Tuple[str,str],str]]]=None)->None:defadd_checkbox_group(self,name:str,title:Optional[str]=None,value:Optional[Iterable[str]]=None,values:Optional[Iterable[Union[Tuple[str,str],str]]]=None)->None:defadd_radio_group(self,name:str,title:Optional[str]=None,value:Optional[str]=None,values:Optional[List[Any]]=None)->None:defadd_default_button(self,name:str,title:Optional[str]=None,value:Optional[Any]=True)->None:Custom ButtonsIn order to allow navigation inside the process, theadd_default_buttonAPI exists to permit creation of buttons.
Implicitly a single button with anOKlabel is added to the User
Task, that when pressed fills thecontext.datain the outgoing
execution token.Withadd_default_buttonwe create custom buttons such asBackandForward, or whatever we need in our process. Unlike the defaultOKbutton, when these are called, they also set in thecontext.datathevaluethat’s assigned to them. This value we
use then further in aGateway, or simple as a condition on the
outgoing edges.The title is optional, and only if missing it’s build either from thenameif all the buttons in the form have unique names, since they
assign a different variable in thecontext.data, or from thevalueif they have overlapping names.SecretsSecrets are files that contain sensitive information are not checked in
the project. In order to make them available to the build, we need to
define them in either~/.adhesive/secrets/SECRET_NAMEor in the
current folder as.adhesive/secrets/SECRET_NAME.In order to make them available, we just use thesecretfunction
that creates the file in the current workspace and deletes it when
exiting. For example here’s how we’re doing the actual publish, creating
the secret inside a docker container:@adhesive.task('^PyPI publish to (.+?)$')defpublish_to_pypi(context,registry):withdocker.inside(context.workspace,context.data.gbs_build_image_name)asw:withsecret(w,"PYPIRC_RELEASE_FILE","/germanium/.pypirc"):w.run(f"python setup.py bdist_wheel upload -r{registry}")Note thedocker.insidethat creates a different workspace.ConfigurationAdhesive supports configuration via its config files, or environment
variables. The values are read in the following order:environment variables:ADHESIVE_XYZ, thenvalues that are in the project config yml file:.adhesive/config.yml, thenvalues configured in the global config yml file:$HOME/.adhesive/config.yml.Currently the following values are defined for configuration:temp_folderdefault value/tmp/adhesive, environment var:ADHESIVE_TEMP_FOLDER.Is where all the build files will be stored.pluginsdefault value[], environment var:ADHESIVE_PLUGINS_LIST.This contains a list of folders, that will be added to thesys.path.
So to create a reusable plugin that will be reused by multiple builds,
you need to simply create a folder with python files, then point to it
in the~/.adhesive/config.yml:plugins:-/path/to/folderThen in the python path you can simply do regular imports.colordefault valueTrue, environment var:ADHESIVE_COLOR.Marks if the logging should use ANSI colors in the terminal. Implicitly
this istrue, but if log parsing is needed, it can make sense to
have it false.log_leveldefault_valueinfo, environment var:ADHESIVE_LOG_LEVEL.How verbose should the logging be on the terminal. Possible values aretrace,debug,info,warning,errorandcritical.pool_sizedefault value is empty, environment var:ADHESIVE_POOL_SIZE.Sets the number of workers that adhesive will use. Defaults to the
number of CPUs if unset.stdoutdefault value is empty, environment var:ADHESIVE_STDOUT.Implicitly for each task, the log is redirected in a different file, and
only shown if the task failed. The redirection can be disabled.parallel_processingdefault value isthread, environment var:ADHESIVE_PARALLEL_PROCESSING.Implicitly tasks are scaled using multiple threads in order to alleviate
waits for I/O. This is useful for times when remote ssh workspaces are
defined in the lanes, so the same connection can be reused for multiple
tasks.This value can be set toprocess, in case the tasks are CPU
intensive. This has the drawback of recreating the connections on
workspaces’ each task execution.Hacking AdhesiveAdhesive builds with itself. In order to do that, you need to checkout
theadhesive-libshared plugin, and configure your local config to
use it:plugins:-/path/to/adhesive-libThen simply run the build using adhesive itself:adhesive