Package: proactive-jupyter-kernel
proactive-jupyter-kernelThe ActiveEon Jupyter Kernel adds a kernel backend to Jupyter. This kernel interfaces directly with the ProActive
scheduler and constructs tasks and workflows to execute them on the fly.With this interface, users can run their code locally and test it using a native python kernel, and by a simple switch to
ProActive kernel, run it on remote public or private infrastructures without having to modify the code. See the example below:1. Installation:1.1 Requirements:Python 2 or 31.2 Using PyPiopen a terminalinstall the ProActive jupyter kernel with the following commands:$pipinstallproactiveproactive-jupyter-kernel--upgrade
$python-mproactive-jupyter-kernel.install1.3 Using source codeopen a terminalclone the repository on your local machine:$gitclonegit@github.com:ow2-proactive/proactive-jupyter-kernel.gitinstall the ProActive jupyter kernel with the following commands:$pipinstallproactive-jupyter-kernel/
$python-mproactive-jupyter-kernel.install2. PlatformYou can use any jupyter platform.
We recommend to usejupyter lab. To launch it from your terminal after
having installed it:$jupyterlabor in daemon mode:$nohupjupyterlab&>/dev/null&When opened, click on the ProActive icon to open a notebook based on the ProActive kernel.3. HelpAs a quick start, we recommend the user to run the#%help()pragma using the following script:#%help()This script gives a brief description of all the different pragmas that the ProActive Kernel provides.To get a more detailed description of a needed pragma, the user can run the following script:#%help(pragma=PRAGMA_NAME)4. Connection4.1 Using connect()If you are trying ProActive for the first time, sign up on thetry platform.
Once you receive your login and password, connect to the trial platform using the#%connect()pragma:#%connect(login=YOUR_LOGIN, password=YOUR_PASSWORD)To connect to another ProActive server host, use the later pragma this way:#%connect(host=YOUR_HOST, [port=YOUR_PORT], login=YOUR_LOGIN, password=YOUR_PASSWORD)Notice that theportparameter is optional. The default connexion port is8080.You can also connect to a distant server by providing itsurlin the following way:#%connect(url=YOUR_SERVER_URL, login=YOUR_LOGIN, password=YOUR_PASSWORD)By providing the completeurlof the server, users can eventually connect through the secure HTTPS protocol.4.2 Using a configuration file:For automatic sign in, create a file namedproactive_config.iniin your notebook working directory.Fill your configuration file according to one of the following two formats:By providing the serverhostandport:[proactive_server]host=YOUR_HOSTport=YOUR_PORT[user]login=YOUR_LOGINpassword=YOUR_PASSWORDBy providing the serverurl:[proactive_server]url=YOUR_SERVER_URL[user]login=YOUR_LOGINpassword=YOUR_PASSWORDSave your changes and restart the ProActive kernel.You can also force the current kernel to connect using any.iniconfig file through the#%connect()pragma:#%connect(path=PATH_TO/YOUR_CONFIG_FILE.ini)(For more information about this format please checkconfigParser)5. Usage5.1 Creating a Python taskTo create a new task, use the pragma#%task()followed by the task implementation script written into a notebook
block code.
To use this pragma, a task name has to be provided at least. Example:#%task(name=myTask)print('Hello world')General usage:#%task(name=TASK_NAME, [language=SCRIPT_LANGUAGE], [dep=[TASK_NAME1,TASK_NAME2,...]], [generic_info=[(KEY1,VAL1), (KEY2,VALUE2),...]], [variables=[(VAR1,VAL1), (VAR2,VALUE2),...]], [export=[VAR_NAME1,VAR_NAME2,...]], [import=[VAR_NAME1,VAR_NAME2,...]], [path=IMPLEMENTATION_FILE_PATH])\n'Users can also provide more information about the task using the pragma's options. In the following, we give more
details about the possible options:5.1.1 LanguageThelanguageparameter is needed when the task script is not written in native Python. If not provided, Python will be
selected as the default language.
The supported programming languages are:Linux_BashWindows_CmdDockerComposeScalawGroovyJavascriptJythonPythonRubyPerlPowerShellRHere is an example that shows a task implementation written inLinux_Bash:#%task(name=myTask, language=Linux_Bash)echo'Hello, World!'5.1.2 DependenciesOne of the most important notions in workflows is the dependencies between tasks. To specify this information, use thedepparameter. Its value should be a list of all tasks on which the new task depends. Example:#%task(name=myTask,dep=[parentTask1,parentTask2])print('Hello world')5.1.3 VariablesTo specifytask variables,
you should provide thevariablesparameter. Its value should be a list of tuples(key,value)that corresponds to
the names and adequate values of the corresponding task variables. Example:#%task(name=myTask, variables=[(var1,value1),(var2,value2)])print('Hello world')5.1.4 Generic informationTo specify the values of some advanced ProActive variables calledgeneric_information, you should
provide thegeneric_infoparameter. Its value should be a list of tuples(key,value)that corresponds to the names
and adequate values of the Generic Information. Example:#%task(name=myTask, generic_info=[(var1,value1),(var2,value2)])print('Hello world')5.1.5 Export/import variablesTheexportandimportparameters ensure variables propagation between the different tasks of a workflow.
IfmyTask1variablesvar1andvar2are needed inmyTask2, both pragmas have to specify this information as
follows:myTask1should include anexportparameter with a list of these variable names,myTask2should include animportparameter with a list including the same names.Example:myTask1implementation block would be:#%task(name=myTask1, export=[var1,var2])var1="Hello"var2="ActiveEon!"andmyTask2implementation block would be:#%task(name=myTask2, dep=[myTask1], import[var1,var2])print(var1+" from "+var2)5.1.6 Implementation fileIt is also possible to use an external implementation file to define the task implementation. To do so, the optionpathshould be used.Example:#%task(name=myTask,path=PATH_TO/IMPLEMENTATION_FILE.py)5.2 Importing librariesThe main difference between the ProActive and 'native language' kernels resides in the way the memory is accessed
during blocks execution. In a common native language kernel, the whole script code (all the notebook blocks) is
locally executed in the same shared memory space; whereas the ProActive kernel will execute each created task in an
independent process. In order to facilitate the transition from native language to ProActive kernels, we included the
pragma#%import(). This pragma gives the user the ability to add libraries that are common to all created tasks, and
thus relative distributed processes, that are implemented in the same native script language.The import pragma is used as follows:#%import([language=SCRIPT_LANGUAGE]).Example:#%import(language=Python)importosimportpandasNOTE: If the language is not specified, Python is considered as default language.5.3 Adding a fork environmentTo configure a fork environment for a task, use the#%fork_env()pragma. To do so, you have to provide the name of the
corresponding task and the fork environment implementation.Example:#%fork_env(name=TASK_NAME)containerName='activeeon/dlm3'dockerRunCommand='docker run 'dockerParameters='--rm 'paHomeHost=variables.get("PA_SCHEDULER_HOME")paHomeContainer=variables.get("PA_SCHEDULER_HOME")proActiveHomeVolume='-v '+paHomeHost+':'+paHomeContainer+' 'workspaceHost=localspaceworkspaceContainer=localspaceworkspaceVolume='-v '+localspace+':'+localspace+' 'containerWorkingDirectory='-w '+workspaceContainer+' 'preJavaHomeCmd=dockerRunCommand+dockerParameters+proActiveHomeVolume+workspaceVolume+containerWorkingDirectory+containerNameOr, you can provide the task name and the path of a.pyfile containing the fork environment code:#%fork_env(name=TASK_NAME, path=PATH_TO/FORK_ENV_FILE.py)5.4 Adding a selection scriptTo add a selection script to a task, use the#%selection_script()pragma. To do so, you have to provide the name of
the corresponding task and the selection code implementation.Example:#%selection_script(name=TASK_NAME)selected=TrueOr, you can provide the task name and the path of a.pyfile containing the selection code:#%selection_script(name=TASK_NAME, path=PATH_TO/SELECTION_CODE_FILE.py)5.5 Adding job fork environment and/or selection scriptIf the selection scripts and/or the fork environments are the same for all job tasks, we can add them just once using
thejob_selection_scriptand/or thejob_fork_envpragmas.Usage:For a job selection script, please use:#%job_selection_script([language=SCRIPT_LANGUAGE], [path=./SELECTION_CODE_FILE.py], [force=on/off])For a job fork environment, use:#%job_fork_env([language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py], [force=on/off])Theforceparameter defines whether the pragma has to overwrite the task selection scripts or the fork environment
already set.5.6 Adding pre and/or post scriptsSometimes, specific scripts has to be executed before and/or after a particular task. To do that, the solution providespre_scriptandpost_scriptpragmas.To add a pre-script to a task, please use:#%pre_script(name=TASK_NAME, language=SCRIPT_LANGUAGE, [path=./PRE_SCRIPT_FILE.py])To add a post-script to a task, use:#%post_script(name=TASK_NAME, language=SCRIPT_LANGUAGE, [path=./POST_SCRIPT_FILE.py])5.7 Branch controlThebranchcontrol provides the ability to
choose between two alternative task flows, with the possibility to merge back to a common one.To add a branch control to the current workflow, four specific tasks and one control condition should be added in
accordance with the following order:abranchtask,the related branchingconditionscript,aniftask that should be executed if the result of theconditiontask iftrue,anelsetask that should be executed if the result of theconditiontask iffalse,acontinuationtask that should be executed after theifor theelsetasks.To add abranchtask, you can rely on the following macro:#%branch([name=TASK_NAME], [dep=[TASK_NAME1,TASK_NAME2,...]], [generic_info=[(KEY1,VAL1), (KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])For the branchingconditionscript, use:#%condition()For aniftask, please use:#%if([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])For anelsetask, use:#%else([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])And finally, for thecontinuationtask:#%continuation([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])5.8 Loop controlTheloopcontrol provides the ability to repeat
a set of tasks.To add a loop control to the current workflow, two specific tasks and one control condition should be added
in the following order:astarttask,the related loopingconditionscript,alooptask.For astarttask, use:#%start([name=TASK_NAME], [dep=[TASK_NAME1,TASK_NAME2,...]], [generic_info=[(KEY1,VAL1), (KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])For the loopingconditionscript, use:#%condition()For alooptask, please use:#%loop([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])5.9 Replicate controlThereplicationallows the execution of
multiple tasks in parallel when only one task is defined, and the number of tasks to run could change.Through the ProActive Jupyter Kernel, users can add replicate controls in two main ways, a generic and a straight
forward way.5.9.1 Generic usageTo add a replicate control to the current workflow in the generic method, three specific tasks and one control runs
script should be added according to the following order:asplittask,the related replicationrunsscript,aprocesstask,amergetask.For asplittask, use:#%split([name=TASK_NAME], [dep=[TASK_NAME1,TASK_NAME2,...]], [generic_info=[(KEY1,VAL1), (KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])For the replicationrunsscript, use:#%runs()For aprocesstask, please use:#%process([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])And finally, for amergetask, use:#%merge([name=TASK_NAME], [generic_info=[(KEY1,VAL1),(KEY2,VALUE2),...]], [language=SCRIPT_LANGUAGE], [path=./FORK_ENV_FILE.py])5.9.2 Straight forward usageThe straight forward method to add a replication is most of all useful when the parallelism that should be
implemented is a task parallelism (the generic usage is more adapted to data parallelism).To add a replication to a task, just add the runs control script by providing therunsoption of thetaskpragma.
Example:#%task(name=T2,dep=[T1],runs=3)print("This output should be displayed 3 times ...")NOTE: To construct a valid workflow, straight forward replicated tasks must have one and only one parent task and one
child task at most. More information about replicate validation criteria are availablehere.5.10 Delete a taskTo delete a task from the workflow, the user should run the pragma#%delete_task()in the following way:#%delete_task(name=TASK_NAME)5.11 Create a jobTo create a job, specify job variables and/or job generic information, use the#%job()pragma:#%job(name=JOB_NAME, [generic_info=[(KEY1,VAL1), (KEY2,VALUE2),...]], [variables=[(VAR1,VAL1), (VAR2,VALUE2),...]])NOTE: It is not necessary to create and assign a name explicitly to the job. If not done by the user, this step is
implicitly performed when the job is submitted (check sectionSubmit your job to the schedulerfor more
information).5.12 Visualize jobTo visualize the created workflow, use the#%draw_job()pragma to plot the workflow graph that represents the job
into a separate window:#%draw_job()Two optional parameters can be used to configure the way the kernel plots the workflow graph.inline plotting:If this parameter is set tooff, plotting the workflow graph is done through aMatplotlibexternal window. The default value ison.#%draw_job(inline=off)save the workflow graph locally:To be sure that the workflow is saved into a.pngfile, this option needs to be set toon. The default value isoff.#%draw_job(save=on)Note that the job's name can take one of the following possible values:The parametername's value, if providedThe job's name, if createdThe notebook's name, if the kernel can retrieve itUnnamed_job, otherwiseGeneral usage:#%draw_job([name=JOB_NAME], [inline=off], [save=on])5.13 Export the workflow in dot formatTo export the created workflow into aGraphViz.dotformat, use the#%write_dot()pragma:#%write_dot(name=FILE_NAME)5.14 Import a workflow from a dot fileTo create a workflow according to aGraphViz.dotfile, use the pragma#%import_dot():#%import_dot(path=PATH_TO/FILE_NAME.dot)By default, the workflow will containPythontasks with empty implementation scripts. If you want to modify or add
any information to a specific task, please use, as explained inCreating a Task, the#%task()pragma.5.15 Submit your job to the schedulerTo submit the job to the ProActive Scheduler, the user has to use the#%submit_job()pragma:#%submit_job()If the job is not created, or is not up-to-date, the#%submit_job()creates a new job named as the old one.
To provide a new name, use the same pragma and provide a name as parameter:#%submit_job([name=JOB_NAME])If the job's name is not set, the ProActive kernel uses the current notebook name, if possible, or gives a random one.5.16 List all submitted jobsTo get all submitted job IDs and names, uselist_submitted_jobspragma this way:#%list_submitted_jobs()5.17 Export the workflow in XML formatTo export the created workflow in.xmlformat, use the#%export_xml()pragma:#%export_xml([name=FILENAME])Notice that the.xmlfile will be saved under one of the following names:The parametername's value, if providedThe job's name, if createdThe notebook's name, if the kernel can retrieve itUnnamed_job, otherwise5.18 Get resultsAfter the execution of a ProActive workflow, two outputs can be obtained,results: values that have been saved in thetask result variable,console outputs: classic outputs that have been displayed/printedTo get task results, please use the#%get_task_result()pragma by providing the task name, and either the job ID or
the job name:#%get_task_result([job_id=JOB_ID], [job_name=JOB_NAME], task_name=TASK_NAME)The result(s) of all the tasks of a job can be obtained with the#%get_job_result()pragma, by providing the job name
or the job ID:#%get_job_result([job_id=JOB_ID], [job_name=JOB_NAME])To get and display console outputs of a task, you can use the#%print_task_output()pragma in the following
way:#%print_task_output([job_id=JOB_ID], [job_name=JOB_NAME], task_name=TASK_NAME)Finally, the#%print_job_output()pragma allows to print all job outputs, by providing the job name or the job ID:#%print_job_output([job_id=JOB_ID], [job_name=JOB_NAME])NOTE: If neitherjob_namenor thejob_idare provided, the last submitted job is selected by default.6. Display and use ActiveEon Portals directly in JupyterFinally, to have the hand on more parameters and features, the user should use ActiveEon Studio portals.
The main ones are theResource Manager, theScheduling Portaland theWorkflow Automation.The example below shows how the user can directly monitor his submitted job's execution in the scheduling portal:To show the resource manager portal related to the host you are connected to, just run:#%show_resource_manager([host=YOUR_HOST], [height=HEIGHT_VALUE], [width=WIDTH_VALUE])For the related scheduling portal:#%show_scheduling_portal([host=YOUR_HOST], [height=HEIGHT_VALUE], [width=WIDTH_VALUE])And, for the related workflow automation:#%show_workflow_automation([host=YOUR_HOST], [height=HEIGHT_VALUE], [width=WIDTH_VALUE])NOTE: The parametersheightandwidthallow the user to adjust the size of the window inside the notebook.Current statusFeatures:help: prints all different pragmas/features of the kernelconnect: connects to an ActiveEon server (OPTION: connection using a configuration file)import: import specified libraries to all tasks of a same script languagetask: creates a taskpre_script: sets the pre-script of a taskpost_script: sets the post-script of a taskselection_script: sets the selection script of a taskjob_selection_script: sets the default selection script of a jobfork_env: sets the fork environment scriptjob_fork_env: sets the default fork environment of a jobsplit: creates/modifies a splitting task of a replicate controlruns: creates/modifies the configuration script of a replicate controlprocess: creates/modifies the script of a replicated processing taskmerge: creates/modifies a merging task of a replicate controlstart: creates/modifies a start task of a loop controlloop: creates/modifies a loop task of a loop controlcondition: creates/modifies the condition script of a branch/loop controlbranch: creates/modifies a branch task of a branching controlif: creates/modifies an if task of a branching controlelse: creates/modifies an else task of a branching controlcontinuation: creates/modifies a continuation task of a branching controldelete_task: deletes a task from the workflowjob: creates/renames the jobdraw_job: plot the workflowwrite_dot: writes the workflow in .dot formatimport_dot: imports the workflow from a .dot filesubmit_job: submits the job to the schedulerget_result: gets and prints the job resultsget_job_result: gets and prints the job resultsget_task_result: gets and prints the results of a given taskprint_job_output: gets and prints the job outputsprint_task_output: gets and prints the outputs of a given tasklist_submitted_jobs: gets and prints the ids and names of the submitted jobsexport_xml: exports the workflow in .xml formatshow_resource_manager: opens the ActiveEon resource manager portalshow_scheduling_portal: opens the ActiveEon scheduling portalshow_workflow_automation: opens the ActiveEon workflow automation portalTODOFeatures improvementsexecute in local a pragma free blockadd options import_as_json/export_as_jsonadd draw(on/off), print_result(on/off) options in submit job pragmamultiple pragmas in a block handlingapply selection_script and fork_env to a list of names (tasks)add auto-completeDocumentationadd some examples picturesadd configure pragma section (block, multiblock)