Package: pycompss-player
IntroductionThe PyCOMPSs player (pycompss) provides a tool to use PyCOMPSs within
local machines interactively through docker images. This tool has been
implemented on top ofPyCOMPSs programming
model, and it is being developed by theWorkflows and Distributed Computing
groupof theBarcelona Supercomputing
Center.ContentsQuickstartLicenseQuickstartThere are two ways in which you can get started with PyCOMPSs. You can
perform a local installation by installing thepycompss
package, or you can use it
through our ready-to-use docker image thorugh thispycompss-player
tool.InstallationDependenciespycompss-player currently requires:docker >= 17.12.0-ceInstallation stepsInstall docker (continue with step 2 if already installed)pycompss-player requiresdocker 17.12.0-ceor greater.Follow these instructionsDocker for
Mac.
Or, if you prefer to useHomebrew.Docker for
Ubuntu.Docker for Arch
Linux.Be aware that for some distros the docker package has been renamed
fromdockertodocker-ce. Make sure you install the new
package.Add user to docker group to run pycompss as a non-root user.InstructionsCheck that docker is correctly installeddocker--versiondockerps# this should be empty as no docker processes are yet running.Installdocker-pypython3-mpipinstalldockerInstall pycompss-player:python3-mpipinstallpycompss-playerThis should add the pycompss-player executables (pycompss,compssanddislib) to your path. They can be used indiferently.Warning:The user executable path may not be automatically exported
into thePATHenvironment variable. So, take this into account if
installed with the--userflag, since thepycompss|compsscommand will be unreachable until the path
is exported intoPATH.UsageStartpycompssin your development directoryInitialize the COMPSs infrastructure where your source code will be (you
can re-init anytime). This will allow docker to access your local code
and run it inside the container.Notethat the first time needs to download the docker image from the
registry, and it may take a while.# Without a path it operates on the current working directory.pycompssinit# You can also provide a pathpycompssinit-w/home/user/replace/path/# Or the COMPSs docker image to usepycompssinit-icompss/compss:2.10# Or bothpycompssinit-w/home/user/replace/path/-icompss/compss:2.10Running applicationsFirst clone the PyCOMPSs’ tutorial apps repository:gitclonehttps://github.com/bsc-wdc/tutorial_apps.gitInit the COMPSs environment in the root of the repository. The source
files path are resolved from the init directory which sometimes can be
confusing. As a rule of thumb, initialize the library in a current
directory and check the paths are correct running the file withpython3 path_to/file.py(in this casepython3 python/simple/src/simple.py).cdtutorial_appspycompssinitpycompssrunpython/simple/src/simple.py1The log files of the execution can be found at $HOME/.COMPSs.You can also init the COMPSs environment inside the examples folder.
This will mount the examples directory inside the container so you can
execute it without adding the path:cdpython/simple/srcpycompssinitpycompssrunsimple.py1Running the COMPSs monitorThe COMPSs monitor can be started using thepycompss monitor startcommand. This will start the COMPSs monitoring facility which enables to
check the application status while running. Once started, it will show
the url to open the monitor in your web browser
(http://127.0.0.1:8080/compss-monitor)Reminder: Include the monitor flag in the execution before the
binary to be executed.cdpython/simple/srcpycompssinitpycompssrun--monitor=1000-gsimple.py1If running a notebook, just add the monitoring parameter into the COMPSs
runtime start call.Once finished, it is possible to stop the monitoring facility by using
thepycompss monitor stopcommand.Running Jupyter notebooksNotebooks can be run using thepycompss jupytercommand. Run the
following snippet from the root of the project:cdtutorial_apps/pythonpycompssinitpycompssjupyter./notebooksAn alternative and more flexible way of starting jupyter is using thepycompss runcommand in the following way:pycompssrunjupyter-notebook./notebooks--ip=0.0.0.0--allow-rootAccess your notebook by ctrl-clicking or copy pasting into the browser
the link shown on the CLI (e.g.http://127.0.0.1:8888/?token=TOKEN_VALUE).If the notebook process is not properly closed, you might get the
following warning when trying to start jupyter notebooks again:The port 8888 is already in use, trying another port.To fix it, just restart the pycompss container withpycompss init.Generating the task graphCOMPSs is able to produce the task graph showing the dependencies that
have been respected. In order to producee it, include the graph flag in
the execution command:cdpython/simple/srcpycompssinitpycompssrun--graphsimple.py1Once the application finishes, the graph will be stored into the~\.COMPSs\app_name_XX\monitor\complete_graph.dotfile. This dot file
can be converted to pdf for easier visualilzation through the use of thegengraphparameter:pycompssgengraph.COMPSs/simple.py_01/monitor/complete_graph.dotThe resulting pdf file will be stored into the~\.COMPSs\app_name_XX\monitor\complete_graph.pdffile, that is, the
same folder where the dot file is.Tracing applications or notebooksCOMPSs is able to produce tracing profiles of the application execution
through the use of EXTRAE. In order to enable it, include the tracing
flag in the execution command:cdpython/simple/srcpycompssinitpycompssrun--tracingsimple.py1If running a notebook, just add the tracing parameter into the COMPSs
runtime start call.Once the application finishes, the trace will be stored into the~\.COMPSs\app_name_XX\tracefolder. It can then be analysed with
Paraver.Adding more nodesNote: adding more nodes is still in beta phase. Please report
issues, suggestions, or feature requests onGithub.To add more computing nodes, you can either let docker create more
workers for you or manually create and config a custom node.For docker just issue the desired number of workers to be added. For
example, to add 2 docker workers:pycompsscomponentsaddworker2You can check that both new computing nodes are up with:pycompsscomponentslistIf you want to add a custom node it needs to be reachable through ssh
without user. Moreover, pycompss will try to copy theworking_dirthere, so it needs write permissions for the scp.For example, to add the local machine as a worker node:pycompsscomponentsaddworker'127.0.0.1:6'‘127.0.0.1’: is the IP used for ssh (can also be a hostname like
‘localhost’ as long as it can be resolved).‘6’: desired number of available computing units for the new node.Please be awarethatpycompss componentswill not list your
custom nodes because they are not docker processes and thus it can’t be
verified if they are up and running.Removing existing nodesNote: removing nodes is still in beta phase. Please report issues,
suggestions, or feature requests onGithub.For docker just issue the desired number of workers to be removed. For
example, to remove 2 docker workers:pycompsscomponentsremoveworker2You can check that the workers have been removed with:pycompsscomponentslistIf you want to remove a custom node, you just need to specify its IP and
number of computing units used when defined.pycompsscomponentsremoveworker'127.0.0.1:6'StoppycompssThe infrastructure deployed can be easily stopped and the docker instances
closed with the following command:pycompsskillLicenseApache License Version 2.0Workflows and Distributed ComputingDepartment of Computer ScienceBarcelona Supercomputing Center (http://www.bsc.es)