Package: scorep-jupyter
The Score-P Python Jupyter KernelThis is the Score-P Python Kernel that enables you to execute Python code in Jupyter Notebooks withScore-P.The kernel uses theScore-P Python bindings.Table of ContentInstallationUsageConfigurationBasic ExecutionMulti-Cell ModeWrite ModePresentation of Performance DataLimitationsFuture WorkCitingContactAcknowledgmentsInstallationFor using the kernel you need a proper Score-P installation.From the Score-P Python bindings:You need at least Score-P 5.0, build with--enable-sharedand the gcc compiler plugin.
Please make sure thatscorep-configis in yourPATHvariable.
For Ubuntu LTS systems there is a non-official ppa of Score-P available:https://launchpad.net/~andreasgocht/+archive/ubuntu/scorep.To install the kernel and all dependencies, including the Python bindings use:pip install scorep-jupyter
python -m scorep_jupyter.installYou can also build the kernel from source via:pip install .The kernel will then be installed in your active python environment.
You can select the kernel in Jupyter asscorep-python.UsageConfiguring Score-P in Jupyter%%scorep_envSet up your Score-P environment. For a documentation of Score-P environment variables, see:Score-P Measurement Configuration.%%scorep_python_binding_argumentsSet the Score-P Python bindings arguments. For a documentation of arguments, seeScore-P Python bindings.Executing CellsWithout Score-PYou can execute cells without Score-P as usual:With Score-P%%execute_with_scorepExecutes a cell with Score-P, i.e. it callspython -m scorep <cell code>Multi-Cell ModeYou can also treat multiple cells as one single cell by using the multi cell mode. Therefore you can mark the cells in the order you wish to execute them.%%enable_multicellmodeEnables the multi-cell mode and starts the marking process. Subsequently, "running" cells will not execute them but mark them for execution after%%finalize_multicellmode.%%finalize_multicellmodeStop the marking process and executes all the marked cells.
All the marked cells will be executed with Score-P.%%abort_multicellmodeStops the marking process, without executing the cells.Hints:The%%execute_with_scorepcommand has no effect in the multi cell mode.There is no "unmark" command available but you can abort the multicellmode by the%%abort_multicellmodecommand. Start your marking process again if you have marked your cells in the wrong order.The%%enable_multicellmode,%%finalize_multicellmodeand%%abort_multicellmodecommands should be run in an exclusive cell. Additional code in the cell will be ignored.Write ModeAnalogous to%%writefilecommand in IPykernel, you can convert a set of cells to the Python script which is to be executed with Score-P Python bindings (with settings and environment described in auxillary bash script).%%start_writefile [scriptname]Enables the write mode and starts the marking process. Subsequently, "running" cells will not execute them but mark them for writing into a python file after%%end_writefile.scriptnameisjupyter_to_script.pyby default.%%end_writefileStops the marking process and writes the marked cells in a Python script. Additionally, a bash script will be created for setting the Score-P environment variables, Pyhton bindings arguments and executing the Python script.Hints:Recording a cell containing%%scorep_envor%%scorep_python_binding_argumentswill add the environment variables/Score-P Python bindings to the bash script.Code of a cell which is not to be executed with Score-P (not inside the multicell mode and without%%execute_with_scorep) will be framed withwith scorep.instrumenter.disable()in the Python script to prevent instrumentation.Other cells will be recorded without any changes, except for dropping all magic commands.%%abort_multicellmodewill be ignored in the write mode and will not unmark previous cells from instrumentation.Presentation of Performance DataTo inspect the collected performance data, use tools as Vampir (Trace) or Cube (Profile).LimitationsSerialization Type SupportFor the execution of a cell, the kernel uses the default IPython kernel. For a cell with Score-P it starts a new Python process. Before starting this process, the state of the previous executed cells is persisted usingdilllibrary (https://github.com/uqfoundation/dill). However:dillcannot yet pickle these standard types:
frame, generator, tracebackOverheadWhen dealing with big data structures, there might be a big runtime overhead at the beginning and the end of a Score-P cell. This is due to additional data saving and loading processes for persistency in the background. However this does not affect the actual user code and the Score-P measurements.Future WorkThe kernel is still under development. The following is on the agenda:Check alternative Python implementations (Stackless/PyPy) for better serialization supportPerformance data visualizationsOverhead reductionPRs are welcome.CitingIf you publish some work using the kernel, we would appreciate if you cite the following paper:Werner, E., Manjunath, L., Frenzel, J., & Torge, S. (2021, October).
Bridging between Data Science and Performance Analysis: Tracing of Jupyter Notebooks.
In The First International Conference on AI-ML-Systems (pp. 1-7).
https://dl.acm.org/doi/abs/10.1145/3486001.3486249Additionally, please refer to the Score-P Python bindings, published here:Gocht A., Schöne R., Frenzel J. (2021)
Advanced Python Performance Monitoring with Score-P.
In: Mix H., Niethammer C., Zhou H., Nagel W.E., Resch M.M. (eds) Tools for High Performance Computing 2018 / 2019. Springer, Cham.
https://doi.org/10.1007/978-3-030-66057-4_14orGocht-Zech A., Grund A. and Schöne R. (2021)
Controlling the Runtime Overhead of Python Monitoring with Selective Instrumentation
In: 2021 IEEE/ACM International Workshop on Programming and Performance Visualization Tools (ProTools)
https://doi.org/10.1109/ProTools54808.2021.00008Contactelias.werner@tu-dresden.deAcknowledgmentsThis work was supported by the German Federal Ministry of Education and Research (BMBF, SCADS22B) and the Saxon State Ministry for Science, Culture and Tourism (SMWK) by funding the competence center for Big Data and AI "ScaDS.AI Dresden/Leipzig