Package: idact
Welcome to idact!Idact, orInteractive Data Analysis Convenience Tools, is a Python 3.5+ library
that takes care of several tedious aspects of working with big data
on an HPC cluster.Who is it for?Data scientists or big data enthusiasts, who:Perform computations onJupyter Notebook,
using libraries such asNumPy,pandas,Matplotlib,
orbokeh.Have access to an HPC cluster withSlurmas the job scheduler.Would like to parallelize their computations across many nodes usingDask.distributed, a library
for distributed computing.May find that it takes too much manual effort to deploy Jupyter Notebook
and Dask on the cluster each time they need it.RequirementsPython 3.5+.ClientOperating System: Windows or LinuxRecommended:Jupyter NotebookorJupyterLabClusterOperating System: LinuxJob Scheduler:Slurm Workload ManagerSSH access to a login (head) node.Shared $HOME directory between nodes.Dask.distributedwithbokeh.Jupyter NotebookorJupyterLabInstallationpython -m pip install idactIf you're usingConda, you may want to update
your environment first:conda update --allCode samplesAccessing a clusterCluster can be accessed with a public/private key pair via SSH.fromidactimport*cluster=add_cluster(name="short-cluster-name",user="user",host="login-node.cluster.example.com",port=22,auth=AuthMethod.PUBLIC_KEY,key="~/.ssh/id_rsa",install_key=False)node=cluster.get_access_node()node.connect()Tutorial:01. Connecting to a clusterAllocating and deallocating nodesNodes are allocated as a Slurm job.
Afterwards, they can be used for deployments.importbitmathnodes=cluster.allocate_nodes(nodes=8,cores=12,memory_per_node=bitmath.GiB(120),walltime=Walltime(hours=1,minutes=30),native_args={'--partition':'debug','--account':'data-analysis-group'})try:nodes.wait(timeout=120.0)exceptTimeoutError:nodes.cancel()Tutorial:02. Allocating nodesDeploying Jupyter NotebookJupyter Notebook is deployed on a cluster node,
and made accessible through an SSH tunnel.nb=nodes[0].deploy_notebook()nb.open_in_browser()Tutorial:03. Deploying JupyterDeploying Dask.distributedDask.distributed scheduler and workers are deployed
on cluster nodes, and their dashboards are made available
through SSH tunnels.dd=deploy_dask(nodes[1:])client=dd.get_client()client.submit(...)dd.diagnostics.open_all()Tutorial:04. Deploying Dask,09. Demo analysisManaging cluster configLocal and remote cluster configuration can be saved, loaded,
and copied to and from the cluster.save_environment()load_environment()push_environment(cluster)pull_environment(cluster)Tutorials:01. Connecting to a cluster,05. Configuring idact on a clusterManaging deploymentsDeployment objects can be serialized and copied between running program
instances, local or remote.cluster.push_deployment(nodes)cluster.push_deployment(nb)cluster.push_deployment(dd)cluster.pull_deployments()Tutorials:06. Working on a cluster,07. Adjusting timeoutsQuick deployment appQuick deployment app allocates nodes and deploys Jupyter notebook
from command line:idact-notebook short-cluster-name --nodes 3 --walltime 0:20:00Tutorial:08. Using the quick deployment appDocumentationThe documentation contains detailed API description, tutorial notebooks,
and other helpful information.Documentation - masterDocumentation - developAll versionsSource codeThe source code is available onGitHub.LicenseMIT License.This library was developed under the supervision of Leszek Grzanka, PhD
as a final project of the BEng in Computer Science program
at the Faculty of Computer Science, Electronics and Telecommunications
at AGH University of Science and Technology, Krakow.