High Throughput Computing with Dask: Part 1 - Dask
Channel: E-CAM
Date: 2021-01-21

High-throughput (task-based) computing is a flexible approach to parallelization. It involves splitting a problem into loosely-coupled tasks. A scheduler then orchestrates the parallel execution of those tasks, allowing programs to adaptively scale their resource usage. Individual tasks may themselves be parallelized using MPI or OpenMP, and the high-throughput approach can therefore enable new levels of scalability.

Dask is a powerful Python tool for task-based computing. The Dask library was originally developed to provide parallel and out-of-core versions of common data analysis routines from data analysis packages such as NumPy and Pandas. However, the flexibility and usefulness of the underlying scheduler has led to extensions that enable users to write custom task-based algorithms, and to execute those algorithms on high-performance computing (HPC) resources.

This workshop will be a series of virtual seminars/tutorials on tools in the Dask HPC ecosystem. All sessions will be held online, with a live Zoom for some registered participants and a live YouTube stream for the public.

The seminars will be on:

Dask (21 January 2021, 2pm UTC)
  - Repo URL: https://github.com/jacobtomlinson/dask-video-tutorial-2020
  - Binder URL: https://mybinder.org/v2/gh/jacobtomlinson/dask-video-tutorial-2020/master?urlpath=lab
Dask-Jobqueue (4 February 2021, 2pm UTC): a library that integrates Dask with standard HPC queuing systems, such as SLURM or PBS
Jobqueue-Features (11 February 2021, 2pm UTC): a library that enables functionality aimed at enhancing scalability

https://www.cecam.org/workshop-details/1022