Using Python for Improved Productivity in HPC and Data Science Applications: the Time is Now

Developer productivity for scientific software is essential in order to address changing timelines in scientific computing needs and an increasing set of science problems that can benefit from large-scale computational approaches. Generally, computational approaches in science were considered to be long-standing and less time-sensitive; this made them worthy of multi-year efforts as exemplified by DOE SciDAC projects. There is now a newfound urgency to analyze massive quantities of data and perform large calculations for societal benefits. Such research efforts include the biological and environmental sciences, where the complexity and multiple scales of systems, together with growing dataset sizes from high-throughput experimental methods, have created major bottlenecks [1]. Urgent needs include research in sustainability, genomics [2] and healthcare research such as recent responses to the COVID-19 pandemic [3]. In addition, the emergence of data-driven and AI-driven solutions to these urgent problems has placed a higher premium on the use of highly optimized, parallel software. Improving developer productivity has been a long-standing challenge that is deeply impacted by emerging hardware platforms, workforce expertise, and programming tools. We are currently at a critical point in all three aspects of this challenge. In terms of hardware platforms, the “free ride” of automatic improvements in efficiency and cost from Moore’s Law and Dennard Scaling has ended. Sustained improvements now require specialized, post-Moore accelerators and radically different computer architectures. Therefore, high-performance computing (HPC) software development, which has always been the realm of specialized experts, is now becoming even more highly specialized. However, the development of scientific software for domain science is not performed by HPC experts, but by domain scientists with highly specialized knowledge of their modeled systems and the structure of their data. In addition, the current state of programming tools for scientific computing is inadequate for rapid creation of new applications on new hardware platforms, especially for computational scientists who are not HPC experts. In this paper, we advocate for the newly emerging approach to developing scientific software: an HPC-based Python ecosystem. Python is a widely used high-productivity language, second only to JavaScript in popularity, but far better suited to scientific software than JavaScript. It allows for a fast code production schedule, even for domain scientists who are not HPC experts. The Python community has worked extensively to circumvent Python’s performance obstacles by introducing a rich collection of high performance libraries. Python serves several different purposes in HPC scientific software development. It can be used as a wrapper or “glue” to combine different languages and traditional HPC libraries, and to develop workflows and use data science and machine learning toolkits. There are many Python libraries that can be quickly parallelized or offloaded. Python itself can be extended with another programming model for parallelism, for example, CuPy to accelerate NumPy and Dask with asynchronous task-parallelism across multiple GPUs or nodes. Additionally, Python tools like Jupyter Notebook enable interactive programming to steer computations in ways beyond traditional file I/O [4].