RADICAL-Cybertools: Middleware Building Blocks NSF's Cyberinfrastructure Ecosystem (NSF 1931512) Motivation: Sophisticated and scalable workflows have become essential for advances in computational science. In spite of the many successes of workflow systems, there is an absence of a reasoning framework for end-users to determine which systems to use, when and why. Workflows are increasingly a manifestation of the algorithmic and methodological advances; workflow users and workflow system developers are often the same. Workflow systems must be easily extensible so as to support diverse functionality and the proverbial “last mile customization”. We advance the science of workflows and prevent workflow system “vendor lock-in” by formulating a building blocks approach to middleware for workflow systems, grounded on four design principles: self-sufficiency, interoperability, composability and extensibility. A building block has: (i) one or more modules implementing functionalities to operate on a set of explicitly defined entities; and (ii) two well-defined and stable interfaces, one for input and one for output. Properties of building blocks ●Self-sufficiency: design does not depend on the specificity of other building blocks ●Interoperability: can be used in diverse system architectures without semantic modifications ●Composability: its interfaces enable communication and coordination with other building blocks ●Extensibility: its functionalities and entities can be extended to support new requirements or capabilities The diagram below describes the primary functional requirements for workflow systems. RADICAL-Cybertools are designed as building blocks that span multiple functional levels: (L4) Workflow and Application Description: Requirements and semantics of applications and workflows. (L3) Workload Management System: Applications devoid of semantic context are expressed as workloads. (L2) Task Runtime System (TRS): Execution of the tasks of a workload. (L1) Resource Access Layer: Capabilities, availability and interfaces required by the tasks to be executed. Motivation: Middleware Building Block Primary Functional Levels RADICAL-SAGA (Simple API for Grid Applications): Provides an interoperability layer that lowers the complexity of using distributed infrastructure whilst enhancing sustainability of distributed appli- cations, services and tools, in the form of a Python API. By abstracting away the heterogeneity of the underlying systems, RADICAL-SAGA simplifies access to many distributed cyberinfrastructures such as XSEDE and OSG. (L1-L2) Interface to Resource ExTASY: Enables advanced sampling of complex macromolecules using molecular dynamics. DeepDriveX: Enables scalable and concurrent execution of Machine Learning (ML) and HPC workloads (X) on HPC Platforms. ICEBERG: Enables scalable image analysis on HPC. It allows the integration of image analysis frameworks and algorithms. SCALE-MS: Enables the concurrent execution of adaptive ensemble algorithms. It uses RADICAL-Pilot for workload execution. RADICAL-Cybertools support multiple points of integration, “unifying” conceptual reasoning across otherwise different tools and systems (L4) Workflow Management Systems Ensemble Toolkit (EnTK): Provides the ability to execute flexible combinations of ensemble-based workflows. It promotes “ensembles” to a first class abstraction, managing where and how the ensemble workload is executed. EnTK exposes a pipeline-stage-task programming model as the interface to express ensemble-based workflows. (L3-L4) Workflow / Workload Management SeisFlows ●Supports seismic inversion workflows on HPC machines, at scale ●We integrated SeisFlow: ○with RADICAL-SAGA (L1) to execute compute jobs ○with RADICAL-EnTK (L3) to orchestrate tasks and data staging ATLAS (Panda and Harvester) ●PanDA is a WMS that supports the execution of workflows via pilots ●We integrated Panda and RADICAL-Pilot to improve its scaling on large HPC resources, and integrated Har