Sustainable Architecture Design Principles
for Large Scale Research Infrastructure
Projects
Yuri Demchenko
University of Amsterdam
International Conferences on High Performance Computing andCommunications (HPCC2023)
Melbourne, 13-15 December 2023

1

Outline• Digitalisation of Science and Large Research Infrastructure projects in Europe• ESFRI, EOSC, SLICES• Project management: Use cases and scenarios• Importance of System and Software Engineering Skills• Sustainable Architecture Design principles• Standardisation and Best Practices• Why Multi-Layer Architecture id essential for infrastructure projects?
• Why API definition is not sufficient?
• Other architecture design styles and principles• Including System and Software Architecture Design principles into Master curricula

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

2

Motivation for this (Reflective) Research• Factor 1: Dropping level of understanding Software and System architecture design principles by
the master students: both University of Amsterdam and NTU of Ukraine• Consequently, quality of course projects• Factor 2: Dropping architecture design competences in EU funded projects• Demos and pilots ended up with TRL2-TRL3 (Technology Readiness Level)• Not repeatable, not operational solutions, create “gravity” of technically not-sustainable solutions• Observation 1: Increasing number of blogs on the efficient architecture design principles• Observation 2: Move to sustainable IT and AI must be based on sustainable architecture designprinciples• Growing understanding of competences and skills management in research and industry and to beaddressed by education• Number of European sectoral Skills Blueprints produced in 2018-2022• General competence frameworks: DigComp, EntreComp, ResearchComp, GreenComp

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

3

European Research Area and Initiatives – 2017-2023: Focus
on e-Science and Digital technologies• European Research Area (ERA) is an important area of the European policy development and funding• Supported by main e-Infrastructure projects: GEANT, EGEE, EGI, EOSC (taking off)
• Research Infrastructure (RI) is one of pillar in supporting European science• More than 1800 RIs in Europe• Coordinated by ESFRI (European Strategy Forum on Research Infrastructures)• ESFRI Roadmap 2021 - https://roadmap2021.esfri.eu/• Traditional domains: Social and cultural Innovation, Environment, Health and Food, Energy, Physical Scienceand Engineering• Majority are digital or e-Infrastructure enabled• New domain (since RM2021): DIGIT – Digital and Information Technologies• To experiment, develop and adopt modern digital and data technologies for scientific research (RIs)• SLICES, EBRAINS, SoBigData, OpenAIRE, EUDAT, EGI, EuroHPC• European Open Science Cloud (EOSC) initiative since 2016 to create a common platform for
European RI integration• Currently focused on building European Federated Data Access and Sharing Infrastructure federating
institutional and sectoral Data Management Infrastructures

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

4

European Open Science Cloud (EOSC)• EOSC is an overarching concept and framework to integrate existing RIs and facilitate information anddata exchange between RI, organisations and researchers• First phase 2016-2020 with funded projects 2016-2022•

53 projects in total• EOSC main projects and co-creation activities• EOSCpilot – Initial EOSC architecture and requirements (10 Mln, ended 2018)• EOSChub – FAIR data principles implementation, Technical integration platform, RI marketplace andAPI/services directory (33 Mln, ended 31 March 2021)• EOSCsecretariat https://www.eoscsecretariat.eu/ (10 Mln, ends 31 October 2021)• Establishment of the Governance structure and EU EOSC association• SRIA (Strategic Research and Innovation Agenda) document (195 pages, Final version dated 15 Feb 2021) –Provided long term vision for EOSC services but w/o technically consistent architecture definition• EOSC-Future (2021-2023) – To develop general architecture and key infrastructure services (30 Mln, Ended
30 Oct 2023)• EOSC-Beyond (2024-2027) – Future develop and implement EOSC in operation with user centricity• Built on experience of the past successful initiatives and project• EGEE and WLCG, EGI, RDA (co-founded by EC and NSF), GEANT/TERENA• Provides a model experience for other EU initiatives, such as GAIA-X European Federated Data Cloud
but leadership faded w/o clear architecture definition

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

5

EOSC Conceptual Architecture• EOSC-Core• Minimum architecture elements to enable theFederation• EOSC-Exchange• Evolving Federation to serve the needs of researchcommunities• Widening to the general public and the private sector• EOSC Federation (RIs and e-Infra)• Research and Innovation Community• Minimal Viable EOSC (aka MVE)
• Minimum Federation to bring value to users

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

6

Projects Setup: Teams composition and expertise• Case 1 (reference or successful). Development of cloud based applications for researchand industry (Innovation Action).
The team is composed of specialists from all relevant domains (infrastructure, network,
DevOps/Agile, security, software development, use cases domain) with sufficient experience insystem and software engineering at different levels (architecture, requirement engineering,
integration).
• Case 2 (not sufficient system engineering expertise). Development of the complex research
infrastructure to support experimentation on digital infrastructure technologies.
• Case 3 (lack of system engineering and infrastructure expertise). This is the often case when the
project requires delivering digital infrastructure to support specific domain research, but the project
consortium partnership lacks partners in computer or infrastructure technologies, system andsoftware engineering.
• Case 4 (not sufficient knowledge of background technologies). This scenario can be a variation of
scenarios 2 or 3. The project architect (or person responsible for the architecture development) has
experience from the previous successful RI or e-Infrastructure projects of 5-8 years ago when
performing in a junior position or specific task leader.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

7

Project Development Scenarios for Success• Scenario 1 (Structural architecture driven design). The project team has sufficient experience in domain specific systems from theprevious work or projects, but the new project requires wider expertise.
•

Scenario 2 (Designing Up, DowThe team starts with implementing available solutions that address the main goals of the system or infrastructure, defines theoverall architecture, and requirements to the core components•

To solve the problem of the expertise gap, the team (actually the team leader) looks for cooperation with other projects and invites experts to share expertise.

•

A dedicated team member is assigned to gain new expertise and take over necessary components implementation in the future.

• Scenario 2 (Designing Up, Down, and Out): System implementation is started without consistent and grounded architecture definition but
with a large availability of legacy components and sufficient expertise in ICT systems design from the previous project.
•

The project or team starts with implementing some infrastructure and services islands, often siloed services, based on available expertise and already existing
components.

•

To proceed with the development and implementation, start the structural analysis of the services being developed and put them in the context of the definedsystem architecture following best practices• Scenario 3 (ad-hoc services piloting). The project is started by the community with identified needs for information digital infrastructureservices but without prior experience in building such services, for example, building research infrastructure for social or environmental
sciences, humanities.
•

The project may succeed in user needs studies and defining user requirements, but it may fall short of transforming user requirements to technical system
requirements and corresponding architecture and functional design.

•

The project may end up with the pilot services for demonstration of the proof-of-concept and stand-alone tools but further successful development will requirea professional approach in infrastructure design, what in its turn may also require services re-design.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

8

Why Architecture is Important?
• Long term vision and coordination of cooperating teams• Blueprint for designing, building, and maintaining the complex and interconnected
systems that underpin research infrastructure• Architecture design thinking – What does it mean?
• To be cultivated starting from university education• Provided as professional training• Lack of common background may create confusion• Lack of well defined architecture may result in divergent development• Divergence may turn to convergence in technical development• If a critical point not missed – due to gravity of wrong solutions

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

9

Important Overloaded Terms that may create confusion anddivergence in project development• Architecture concept is itself often understood in different ways by researchers and developers with different
backgrounds and experiences.
• Way to achieve homogeneity is learning by examples• Architecture, reference architecture, framework, reference model• All these terms are in many cases interchangeable, often used together and in combination, but have their own meanings in specificcontexts.

• Architecture diagram, architecture model, functional diagram, process or sequence diagram• Data driven and data centric architecture, models and applications vs storage architectures• Must ensure continuity in data handling and processes management• Relate to workflow driven models• Multi-layer and multi-tier systems in system and applications engineering, and multi-level systems insecurity engineering• Blueprint and Bill of Materials (BOM) as general concepts and similarly named concepts in DevOps and
CI/CD in software engineering.
• Metadata and data modeling definitions as they are defined in industry and research data management
domains against information models and metadata in telecom and Internet service management.
• Security as a general concept and those related to different security domains such as computer
security, network/ Internet security, application security, cryptography, hardware security, operational security,
access control, identity management and trust management.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

10Why Layered/Multi-Layer Architecture is important?
• Services (resources) separation• Independent development and update/evolution: changes in one layer doesn’t affect others layers• Re-use of services• Multi-provider integration• Inter-layer interfaces (also specialized, not only WSDL/SOAP or REST/JSON based)• Services Composition/integration and Orchestration• Simplifies distributed systems integration• Simplifies/structure services management• Service Information model: namespaces, semantics, metadata• Sustainable system evolution• Similar to LTE in 3GPP and 5G/6G

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

11

Standardisation: Past based and Forward looking Best Practices• Internet Architecture: TCP/IP stack• Telecom OSI (Open System Interconnection)• TeleManagement Forum• ITU-T• ITU Q.4068 : Open application program interfaces (APIs) for interoperable testbed federations• ISO and IEEE
• Service Oriented Architecture (SOA)• NIST Cloud Computing Reference Architecture• NIST Big Data Architecture

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

12

TCP/IP - Internet Protocol and Transmission Control Protocol

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

13

Internet Protocols “narrow waist” - Hourglass• Source: 2001 Presentation by Steve Deering
HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

14

Example Multi-layer NIST Cloud Computing Reference
Architecture• Formal definition of the
Cloud Computing
technology and corearchitecture in 2008 built
the industry• 5 Properties:• On-demand selfservice• Broad network access• Resource pooling• Rapid elasticity• Measured Service• 3 main service models:
IaaS, PaaS, SaaS
• Deployment models••••

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

Private clouds
Public clouds
Hybrid clouds
Community clouds

15

Sustainable Architecture Design Principles (SADP)• Infrastructure related• Service Architecture related• Data Management Infrastructure related• Security and compliance design principles• Project Management and DevOps• Agile development practices

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

16

Sustainable Architecture Design Principles
General architecture design principles•••

Layered architecture design for services and mechanisms, including inter-layer interfaces, including cross-layer services and mechanisms definition that are typically defined as service planes, forexample, management plane, security plane, data management plane.
Multi-tier services and infrastructure design, including combined multi-layer and multi-tier systems that may use or apply different architectural and layered solutions.
Application Programming Interfaces (API) for composable services that must be supported by consistent (and fully qualified API metadata and namespaces definition).

Service architecture related•••

Service Oriented Architecture (SOA) and Microservices Architecture (MSA) that is supported with the different VM and container solutions and/or platforms.
Cloud powered, cloud based and cloud native design principles that require knowledge of the modern cloud architecture and cloud platform, both Open Source and public clouds (at least Amazon
Web Services, Microsoft Azure, and Google Cloud Platform). This also includes such powerful cloud based mechanism as Virtual Private Cloud (VPC) that provide VPN based secure environment for
multi-tenant customer applications.
Service lifecycle management model that should include all necessary services to support lifecycle stages in the context of specific services. This also includes services composition andorchestration for services deployment and operation.

Data infrastructure and services related•••

Big Data computation models and supporting platform, distributed and highly scalable systems, in particular Hadoop ecosystem and NoSQL databases.
Data management infrastructure and services that should cover two domains: services data (mostly related to management plane) and business or research data produced as a result of business
operation or scientific research.
Services and data management continuity in IoT/sensor networks, edge, cloud, data-driven applications that also include 5G/6G Radio Access Network (RAN), edge and cloud convergence.

Security and compliance design principles••

Security architecture and security services lifecycle management which are well defined by numerous standards and supported by the major infrastructure development frameworks; also security
services have their own multi-layer architecture (can also be referred as security plane), their integration with the main infrastructure services, including data infrastructure) is realized via API calls
and consistent definitions of the security roles, access control policies and credentials and secrets management.
Compliance frameworks that define requirements to and recommendations for secure services and infrastructure design and operation. Cloud Security Alliance (CSA) and Compliance Assessment
Initiative Questionnaire (CAIQ) provide the best overview of all important standards and regulations to ensure systems security and compliance, and data protection.

Project Management and DevOps•••

DevOps and SRE (Site Reliability Engineering) practices applied to system and services engineering and operation. This should also include continuous monitoring and optimisation on multiple user centric and business-centric SLI/KPI (Service Level/Key Performance Indicators).
DevSecOps that extended the DevOps model and practices by addressing security aspects during the whole system/services lifecycle, intending to address “Security by Design“ concept (however
not yet fully developed)
General compliance with the project management principles, models and procedures applied to infrastructure, services, and data handling and analytics.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

17

SADP: General architecture design principles
General architecture design principles• Layered architecture design for services and mechanisms, including inter-layer interfaces. Including crosslayer services and mechanisms definition that are typically defined as service planes, for example,
management plane, security plane, data management plane.
• Multi-tier services and infrastructure design, including combined multi-layer and multi-tier systems that
may use or apply different architectural and layered solutions.
• Application Programming Interfaces (API) for composable services that must be supported by consistent(and fully qualified API metadata and namespaces definition).

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

18

SADP: Service architecture related
Service architecture related• Service Oriented Architecture (SOA) and Microservices Architecture (MSA) that is supported with thedifferent VM and container solutions and/or platforms.
• Cloud powered, cloud based and cloud native design principles that leverages (require knowledge) of themodern cloud architecture and cloud platform, both Open Source and public clouds (at least Amazon Web
Services, Microsoft Azure, and Google Cloud Platform). This also includes such powerful cloud based
mechanism as Virtual Private Cloud (VPC) that provide VPN based secure environment for multi-tenantcustomer applications.
• Service lifecycle management model that should include all necessary services to support lifecycle stages
in the context of specific services. This also includes services composition and orchestration for services
deployment and operation.
• Operations and management support: SLA, monitoring, audit, (certification)

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

19

SADP: Data infrastructure and services related
Data infrastructure and services related• Big Data computation models and supporting platform, distributed and highly scalable systems, inparticular Hadoop ecosystem and NoSQL databases.
• Data management infrastructure and services that should cover two domains: services data (mostly
related to management plane) and business or research data produced as a result of business operation orscientific research.
• Services and data management continuity in IoT/sensor networks, edge, cloud, data-driven applications
that also include 5G/6G Radio Access Network (RAN), edge and cloud convergence: data ID, data models,data linkage and lineage for different types of data: experiment (stimulus and result), configuration,
metadata, process models

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

20

SADP: Security and compliance design principles
Security and compliance design principles• Security architecture and security services lifecycle management which are well defined by numerous
standards and supported by the major infrastructure development frameworks. Although security services
have their own multi-layer architecture (can also be referred as security plane), their integration with themain infrastructure services, including data infrastructure) is realized via API calls and consistentdefinitions of the security roles, access control policies and credentials and secrets management.
• Compliance frameworks that define requirements to and recommendations for secure services andinfrastructure design and operation. Cloud Security Alliance (CSA) and Compliance Assessment Initiative
Questionnaire (CAIQ) provide the best overview of all important standards and regulations to ensure
systems security and compliance, and data protection.
• Industry best practices adopted: zero-trust access control, Federated AuthN, AuthZ, Identity Management,
session and security context management

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

21

SADP: Project Management and DevOps
Project Management and DevOps• DevOps and SRE (Site Reliability Engineering) practices applied to system and services engineering andoperation. This should also include continuous monitoring and optimisation on multiple user-centric andbusiness-centric SLI/KPI (Service Level/Key Performance Indicators).
• General compliance with the project management principles, models and procedures applied toinfrastructure, services, and data handling and analytics, e.g. IEEE PMI.
• DevSecOps that extended the DevOps model and practices by addressing security aspects during thewhole system/services lifecycle, intending to address “Security by Design“ concept (however not yet fully
developed)
DevOps culture controversy:• Agile works with tightly cooperated and often compact located team to enable skills and experience sharing, continuous
development steering• May work bad for distributed part time team• “Fail fast” principle lowers importance of professional architecture design• User stories doesn’t substitute consistent Requirements Engineering

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

22

SADP In Education and Training• SADP and architecture design patterns in the course Big Data
Infrastructure Technologies for Data Analytics (BDIT4DA)• NTUU Igor Sikorski “Kyiv Polytechnic Institute” and Vodafone Ukraine Data
Science Academy• SADP and cloud based design patterns• DevOps and Cloud based Software Engineering at the University of
Amsterdam (UvA)• SLICES Academy: Starting with Summer School and providing online
MOOC courses on demand• SADP and Experimental Data Management

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

23

Big Data Infrastructure Technologies for Data Analytics MasterCourse (NTUU “KPI” & UvA)
• Module 1: Introduction to the course. Cloud Computing foundation. Cloud service models, cloud resources.
• Module 2: Big Data architecture framework, cloud based Big Data services and platforms.
• Module 3: Big Data Algorithms: MapReduce, Pregel. Hadoop platform and components for Big Data analytics:
HDFS, YARN, MapReduce, HBase, Pig, Hive, others.
• Module 4: Data Streams and Streaming Analytics. Kafka, Flume. Spark architecture and popular Sparkplatforms, DataBricks.
• Module 5: SQL and NoSQL Databases. CAP Theorem. Modern large scale databases AWS Aurora, Azure
CosmosDB, Google Spanner.
• Module 6: Data Management and Governance. Research Data Management and FAIR data principles in data
management.
• Module 7: Big Data Security and Compliance. Cloud compliance standards and cloud provider services
assessment.
• Module 8: Platforms and tools for Data Analytics and NL pipeline automation (such as AWS SageMaker or
Azure MLOps, supported with Data Lakes)• Module 9: Managing Data Science Projects. Research methods and project organisation. Data Science
Process Models, DataOps and MLOps.
• Module 10: Sustainable Architecture Design principles and cloud based design patterns, which will be
extended with discussed in this paper and design patterns.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

24

Big Data Infrastructure Components and Course Modules
S9

S10• Module 9: Platforms and tools for
Data Analytics and NL pipeline
automation
(such
as
AWS
SageMaker or Azure MLOps,
supported with Data Lakes)• Module 8: Managing Data Science
Projects. Research methods andproject organisation. Data Science
Process Models, DataOps andMLOps.
• Module
10:
Sustainable
Architecture Design principles andcloud based design patterns, which
will be extended with discussed inthis paper and design patterns.

S8

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

25

Importance of Data Management topics in University
Curricula• Data Management and Governance (DMG) and corresponding infrastructureservices are important for data driven and data centric architecture models• FAIR data principles: Research data must be Findable, Accessible, Interoperable,
Reusable• Metadata and namespace management• Lineage and provenance, linked data
• PID (Persistent Identifier) and data discoverability

HPCC2023, 13-15 Dec 2023

Technical implementation of the FAIR
data principles requires comprehensive
data management infrastructure to
support• Data Storage• Metadata Registries• Data publication• Data discovery• Linked data and data lineage(provenance)• Multiple datasets access for analysis

Sustainable Architecture Design Principles

26

DevOps and Cloud based Software Engineering (DevOps4SE)• DevSecOps; Secure Software Development Lifecycle Management (SDLM);
cloud based tools for secure software development and testing• Extended with compliance and Risk assessment to cover the whole product lifecycle• Data Science project management and DataOps/MLOps processes and platforms• MLOps maturity levels• Sustainable Architecture Design principles and cloud based design patterns

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

27

Future work and development• Further extensive study of architecture styles and principles• Link to TOGAF and Enterprise Architecture Model• SADP and Legacy solutions + “gravity” of siloed design• Following and contributing to professional blogs like Medium, SCUB LAB• Identify new architecture design principles in such areas as 5G/6G, AI and LLM/GenAI enabled• Develop SADP related topics for different Software and System Engineering courses• Run seminars and tutorials at conferences and professional events• Possibly write a (text)book

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

28

Additional Information (for discussion)• SADP and Architecture styles illustrated• FAIR data principles

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

29

SADP Illustrated• Multi-layer architecture style• 3 and multi/N tiers models• Microservices• Web-queue-worker• Event driven• Big Data
• HPC/Big Compute

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

30

N-Tier Architecture• N-tier is a traditional architecture forenterprise applications. Dependencies aremanaged by dividing the application
into layers that perform logical functions,such as presentation, business logic, anddata access.
• A layer can only call into layers that sit
below it. However, this horizontal layering
can be a liability.
• It can be hard to introduce changes in one
part of the application without touching
the rest of the application. That makes
frequent updates a challenge, limiting how
quickly new features can be added.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

31

3/N tier Architecture Example for Cloud based Applications• Traditional architecture for enterprise applications• Presentation/Web tier – Business Logic (application) – Datatier (assets)

HPCC2023, 13-15 Dec 2023
Sustainable Architecture Design Principles
https://learn.microsoft.com/en-us/azure/architecture/guide/architecture-styles/n-tier

32

Layers and Tiers
Using cakes metaphor• Layers are defined to separate
functionalities and defineinterlayer interfaces• Multi-layer cloud architecture• Tiers are defined for physicallyseparate functionalities that
communicate via API• 3 tier applications architecture• Presentation tier
• Business application/logic tier
• Data access tier

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

33

Frontend and Backend services – General view•

Using Radicore components in a front-end website, Using Radicore components in a front-end website
Posted on 1st June 2009 by Tony Marston
https://www.tonymarston.net/php-mysql/radicore-front-end.html

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

34

Azure Well-Architected Framework• Architecture types• Design principles• Best practices• Azure Well-Architected Security Pillar
https://learn.microsoft.com/en-us/azure/wellarchitected/security/overview• More design patterns

https://learn.microsoft.com/en-us/azure/architecture/guide/
https://learn.microsoft.com/en-us/azure/architecture/patterns/
https://learn.microsoft.com/enus/azure/architecture/browse/?filter=reference-architecture

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

35

Web-Queue-Worker• For a purely PaaS solution, consider a WebQueue-Worker architecture.
• In this style, the application has a web frontend that handles HTTP requests and aback-end worker that performs CPUintensive tasks or long-running operations.
• The front end communicates to the worker
through an asynchronous message queue.

• Web-queue-worker is suitable for relatively
simple domains with some resourceintensive tasks.
• The use of managed services simplifies
deployment and operations. But withcomplex domains, it can be hard to manage
dependencies.
• The front end and the worker can easily
become large, monolithic components thatare hard to maintain and update.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

36

Microservices• A microservices application iscomposed of many small,independent services. Each service
implements a single businesscapability. Services are loosely
coupled, communicating throughAPI contracts.
• Each service can be built by asmall, focused development team.
Individual services can be deployed
without a lot of coordination
between teams, which encourages
frequent updates.
• A microservice architecture is more
complex to build and manage thaneither N-tier or web-queue-worker.
It requires a mature development
and DevOps culture.
• But done right, this style can leadto higher release velocity, fasterinnovation, and a more resilientarchitecture.
HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

37

Event-Driven Architectures• Event-Driven Architectures use apublish-subscribe (pub-sub) model, where
producers publish events, and consumers
subscribe to them.
• The producers are independent from theconsumers, and consumers are
independent from each other.
• Consider an event-driven architecture forapplications that ingest and process alarge volume of data with very low latency,such as IoT solutions.
• The style is also useful when different
subsystems must perform different typesof processing on the same event data.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

38

Azure Big Data Architecture Style

https://learn.microsoft.com/en-us/azure/architecture/guide/architecture-styles/big-data•••

Azure includes many services that can be used in a big data architecture. They fall roughly into two
categories:
Managed services, including Azure Data Lake Store, Azure Data Lake Analytics, Azure Synapse Analytics,
Azure Stream Analytics, Azure Event Hubs, Azure IoT Hub, and Azure Data Factory.
Open source technologies based on the Apache Hadoop platform, including HDFS, HBase, Hive, Spark,
Oozie, Sqoop, and Kafka. These technologies are available on Azure in the Azure HDInsight service.

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

39

FAIR Data Principles: Metadata Management(GO FAIR recommendations)Findable:

Accessible:• F1 (meta)data are assigned a globally unique andpersistent identifier;• A1 (meta)data are retrievable by their identifier using astandardized communications protocol;• A1.1 the protocol is open, free, and universally• F2 data are described with rich metadata;
implementable;• F3 metadata clearly and explicitly include the identifier of• A1.2 the protocol allows for an authentication andthe data it describes;
authorization procedure, where necessary;• F4 (meta)data are registered or indexed in a searchable • A2 metadata are accessible, even when the data are noresource;
longer available;

Interoperable:

Reusable:• I1. (meta)data use a formal, accessible, shared, and• R1 meta(data) are richly described with a plurality ofbroadly applicable language for knowledge representation. accurate and relevant attributes;• I2. (meta)data use vocabularies that follow FAIR
• R1.1 (meta)data are released with a clear and accessible
principles;data usage license;• I3. (meta)data include qualified references to other(meta)data;

HPCC2023, 13-15 Dec 2023

• R1.2 (meta)data are associated with detailed provenance;• R1.3 (meta)data meet domain-relevant community
standards;

Sustainable Architecture Design Principles

40

FAIR from the technical point of view – Required
Infrastructure functionality• Findable• Metadata and PDI – infrastructure and tools• Metadata Registries and handles resolution, API• Policies and SLA• Accessible
•
•
•
•
•

Repositories and data storage: infrastructure and management
Policy and access control: infrastructure and API management
Data access protocols
Usage Policy and Sovereignty
Data protection, compliance, privacy and GDPR

• Interoperable• Standard data formats• Metadata Registries and API• FAIR maturity level and certification• Reusable• Data provenance and lineage• Preservation• Metadata, PID and API – linked or embedded into datasets

HPCC2023, 13-15 Dec 2023

Sustainable Architecture Design Principles

Technical implementation of the
FAIR data principles requires
comprehensive data management
infrastructure to support• Data Storage• Metadata Registries• Data publication• Data discovery• Linked data and data lineage(provenance)• Multiple datasets access foranalysis41