Quality for
DevOps teams
Rik Marselis
25 September 2023
TMAP: the body of knowledge for
quality engineering & testing in IT delivery© 2023 Sogeti. All rights reserved.

1Challenges of today’s IT delivery
The business demands:• Deliver business value• Deliver quality at speed

Value&Speed

The team challenges are:• Quality engineering is everyone's responsibility• QA & testing is integrated in process and people
The focus is:• Organize high-performing cross-functional teams (you build it, you run it!)• Automate everything (as long as it is useful)© 2023 Sogeti. All rights reserved.

2

High-performance IT delivery
is an approachthat enables cross-functional teams

to continuously improve
the products, processes and people

that are required todeliver value

to the end users.
© 2023 Sogeti. All rights reserved.

3

Rik Marselis

Principal Quality Consultant

1980

2007

2018

2008

2009 201220202012 2014© 2023 Sogeti. All rights reserved.

4

ISTQB
International
SoftwareTesting
ExcellenceAward2022

EuroSTAR
2023
Testing
Excellence
Award

Olivier Denoo presents the award
at the ISTQB general assemblyin Marrakech on 20 October 2022

Bob van de Burgt presents the award
at the EuroSTAR Awards Gala nightin Antwerp on 15 June 2023
© 2023 Sogeti. All rights reserved.

5

3 people will
win a copy ofthis TMAP book!
TMAP: the body of knowledge forquality engineering & testing in IT delivery© 2023 Sogeti. All rights reserved.

6

Waterfall

V-modelFigure 7.1 from the book Quality for DevOps teams

Scrum

DevOpsDemand-Supply

Agile mindset

SAFe®

© 2023 Sogeti. All rights reserved.

7

DevOps – highlights
The six DevOps principles:

1. Customer-centric action
2. Create with the end in mind
3. End-to-end responsibility
4. Cross-functional autonomous teams

5. Continuous improvement
6. Automate everything you can

The six fundamental DevOps activities

DevOps is a cross-functional systems engineering culture that aims at unifying
systems development (Dev) and systems operations (Ops) with the ability to create anddeliver fast, cheap, flexible and with adequate quality, whereby the team as a whole is
responsible for the quality. Other areas of expertise, such as business analysis and quality
assurance (including testing) are usually integrated in the team. A DevOps culture has an
Agile mindset that can be supported/implemented by e.g. the Scrum framework.

© 2023 Sogeti. All rights reserved.

8

High-performance IT delivery
is an approachthat enables cross-functional teams

to continuously improve
the products, processes and people

that are required todeliver value

to the end users.
© 2023 Sogeti. All rights reserved.

9

Business value depends on the quality of products, processes & people

Quality
Products

PeopleProcesses© 2023 Sogeti. All rights reserved.

10

High-performance IT delivery
is an approachthat enables cross-functional teams

to continuously improvethe products, processes and people

that are required todeliver value

to the end users.
© 2023 Sogeti. All rights reserved.

11The starting point for DevOps:

And for Agilein general

A cross-functional teamtakes joint responsibility

TEAM:
TogetherEveryone
Achieves
More

Aligned with the DevOps culture of IT delivery – where roles are important and functions
are not – keep in mind that if in TMAP you see a term such as developer or tester, werefer to the role of a team member at a specific moment in time, not to a function.

Example of a cross-functional
high-performance IT delivery team© 2023 Sogeti. All rights reserved.

12

High-performance IT delivery
is an approachthat enables cross-functional teams

to continuously improve
the products, processes and people

that are required todeliver value

to the end users.
© 2023 Sogeti. All rights reserved.

13

The starting point of organizations is to deliver business value© 2023 Sogeti. All rights reserved.

14

Our “VOICE model” for delivering business value

Build quality in by
applying relevant
quality measures
Figure 3.1 from the book Quality for DevOps teams

Measuring indicators=(mainly) testingSupply information
based onmeasuring indicators© 2023 Sogeti. All rights reserved.

15

Menti-question
What words do you think of for the concept of“Quality Engineering”© 2023 Sogeti. All rights reserved.

16

Quality Engineering (definition)

Quality Engineering
is about team members and their stakeholders
taking joint responsibility
to continuously deliver IT systems
with the right quality at the right momentto the businesspeople and their customers.
It is a principle of software engineering
concerned with applying quality measures
to assure the quality of IT systems.

© 2023 Sogeti. All rights reserved.

17

A Quality Measure isa group of activities
that is aimed at achieving
a certain level of quality.
© 2023 Sogeti. All rights reserved.

18

Quality Engineering & Testing support delivering business value

Quality Engineering“Preventive”
Build quality in,quality measures right first time

Testing“Detective”
Provide information
quality measures about quality & risks“Corrective”
Pro-active quality
quality measures improvement© 2023 Sogeti. All rights reserved.

19

Quality engineering consists of many activities ➔ grouped in topics

Multi-teamscope

Team scope

Organizing

Performing

Aimed at:orchestrating, arranging,planning, preparing andcontrolling

Aimed at:operational
QA & testing
activities.

Figure 11.2 from the book Quality for DevOps teams© 2023 Sogeti. All rights reserved.

20

How quality engineering & testing topics align with DevOps activities

Figures 1.4 and 11.2 from the book Quality for DevOps teams© 2023 Sogeti. All rights reserved.

21

Continuous improvement of products

Products
Processes

PeoplePreventive quality measure

Example: refinements© 2023 Sogeti. All rights reserved.

22

Build quality in from the start, do proper refinements
One concept that can be used is “perspective based reviewing”
Also known as the 4-amigos approach
Every role of the team is involved in the review

Advantages:- All relevant aspects are considered- Unclear information is cleared up instantly- All team members have a common understanding© 2023 Sogeti. All rights reserved.

23

Continuous improvement of products

ProductsProcesses

PeoplePreventive, detective andcorrective quality measures

Example: align quality measures withrisk level© 2023 Sogeti. All rights reserved.

24Build quality in from the start, aligned with risk level• Do a quality risk analysis• Low risk ➔ one developer creates the code, another reviews• High risk ➔ pair programming, two developers create code together• This way the effort of development of the product is aligned with thequality level needed to provide business value• And what about “No Risk” ?
➔ No testing, but also…
➔ No development!!! (no risk means nobody really needs it!)© 2023 Sogeti. All rights reserved.

25

Continuous improvement of processes

Products
Processes

PeopleDetective quality measure

Example: Testing© 2023 Sogeti. All rights reserved.

26

Definition of testing within business delivery & quality engineering

Testing consists of verification, validation and exploration activities
that provide information about the quality and the related risks,
to establish the level of confidence
that a test object will be able to deliver the pursued business value.
Figure 5.6 from the book Quality for DevOps teams© 2023 Sogeti. All rights reserved.

27

Static versus Dynamic testing(examples)

Perspective based reviewing“Running” the system© 2023 Sogeti. All rights reserved.

28

Testing new & changed versus unchanged deliverables

Testing new and changeddeliverablesTesting unchanged
deliverables

Testing fixed problems

All three varieties of testing are important!!
© 2023 Sogeti. All rights reserved.

29

The process of testing: goal, test design and test execution
Goal

Test execution

Test design

What is the most important part of a test case?
The expected outcome!! (which is compared to the actual outcome)Figure 43.3 from the book Quality for DevOps teams© 2023 Sogeti. All rights reserved.

30

Test design approaches and test design techniques

Figures 43.1, 45.4 and 47.1 from the book© 2023 Sogeti. All rights reserved.

31

Continuous improvement of processes

ProductsProcesses

PeoplePreventive and detective
quality measures

Example: automate tests in yourpipeline© 2023 Sogeti. All rights reserved.

32

Continuous feedback & shift left

e.g. Test
DrivenDevelopment© 2023 Sogeti. All rights reserved.

33

Quality at speed ➔ Automate everything (as long as it is useful)

An automated pipeline enables:

Figure 6.1 from the book

• Elimination of errors due to manual task execution• Provisioning of standardized development feedback loops• Enabling of fast product iterations© 2023 Sogeti. All rights reserved.

34The testing pyramid

Note: When in automated regression testing
not many anomalies are found this is good,
because regression testing is mainly tocheck if no unexpected changes were madeto the system.
Figure 32.2 from the book Quality for DevOps teams

automated testing (as long as it is useful)Testing should be automated asmuch as possible, and testingshould be done as early aspossible. But manual testing also
has value!
© 2023 Sogeti. All rights reserved.

35

It’s seldom wise toautomate nothing…
It’s NEVER wise toautomate everything!
© 2023 Sogeti. All rights reserved.

36

Continuous improvement of people

ProductsProcesses

PeoplePreventive and detective
quality measure

Example: Agile at scale …
Quality orchestration© 2023 Sogeti. All rights reserved.

37

Agile at scale (using the scaled agile framework as an example)
Entireorganisation
End-to-end
business solution
Multiple teamsTeam© 2023 Sogeti. All rights reserved.

38

End-to-end quality orchestrator
Responsible for organizing end-to-end quality across multiple teams.
Can take care of governance together with, or instead of, the scrum-of-scrums.
Takes responsibility for making the end-to-end testing a recognizable, transparent, efficient andeffective process.
Part of a virtual team or support team for end-to-end testing.

(more details about the quality orchestrator can be found in the syllabus for the “TMAP:Organizing built-in quality at scale” training & certification)© 2023 Sogeti. All rights reserved.

39

End-to-end testing of multiple systems that together support
one business process.
Business process

System A

System B

System CSystem D

System E

Scopeof endto-endtest

Scope of testing of
one team
Note: other teams have thescope of systems A, B, Dand E. There is one end-toend test scope.

Teams often don't have the knowledge, time and resources to properly organize and perform
end-to-end tests. A separate dedicated team can organize and perform end-to-end-regression
tests on demand of a team, for example after a change is integrated in the main branch.
Figure 6.1 of the book© 2023 Sogeti. All rights reserved.

40

CI is team-focus (one build pipeline per team)CD is multiple team focus (one release pipeline per business solution)
Team 1Team 2Team 3

Figure 6.1 from the book© 2023 Sogeti. All rights reserved.

41

End-to-end testing: what’s your scope?
End-to-end testing is a specific test variety where the test object has a large scope.
Often people project the test object of an end-to-end-test as a simple chain of systems.

But in reality …

Or even …

© 2023 Sogeti. All rights reserved.

42

End-to-end testing: practical tip for complex situations
Determining end-to-end test cases may seem very difficult, especially in situations where acomplex mesh of systems is the test object.

However, the way to approach this is to start defining the desired outcomes that are needed tobuild confidence in the end-to-end-process.
Then work backwards towards the start of the chain of systems, determining what inputs areneeded to get the desired outcome. This generally will result in more focused testing and a much
lower number of test cases.

Desiredoutcome© 2023 Sogeti. All rights reserved.

43

Wrap-up
Goals:
Deliver business value
Deliver quality at speed
How:• Collaborate within and across cross-functional teams• Apply quality measures© 2023 Sogeti. All rights reserved.

44

Wrap-up: To deliver business value ➔build quality in and supply information to support confidence

Build quality inFigure 3.1 from the book Quality for DevOps teams

Supply information

Get confidence© 2023 Sogeti. All rights reserved.

45

Questions&Answers
Do you want to know more?

Find the book at www.ict-books.com
or in your local or online bookstore

Whowins acopy ofthebook???

© 2023 Sogeti. All rights reserved.

46

www.TMAP.net ➔ body of knowledge for QE&T

Dozens of practical and usefuldownloads, such as templates© 2023 Sogeti. All rights reserved.

47

The TMAP certification scheme
Training providers
in many countries

TMAP: Highperformance
qualityengineering
TMAP: Quality
for crossfunctional teams

TMAP:
Organizing builtin quality
at scale

Focused on people in IT delivery teams

New2023

TMAP: Quality
engineering for
SAP

Focused on people related tobusiness and maintenance

Info about TMAP training & certification at:

www.TMAP.net/page/tmap-training
www.TMAPcert.com

Exams provided by© 2023 Sogeti. All rights reserved.

48

TMAP:
body of knowledge forquality engineeringand testing
Feel free to contact me on:
Rik.Marselis@Sogeti.com

About Sogeti

TMAP website:
www.tmap.net
Sogeti academy:
academy.sogeti.nl

Sogeti is a leading provider of technology and engineering services. Sogeti delivers
solutions that enable digital transformation and offers cutting-edge expertise in Cloud,Cybersecurity, Digital Manufacturing, Digital Assurance & Testing, and emerging
technologies. Sogeti combines agility and speed of implementation with strong technology
supplier partnerships, world class methodologies and its global delivery model,Rightshore®. Sogeti brings together more than 25,000 professionals in 15 countries,
based in over 100 locations in Europe, USA and India. Sogeti is a wholly-owned subsidiaryof Capgemini SE, listed on the Paris Stock Exchange.
Learn more about us atwww.sogeti.com

This message contains information that may be privileged orconfidential and is the property of the Capgemini Group.

© 2023 Sogeti. All rights reserved.

49