The future
up closeWORLD QUALITY REPORT
15th Edition 2023-24

I N A S S OCI AT ION WI T H :WORLDQUALITYREPORT2 0 2 3 - 2 4
Regional and country reports
are available online fromwww.worldqualityreport.com
or from Capgemini, Sogeti, and
OpenText local offices covering:North America, United Kingdom,France, Germany, Benelux,
Southern Europe (covering Italy,
Spain, and Portugal) Eastern
Europe (covering Poland, Romania,
Hungary, and the Czech Republic)
China, Australia and New Zealand,
Japan, and the Nordic region.

CONTENTS
INTRODUCTION

04

EXECUTIVE SUMMARY06

KEY RECOMMENDATIONS08

CURRENT TRENDS INQUALITY ENGINEERING & TESTING

11

Business Assurance

12

Agile Quality Management

17

QE Lifecycle Automation

22

AI, the Future of QE

28

Quality Ecosystem

35

Digital Core Reliability

42Intelligent Product Testing48

Quality & Sustainability

53

SECTOR ANALYSIS

60

Automotive

62

Consumer Products, Retail,and Distribution

65

Energy, Utilities, Natural Resources,
and Chemicals

68

Financial Services

71

Healthcare and Life Sciences

73

Manufacturing

76

Public Sector

79

Telco & Media82

ABOUT THE STUDY87

ABOUT THE SPONSORS

92

INTRODUCTION
Welcome to the very special 15th edition of the World Quality Report, which isrecognized as the industry’s largest research study on quality engineering (QE). The
World Quality Report provides a comprehensive assessment of the current stateof quality engineering practices around the world and across different industries.
Over the last 15 years, this report has examined the most important trends anddevelopments in quality engineering and testing by surveying 1,750 executives to
put together the most interesting firsthand insights. It’s a great honor for us here
at Capgemini and Sogeti to publish this report, along with our strategic technology
partner OpenText. We have made every effort to cover topics that are as wide andfar-reaching as possible to give you an in-depth view of the latest trends, challenges,
transformative initiatives, and disruptions shaping them. In this report, you will see
our key findings and recommendations for several focus areas: business assurance,
agile quality management, QE lifecycle automation, AI, quality ecosystem, intelligent
product testing, and sustainability. The expert findings are further accentuated bycommentary, examples, and best practices from 10 senior executives from various
Fortune 500 organizations, who participated in our deep-dive interview series aroundthese topics.
CHANGING TIMES AGAINMARK BUENEN
Global Leader, Quality Engineering& Testing, Capgemini & Sogeti

In the past editions, we saw an immense drive for faster and more impactful IT
developments and digital transformation. We also saw an even greater accelerationof that agenda on a global scale across all industries. Development of unified digitalplatforms, Agile solution delivery, hyper-automation, advanced analytics and AI, andbusiness and IT fusion were required to push organizations forward. Having the rightlevel of focus on quality was thus even more critical.
This year we see a continuation of these needs plus the emergence of a true gamechanger in the field of software and quality engineering: Generative AI adoption toaugment our engineering skills, accelerated like never before. The lack of focus on
quality seen in the last few years is becoming more visible now and has brought backthe emphasis on the Hybrid Testing Center of Excellence (TCoE) model, indicating
somewhat of a reversal trend. Business Assurance is the name of the game, and QE
evangelists need to truly step up and elevate their real purpose to focus on assuringbusiness outcomes.
TOP, DOWN, ALL AROUNDSATHISH NATARAJAN
Group Vice President, Quality
Engineering & Testing, Capgemini& Sogeti, Americas

4

World Quality Report I 2023-24

The future and what it holds for quality engineering and testing is essentially twodimensional. One view is of the big picture — macro dynamics and interdisciplinarysynergies. The other is a deep dive into the nuances and micro shifts in the QE space. This
year’s World Quality Report offers insights through both lenses: a pull-back panoramaof the future as well as the inner workings of minor developments and moving pieces.
A TEAM EFFORTThis report wouldn’t have been possible without the significant contributions of many
people. If you are one of the 1,750 executives across 32 countries who took part in thisyear’s survey, we would like to thank you for your time and contribution. Your input wasinvaluable in helping us gauge the prevailing moods and trends. We also have a special
callout to the industry expert panel, whose insights have been valuable in illustrating
the broader themes. We thank our partners at OpenText, and our lead authors, sector,
and regional subject matter experts (SMEs) at Capgemini and Sogeti, who together

analyzed, interpreted, and provided expert commentary on the research data and interviews to build this report. In addition,
we thank the report’s production team: much work takes place behind the scenes to ensure this annual exercise bears fruit.
Finally, we thank you, our readers. It’s your own experience and interest that gives the World Quality Report its reason for being.
As ever, we hope this year’s edition makes a rewarding contribution to your continuing efforts in software quality assurance, andthat you can take advantage of these findings and recommendations to shape your quality engineering strategy, and perhaps evento challenge some of your current thinking.

OpenText is excited to deliver the 15th edition of the World Quality Report.
The ever-evolving global environment we face has highlighted the importance of
emerging trends showcased in prior editions of the World Quality Report, from the rapid
organizational adoption of cloud-based quality ecosystems to widespread investment
in Artificial Intelligence (AI) solutions, all of which promise an exponential increase inproductivity and efficiency.
This edition shines a light on AI - the future of quality engineering. More than half oforganizations anticipate that AI adoption will lead to higher productivity and increased
velocity of development, and they plan to continue the investment towards executionof scaling the quality engineering process accordingly. Of course, AI also comes withchallenges, such as 33% of respondents believe sustainability concerns and lack ofprivacy will present roadblocks to maximizing the value of AI in their organizations.
The organization that can overcome these roadblocks will be well positioned to be
an industry leader going forward.
Once organizations overcome the challenges and mitigate the risks associated withAI, the opportunities and applications are endless. From performance engineeringand autonomous testing to data generation and root cause analysis, AI is poised
to permanently alter our understanding of what is possible. No matter how you
decide to implement AI, the key will be making AI a part of your organization’s DNA,
which will require careful thought and analysis for prioritization along withunambiguous KPIs measuring tangible outputs to drive successful quality engineeringtransformation with AI.

MUHI MAJZOUB
Chief Product Officer,
OpenText

As AI becomes ubiquitous, it will create the need for organizations to build holistic
quality ecosystems in which data is simultaneously highly visible and secure for every
part of the organization to see and interact with. Moreover, 64% of respondents
recognize quality ecosystem sustainability as a priority, not just for themselves but
for customers and investors as well, with the economic, social, and environmental
benefits being profound.
It is critical that organizations build a robust quality ecosystem now, because soon,
having one will not be optional, it will be a necessity. Organizations must invest in
workforce transformation by building subject matter expert and chaos engineering
capabilities within quality engineering organizations to embrace and support the
emerging trends in digital and cloud migration strategies of the customers. They will
also have to accelerate the inclusion of cloud and infrastructure testing as part of thesoftware development lifecycle to improve on resiliency, security, redundancy, and data
recovery. Lastly, they will also need to build systems resistant and resilient to failure byadopting stringent practices to test the impact of failure events on cloud and continue
to give extreme high priority to the performance and resiliency of applications on cloud.
We invite you to delve into the report to discover the many challenges and opportunities
that organizations will face in their endless pursuit of quality excellence and get a
glimpse of the future up close.
Finally, I would like to express my appreciation and thanks to Capgemini, and to everyone
who has worked so diligently to create this edition of the World Quality Report.

5

EXECUTIVE SUMMARYWorld Quality Report 2023-24
EXECUTIVE INTRODUCTION:EMBRACING QUALITY
TRANSFORMATION FORSUSTAINABLE SUCCESSJEAN-BAPTISTE BONNET

JEFF SPEVACEK

Offer Leader, Intelligent Testing forConnected Products,
Capgemini Engineering, France

Head of QE&T,
Financial Services,
Capgemini North America

KAVITHA VELCHALASHWINI NADIGER

Portfolio Manager, Testing Deal
Architect for Global Sales
Support Team, QE&T,
Capgemini Financial Services

Vice President,
Portfolio COE Headfor QE&T,
Capgemini North America

ANISH BEHANANJEBA ABRAHAM

Head of QE&T, Capgemini UK

Practice Vice President,
QE&T, Sogeti USA

ANDREW FULLEN
Head of Innovation and Technology,
Sogeti UK

6

World Quality Report I 2023-24

In the constantly changing world of business, thepursuit of quality has become a key factor in achieving
sustainable, long-term success. It’s now more
important than ever to develop a systematic approach
to identifying business risks and crucial priority areas.
This 15th edition of the World Quality Report explores
new and evolving approaches in quality engineering(QE), and provides an in-depth view of the latest
trends, challenges, transformative initiatives, anddisruptions shaping them.
Quality is no longer a mere checkpoint; it’s a serious
boardroom agenda and part of a culture thatorganizations are increasingly embracing. As 38% ofbusinesses shift their focus from volume to value,
there’s a growing recognition of the importance ofquality in delivering value to customers. Notably, 67%
have incorporated quality assurance (QA) into thecore of their business operations, ensuring that theirtechnology investments align with quality standards.
Furthermore, almost all organizations havetransitioned from conventional testing to agile
quality management. Evidently, they understand thenecessity of adapting to the fast-paced digital world.
An agile quality culture is permeating organizations,albeit often at an individual level rather than ata holistic program level. Many organizations are
adopting a hybrid mode of Agile. In fact, 70% oforganizations still see value in having a traditional
Testing Center of Excellence (TCoE), indicating
somewhat of a reversal trend.
What is clear is the extended knowledge and skills
that are required from the QE experts who operate inagile teams. Coding skills in particular (C#, Java, SQL,
Python), and business-driven development (BDD)
and test-driven development (TDD) competencies,
are in demand.

It is also evident that operating in a full agile model does stillcreate some serious challenges for many organizations. The
most important agile quality challenges are a lack of knowledgeof techniques and inadequate support for testing processes.
To address these challenges, a shift towards product-based
quality engineering setup is recommended, with a focus on
customer-centric design thinking, artificial intelligence andmachine learning (AI/ML) adoption, and aligning to a businessassurance mindset.
Objectives such as improved customer experience, risk
reduction, increased QA efficiency, and reduced testing effort
drive automation in quality. While quality automation is certainlyon the rise, organizations still face critical challenges. An averageof 27% report challenges related to dealing with legacy systems
and the fast-changing application landscape.
Artificial intelligence (AI), and especially the application ofgenerative AI, is poised to be a game-changer in quality
engineering. It is fundamentally redefining the art of thepossible. The expected outcomes of generative AI are sure todrive higher productivity and efficiency gains. Almost 77% oforganizations are reportedly investing in AI solutions to drive
the quality transformation agenda. However, concerns related
to security, privacy, and biased outcomes need to be addressed.
A significant percentage (31%) remains skeptical about
the value of AI in QA, emphasizing the importance of anincremental approach.
Quality ecosystems are evolving rapidly, with a decisive shift
in cloud adoption for test environments and the emergenceof site reliability engineering (SRE). We are witnessing themetamorphosis of cloud testing, with over 82% of theorganizations asserting the need for a cloud testing strategy
as they modernize their application portfolios. Still, challenges
persist in areas around test data provisioning for CI/CD pipelines
without reliance on production data.

of testing is about delivering the best customer experiences anddigital happiness, protecting the brand image, and ultimately
assuring successful business outcomes. This requires more focus
on end-to-end customer journey validation and a purposefulcollaboration with business to maximize impact. We will soon
start to see newer approaches such as test isolation and contract
testing become mainstream to sidestep the roadblocks in endto-end business process testing.
The interconnected product landscape brings a level ofcomplexity never seen before. Demanding customer
experiences, interoperability needs, regulatory pressures,constantly changing guidelines, and cyber threats warrant amuch more rigorous testing focus. There is a genuine needto reinvent the product testing approach for the globallyinterconnected spaces we interact in. Hyper-personalization
for unique user experiences necessitates exhaustive testing tovalidate all possible combinations, which is impossible to achieve.
This is where AI takes center stage to drive maximum value,
a key lever being an intelligent testing strategy.
Lastly, organizations recognize the role of QA in driving thesustainability agenda. It is very encouraging to learn that 73%
of the respondents showed positive intent to become active
in driving sustainability best practices in their organizations.
A good first step towards making an impact is to own theprocess of bringing in transparency in measuring, reporting,and monitoring. They must act as custodians of sustainability andnot allow greenwashing practices, which we are unfortunately
starting to witness.
In conclusion, succeeding in quality engineering and assurance
requires embracing change, fostering a culture of quality,
and leveraging advancements such as AI and automation.
As organizations continue to evolve, a commitment to quality
ensures sustainable success now and in the future.
Ready to see the future up close? Let’s take a look.

For most industries, if not all, the digital core is at the heart of how
you do business. More than ever before, the strategic purpose
7

KEY RECOMMENDATIONSWorld Quality Report 2023-24• AGILE QUALITYMANAGEMENT•

Invest in robust training programs: Organizations
must adopt a build-or-buy mentality. Developing arobust training program reduces overall costs whilesetting your organization up to continue to adapt
to the industry’s ever-changing technology andskillset requirements.

•

Integrate with DevOps/DevSecOps: Forging aseamless integration with DevOps/DevSecOps
practices through close collaborations can ensure
the smooth incorporation of quality engineering
assets into CI/CD pipelines.

•

Modernize your QE organization: To empower
individual agile aligned teams to be more
self-sufficient.

• BUSINESS ASSURANCE•

Leverage the testing function to deliver businessoutcomes. This will ensure that business operation
processes are aligned with business standards
and objectives.

•

Use value stream mapping to strengthen businessassurance if you want to create the right conditions
for improved business results and long-term success.

•Embrace change and foster a culture of continuous
improvement. Prioritize continuous improvement
to drive incremental change for sustainable success.

8

World Quality Report I 2023-24

• QE LIFECYCLE AUTOMATION• DIGITAL CORE RELIABILITY•

Continue to expand on what works – don’t adopt new
automation technologies or platforms just because they
are new or trendy.

•

Use newer approaches like test isolation, contract testing
etc., to drive more segmentation and higher automatedtest execution.

•

Automation can be prioritized against legacy systems– focus on automating the test scripts and technologies
which yield a larger and faster return on investment first.

•

Utilize service virtualization to overcome challenges
around multiple applications, environments, anddata dependencies.

•

Upskill teams to use AI and to evaluate AI systems– Develop training plans and career pathways so thatyour organization knows how to leverage and assess
these systems.

•

QA teams of the future will need to have a mix of skills– strong testing skills need to be augmented with somedomain and platform skills to drive efficiencies.

•

Set up an Automation Marketplace platform – Let
your automation engineers own the development andmaintenance of the scripts while other teams integratethem into their daily routines.

• AI, THE FUTURE OF QE•••

Organizations need to continue the investment towards
scaling the Quality Engineering process, skills, andbandwidth accordingly.
Implementation of AI use cases in Quality Engineering will
require careful thought and analysis for prioritization
along with unambiguous KPIs measuring tangible outputs
to drive successful transformation with AI.

••

Invest in AI solutions for test prioritization and test caseselection to drive maximum value from intelligent testing.

•

Focus on an E2E testing approach to ensure seamlesscustomer experience through effective utilization of
abstraction of various architecture layers.

•

Give equal priority to the non-functional aspects of thesystem as the functional aspects as they equally influence
the end-user experience.

• QUALITY & SUSTAINABILITY•

Have a set of key KPIs or metrics that can be easily
understood across the organization. Ensure regularreviews of the KPIs, and ensure they remain effective indriving improvements.

•

Accelerate the inclusion of cloud and infrastructure
testing as part of the software development lifecycle,
to improve on resiliency, security, redundancy,and data recovery.

KPIs need to be considered not just in QE&T but throughrequirements gathering, production support, and working
with external vendors.

•Continue to consider intelligent integrations to makethe best use of commercial off-the-shelf and cloud-native
tooling – as a hybrid strategy for non-production
environment provisioning.

Sustainability needs to be part of the whole lifecycle,
included as a non-functional requirement, tested-for as arequirement, and reported on against the success (or not)of that compliance.

•

Know your green success factors and ensure they are partof your strategy, your education, your procurement, andyour hiring processes.

An iterative approach towards chosen prioritized use
cases, along with an MVP strategy is recommended as
there is going to be considerable experimentation aroundAI solutions in the near future with multiple unknowns, thatwill get uncovered during the journey.

• QUALITY ECOSYSTEM•• INTELLIGENT PRODUCT TESTING•

Adopt stringent practices to assess the impact of failure
events on cloud, and continue to give extremely high priority
to the performance and resiliency of applications on cloud.

•Invest in workforce transformation in building SRE
and Chaos Engineering capabilities within quality
engineering organizations.

910World Quality Report I 2023-24

CURRENT TRENDS INQUALITY ENGINEERING & TESTING
Business Assurance

12

Agile Quality Management

17

QE Lifecycle Automation

22

AI, the Future of QE

28

Quality Ecosystem

35

Digital Core Reliability

42Intelligent Product Testing48

Quality & Sustainability

53

11

BUSINESS ASSURANCE• ASHWINI NADIGER
Vice President, Portfolio CoE Head for Quality
Engineering & Testing, Capgemini North America• KYLE ABRAHAM
Associate Vice President, Quality Engineering &Testing, Capgemini USA“The first thing is, really listening to our
product teams that represent the business.
These product teams not only have productowners, but also our business transformation
teams that do the end-to-end
program-level introduction. Everything for
us is about our guest experience at each ofour destinations.”
FABIAN RODRIGUEZ• KAUSHIK MUKHERJEEAssociate Vice President, Quality Engineering &Vice President, Application Delivery & Strategy |Digital & Technology, Universal Destinations & Experiences,
NBC Universal

Testing, Capgemini USA• VICTORIA VOINIGESCU
Sr. Product Manager, OpenText• OREN ZIV
Product Manager, OpenText“Business results are assessed by ensuring
the reliability of the orchestration layer,
cost-effectiveness in resource allocation,
automation, and defect prevention strategies.
Some of the challenges in achieving theseobjectives are: keeping up with rapidly
evolving technologies and platforms, the scaleand complexity of the orchestration layer(as we continue to grow rapidly), and releasing
new features quickly while also ensuring
quality standards.”
VYSHALI REDDY
Senior Director, Mobile Quality Engineering (MQE),
Charter Communications

12

World Quality Report I 2023-24

Business Assurance

Today’s dynamic digital age has steered a host of new opportunities toward us. However, managing multiple streams in complex
business projects and delivering the highest quality in omnichannel customer experiences is a herculean task. A business assurance
practice is a necessity today.
We live in a world where a single social media post could cause an irreversible dent in an organization’s brand image and reputation.
And one lapse in business judgment could hamper customer satisfaction at large, and undo achievements gathered over the years.
It’s a tightrope, and businesses must maintain a fine balance.

BUSINESS ASSURANCE FOR BRAND PROTECTION, RELIABILITY,
DIGITAL EXPERIENCE, AND POSITIVE BUSINESS OUTCOMES
The last few decades have transformed the scope of testing,as well as the definition of business assurance practices.
Business assurance has now become an integral part of strategic
business functions and is considered imperative for long-term
organizational success.
Business assurance encompasses a systematic approach todetermining business risks and focusing on what really mattersthe most. It consists of a comprehensive plan, methodologies,
and approaches to ensure that business operation processes andoutcomes are aligned with business standards and objectives.

Organizations are finally recognizing business assurance asa necessity, and we think that is driving a fundamental shift
in business culture today. We see more and more businesses
adopting comprehensive business assurance practices tomitigate risks, ensure quality, promise reliability, maintaincompliance, and safeguard the organization’s reputation andfinancial stability.
Businesses are even diverging from traditional, set ways offunctioning to adapt to business assurance principles, whichis reassuring.

In a volatile business landscape that demands full control,
business assurance can help new products and services launchahead of the competition with a consistent level of quality.

A SHIFT IN MINDSET: VALUE OVER VOLUME
FIG 1

Optimization of IT lifecycle

WQR 2023- Global Results

Which of the following best describes how much use you make of Value Stream Mapping
and related techniques in optimizing the IT lifecycle?
Conducting a pilot

71%

Currently using

14%

Planning stage (pre-pilot)

12%

Tried and no longer using

2%

On the roadmap in the next 12 months1%

Not considering

0%

TOTAL n = 1750

13

One of the key trends of this year’s survey is the rapid adoptionof a business outcome-driven approach across enterprises. Manybusinesses are now shifting from mere output to a results-driven
mindset with value stream mapping (VSM).
This has brought an increased focus on knowledge thathelps businesses carve out their niche and differentiate fromcompetitors. Value stream mapping is one of the key business

FIG 2

Driving business outcomes

drivers focusing on customer-centricity, adding value to thecustomer, and enhancing customer experience.
In this year’s WQR survey, we find a sharp increase in the
adoption of VSM and related techniques in optimizing the IT
life cycle - 71%, which is up by 11 % from last year.

WQR 2023- Global Results

How do you drive business outcome in your projects?
Build delivery pods and drive large-scaletransformation

64%

Drive the behavior skill or capability to
get a measurable business result

58%

Build and sustain customer trust

49%

Build capabilities aligned with customer needs

46%

Enable responsive Business Governance

45%

Drive value over volume

TOTAL n = 700

Organizations are prioritizing value over volume, indicatingthat they are harnessing non-linear growth. VSM helps deliver
business outcomes through a systematic approach. It defines,
visualizes, and analyzes processes, and optimizes them by
eliminating inefficiencies. It aligns all operational activities withcustomer needs and strategic goals.
This preference for value has also been key for over 64% oforganizations actively undertaking product-oriented delivery
models. This is driving large-scale transformations and has
enabled 46% of organizations to build specialized capabilitiesaligned with customer needs.

14

World Quality Report I 2023-24

38%

Base: Those Who Selected "Chief Information Oﬃcer AND Chief Marketing Oﬃcer OR
Chief Digital Oﬃcer AND Chief Technology Oﬃcer/ Product Head"

Importantly, another positive step in this direction: there’s
more acknowledgment that business assurance is not a onestep effort, but a continuous journey to ensure long-term
organizational success.
The fact that 58% of organizations are experiencing a shift inmindset and behaviors to get measurable business results, and
49% are building and sustaining customer trust by going beyond
conventional testing KPIs, affirms this.

Business Assurance

FIG 3

QE involvement in Business Assurance testing

WQR 2023- Global Results

What is the involvement of QE in performing Business Assurance testing?
Business Operations-driven: UAT done by Operations
team as proxy for Business

34%

54%

11%

Business & Testing Collaboration: Balanced UAT done
collaboratively by Business, BAs and Testers

30%

56%

14%

Testing-driven: UAT done by Testers & IT BAs with limited
Business participation

29%

54%

17%

Business-driven: UAT done primarily by Business who have astrong understanding of the applications

28%

53%

19%

Owner

Involved

Not involved

Question sample size = 314

Base: Those Who Selected "QA Testing/ Testing Manager AND
Quality Engineers". Don't know excluded from base.

IS THE CULTURE SHIFT AFFECTING
THE ROLE OF A QE FUNCTION?
Interestingly, along with a change in priorities, the culture shift
is also affecting the role of quality engineering (QE) and its
function within the business.
We found that 56% of organizations are now successfully
fostering collaboration between their business and testingteams to achieve a balanced approach to user acceptance testing(UAT). This shows that both business stakeholders and testers
are working together during the UAT process to drive value.
These findings also indicate how QE has evolved and integrated
with business to successfully deliver business outcomes bymeeting business standards and objectives and raising overall
quality standards.

TAKING A RELENTLESS ROAD TO CONTINUOUS IMPROVEMENT
It is encouraging to see that organizations are embracing
the culture of QE and continuous improvement to remain
effective, relevant, and aligned with the changing needs of thebusiness environment.
Having said that, driving this improvement is not only about
making incremental changes but it’s also a strategic approach.
Interestingly, this year’s survey shows that 56% of theorganizations are aligning strategic business goals, products,

and value streams, and 53% of organizations are leveraging factbased, measurable methods and monitoring.
Overall, 49% of the organizations are measuring the numerous
benefits of continuous improvement, including increased
efficiency, quality, innovation, and customer satisfaction.
Additionally, 38% of organizations are prioritizing continuous
improvement by changing their business mindset and driving
incremental change to be better positioned to adapt, thrive,and achieve their long-term goals.

15

FIG 4

Top three considerations when driving continuous improvement

WQR 2023- Global Results

What are the top three most important considerations when driving
continuous improvement?
Ability to align between strategic business goals, products,
value streams or portfolios

56%

Leverage fact-based, measurable methods and monitoring

53%

Ability to measure eﬃciency of process

49%

Business mindset and thinking

38%

Drive incremental change

38%

Continuous transformation and innovation

36%

Sustainable and repeatable process improvement

31%

TOTAL n = 1750

RECOMMENDATIONS AND SUMMARY
1. Leverage the testing function to deliver businessoutcomes. This will enable a fundamental shift in theorganization’s culture and mindset to ensure that business
operation processes are aligned with business standards
and objectives.
2. Use value stream mapping to strengthen businessassurance. Align processes with customer objectives tomitigate risks, protect reputation, and further drive quality,
reliability, customer satisfaction, and financial stability.
Collectively, these factors create the conditions for improved
business results and long-term success.

16

World Quality Report I 2023-24

3. Embrace change and foster a culture of continuous
improvement. Prioritize continuous improvement to driveincremental change for sustainable success. Set goals,
evaluate solutions, and iterate for ongoing growth.
The scope of business assurance is massive and there is noone-size-fits-all solution when it comes to its applicationin enterprise processes. However, with mindful strategies
that are hyper-focused on business outcomes, we can create
thriving ecosystems that are ready to take on the future.

AGILE QUALITY MANAGEMENT• JEFF SPEVACEKHead of Quality Engineering & Testing, Financial
Services, Capgemini North America• KAVITHA VELCHAL
Portfolio Manager, Testing Deal Architect for
Global Sales Support Team, Quality Engineering &Testing, Capgemini Financial Services• RAMA YAMMANURU
Principal, Quality Engineering & Testing,
Capgemini USA• PARINITA PATANKAR
Vice President, Quality Engineering & Testing,
Sogeti India“Quality Engineering needs to take ona “Quality as Code” mentality. Instillinga higher degree of development capabilities
into the team, so that they can moreclosely integrate with the overall
development lifecycle”.
IFTEKHAR IVAAN
Head of Quality Engineering of a Major Global Bank

• BARRY WESTON
Delivery Head of Quality Engineering & Testing,
Sogeti UK• HITESH PATEL
Region Practice Manager, Quality Engineering &Testing, Sogeti USA• AYAL COHEN
Director, Product Management Functional Testing,
OpenText• MOTTI FINE

“Quality requires more attention than ever
because you are now on a massive growthpath. People are moving towards moredigitalization and you must ensure what you
release or what you give to customers is of
the highest quality”.

Product Manager, OpenText• OREN ZIV
Product Manager, OpenText

SHIVAPRAKASH MOOLYAManager, Quality Engineering, BUPA (UK)17

The concept of agile organizations has been a part of boardroom conversation over the past decade. Businesses continue topursue the goal of remaining relevant in volatile, uncertain, complex, and ambivalent environments. Over time, this concept has
evolved into much more than a mere development methodology. It has become a way of thinking and a mindset that emphasizes
continuous
improvement, adaptability, and customer-centricity. This evolution is clearly represented in the trends shaping agile
Introduction
quality management (AQM) practices today.
Last year, we noticed a paradigm shift in AQM through the emphasis placed on embracing agile principles rather than merely
implementing agile methodologies. This year, we try to uncover how deep this understanding of agility runs.

ARE BUSINESSES ‘AGILE’ READY?
FIG 5

Setup of QE organization to support agile projects

WQR 2023- Global Results

How is your Quality Engineering organization set up to support agile projects today?

70%

Traditional Testing
Center of Excellence

TOTAL n = 300

28%

Product Aligned QE(combination of
Capability Units with
POD aligned teams)

2%

Fully Dev
Aligned

No formal
structure / ad hoc

Base: Those Who Selected "IT Directors"

The adoption of agile principles across the industry is beyond nascent stages and most organizations today are well into theiragile and DevOps journey. However, the approach quality engineering organizations have taken varies; some have taken a ground-up
approach, focusing first on workforce transformation, while others have taken a top-down approach, first choosing to re-structure
their quality engineering (QE) organization from a process standpoint.
Organizations are now prioritizing development skills over traditional testing skills as the most critical skills for quality engineers.
Development-focused skills like C#/Java/SQL/Python and CI/CD are all ranked in the top 5, while traditional testing skills like
automation and performance tooling ranked at the bottom of the results.
Many organizations are undergoing some form of workforce transformation, evolving their traditional quality engineers intofull-stack quality engineers. When looking at QE organizations through the lenses of organization setup and critical skills, it can beinterpreted that most organizations are taking a grassroots or reactionary approach – choosing to upskill their team first and thenfocusing on change at an overall organization level.

18

World Quality Report I 2023-24

Agile Quality Management Office

WE SEE THE QUALITY ENGINEER’S ECOSYSTEM EVOLVING
FIG 6

Top ﬁve most critical skills for Quality Engineering Associates

WQR 2023- Global Results

Please select the ﬁve (5) most critical skills for your Quality Engineering Associates.
C#/Java/SQL/Python

42%

CI/CD, Orchestration

39%

BDD / TDD

38%

ETL / Big Data e.g. Hadoop fundamentals

35%

Open source testing solutions

33%

AI / ML

32%

Cloud fundamentals (OCP, AWS, GCP or Azure)

29%

Automation Test tooling
Performance Test tooling

TOTAL n = 580

28%
24%

Base: Those Who Selected "VP Applications/ Product Owner
AND QA Testing/ Testing Manager AND Quality Engineers"

There is a clear trend that suggests a parallel evolution in the skills makeup of quality engineers. Initially automation-centric, therole has shifted towards a more development-oriented approach.
A quality engineer today is expected to support in-sprint development activities, with respondents citing development skills suchas SQL/Python, CICD/Orchestration, and C#/Java as the most critical skills for quality engineers today.
The global survey of agile-oriented organizations revealed that 42% of the respondents felt scripting language like C#/Java/SQL/
Python is the most critical skill required by today’s quality engineer, 39% voted for CI/CD and orchestration closely followed by
BDD/TDD at 38%; compared to just 28% for automation tooling and 24% for performance tooling. While skills like ETL, opensource testing solutions, Artificial Intelligence (AI)/Machine Learning (ML), cloud, and tooling were considered important; theabove-mentioned three stood ahead of the rest.
We feel this is in alignment with the industry’s continued focus on speed to market; as quality engineers continue to adopt moreof a developer mindset, their ability to introduce automation earlier in the lifecycle increases as is their ability to troubleshoot andeven remediate defects on a limited basis.

19

HOWEVER, WITH EACH INDUSTRY TREND,
A NEW SET OF CHALLENGES EMERGE
Major challenges faced by quality engineers in Agile projects

FIG 7

WQR 2023- All Countries Results

64%

68%

77%
64%

36%

50%

66%
55%

57%
50%

71%
73%

68%

70%

50%

57%

59%

29%

32%

36%

TOTAL
n = 315

Aus + NZ
n = 14

Canadan = 14China
n = 11

Francen = 29Netherlands
n = 19Germanyn = 22UK
n = 22

18%

36%

62%
59%

79%
82%
64%
68%
64%

60%
52%

50%

57%

63%

71%

73%
68%

62%

60%

64%
64%
68%
54%
41%

63%
64%
59%

77%

79%

86%

What are the major challenges faced by quality engineers in your Agile project?

USAn = 56Lack of time totest or automate

Lack of coding
skillsLack of supporting
testing processes

Limited career
path for testers

Lack ofknowledge ofAgile techniques

Braziln = 14

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers" At S1Each region appears to have its own set of challenges whichaligns with how the QE organizations are set up today. The
US and Canada face similar primary challenges, with >65%
of respondents confirming a general lack of supporting test
processes; although 86% in Canada confirming a lack of coding
skills, and 70% of US respondents confirming a limited career
path. This is indicative that organizations are attempting tooperate in traditional Testing Center of Excellence (TCoE) models
even today. To stay competitive in the market, organizations arepushing to get new features and enhancements out faster, whichputs a greater focus on speed to market instead of a long-termview on maintainability.

On the contrary, other regions like the UK reported a lack ofknowledge of agile techniques (77%) as the most common
challenge. This could indicate the speed at which theseorganizations moved into a product-aligned QE model whilestill trying to utilize their traditional testers.
Global survey results indicate all the above-mentioned challenges
need to be addressed as more and more organizations “embrace”
agility in the true sense.

SHIFTING TO A PRODUCT-BASED QUALITY
ENGINEERING SETUP MIGHT BE KEY TO OPTIMAL AGILE QUALITYMANAGEMENT PRACTICES
Many quality engineering organizations predicted the evolution and took a more proactive stance by shifting from a traditional‘Testing Center of Excellence’ approach to a product-based quality engineering setup; however, most organizations are still in theprocess of truly integrating their quality engineering teams into an agile-centric model. As shown in the diagram above, only 4% of
respondents report that more than 50% of their quality engineering teams are operating in a more agile-centric pod-based model.

20

World Quality Report I 2023-24

Agile Quality Management Office

The product-aligned QE organization setup enables organizations to diverge from centralized automation, test data, and performance
testing teams to nimbler ‘capability units’, relieving the overarching burden on quality engineers. Apart from this, businesses might
be able to maximize outputs through:1. Customer-centric design thinking, which emphasizes rapid feedback and improves a business’s agility to inculcate an innovation
mindset, aligned with customer needs.
2. Incorporating Artificial Intelligence and Machine Learning in test automation in a business tech ecosystem can also directly
contribute to higher-quality outcomes by predicting issues, analyzing data, and enhancing testing efficiencies.
3. Embracing agile practices across the organization to promote better alignment and enabling seamless collaborationto achieve goals.

FIG 8

Percentage of QE associates who are Agile POD aligned

WQR 2023- Global Results

What percentage of your Quality Engineering associates are Agile POD aligned?
0% None1-25%

59%25-50%
50-75%
More than 75%

TOTAL n = 1005

37%
4%

Average %TOTAL n = 1005

29%

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/ Testing Manager
AND Quality Engineers AND Chief Information Oﬃcer”

CONSIDERING THE CURRENT LANDSCAPE, WE HAVE A FEW RECOMMENDATIONS FORENSURING AGILE ADHERENCE IN BUSINESS OPERATIONS.
Invest in robust training programs: Organizations must
adopt a build-or-buy mentality – either looking for new talent
in the market or upskilling their current workforce to meetthe evolving demands of the business. Replacing the existing
workforce will yield faster results but comes at the drawbackof increased cost and reduced sustainability. Developing arobust training program reduces overall costs while setting
your organization up to continue to adapt to the industry’s
ever-changing technology and skillset requirements.
Integrate with DevOps/DevSecOps: Forging a seamlessintegration with DevOps/DevSecOps practices throughclose collaborations can ensure smooth incorporation ofNow, more than ever, the role of quality engineering iselevating – quality engineering teams are asked to take on amore development-focused persona and build utilities for the
rest of the organization to leverage rather than focusing solely
on safeguarding quality. The evolution of agile practices, thequality engineering assets into CI/CD pipelines by identifying
and rectifying issues promptly, enabling rapid and secure
software delivery, and enhancing overall product quality andcustomer satisfaction.
Modernize your QE organization: Establish lightweightcapability units for test process and governance, test
automation, performance engineering, test data management,
and test environment management to establish enterpriseframeworks and tooling platforms. This will enable a capability
for all approaches, empowering individual agile aligned teams
to be more self-sufficient.

integration of AI and ML, and the synergy between DevOps andagile are transforming quality engineering in infinitely futuristic
ways. The question is, when will organizations embrace thesechanges en-masse, adopt proactive strategies, and make thema norm? Next year’s report will probably unveil the answer.
21

QE LIFECYCLE AUTOMATION• ANDREW FULLEN, Head of Innovation andTechnology, Sogeti UK

Humankind has always been in the pursuit of simplifying work,and automation is probably one of the best fruits of this pursuit.

• JEFF SPEVACEK, Head of Quality Engineering &Testing, Financial Services, Capgemini North America

Today, as we notice accelerated paces of delivery that needlightning-fast development speeds and adherence to quality
benchmarks - QE Automation holds the promise to deliver
seamless results.

• CHAITALI LAMBAT, AI & Automation Lead, Quality
Engineering & Testing, Sogeti India• AYAL COHEN, Director, Product Management
Functional Testing, OpenText• JULIAN FISH, Director, Product Management,
OpenText• VICKY GIAVELLI, Director, ProductManagement, Performance Engineering and
Virtualization, OpenText

This year, we decided to probe its potential and extend our
focus beyond functional automation and encompass the entire
quality engineering (QE) lifecycle. We attempted to analyze
the challenges in non-traditional automation areas to mirror
those in conventional ones and to assess the far-reaching
benefits of automation.
And even though we are charting unfamiliar territories like
Artificial Intelligence (AI) and Machine Learning (ML) – we would
echo what the English author Douglas Adams once said, “Don’t
panic,” because we believe that with automation, we learn, we
adapt, we benefit, and we will deliver.
Let us dive into the findings of this year’s World Quality Report(WQR) survey and its revelations for the near future.

WHAT IS EVERYONE EXPECTING OUT OF AUTOMATION?
Top beneﬁts needed from automation

FIG 9

WQR 2023- Global Results

What are the top three beneﬁts you need automation to deliver
to your organization?
54%

52%

51%

50%

50%
42%

Risk
reduction

Improve test
eﬃciency

TOTAL n = 1750

22

World Quality Report I 2023-24

Reduce live
defects

Reducetesting eﬀort

Improve customer
experience

Reduce time
to market

QE Lifecycle Automation

It is interesting to note that the top benefits from automation
this year include risk reduction (54% of respondents), improved
test efficiency (52% of respondents), and reduced live defects(51% of respondents). We were surprised to learn that reduced
time to market held the least priority at just 42% of respondents– while it had topped the list last year. We think the reason behind
this could be that organizations are already moving as fast as
they can and seeing less benefit in cutting time further. It is alsointeresting to note that in a year where many organizations are
freezing or reducing their IT spending, improved test efficiency
and reduced testing effort came up as key priorities. This shift
is a clear indication that organizations are starting to achieve
their speed-to-market goals, and now more focused on reducing
technical debt and overall spending on IT instead.
So, where are organizations placing their bets now?

ALL EYES ON AI
FIG 10Top three focus areas for automation

WQR 2023- Global Results

Which are the top three focus areas for automation in yourorganization over the 12 months?
Testing AI

55%

Mobile40%

Security

36%

Audit and Compliance30%

Performance

27%

Accessibility

26%

Environment Provisioning

25%

Data creation

23%

Requirements

20%

Functional

18%

TOTAL n = 1435

Base: Those Who Selected "Chief Information Oﬃcer AND VP Applications/ Product Owner
AND IT Directiors AND Chief Marketing Oﬃcer OR Chief Digital Oﬃcer AND Chief Technology Oﬃcer/
Product Head AND VP OR Director Of Research AND Development/ Engineering"23

We were expecting AI to steal the limelight this year since itis the buzzword in the boardroom these days. We think that
AI is the new tide that needs to be ridden with caution, whichmeans quality engineering and testing (QE&T) teams need tounderstand how AI-based tools work for them, how they can dotheir jobs better, and bring better outcomes for their customers.

areas. Perhaps, this is an area where we can expect to see AI
tools becoming a key part of automation toolkits.

More than 50% of the respondents were eager to see testing
AI with automation, which is well ahead of the other top focus
areas like security and mobile.

Closely following this, were security and mobile. With 36% of
the respondents voting for security to remain a key focus area,
we predict that risk reduction will continue to be a key benefit
of automation. In a globally digitized economy, we are seeing
end-user interactions dominating mobile devices, and mobilefirst platforms taking precedence, as they will continue to do
for the foreseeable future.

Interestingly, we found that functional (18% of respondents)
and requirements (20% of respondents) automation were of
the least focus for organizations, presumably because of thechallenges in automating and regular updating involved in these

Another noteworthy trend was how respondents chose
non-functional (performance and accessibility) overfunctional automation features, indicating more focus on
customer experience.

BUT EVEN AS AUTOMATION CONTINUES TO RUN DEEP IN
ORGANIZATIONAL ECOSYSTEMS, THERE ARE DILEMMAS TO DECODE
FIG 11Main roadblocks to automation

WQR 2023- Global Results

When it comes to doing more with automation, what are the two mainthings holding your teams back?
Legacy systems and decisions

34%

Lack of skills

30%

Collaboration issues

18%

Lack of tools

18%

Complexity

16%

Lack of a strategy

15%

Lack of data

13%

Requirements

13%

Lack of time

12%

Lack of requirements

12%

Lack of environments

11%

Lack of money

TOTAL n = 615

24World Quality Report I 2023-24

6%

Base: Those Who Selected "IT Directors AND QA Testing/ Testing Manager AND Quality Engineers"

QE Lifecycle Automation

The WQR survey suggested that legacy systems and decisions (34% of respondents), followed by the lack of skills (30% of respondents)
as the two things holding back automation. There is a continued need for developing technical skills to deal with both existing andupcoming technologies. But it was heartening to see that only 6% of the respondents claimed ‘lack of money’ as an issue, making
it evident that organizations are ready to invest in automation when the strategy is right and return on investment is attractive.

WE ALSO SEE AN URGENCY TO OPTIMIZE TEST SUITESFIG 12Integration of automation test suites into pipeline

WQR 2023- Global Results

What percentage of your automation test suites (smoke + regression) are integrated into yourOrchestration / lifecycle / delivery pipeline?

0%0%

1 - 25%29%

25 - 50%68%

50 - 75%3%More than 75%

Q10.
Average %TOTAL n = 315

TOTAL n = 315

31%

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

Overall, the survey revealed that only 3% of respondents’ organizations have more than half of their suites integrated which may be
due to diminishing returns or lack of QE access to orchestration instances. This comes as a surprise since most of the organizations
now have some automated test suites integrated into their pipelines for smoke and regression testing.

“This shift-left mentality has also given testers the opportunity to shift their
testing farther left and lean more on automation as an early to potentially only
solution. The idea of in-sprint automation vs. regression-only automation has
become a reality”.
CHAD FETTER
Director of Quality Strategy and Delivery, Medica

25

FIG 13

Challenges preventing integration into DevOps / DevSecOps pipeline

WQR 2023- Global Results

What are the most impactful challenges preventing your Quality Engineering organization from
integrating into the DevOps/DevSecOps pipeline?
Quality Engineering team doesn't
have access to enterprise CI instance58%

We are still deﬁning our DevOps/
DevSecOps roadmap

51%

Test Data availability

44%

Lack of skillset

37%

Test environment availability

36%

Limited Automation

36%

Management support

36%

TOTAL n = 750

58% of the respondents stated that quality engineering teams
still do not have access to an enterprise CI for instance. This may
be due to a lack of knowledge of legacy systems, stability issuesof test scripts and high maintenance. Especially, since we knowthat increasing QE integrations into DevOps/DevSecOps lifecycle

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Oﬃcer/ Product Head"requires greater collaboration between the development,
quality engineering, and DevOps/DevSecOps teams. What
was worth noting was that the more senior respondents (thosefurthest from the actual automation) reported higher levels of
integration than those solely responsible for the work.

“If you look at productivity, hyper-automation, zero-touch, and full end-to-end
lifecycle automation, people still struggle to get there. I look at AI-augmented
tool ecosystems that really help us automate and make your testing life cycle
faster and better from a productivity side”.
GANESH PRACTOOR
Director - Quality Engineering Group at one of the world’s leading financial organizations

INABILITY TO ADOPT ADVANCED AUTOMATION SOLUTIONS
Despite the promises of low code/no code and automation-based frameworks, only 19% of respondents are currently using suchtools, with nearly 70% of respondents at the stage of testing a pilot. While low-code type tools can allow an easier entry point
into automation, there is still a long way to go before they truly challenge more traditional and feature-rich automation options,especially in terms of productivity and efficiency.
26

World Quality Report I 2023-24

QE Lifecycle Automation

FIG 14Use of advance automation solutions

WQR 2023- Global Results

Are you using or plan to use advance automation solutions like Low Code/No Code or AI-based automation frameworks?
69%

19%
10%
2%
Conducting
a pilot

Currently
using

Planning
stage(pre-pilot)

On theroadmap in thenext 12 months

TOTAL n = 315

Notconsidering

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

RECOMMENDATIONS AND SUMMARY
While delivering good systems with quality is a priority, italso must be done in a timely manner. Quality engineeringand testing teams must adapt to working with AI and testing
AI systems. Even as lawmakers address the challenges of AI
systems, there are already legal requirements to prove AI
systems have been tested and are fair.

•

Automation can be prioritized against legacy systems– focus on automating the test scripts and technologies
which yield a larger and faster return on investment first.
Once you have achieved the desired coverage, capability
and speed to market goals; then move on to the morechallenging technologies.

Considering legacy systems and decisions still have a major
impact on automation teams everywhere. It is importantto equip teams with the right skills to understand legacy
systems and technologies, while also adapting and adopting
AI systems – which by no means will be easy.

•

Upskill teams to use AI and to test AI systems – AI-based
systems are here to stay, whether used for testing or
the focus of testing. Develop training plans and career
pathways so that your organization knows how toleverage and test these systems.

Our research leads us to the following recommendations toincrease the value from your quality automation initiatives:•

Set up an Automation Marketplace platform – enables
an “automation as an asset” approach, empoweringdevelopment, infra and business communities to leverage
automation test scripts which have been created. Your
automation engineers own the development andmaintenance of the scripts while other teams integratethem into their daily routine.

•

Continue to expand on what works – don’t adopt new
automation technologies or platforms just because
they are new or trendy. There is elegance in simplicity;
continue using the frameworks and solutions that areproviding value. When new solutions arise, evaluate themfirst before pushing for full adoption.

27

AI, THE FUTURE OF QE• JEBA ABRAHAM
Practice Vice President, Quality Engineering &Testing, Sogeti USA• CHAITALI LAMBATAI & Automation Lead, Quality Engineering &Testing, Sogeti India“To be very simplistic, my experience andmy view is that AI can do wonders for you,but not before you make yourself mature.
You cannot jump in there prematurely because
AI wants to take you from the mature levelto the next level of excellence, not from theground level. There are several use cases,
real good use cases of AI implementation,
but none of them will be helpful if you are
not good at your basics”.

• SAI GRANDHI
Associate Vice President, Quality Engineering &Testing, Sogeti USA

RAJESH NATARAJAN
Senior Director, Quality Engineering at Hiscox• UDI WEINBERG
Director, Product Management, OpenText• AYAL COHEN
Director, Product Management Functional
Testing, OpenText“We will need skill augmentation, we willneed people who can operate bots, who know
about the technology... The human workforce
will not become obsolete with AI, instead, theexpected skillset for them is going to change”.
AMITA KARADKHEDKAR
Senior Vice President of Quality Engineering of a Major Global Bank

28

World Quality Report I 2023-24

AI & the future of QE

Artificial Intelligence (AI) is the most exciting technological advancement today. While the concept of AI is not exactly new,Generative AI is a game changer and has brought in an exciting revolution in how AI is applied across industries and organizations.
So much so that one of the biggest pioneers of the technology industry, Bill Gates, in his blog ’The Age of AI has begun’,**he says, “The development of AI is as fundamental as the creation of the microprocessor, the personal computer, the Internet,
and the mobile phone. It will change the way people work, learn, travel, get health care, and communicate with each other. Industries
across the globe will reorient around it. Businesses will distinguish themselves by how well they use it.” So, what does this mean
for the future of quality engineering (QE)? Let’s dive right in.

HOW IS THE SCENE UNRAVELING?
We have been experimenting with AI to improve quality outcomes for a while now. However, with the advent of Generative AI, we
see for the first time that higher productivity is the primary outcome organizations are using AI for, and we are not really surprised.

FIG 15

Quality outcomes driven by AI

WQR 2023- Global Results

What are the quality outcomes your organization is utilizing AI to drive?
Higher Productivity

65%

More Velocity

53%

Improved Customer Experience
Better Reliability
Lesser Defects

TOTAL n = 750

41%
33%
29%

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

It’s heartening to see data that shows us that 77% of organizations today consistently invest in AI and utilize it to optimize QA
processes. We believe that this is directly linked to the fact that the trust in training data for AI solutions is very high, which in turnreflects the robust infrastructure and processes that organizations have developed over the years to collect continuous telemetry
from all parts of the quality engineering process.

Reference:** https://www.gatesnotes.com/The-Age-of-AI-Has-Begun?WT.mc_id=20230326130000_Artificial-Intelligence_BG-LI_&WT.tsrc=BGLI

29

FIG 16

Attitude Statements

WQR 2023- Global Results

Please indicate how much you agree or disagree with the below statements with regards to the use of AI
in your organization.
We consistently invest in AI solutions toimprove quality processes

77%

We actively utilize AI to optimize our
testing processes

74%

I have conﬁdence that the data I use totrain my AI algorithms/platform is correct
and accurate

74%

My organization is willing to act on
intelligence provided by the AI platform

71%

Business owners trust the intelligence
provided by the AI platform

68%

I have identiﬁed processes/applications
that can beneﬁt from AI

TOTAL n = 1750

When we deep dive into what
quality engineering use cases
those AI investments are
driving, we found that the
majority of organizations areusing AI towards building andimproving test scope alongwith improving performanceengineering, as well as the test
ecosystem overall.

30

World Quality Report I 2023-24

64%

AI & the future of QE

FIG 17Frequency of the use of AI in use cases

WQR 2023- Global Results

How frequently, if at all, has your organization used AI for the following use cases?
Autonomous testing (scope discovery)

55%

32%

13%

Performance testing or Test prioritization

53%

36%

11%

Performance testing/engineering

51%

37%

12%

Test environment provision

50%

38%

11%

Contextual test data provisioning/generation

47%

38%

15%

Cross analysis of test results
and defect prediction

44%

37%

19%

Self-healing & test execution

40%

46%

14%

Event correlation and root cause
analysis of issues

40%

52%

9%

Automated scope selection(impact analysis)

37%

49%

14%

Test case design (automated & manual)

37%

52%

12%

Used a lot

Used on some occasions

Have not used

Total sample size = 1750

TRENDS THAT CAUGHT US BY SURPRISE
That said, there are some big changes in trends we needto delve deeper into. Reliability and fewer defects are atthe very end of the list, which came as a big surprise for us.
Improving the reliability of tests (be it rooting out stale tests
or self-healing automated tests) and reducing the numberof defects (defect analysis, defect prediction, risky codeanalysis, etc.) have been focus areas for the application of
AI to improve quality engineering for a while now. So, while
it wasn’t completely a surprise to see productivity be the top
outcome organizations were aiming for, we didn’t expect tosee these two at the very bottom of the list. What’s possibly
causing this?

WE THINK THERE COULD BE TWO FACTORS AT PLAY HERE:
1. With the availability of multiple large language models (LLM), the ability to achieve much higher productivity is possible now
without too much effort. Higher productivity naturally leads to higher speed. Higher productivity and speed in the quality engineering
process makes it possible to ship code more frequently with a high customer experience. Generative AI has drastically upended thecost vs. benefit equation for these use cases, effectively making them low hanging fruits, ripe for harvest.

31

2. With Agile and DevOps practices being adopted across organizations, there is more continuous testing with multiple philosophies
like “fail fast” and “perpetual beta” increasing the tolerance for defects being found, as long as they can be fixed quickly and efficiently.
That said, it makes us wonder what (if any) additional changes we would be seeing in the survey next year, with the rapid evolutionof AI technologies. This is definitely something to watch out for.

SOME OLD ROADBLOCKS, SOME NEW
The survey data also suggests that there are multiple challenges for organizations to overcome while driving quality outcomesthrough AI. The roadblocks range from concerns on privacy and compliance to a lack of resilience and dealing with hallucinations andbias. We feel that the roadblocks to large scale AI adoption in QE haven’t changed too much over the years. Although organizations
have learnt to deal with these roadblocks better, they haven’t all been completely overcome yet. With 77% of organizations investingin AI solutions, we feel that those solutions are coming very soon in the near future.

FIG 18Biggest roadblocks of Generative AI to improve quality

WQR 2023- Global Results

What are the biggest roadblocks/ concerns on utilizing Generative AI to improve quality?
Lack of privacy

35%

Sustainability concerns

34%

Lack of resilience33%

We don't recognise value in adopting AI

31%

Integrations with existing tools

30%

Regulations and compliance29%

Scalability (hidden cost of use cases)

27%

Lack of accuracy (hallucinations)24%

Lack of context

TOTAL n = 750

17%

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

But, interestingly, 31% of organizations still don’t see the value in adopting AI yet. For this group, we assume that the costs/
risks outweigh the benefits and that is a bigger roadblock to overcome. It will probably require sustained investment/effort tobreak past the barrier.

32

World Quality Report I 2023-24

AI & the future of QE

WHAT WILL IT TAKE TO DRIVE SUCCESS IN THIS NEW AGE?
With the amount of investment in AI, there is bound to be a large influx of AI projects across organizations. Successfully testingthese solutions and deploying them while ensuring high quality will be critical for driving success in the future.

FIG 19Plans for testing of AI-based solutions

WQR 2023- Global Results

How is your organization planning for testing of AI-based solutions?
Focus on requirements to explainability,transparency, robustness, privacy

75%

More focus on data science skills as thesystem learns from its mistakes as it goes
and it is virtually impossible to reproduce a bug

54%

Test Data generation to cater to thelarge volumes of test datathat would be required

48%

Updated test approach (processes,
methodology) as the system issophisticated and hard to "break"
Additional skillsets in the QA team

35%

23%

We don't have any plans to test
AI-based solutions

TOTAL n = 750

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Oﬃcer/ Product Head"Our respondents overwhelmingly plan to focus on requirements related to transparency, robustness, and privacy of these upcoming
AI solutions along with a continued investment in data science skills within their quality teams.
The other thing we see from the survey is that there is no clear winner when it comes to where organizations want to use AI to
improve quality outcomes. Of course, AI can bring benefits to every tenet of the quality engineering lifecycle. We, however, feelthat the approach needs to be treated with caution given the risk of things quickly going too far down the wrong path when it
comes to AI solutions, unless they are robust enough to begin with.

33

FIG 20

Attitude Statements

WQR 2023- Global Results

Please indicate how much you agree or disagree with the following statements regarding your plans for
artiﬁcial intelligence and testing.
AI systems are going to be used to help
us optimize our test scope and increase velocity

79%

We will use AI to speed up our test design78%

AI will be used to test data76%

We will use AI for more data analysis ofuser feedback and production telemetry
for maximum business impact

76%

We will leverage AI technologies to build
self-healing test automation74%

We intend to use AI for test results
analysis and defect prediction

73%

TOTAL n = 750

Base: Those Who Selected "VP Applications/ Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

RECOMMENDATIONS AND SUMMARY•

Higher productivity driven by AI will rapidly increase the development speed and organizations need to continue theinvestment towards scaling the quality engineering process, skills, and bandwidth accordingly.

•

Implementation of AI use cases in quality engineering will require careful thought and analysis for prioritization alongwith unambiguous KPIs measuring tangible outputs to drive successful transformation with AI.

•

An iterative approach towards chosen/prioritized use cases along with an MVP strategy is recommended as there isgoing to be considerable experimentation around AI solutions in the near future with multiple unknowns that will get
uncovered during the journey.

34

World Quality Report I 2023-24

QUALITY ECOSYSTEM• ASHWINI NADIGER
Vice President, Portfolio COE Head for Quality
Engineering & Testing, Capgemini North America• ANISH BEHANAN,
Head of Quality Engineering & Testing,
Capgemini UK• PADMAJA ALAPATI,
Practice Lead, Quality Engineering & Testing,
Sogeti India• DEEPA TALWARIA
Director, QEP Offer Lead, Quality Engineering &Testing, Capgemini USA“Rapid technological advancements require
QE teams to keep pace with testing new
platforms, devices, and technologies. QE
teams will need to invest in upskilling,
and adopting advanced testing tools andmethodologies while maintaining flexibility
to adapt to the evolving tech landscape”.
VYSHALI REDDY
Senior Director, Mobile Quality Engineering (MQE),
Charter Communications• KYLE ABRAHAM
Associate Vice President, Quality Engineering& Testing, Capgemini USA• LIOR URBANI,
Functional Testing Product, Manager, OpenText• AYAL COHEN,
Director, Product Management Functional
Testing, OpenText“It’s about testing to make sure that it
works. Optimizing the customer journey
and making sure that customers have an
overall good experience and not just an
accurate experience is important. So, whilst
its obviously important that our customers
get the right offer, we also want thecustomer experience to be as easy tonavigate as possible”.
MARK PAIN
Head of Product Delivery, Volkswagen Financial Services, UK

35

In the last few chapters, we pieced together various elements of the quality engineering (QE) landscape. However, to understandthe ‘quality ecosystem’ in depth let us take a step back – or two and look at the bigger picture. The quality ecosystem comprises all
the capabilities that enable effective testing of applications. This includes, but is not limited to, functions such as test environment
management, test data management, etc.
Even as organizations continue to test and expand the quality ecosystem capabilities to hyper-focus on customer needs, we look at
how emerging industry trends such as cloud testing and mobile testing impact the overall ecosystem. With site reliability engineering(SRE) increasingly being recognized as a necessary capability, and the eventual movement of workloads to cloud, we dive deeper
to further understand the role of cloud testing and SRE in building exceptionally reliable and resilient software for the future.

WITNESSING THE METAMORPHOSIS OF CLOUD TESTINGFIG 21

Inclusion of cloud and infrastructure testing

WQR 2023- Global Results

Does your organization include cloud and infrastructure testing as partof the development life cycle?

Yes, for all the projects where applications are hosted on
cloud, this is mandatory

82%

No, this is not followed as a standard practice18%

There is no awareness of the requirement of cloud testingand its beneﬁts

0%

There is no requirement to do this as a separate function

0%

TOTAL n = 315

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

Last year, our World Quality Report (WQR) reported that 57% of organizations considered cloud testing as a part of the softwaredevelopment lifecycle but not as a mandatory function. This year, however, 82% of the survey respondents highlighted cloud testing
as mandatory for applications on cloud. This highlights a positive and decisive shift in the testing strategy that organizations aretaking on cloud and infrastructure testing. It also demonstrates how important it is to test cloud-related features for functionaland non-functional aspects of applications. This change in thinking is a result of organizations realizing that movement to cloudalone does not make the system available and reliable.

36

World Quality Report I 2023-24

Quality Ecosystem

FIG 22End-to-end tooling plans for cloud testing

WQR 2023- Global Results

Which, if any, of the following end-to-end tooling plans for cloud testing do you have?

80
70

71%
63%

60
50

50%

46%

40

38%

30
20
10

1%

0

Cloud Native
tooling

Project Speciﬁc
decisionEnterprise-wide
COTS tooling

TOTAL n = 315

Open-source
tooling preferred

Integrated COTS andOpen source tooling

None ofthese

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"This year’s survey also highlighted that 71% of organizations prefer cloud native tooling (while only 50% prefer commercial off-the-shelftooling) to perform cloud testing, thus, alluding to the ease of using the former when workloads are using a specific cloud platform.

FIG 23Cloud platform strategy for provisoning non-production environments

WQR 2023- Global Results

What is your cloud platform strategy for provisioning non-production environments?
58%

Hybrid (On-Prem +Single Cloud) Environments

TOTAL n = 315

19%

12%

11%

Single Preferred
Cloud Platform

Multi - Premise (On-Prem +Multi-Cloud) Environments

Multi Cloud
Environments

Base: Those Who Selected "QA Testing/ Testing Manager/ Quality Engineers"37Interestingly, a hybrid cloud strategy seems like another upcoming trend since many organizations are using this approach fortheir non-production environments over single cloud platforms. We think the shift to a hybrid cloud strategy is from a multi-cloud
and a multi-premise model point-of-view and is driven by cost-related decisions. Nonetheless, with almost 58% of organizations
adopting a hybrid cloud (single cloud platform along with on-premises environments) strategy this year, we think this trend stands
to become the mainstream in the next few years, even though 19% of organizations still prefer single cloud platforms to provision
non-production workloads.
But as more organizations jump onto the cloud wagon, a question looms large – what is the point of an immaculate cloud-hosted
site, if it’s unreliable?

EMERGENCE OF SITE RELIABILITY AND CHAOS ENGINEERINGThe applications hosted on cloud could be impacted by the unexpected turbulence and outages of the cloud services. An hour-long
Microsoft Exchange Online outage in March 2023 and a fire that wreaked havoc for Google Cloud users in Europe are a few major
examples in recent times. Such incidents have given way to focus on site reliability and chaos engineering.

FIG 24

Embracing the transformation that SRE can bring

WQR 2023- Global Results

How are your QA teams embracing the transformation that SRE can bring indesigning performant software and scalable practices?

69%

23%

4%

4%

Increasing
automation andintegration

Prioritizing QA eﬀorts
to most impactful
areas through SLIs

Integrate QA into
engineering, development,
and operations

Refocusing tests
through SLIs and error
budgets

TOTAL n = 1040

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers AND IT Directors AND Chief Information Oﬃcer"

Site reliability engineering (SRE) applies engineering to operations, aiding automation, and integration from a quality perspective,
and 69% in this year’s survey emphasized this. While organizations seem to recognize that SRE offers transformative potential indesign, scalability, and its features offer enhanced automation and integration to boost app reliability, the lack of a skilled workforce
remains a pressing challenge to tackle.

38

World Quality Report I 2023-24

Quality Ecosystem

FIG 25

Importance of testing impact of failure events

WQR 2023- Global Results

How important is it in your organization to test the impact of failure events on applications
running on cloud especially using chaos engineering?

TOTAL n = 1040

0%

24%

71%

4%

Not at allimportantSomewhat
importantVeryimportantEssential

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers AND IT Directors AND Chief Information Oﬃcer"

Simultaneously, we see a rise in the importance of testing failure in cloud events, with 71% of respondents calling it ‘very
important.’ This brings chaos engineering into the scene. Most Quality Assurance (QA) organizations today are adopting it
along with performance engineering to validate if applications hosted on cloud can handle any unplanned infrastructure
outages, service, or configuration changes gracefully without impacting the end-user experience significantly. Hence there is anincreasing need for QA organizations to adopt chaos engineering practices and tools to support their customers.

39

OTHER TRENDS THAT HAVE OUR ATTENTION: ENTERPRISELEVEL TEST DATA STRATEGY, MOBILE APPLICATION TESTING, AI(ARTIFICIAL INTELLIGENCE) AND MORE
FIG 26Adoption of enterprise-wide data provisioning strategy

WQR 2023- Global Results

In your organization, is there an enterprise-wide data provisioning strategy
adopted as a standard across all line of business / projects?

69%

18%
12%
1%

Yes - a strategy isfully in place

Yes - a strategy iscurrently being
adopted/almost ready

TOTAL n = 1040

Yes - we have juststarted developing/
adopting a strategy

No - we do not havea strategy in place
yet but we plan to in
the next 12 months0%
No - we do nothave a strategy inplace yet but we plan
to in the next
2-3 years

0%No - we do not havea strategy in place
yet but plan to in
the next 4-5 years

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers AND IT Directors AND Chief Information Oﬃcer"

It is understood that effective test data management is vital in the test ecosystem. The growing demand for end-to-end
automation highlights the need for abundant, on-demand test data. This year, 69% of organizations adopted or prepared for an
enterprise-wide test data provisioning strategy— a significant leap from last year’s 31%. Data privacy remains crucial, yet 61% oforganizations overlook integrating test data provisioning with delivery pipelines. This challenge hinders the provision of suitable
test data at the pace of automation.

40

World Quality Report I 2023-24

Quality Ecosystem

FIG 27Making mobile application testing more eﬀective

WQR 2023- Global Results

What steps are being taken by your organization to make mobile application testing more eﬀective?

AI and Data Science based approaches

76%

Include as part of DevOps toolchains

41%

Geolocation testing for enhanced global performance

35%

Integration with cloud-based mobile device labs

28%

Higher mobile test automation

20%

TOTAL n = 315

Base: Those Who Selected "QA Testing/ Testing Manager/ Quality Engineers"

And talking about data management in an ever-increasing digitized world without mentioning mobile app testing would be unmerited.
It’s no surprise that 76% of organizations are adopting mobile app testing to optimize testing across countless device variants
while employing AI and data science. Cloud-based automated testing with AI/ML techniques has further refined testing efforts
for diverse mobile devices worldwide.

WHAT SHOULD ORGANIZATIONS FOCUS ON?
It is great to see that more organizations are moving non-production environments to cloud even though it is moreconcentrated on a hybrid strategy. Organizations increasingly embracing SRE and leveraging chaos engineering and AI-data
science approaches to specialized testing is also a move in the right direction.

Our recommendations and summary:•Accelerate the inclusion of cloud and infrastructure testing as part of the software development lifecycle, to improve
resiliency, security, redundancy, and data recovery.

•

Continue to consider intelligent integrations (including customizing specific features) to make the best use ofcommercial off-the-shelf and cloud native tooling – as a hybrid strategy for non-production environment provisioning.

•

Adopt stringent practices to test the impact of failure events on cloud and continue to give extremely high priority
to the performance and resiliency of applications on cloud.

•Invest on workforce transformation in building SRE and chaos engineering capabilities within quality engineeringorganizations to embrace and support the emerging trends in digital and cloud migration strategies of the customers.

41

In the last few decades, the world has undergone atransformative journey propelled by an unparalleled force:
digitalization. This sweeping phenomenon, characterized by the
integration of digital technologies into various aspects of humanlife and industries, has reshaped our societies, economies, andinteractions at a pace that defies historical norms.

DIGITAL CORE RELIABILITY• JEBA ABRAHAM, Practice Vice President,
Quality Engineering & Testing, Sogeti USA

When the COVID-19 pandemic broke out, much of theworld moved online, further accelerating digitalization at a
breakneck speed*.

• STEFAN GERSTNER, Vice President, Quality
Engineering & Testing, Sogeti• KHUSRO SIDDIQUI, Director, SAP Business
Assurance, Capgemini USA• SHASHANK LOKHANDE, Senior Director, Quality
Engineering & Testing, Sogeti India

The accelerated adoption of digital technologies, primarily
driven by the necessity to keep functioning during a global
pandemic has transformed our world forever. A simple metricis digital payments. In Asia alone, the number of digital payments
has increased by 300% between 2017 and 2023**. Needless tosay, the digital core - technology platforms and applications
that enable organizations to transform into digital businesses- requires thorough testing to enable success in the digitalmarketplace of the future.

UNEARTHING THE CORE OF IT ALL: TESTING, PROCESSES,
AUTOMATION, AND MORE
We looked at quality engineering and testing trends across organizations around the digital core. Some of the findings were expected– like the most important focus areas when testing digital solutions are customer experience and system performance (directly
impacting the usability of the platform/solution). This reiterates the fact that the digital core is all about building experiences andsolutions, delighting, and retaining customers and driving growth. It also underlines the importance of being able to act in a swift
way on market opportunities while ensuring clients’ digital happiness.

Most important when testing 'digital core' solutions

FIG 28WQR 2023- Global Results

What is most important to your organization when testing 'digital core' solutions?
52%

47%

46%

46%
37%

CustomerExperience

Systemperformance

Data
Validation

Speed ofdeployment

TOTAL n = 750

References:*https://www.oecd.org/digital/digital-economy-outlook-covid.pdf** https://www.statista.com/outlook/dmo/fintech/asia#transaction-value

42

World Quality Report I 2023-24

37%

Compliance Roles/Security
relatedand workﬂow
requirements
validations

37%

37%

Good enough Accessibility
end-to-end
andbusiness
Localization
process (no High
/Critical defects)

Base: Those Who Selected "VP Applications/ Product Owner AND QA/ Testing Manager/
Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

Digital Core Reliability

Another trend that was expected was the focus on end-to-end business processes. Organizations have arranged testing teams
for maximum support towards end-to-end (E2E) testing, with ~50% of them having dedicated quality analysis (QA) teams testing
digital core applications. To top it all off, 35% of organizations have their business users testing digital core solutions to ensure E2E
quality. While the close involvement of business users clearly adds value, further professionalization is required to optimize theirinvolvement and gain maximum benefit.

FIG 29Involvement in ensuring the quality of the end-to-end solution

WQR 2023- Global Results

Who is involved in ensuring the quality of the end-to-end solution?

49%

39%

39%

35%

34%

Dedicated"digital core"
QA team

Enterprise
wide Testing
CoE

Product/
Feature teams

Business
Users

External
Systems
Implementor

TOTAL n = 1750

E2E testing is also extremely complex and is one of the top challenges while testing digital core applications, but more on that later.
Another trend observed was the pervasive usage of automation across the entire lifecycle of quality. In fact, more than half of theorganizations use automation across every step of the lifecycle – from test scope selection to analysis and reporting.

“It is vital for Volkswagen Financial Services (VWFS) that our digital products arerobust, as we are representing more than just VWFS.
VWFS does have its own market presence, but fundamentally, most consumers
will consider the car brand as the brand. So, quality is important, and as you would
expect, those car brands are naturally keen to ensure that their customer experienceis outstanding, and anything we do as the financial services captive reflects on
their brand level too”.
MARK PAIN
Head of Product Delivery, Volkswagen Financial Services, UK

43

FIG 30

Where automated solutions are used for 'digital core' testing

WQR 2023- Global Results

Where do you use automated solutions for your 'digital core' testing?
Conﬁguration Validation

59%

Analysis & Reporting

59%

Environments & Deployments

56%

Impact Analysis & Test Scoping

55%

Workload modelling for Performance Testing

55%

55%

Test Prioritization

55%

Test Case Generation

54%

Test Execution

53%

Automated Test Case creation

52%

Test Data Provisioning/Generation

50%

TOTAL n = 750

44

World Quality Report I 2023-24

Base: Those Who Selected "VP Applications/ Product Owner AND QA/ Testing Manager/
Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

Digital Core Reliability

BUT HOW MUCH IS AUTOMATION TRULY CONTRIBUTING?
Although automation is pervasive across the lifecycle, we observed that the usage of test automation still has a long way to go.
About 90% of organizations have <40% automation coverage of their testing and 26% of them have 20% or less coverage.

FIG 31

Level of automation of 'digital core' testing

WQR 2023- Global Results

How much of your 'digital core' testing is automated?

64%

26%

9%
1%

0%/None1 - 20%21 - 40%

41 - 60%

61 - 80%

More than 80%

TOTAL n = 1750

This clearly is due to the same top challenges around testing digital core solutions - the complexity of the environment owing to
the mix of tools, dependencies related to environment and data availability. When it is hard to test, it is even harder to automate.
There’s also another contradiction we need to address - while 31% of organizations feel the pressure to keep up with the pace ofdevelopment teams developing digital core solutions, 69% of organizations do not feel the pressure. With <40% of automation
coverage and digital core solutions becoming more SaaS and less customized, it is a bit of an enigma to unravel. Why don’t organizations
feel the pressure to keep up? Is that because they have large QA teams rushing to complete all testing manually? Or are teams
stressed by the number and frequency of code drops coming in for testing?

WHAT’S STOPPING GUARANTEED QUALITY FOR DIGITAL
CORE SOLUTIONS?
We saw earlier that there is a clear focus on end-to-end testing of business processes. And this is highly relevant for the mostimportant objective of testing digital solutions – customer experience. However, we identified numerous challenges to ensuring
quality of digital core solutions.

45

Key challenges faced while ensuring quality of'digital core' solutions

FIG 32WQR 2023- Global Results

What are the key challenges you face while ensuring quality of 'digital core' solutions(ERP, Salesforce etc.)?

41%

Complexity
due to multiple
tools in the mix

38%

Complexity intesting businessprocesses
end-to-end

TOTAL n = 750

36%

Environmental
challenges

34%

33%

Test Data
comparable to
PROD

Gaps in Domain
expertise

31%

Pressure tokeep up with
development
velocity

27%

Impact analysis
and Test Scoping
for updates/
upgradesBase: Those Who Selected "VP Applications/ Product Owner AND QA/ Testing Manager/
Quality Engineers AND Chief Technology Oﬃcer/ Product Head"Complexity around tools, environmental and data dependencies are still prevalent. There is also additional complexity in testing
business processes end-to-end. This might be due to the dependencies on multiple interconnected systems in complex landscapes,
which is quite typical when testing digital core solutions. This could also be due to the number of people/teams involved inensuring E2E quality through testing process chains and handing off the testing. Determining if identified issues are real issues
is another challenge in such environments.

ENSURING QUALITY OF DIGITAL CORE SOLUTIONS DEMANDSNEWER STRATEGIES AND APPROACHES
When it comes to test automation, replicating what functional testers do manually step by step through a tool might not be thebest possible approach. Trying to break the end-to-end tests into more manageable pieces with a level of contract testing could
be a more sustainable approach for building automated tests.
The other important success factor is to have the right skills in QA Teams. When we looked at the data, quality assurance skills wererated as the most important skillset over domain or platform skills required for testing digital core solutions. This datapoint feels
like bit of an outlier when compared to two other datapoints – 35% of organizations still utilize business users in validating digitalcore solutions and 33% of organizations pointed to gaps in domain expertise as a challenge to overcome when testing digital coresolutions. The truth is while solid testing and quality assurance skills are sought after due to the nature of digital core solutions,domain expertise remains invaluable.

46

World Quality Report I 2023-24

Digital Core Reliability

FIG 33

Most important skillset to testers when testing'digital core' solutions

WQR 2023- Global Results

What is the most important skillset to the testers in your QA organization when
testing 'digital core' solutions?
64%

21%

Quality
Assurance

Knowledge oftechnology platform

TOTAL n = 750

9%7%

Business domain
knowledge

CodingskillsBase: Those Who Selected "VP Applications/ Product Owner AND QA/ Testing Manager/
Quality Engineers AND Chief Technology Oﬃcer/ Product Head"

Organizations need to find a way to integrate quality assurance skills and domain skills into teams. There is more than one way toachieve this of course – building and sustaining stable teams over longer durations for functional areas, getting business users totest with QA teams, and upskilling QA teams on domain knowledge are some to be highlighted.

“You have to test with the right code and the right data, and the biggest shift
we recently made is testing with production data. One of the biggest realizations
we had was no matter what you test if you are not testing with prod data,
there are going to be big gaps”.
RAJESH NATARAJAN
Senior Director, Quality Engineering at Hiscox

RECOMMENDATIONS AND SUMMARY
There is a lot of room to grow automated testing coverage and further build customer delight via seamless experiences.
These are our recommendations to organizations building and deploying digital core solutions:
1. Use newer approaches like test isolation, contract testing, etc., to drive more segmentation and higher automated test execution– this will solve the challenge around domain skills as well as relieve teams of stress to keep up with the development speed.
2. Utilize service virtualization to overcome challenges around multiple applications, environments, and data dependencies.
Service virtualization combined with test automation will really move the needle in the future.
3. QA teams of the future will need to have a mix of skills – strong testing skills will always be needed and will form the core.
However, they will need to be augmented with some domain and platform skills to drive efficiencies.

47

INTELLIGENT PRODUCT TESTING• JEAN-BAPTISTE BONNET
Offer Leader, Intelligent Testing
for Connected Products, Capgemini
Engineering, France• NARENDRA VAISHAMPAYAN
Director, IoT Quality Engineering & Testing,
Sogeti India• ATUL JADHAV
Director ERD, Engineering, Capgemini India• VIPUL MOHINDRA
Director, Sales Operations - MFG, Auto, Aero,
Capgemini North America ERD• UDI WEINBERG
Director, Product Management, OpenText• OREN ZIV
Product Manager, OpenText
48

World Quality Report I 2023-24

Intelligent Product Testing

Point at any car you see on the road, chances are it features
a complex infotainment system, maybe additional connected
medical devices, wellness devices, or even appliances, whichmight all be connected to mobile apps. Products today are an
extension of us.
Over the last decade, the product landscape has transformed
in unimaginable ways. We have long moved from stand-alone
products to upgraded globally interconnected products.
Naturally, this evolution has introduced ‘a never seen before’level of complexity.
A lot is at stake. An unsavory user experience can not only
jeopardize a product launch but also damage a brand image. And
it is not just about customer experiences - regulatory pressures,changing guidelines, and cyber threats all suggest the need forrobust product testing mechanisms.
There is even a need to re-invent the product testing approachfor this globally interconnected world we now live in.

SO, WHAT REALLY GOES INTO
CREATING THE PERFECTINTELLIGENT PRODUCT TESTINGSYSTEM ECOSYSTEM?
This year’s quality survey report highlighted three main activities:
1. Ensuring seamless integration with the cloud ecosystem
2. Ensuring the security of the product3. Ensuring a perfect user experience
What was interesting to note in the results was that although
the trend in the importance of testing activities was within avery narrow range, cloud testing came up as the most prominent
factor with 50% of respondents placing high importance on
it. In fact, some considered it as important as performance
testing itself. This further validated the predominance ofinterconnected products.
Tending to cybersecurity needs came up as another important
activity (47% of respondents) which makes perfect sense
considering adherence to future regulations, or deterring threats
for connected products is paramount for the overall robustnessof the ecosystem. A few other activities such as functional
testing (48%), accessibility testing (46%), and embedded testing(46%), also showed up as relevant testing areas.

“We must provide robust solutions for ourcustomers because that is an expectationfrom a regulatory perspective, but more
importantly, we want to ensure a good
customer experience”.
MARK PAIN
Head of Product Delivery, Volkswagen Financial Services, UK

49

FROM CHIP TO CLOUD
While the respondents acknowledged the need for an evolved testing ecosystem, we were surprised to learn that only 36% thought
end-to-end (E2E) testing was necessary. It has been established that the value of connected products is distributed throughout
the entire chain, and its resilience depends on the weakest link in the chain, and yet, the overall value of E2E testing has not been
fully recognized. One reason we suspect could be that it requires significant investment in test benches, hardware, and cloudinfrastructure for E2E testing.
An E2E approach is frequently viewed through the abstraction of various layers of the system, such as embedded systems, cloud, andconnectivity. However, in our experience, we believe this is going to have a minor impact on intelligent product testing in the future.

FIG 34

Importance of test activities to industry

WQR 2023 - Global Results

Please rate each of the test activities below in terms of how important they are to your industry.

50%

50%

48%

47%

46%

46%

Cloud
Testing

Performance
TestingFunctional
Testing

Cybersecurity
Testing

Accessibility
Testing

Embedded
Testing

45%

43%

36%

36%

35%

33%

Interoperability
Testing

Visual Testing

Connectivity Testing

E2E Testing

Regulatory
Compliance Testing

Acceptance
Testing

TOTAL n = 315

Base: Those Who Selected "QA/ Testing Manager/ Quality Engineers"

WHAT ARE THE MASSES EXPECTING FROM INTELLIGENT TESTING?
One of the significant trends observed in the WQR 2023 survey isthe expectation for improving the test definition. The increasing
complexity of products, along with hyper-personalization tocreate a unique user experience, necessitates passing millionsof test cases to achieve the perfect user experience. Realisticallyspeaking, it’s not possible to determine all possible combinations.
According to respondents, the critical aspect of intelligent
testing is test-case definition, with test case selection (52%)
followed by a prioritization of test cases (50%). It emphasizes
that testing approaches need to constantly evolve to prevent
systemic failures. Following the two main expected outcomes,
the test data and test case generation are aligned (at 45%
50

World Quality Report I 2023-24

and 42% respectively), with the required improvement of thetesting approaches.
Another key finding was that 47% of respondents consider
root cause analysis to be of high priority. This denotes a shift
to the left and a willingness to use test results not only to assess
product quality but also to provide R&D with insights throughout
the development process.
Expectations regarding intelligent product testing clearly focus
on the key elements of the user experience, such as functionality,interoperability, and cybersecurity. Additionally, respondents
have the highest expectations of improved test coverage.

Quality & Sustainability

Importance of implementing testing use cases as part ofIntelligent Product Testing

FIG 35WQR 2023 - Global Results

Please rate each of testing use cases below in terms of how important they are to be implemented
as part of Intelligent Product Testing?
52%

50%

47%

45%

42%

39%

37%

33%

15%

Test caseselection

Test caseprioritization

Root cause
analysis

Test data
generation

Automated
test casegeneration

Test caseminimization

Test caseadequacyCodedefectprediction

Test
execution
analysis

TOTAL n = 615

Base: Those Who Selected "IT Directors AND QA Testing/ Testing Manager AND Quality Engineers"“It is now about automating everything that is a part of the testing life cycle,
is taking your time, and needs to be done repeatedly”.
RAJ SINGH
Vice President , Head of Wholesale IT QA of a Leading Investment Bank and Brokerage Group51

AI TAKES THE CENTER STAGE, BUT AT WHAT COST?
As previously discussed, there are high expectations for use
cases related to test plan definition, especially in terms ofprioritization, selection, and automatic generation of test cases.

43% of respondents consider integrating data platform
technologies like Hadoop or MapReduce to be essential to theircore competencies.

Artificial Intelligence (AI) and Generative Artificial Intelligence(Gen AI) are at the core of all these technologies. The analysisof the WQR 2023 responses indicates that the respondents arewell aware of the significant impact that these new technologies
will have on their activities, and they need to incorporate theseskills into their quality engineering teams.

As expected, AI-related technologies are the most important,with 44% of respondents believing that they will need to use bothsupervised and unsupervised machine learning technologies intheir activities. Furthermore, to a lesser extent (42%), natural
language processing techniques are also required.

Data is the main area of interest for the respondents. Hyperautomation of testing and expanding test cases to cover
the entire product domain results in a significant increase in
the volume of data, that needs to be stored and processed.

FIG 36

For product testers, the latest findings suggest that automation
toolchain skills and programming languages are now consideredmainstream and no longer considered core competencies.

Importance for the skills required for Intelligent
Product Testing

WQR 2023 - Global Results

Please rate each of the items below in terms of how important they are for the skills required
for Intelligent Product Testing.
Machine Learning models (Supervised, Unsupervised)

44%

Data Platform (Hadoop, MapReduce etc.)

43%

Natural Language Processing

42%

Automation tool chain (Robot, cucumber etc.)

34%

Programming Languages (Python, Java, C#, R etc.)

31%

TOTAL n = 435

Base: Those Who Selected "VP Applications/ Product Owner AND Chief Technology Oﬃcer/ Product Head"

RECOMMENDATIONS AND SUMMARY
There is a lot at stake in terms of interconnected products or services, as other studies have indicated that a vast majority of
consumers abandon the product or service in question if they are disappointed. Our recommendations for organizations building
connected products of the future1. Invest in AI solutions for test prioritization and test case selection to drive maximum value from intelligent testing, against
the millions of possible combinations in today’s hyperpersonalized connected product environment.
2. Focus on an E2E testing approach to ensure seamless customer experience through effective utilization of abstraction ofvarious architecture layers (embedded software, cloud, and connectivity).
3. Give equal priority to the non-functional aspects (performance, security, scalability, usability) of the system as the functional
aspects as they equally influence the end-user experience.

52

World Quality Report I 2023-24

QUALITY & SUSTAINABILITY• ANDREW FULLEN
Head of Innovation and Technology, Sogeti UK• ANTOINE AYMER
CTO, Quality Engineering & Testing, Sogeti• MAHESHWAR KANITKAR
Vice President, Global Quality Engineering &“Sustainability is one of our key objectives.
Sustainability is extremely important.
Really knowing the impact that our actions
have is critical”.

Testing Sales Leader, Sogeti Global
GANESH PRACTOOR• RAJARAO TADIMETY

Director - Quality Engineering Group at one of the world’s
leading financial organizations

Associate Vice President, Quality Engineering &Testing, Capgemini USA• JENNA FELDMAN
Director, Corporate Citizenship, OpenText“We look at sustainability as reuse and it’s
not just within our QE practice, but it’s how
we reuse it across our environments. So,
we’re taking certain platforms and sayingthat this is the baseline for our product,
which then becomes the baseline for all of
our other parks”.
FABIAN RODRIGUEZ
Vice President, Application Delivery & Strategy | Digital &Technology, Universal Destinations & Experiences, NBC Universal

53

A constant challenge reported to us over the years by the World Quality Report (WQR) editions revolves around managing various
tech environments, and yet there is a far more serious environmental concern - our planet.
Thankfully, sustainable IT is increasingly becoming a boardroom discussion topic highlighting how much of an impact IT really hason the environment.
This year, we wanted to deep dive and gauge how far the ideas of sustainability and environmental impact have made their way
into quality engineering and testing (QE&T) landscape.
In this chapter, we identify some pressing questions and observations around the impact of QE&T on sustainability and understandwhere we are truly headed as an industry.
1. What role does quality engineering play in reducing the environmental impact of IT programs and systems? Especially while
reports show that data centers and cloud contribute more to greenhouse emissions than the aviation sector.
2. Understanding the enormity of influence is key to reducing the impact of IT. It may be through being part of an organizationwide program or by helping track emerging metrics.
3. What are people currently engaged in? What are their future plans? And what motivates these actions - is it driven by financial
considerations or a sense of moral obligation?
Let us dive into some of the trends we received from this year’s WQR survey results.

FIG 37

QE's contribution to the sustainability agenda

WQR 2023- Global Results

Which of the following best describes how the Quality Engineering (QE) contributes to the sustainability
agenda of an organization?
67%

Very active

TOTAL n = 1435

54

World Quality Report I 2023-24

24%

6%

2%

ActiveA key driver

Passive

Not involved
at allBase: Those Who Selected "Chief Information Oﬃcer AND VP Applications/ Product Owner
AND IT Directors AND Chief Marketing Oﬃcer AND Chief Digital Oﬃcer And Chief Technology Oﬃcer/
Product Head AND VP OR Directior Of Research AND Development/ Engineering"Quality & Sustainability

IS QUALITY ENGINEERING (QE) REALLY CONTRIBUTING TO
BUSINESSES’ SUSTAINABILITY AGENDAS?
The vast majority (97%) felt QE was active or continually active in driving the sustainability agenda in their organizations. We saw
73% of respondents describing themselves as highly active, or key enablers in driving sustainability best practices in their organizations.
Such a high degree of engagement bodes well for delivering on the promises of a sustainable IT world.

FIG 38

Priority given to sustainability in QE&T

WQR 2023- Global Results

Is sustainability in QE&T a priority for your organization?

TOTAL n = 315

64%

59%

44%

38%

17%

It's
importantto ourcustomers
and/orinvestors

It's
importantto us
and our
employees

We'll lookat it
sometimein thefutureWe'll lookat it inthe next
12 months

It's theright thing
to doIt’s notrelevant
to us

N/A

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

But our findings were not rid of contradictions - two-thirds said that sustainability was important to their customers and investors,
but worryingly, nearly half said they would look at it sometime in the future, with only a third planning to look at it in the next year.
Half of those who responded positively called out vendor compliance with green frameworks and regulations as key drivers offocus in this area. We believe this is a trend that will grow as more governments implement green credentials, especially in areas
like procurement of services and supplies.

55

HOW ARE WE INTEGRATING SUSTAINABLE PRACTICES INTO QE&T
PROCESSES TODAY?
For most respondents (63%), the priority over the next 12 months was to understand how they could monitor and report on green
metrics within their QE testing processes. It certainly will take time to find the right metrics for organizations.

FIG 39Focus for sustainability practices in QE&T processes

WQR 2023- Global Results

What areas will you focus on in the next 12 months to incorporate sustainable practices into
your Quality Engineering and Testing (QE&T) processes?
63%

51%

49%

43%

36%

9%

Monitoring,
Reporting, KPIs,
and Green/
Carbon Impact

Reliability ofsystems andenvironments

Performanceeﬃciency

Vendorcompliance with
Green /
Sustainability
Frameworks /Regulations

Optimizationand cost control
of processes

If it doesn't
cost too muchwe’ll
do someTOTAL n = 315

None

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

We had expected the prioritization of process optimization and cost-saving efforts to receive a higher response rate than itdid (36%). We were surprised to learn that only 16% saw an economic benefit as being a driving factor. It is reassuring, in a way,that economic benefit is not seen as the key reason to focus on sustainability.

56

World Quality Report I 2023-24

Quality & Sustainability

FIG 40

Most important QE&T contribution to sustainability beneﬁts

WQR 2023- Global Results

What is the most important beneﬁt of sustainability that QE&T can contribute to?

55%

21%
16%
8%

Environmental (durability,
greenhouse gas emissions
reductions and recycling/waste
reduction)

Social (including our status asa socially responsible org)

Economic (cost and energy
savings)

Human (non-bias, inclusive, fairsystems)

TOTAL n =1750

It was promising to learn that over half of the respondents felt that the most important benefit to come from a focus on sustainability
was the benefit to the environment. Less promising though was that 21% of replies admitted they were only focusing on sustainability
because it “looked good for the organization”.

57

FIG 41

Frequency of testing for sustainability attributes

WQR 2023- Global Results

How frequently are your teams testing for the following sustainability attributes for Environments& Testing & Development & Production?
11%

9%

7%

7%

7%

6%

6%

Modularity
andre-usability

Usability

Resourceeﬃciency

Cost of Energy

Resource
reliability

Energyeﬃciency

Performanceeﬃciency

TOTAL n = 315

Base: Those Who Selected "QA Testing/ Testing Manager AND Quality Engineers"

When it came to the frequency of testing for sustainability attributes - modularity, and re-usability were clear winners, followedby usability. These are good engineering practices that align with sustainability practices. A win-win situation that hopefully willcontinue to drive best practices and make a difference to sustainability.

58

World Quality Report I 2023-24

Quality & Sustainability

RECOMMENDATIONSAND SUMMARYCan QE make a difference? Yes, it can. But not alone.
It needs to be part of an organization-wide agenda, withdefined priorities and periodic reporting of targets. QE
needs to learn to measure and report new metrics andbring its existing engineering view to look at how things
can be done better, and how its existing approach tometrics can be used. It is an incredibly positive sign thatmany of those who were interviewed are committed tomaking IT greener and more sustainable, but we also
must be cautious. Greenwashing exists and there is
far more to being green than being able to ‘turn off alight switch’ or report on your impact. Moving to green
electricity is a positive step but what comes after that?
Our research leads us to the following recommendations
to increase the value from your quality automation
initiatives:• Have a set of key KPIs or metrics that can be easily
understood across the organization. Reward innovationand improvements in the KPIs. Ensure regular reviewsof the KPIs, and ensure they remain effective in driving
improvements.
• KPIs need to be considered not just in QE&T but
through requirements gathering, production support,
and working with external vendors. They need to be
included in your strategies and roadmaps. They alsoneed to be realistic and achievable.
• Sustainability needs to be part of the whole
lifecycle, included as a non-functional requirement,
tested for as a requirement, and reported on against
the success (or not) of that compliance.
• Know your green success factors and ensure
they are part of your strategy, your education, yourprocurement, and your hiring processes so they
become an integral part of how you operate.
A final thought. Writing data to the cloud and retrieving
it is an energy-intensive process. Good practices such asdata hygiene and reducing unnecessary data storage
can be good for you, for your customers, for privacy
compliance, and for our planet.

59

SECTORANALYSIS

60

World Quality Report I 2023-24

Automotive

62

Consumer Products, Retail,and Distribution

65

Energy, Utilities, Natural Resources,
and Chemicals

69

Financial Services

71

Healthcare and Life Sciences

73

Manufacturing

76

Public Sector

79Technology, Media, and Telecoms

82

61

QUALITY IN MOTION:
How evolution in hardware and software is
impacting the automotive industry• AXEL SCHOENWALD
Sales Leader, Automotive, Sogeti Germany• ANDY HOWARD
Vice President, Head of Automotive,
Capgemini Americas

MAKE SPACE, DATA. SOFTWAREIS NOW A CLOSELY GUARDED
SECRET TOO!
Data reliance and the need for substantial research anddevelopment are prompting reputed original equipment
manufacturers (OEMs) to transition from a dealer-dependent
model to an agency model for better customer data management.
The modern automotive landscape places a premium
on software as the primary driver of innovation leading
automotive manufacturers to invest in in-house softwaredevelopment teams converging the embedded software andIT software competencies. This move has prompted cloud andquality management experts to harness their knowledge ofelectrification, automation, and digitization, and elevate thecompany’s overall software competencies. It also responds to
the rapidly increasing competition in the automotive industry,wherein we see many players seeking to reinvent the holistic
mobility experience.

• FRANCK DESAULTY
Automotive Solution Director, Capgemini Engineering

Nonetheless, this shift amplifies the demand for quality, withcustomers expecting automotive companies to hit it out ofthe park with every product they release into the market. The
aim is to guarantee each component seamlessly integrates
with another. Beyond rigorous quality checks, having experts
advocate for a customer’s needs helps the OEMs focus on theuser, rather than just the functional aspects of the softwareand the hardware.

Speed, lifestyle, and exclusivity define and drive the automotive
industry, which has witnessed some incredible technologicaltransformations over the years. And it continues to innovateand evolve its product range, which isn’t limited to cars anymore.

SO, WHAT DOES THAT MEAN
FOR QUALITY ENGINEERS IN THE
AUTOMOTIVE INDUSTRY?

Today, it encompasses an array of mobility services that often
involve two interdependent, yet different areas - firstly, thesales and after-sales services; and secondly, the connectivity
and autonomous mobility services. The former revolves aroundselling the car and the latter is related to experiential elements
including autonomous driving, connected car services, etc. thatturn your car into a mobile on wheels!
Supported and regulated by international safety standards andsustainability mandates, we are seeing a choice of automotives
like never before. But what sets this industry apart is that when
you buy a product, you’re not only paying for the product but
also for the guarantee that it will serve you the way and when
you need it.
For the first time, we are also seeing a trickle-down effect of thison public transport- a case in point is the launch of Germany’s
unlimited railway access pass for citizens and visitors this year.
Things definitely seem like they are changing for the better, but
as we witness these dynamic shifts we must ask – what mustremain constant and why?
62

World Quality Report I 2023-24

Today, quality engineering is largely being shaped by two aspects:
1. Time-to-market: Aligning the start of production to software
readiness to allow testing to begin as soon as the build starts
and prevent losses due to a mismatch of timelines.
2. Quality before customer: A customer-centric approach toensure the car works as the customer expects it to work.
A quality engineer today must have a comprehensive
understanding of software testing and hardware engineering
ensuring a holistic perspective and end-to-end testing for
efficient execution. This approach is vital for delivering the best
possible customer experience. However, quality testing is not
only reliant on an experienced quality engineer but also on theinfrastructure required for testing – an end-to-end test factory/
ecosystem and a virtual testing environment that carries out
simulations using real-life scenarios that evaluate every pointof concern in the vehicle.

63

IS EV THE NEXT BIG THING?
AND HOW IS SUSTAINABILITYIMPACTING QUALITY?
Despite the increasing popularity of electric vehicles, innovators
and OEMs are still unsure whether EVs are the final step,or whether gas motors and other alternatives like hydrogen
will prevail. Experts are of the opinion that when it comes toembracing sustainability in the automotive industry, companies
need to focus on the idea of mobility and the lifestyle of thecustomer, rather than the car itself.
Given that manufacturing EVs requires cleaner power than wecurrently have, there is an increased push for harnessing green
alternative power sources like solar and wind energy. But justlike Rome wasn’t built in a day, addressing a customer’s needsmeans stepping in for the long game. Initiatives like the freecombustion car rental policy by Volkswagen go a long way inhelping customers experience the best of both worlds whilelargely curbing carbon emissions.
However, while most discussions still revolve around the futureof mobility, the discussion for quality must not take a back seat.
Now more than ever, with the automotive industry focusing on
innovation, quality testing is paramount, paving the way for asafer, more efficient, and eco-friendly future.

Survey Watch

82%

Respondents said they willconsistently invest in AI solutions
to improve quality processes

68%

Respondents chose Embedded testing
as the most important test activity totheir industry

41%

Respondents said lack of skills is the
major roadblock to automation

47%

Respondents named C#/Java/SQL/
Python as the top skill for QE associates

64

World Quality Report I 2023-24

QE IN THE CONSUMERSECTORA Glimpse into Our Future ‘Shelves’• LAUREN OUART
Vice President, Global Account Executive,
Capgemini USA
The global pandemic and ensuing disruptive geopolitical events
fundamentally changed everything about retail and consumer
products. The industry has had to constantly reinvent itself tomake up for the unpredictability of the supply chain, demand
forecasting, and more.
Even as erratic global developments, quality, and IT
transformations gripped the globe, organizations in the
consumer sector stayed resilient and bounced back by infusing
innovation into how they work.
Over the past three years, the focus zeroed in on manufacturing
products more efficiently to recover profits lost during the
pandemic. While in the short run, a steep price-hike strategy
seemed to have helped, it has proved to be counter-productivein the longer run. Another challenge that the industry struggledwith in the past and which remains is the availability and reliabilityof consumer data that can be put through AI to get conclusiveand decisive insights for manufacturing.

THE WELLNESS TRIANGLE –CONSUMER PRODUCT, THE
INDUSTRY AND HEALTHAs the world fought the COVID-19 virus, the importance ofmental and physical health seeped into everyone’s consciousness,
which has had a direct impact on the demand for connected
products/wearables like smart fitness trackers. Concurrently,
we saw a rise in citizen developers who employed Low-CodeNo-Code platforms to chase innovation and propel the world
forward amidst uncertainty. In response to this development,
most organizations have begun adopting advanced automation
solutions and have even proceeded to the pilot testing phase.
While a weariness in the market has set in, which mostly stems
from having to tackle constant changes, it is consumer trends andinnovations like these that made and continue to make things
fast-paced, thrilling, and dynamic!

OUT OF STOCK BECAME A
TREND IN ITSELF!
No matter which part of the world you lived in, all of us dealt withfrenzied panic-induced buying during the pandemic. It is 2023
and we still feel the lingering effects of the global shortages
we experienced.
We think this is also where quality started being hailed as the‘Robin Hood’ of production – a critical game-changer, often
perceived as a challenge. However, an additional challenge
surfaced – tight supply chains and tighter timelines. This isbeing addressed through the recognition that the quality ofa product is inversely proportional to the delay in products
hitting the market and the waste generated. The productionof better-quality goods implies less energy usage, less scrap,
and less pollution even with the same production levels! This
also had a direct cascading effect on IT, manifesting as a surge inthe use of cutting-edge AI-based tech, and a renewed interestin sustainability.

BACK TO THE FUTURE WITH AI!
We now see organizations trying to leverage the power of AI,
however, not many of them have established AI at scale yet,and some still only employ it in pockets. Many have strategicallyrefused to employ “technology for technology’s sake,” and all
decisions are being driven by the desired business outcomes.
We see Web 3.0 technology, especially the metaverse as
another promising tool that can brilliantly execute efficientmanufacturing and negate the need for price hikes. Its ability
to create an immersive digital environment would enable teams
to run plant simulations that tackle the problem of low-supply
and high-demand situations. Imagine being able to create an
entire factory in the metaverse and recreating multiple scenariosthat can reduce your time, labor, and costs associated withreal-world experimentations!
All of this hinges on the quality of data. Rapidly changing
customer trends, tastes and preferences, and global instability,
are a challenge to the availability and reliability of consumer data.
The predictability of questions like which product will consumers
want and when and where do they want it delivered, requires
extensive research by market experts. The resultant surplus ofoperational and strategic data is impacting how organizations
streamline their resources and efforts, leading them to developa ‘divide and rule’ policy without really working in silos. Someexperts are forecasting a rise in the adoption of a federated
model of working, which focuses on striking a balance between
a centralized and decentralized approach. Multiple teams orunits work independently but collaboratively towards a sharedgoal, yet each agile team is responsible for overseeing the qualityof its own product or processes. Under such a model, eachfunction becomes closely intertwined and interdependent,
which makes developing multidisciplinary skills a must, especiallyfor quality engineers who need to wear multiple hats of coders
as well as testers.
65

66World Quality Report I 2023-24

SUMMARY
There’s a tremendous scope for innovation in the consumer
industry, with new products being developed and processes
being perfected every day. According to the 2023 WQRsurvey, higher productivity and an improved customer
experience are the quality outcomes organizations arefocusing on. The pandemic redefined everything from whata physical store meant and could do, to reinventing the retailexperience for customers. Even the first step of delivering an
end-to-end customer experience has transformed, with sales
and branding leveraging the tech tools at their disposal. While
we are still figuring out the ethical implications of every step
we take toward the future, the shopping cart can’t stop rolling– it’s the consumer industry after all!

Survey Watch

80%

Respondents said they will consistently
invest in AI to improve QE processes

50%

Organizations say only 1%-25% of
their organizations have adopted
Quality Engineering

76%

Organizations are using traditional
testing centres of excellence to support
agile projects

73%

Respondents said it is very importantto test the impact of failure events on
applications running on cloud especiallyusing chaos engineering

67

ENERGY AND UTILITIES :FUELED BY QUALITY
Probing the potential of the Energy andUtilities QE ecosystem• MANISH SHARMA
Vice President, Market Unit Leader, Energy,
Capgemini USA• DRU K. HAWKINSGroup Vice President, Energy Sector, Capgemini USA• JEFF LARSEN
Vice President, Natural Resources and Chemicals
Industry Sector, Capgemini USA

We could have the most brilliant ideas, but without energy, theyremain unactionable. Be it manufacturing consumer goods,
finding cures for diseases, or creating technologies like Web
3.0 and blockchain modules, the energy and utilities sector isthe driver that makes it all happen.
In the post-pandemic era, we saw this sector adopting market
trends like cloud integration, digitization, and using decentralized
modes of IT processes.
This year, we are witnessing a shift towards artificial intelligence(AI), automation, and an increasingly agile approach to quality
engineering (QE). So where is the conversation headed?

DIFFERENT BATTLES,COMMON TOOLSWe see that critical functions like enhancing the performanceof power plants, reducing energy losses during transmission, or
improving the reliability of utility services are now being madepossible due to smart QE models.
However, the utilization of QE seems to vary significantly. Whilesome organizations focus on using QE throughout the value
chain, there are others that apply it individually on downstream,
upstream, or sometimes mid-stream functions.

68

World Quality Report I 2023-24

Interestingly, this split usage is the most evident among the threemajor sub-sectors: oil and gas companies, oilfield services (OFS)
companies, and engineering, procurement, and construction(EPC) companies that have drastically unique testing landscapes.
Oil and gas companies earlier developed their own testing
systems, but later resorted to commercial ones that could
be purchased and automated as per enterprise-specificrequirements. While this seemed lucrative initially, commercial
testing products failed to integrate functional/regression
testing, create test beds, or even automate test scripts todrive the desired results. These companies eventually resorted
to adding a layer on top of off-market products whichchanged the game!
On the other hand, when it comes to OFS, their core focus is
to continuously build technology products for the oil and gas
companies, and they operate in a different testing ecosystem.
Their side of the story would involve functions like building thedrilling platforms, providing real-time data integration, andworking with sophisticated algorithms to test simulations foroil and gas companies.
Then we have the third counterpart – EPC which are pure-playconstruction companies and might not even develop their own
software products and typically use OEM (original equipment
manufacturer) products. For them, testing is more on the coreengineering side and might often include creating simulations
to validate the integrity of the actual physical models they build.
But in all these interrelated sub-sectors, data is thecommon denominator.

DATA – THE UNIFIER
One of the major challenges the energy and utilities sector faces
today is the amount of data it must deal with. Additionally, it
must also be analyzed in real-time to reach optimal businessdecisions.
The focus today is to create data analytics tools that could, forinstance, extract real-time data from drills to analyze seismic
activities in the layers and sediments below the surface to makethe exploration more effective, and so on.
And we are seeing several technology providers working withintelligent software solutions to aid these quests for oil andgas companies. But again, when it comes to data, moving
legacy systems to the cloud and making them work with native
capabilities remains challenging.
Having said that, today we see AI going a step further to drive
quality and redefine how data is being leveraged in this space.

69

STRIKING GOLDAI and Gen AI are being widely accepted as quality control tools.
As expected, it is being used to assess and mitigate potential
risks and their impact on the quality of products.
Speaking of AI, we explored the use of digital twins in the
manufacturing chapter, and we think it could add a lotof value in this sector too. It might be uniquely beneficial
to EPC and OFS companies, where they can use AI to create
digital twins.
This could offer a platform for engineers to collaborate andtest product hardware to certify the possibility of successful
integration of systems and software before developing the
product physically – thus efficiently managing cost, time,and resources.
It might also play a hand in weaving sustainable practices within
the energy and utilities ecosystem.

DELIVERING BIG RESULTS,
BUT NOT AT THE COST OF
THE PLANET!
There’s no denying that if there is any industry that continues tobe a major concern for the environment – it is this one. But wethink that effective QE can truly transform how sustainability
is viewed, adapted, and integrated in this sector.

BECKONING A NEW ERAAs the biggest players adopt an agile approach to streamlining
their business operations, we predict other companies tofollow suit. With the advent of AI and Gen AI, we also see a
lot of potential to transform QE at a fundamental level within
the industry and not only mitigate potential risks but go a stepfurther to eradicate them completely.

Off the top of our heads, here are a few ways a robust QE system
can enable sustainability:

Survey Watch· Identify sources of waste heat whilst suggesting potential
recovery solutions.
· Optimize transport routes or operations and monitor air quality
as a part of bigger greening projects.
· Determine appropriate equipment that is energy-efficient or
can further create equipment that might have a similar impact.
We are also seeing ample use cases where AI solutions like open
cascade and digital twins are being used to actively contributeto carbon capture and storage technologies which can also be
remotely operated with minimal human intervention, ensuringconsistent and reliable performance. Most importantly, it isheartening to see how IT is playing a crucial role in improving
the accuracy of sustainability reporting data, addressing thechallenge of greenwashing, and aiding in regulatory compliance.

70

World Quality Report I 2023-24

28%

Organizations site collaboration
issues and lack of skills as major
roadblocks to automation

71%

Organizations believe QE plays anactive role in contributing to thesustainability agenda

55%

Organizations are using AI todrive velocity

71%

Organizations will use AI for
more analysis of user feedback andproduction telemetry

THE FINANCIAL SERVICES
INDUSTRY – INVESTING IN
DELIVERING SUCCESS

and AI, in some shape or form. With higher productivity andspeed being the top two contenders for AI-driven quality
outcomes (WQR Survey, 2023), there are some organizations
that are still exploring the potential benefits of employing AI.
There are some organizations that are still exploring thepotential benefits of employing AI. Their approach to the samedepends on two aspects:
1. The level of maturity of an organization, which decides its
willingness to embrace the latest trends and move ahead ofcompanies that lag.

• ANAND MOORTHY
Vice President & Head of Quality Engineering &Testing, Financial Services, Capgemini• LISHA RAKESH
Senior Director, Financial Services, Capgemini India• NILESH YEOLA
Senior Director, Quality Engineering & Testing,
Financial Services, Capgemini India

From introducing the concept of the barter system to regulatingcomplex business and global transactions, the financial services
sector has come a long way today. In last year’s report, weestablished that even though Artificial Intelligence being in
the fray has made transactions simpler, the discussion over its
overall integration in the end-to-end processes largely remains
complex. This year, we are trying to assess and explore if
we are doing enough to keep our financial systems andprocesses safe and the skills that will help Quality Engineers
stay relevant tomorrow.

UNTAPPED RICHES TO BE
UNLOCKED

2. Data security and privacy concerns and the readinessto mitigate them, that largely determine the extent of AI andcloud adoption.

The pace of full-scale migration of applications to the cloudhas not yet seen significant acceleration. Governments andinternational institutions are engaging in discussions about thepolicies and compliance necessary for regulating the use of AI.
While it remains to be seen how the FS sector tackles extra
regulations, increased collaboration with cloud providers tonavigate heavy regulations may help.
Nevertheless, most organizations have begun implementing
it internally and have shown interest in the opportunities thatarise by embracing AI. While the insurance industry is showing
growing enthusiasm about the benefits of Gen AI in process
automation and customer engagement, most banks are
adopting automation at an accelerated pace to reduce risks andlive defects. With a lot of innovation queued up for the future,
one key challenge lies in arranging the right test environments
and the right test data to establish a successful ecosystem for
quality testing. This demands major infrastructural changes likethe transformation of legacy systems to modern applications
and addressing the skill gap – both challenges being identified
as impediments to automation and QE in this year’s WQR survey.
Consequently, it encourages major infrastructural changes andequips a single tester or quality engineer with a diverse skill set
that encompasses both quality testing and automation domains.

In the quality landscape, the focus that was initially on automating
functional testing has now shifted to non-functional testing,prompting us to revisit the basics of business and modernize
them. The next chapter belongs to Generative AI and its
effects on lifecycle automation. But how important is Quality
Engineering in a world that is increasingly becoming reliant on AI?
The financial services industry, traditionally one of the mostheavily regulated industries in the world, is displaying a growing
appetite for AI. Every organization is talking about quality

71

“SUSTAINABILITY?
IT’S A CHECKBOX IN EVERYPROPOSAL!”A substantial area of progress has also been made in GreenIT and sustainability through infrastructural optimization andautomation. With an environmental impact generated by IT,
and a carbon footprint larger than that of the airline industry,
switching to Green IT has become an important investment andstrategic discussion, with its overall role in quality testing yet
to be explored.
Quality is not a mere function, but a part of the bigger picture.
The FS industry has been observing steady growth in the
direction of digital transformation, yet every step needs to bevetted to prove the positive effects of automation. Despite
multiple challenges and milestones that are yet to be achieved,
we can start with a comprehensive top-down approach toidentify areas for automation in the process lifecycle. It’s ajourney of continuous improvement and exploration, and our
only trajectory from hereon is upwards!
72

World Quality Report I 2023-24

Survey Watch

75%

Respondents said they are using orplan to use advanced automation
solutions like No Code/ Low Code or AI
based automation frameworks

50%

Respondents said they have used AI
a lot for test environment provisions

80%

Respondents say that 25%-50% of theirautomated testing is integrated todelivery pipelines

82%

Organizations support agile
projects through a traditional Testing
Center of Excellence

QUALITY ENGINEERINGIN HEALTHCARE AND LIFESCIENCES: A BOOSTER
SHOT

NAVIGATING THE SPEED-SAFETY
TIGHTROPE
A significant industry challenge is striking the delicate balance
between expediting product development and adhering
to stringent safety and regulatory standards. This resultsin extended processes for Installation Qualification (IQ),
Operational Qualification (OQ), and Performance Qualification(PQ) in both product and technology domains.

• AZFAR MALLICK
Executive Vice President, Life Sciences,
Capgemini Americas

The Quality Engineering (QE) function is tasked with reducing
time-to-market and program costs through the implementationof AI-driven scientific techniques and innovations that streamline
the scope and execution of quality-related activities, fostering
an automation-centric culture.

• SUTHARSON VEERAVALLIAssociate Vice President - Strategic Solutions and
Growth, Life Sciences, Capgemini Americas

However, it is worth noting that the full range of benefits
has not yet been realized. There remains significant room
for improvement in implementing a comprehensive end-toend approach to Quality Engineering (QE). While pockets
of excellence exist across the enterprise, there are many
opportunities for standardization and scalability.

• SHEETAL CHAWLA
Executive Vice President & Managing Director,
Head of Life Sciences and Northeast Region,
Capgemini Americas

SLOW AND STEADY SEEMS TO
BE THE APPROACH OF CHOICE
The Healthcare and life sciences industry is heavily regulated bystringent GXP guidelines (Good Manufacturing Practices, Good
Clinical Practices, etc.). These regulations, although vital, canfrequently add complexity, especially when seen through thelens of Quality Engineering (QE). This tendency to err on the sideof caution has long been the norm, and we see no significant
shift in this mindset even today.

According to the United Nations, the average worldwide
lifespan for adults has nearly doubled since 1950, largely due togroundbreaking innovations in patient-centered treatments andwellness. The Healthcare and Life Sciences industry has played apivotal role in advancing human well-being through continuousand transformative changes, with technology as its core driver.
The industry continues to make remarkable advances in cuttingedge fields like mRNA technology, which is speeding up biologics,
cell and gene therapies, decentralized clinical trials (DCT),
connected health, and precision medicine.
Quality is the most important topic for the industry that is built
on safety and efficacy. As the industry evolves, so does the roleof quality engineering (QE) within it.

Unlike other industries that readily embrace disruptive
technologies, Healthcare and life sciences tend to proceedwith caution. There is a consistent preference for substantial
proof of the new approach’s value before committing tosignificant changes, which places a considerable burden on the“Proof of Value” concept.
Based on our experience, even industry leaders in this field
exhibit a high degree of skepticism and prudence when
considering something new. They often opt for small-scale
test runs to solidify their understanding of the value inpractical application.
This approach frequently leads to two main challenges. Firstly,
the pace of implementing innovative ideas and tools in Quality
Engineering (QE) is notably slow, resulting in a low speed torealize value. Secondly, there are often hurdles in transitioning
from the proof-of-value stage to broader enterprise adoption.
Furthermore, the industry relies heavily on domain-specificenterprise off-the-shelf applications, hindering the cohesive
deployment of innovative ideas.
73

74World Quality Report I 2023-24

EMBRACING INEVITABLE
CHANGE
There are a few critical trends in quality engineeringthat are gaining prominence.
AI’s ascension was predicted, and as expected,
we see AI, particularly Generative AI, gaining
prominence. In the quality engineering space,
optimizing the scope of quality activities asprescribed by AI is gaining momentum. We are
now witnessing broader adoption across theenterprise following the successful ‘proof of value’phase. Conversations and pilot projects focusedon Generative AI, particularly in the context
of generating test requirements, are gaining
significant momentum.
Quality Engineering (QE) is playing an increasingly
vital role in empowering business users to enhancetheir Performance Qualification (PQ) efforts,
enabling them to shift their attention back to corebusiness operations. QE is anticipated to introduce
a level of business acumen, acting as a surrogate
for business users, streamlining efforts, andproviding tailored guidance to drive product andsoftware quality.
Sustainability principles are gradually transitioning
from strategic planning to tangible initiatives. Whilesustainability within technology programs is gaining
recognition among senior stakeholders, it remains
in its infancy at the operational and tactical levels.
We are currently observing preliminary discussions
regarding the potential involvement of quality
engineers in integrating sustainability considerations
into quality-related activities.
Despite its initial stages, we anticipate that the roleof quality engineers in this regard will become moredefined in the years ahead.

DIAGNOSING THE
DRIVERS OF FUTURE
While the pace of Quality Engineering (QE) transformation inthis industry may be gradual, the direction is unquestionably
clear. We recognize that maintaining a delicate balance between
safety, compliance, and progress remains an ongoing challenge,
and QE plays a pivotal role in this endeavor. The influence of QE
in Healthcare and Life Sciences is steadily expanding into areas
traditionally outside its scope. This necessitates the developmentof an enterprise-wide quality strategy, comprehensive quality
education, increased collaboration across the organization, andthe effective utilization of the latest in technology.
At the heart of this transformation are skilled quality engineers
who possess a deep understanding of the industry, regulations,and technological solutions. They are capable of crafting industryspecific models, rigorously evaluating them on a smaller scale,and ultimately implementing them across the entire enterpriseto provide high-quality, safe solutions to patients worldwide.

Survey Watch

81%

Organizations have confidence inthe accuracy of data used to train
AI platforms

56%

Organizations have occasionallyused AI for test case design

58%

Organizations use Automation to
improve test efficiency

56%

Respondents stated BDD/TDD as
the most critical skill for Quality
Engineering associates

75

EVOLUTION OF QUALITY
IN THE MANUFACTURING
INDUSTRY• ABHISHEK DHIR
Vice President, Head of Manufacturing sector,
Sogeti Americas• TOM VAN DER VEN
Lead Consultant, High Tech, Capgemini NL• SHASHANK MANE
Sales Leader, Manufacturing sector, Capgemini Americas• ATUL JADHAV
Director ERD, Engineering, Capgemini India

The manufacturing industry runs on hundreds of precariouslyinterlinked functions and a discrepancy of even 0.001-millimeters
can wreak havoc.
So naturally, quality by default becomes central to it all.
Manufacturers today are expected to maintain brand legacies
while adapting to the latest trends and ensure rigorous standardsof reliability, performance, and safety while optimizing resourceuse, which is a herculean task!
As the manufacturing industry experiences profound shifts,
driven by the convergence of operational technology (OT) andinformation technology (IT), we are seeing the landscape of
quality engineering (QE) evolving in a whole new direction.

SO, HOW IS MANUFACTURING
FARING THIS YEAR?
Every organization has its own set of hyper-customized quality
checks. Where QE cannot replace quality checks it is helping
in reducing the amount of work and blockages instead.
Test engineering today is remodeling and redefining test
design techniques.

76

World Quality Report I 2023-24

Over the years, we have notedthat despite the need for rapidtransformations, manufacturing takes
on innovation and modern technology
with skepticism.
Even when it came to digitization, thefocus was more on digitizing existingprocesses, to optimize or speed themup, rather than transforming entire
process lifecycles.
If it were not for the COVID-19 pandemic,
the manufacturing sector would nothave been propelled to adopt digital
innovation. Amid the pandemic,
manufacturing sites finally embraced
Jira, and Kanban boards to optimize
repair tools and processes. Automation
extended to lunch orders, spurringinterest in broader applications like CI/
CD (Continuous Integration andContinuous Delivery) and Selenium.
Having said this, we understand theskepticism because we have worked
with century-old brands that havebeen world leaders in specific productcategories. Here, replacing entire
systems wouldn’t just be difficult, butalso counterproductive! Old legacy
systems often have extremely uniquecharacteristics that cannot becompletely reproduced.

77

THIS EXPLAINS THE RAMPANTUSE OF DIGITAL TWINS (DT) IN
MANUFACTURING
We touched upon it last year as well, and we only see its usage
growing with time. DT replicates processes digitally which allows
manufacturers to experiment and test innovations without itinterfering with existing production.
Organizations now realize that testing requires changes andupdates - and building something from scratch is a timeconsuming, expensive, and sometimes impossible process.
You cannot always test specifics. You cannot crash an airplane
10 times to test how durable it is, and a hybrid (part physical,
part digital) setup could be a lifesaver in such cases.
It allows you to note how a specific piece of hardware PLC willinteract when you scale it up into a big system and then the DT
can do the rest of the work - making it a near-perfect solution.

WE CAN SEE SIMILAR TRENDS
FOR AUTOMATION, ARTIFICIALINTELLIGENCE (AI) AND
GENERATIVE AI AS WELL
AI is being widely used to create test cases, mock situations
and find defects in systems. By exponentially expanding theworld of test cases and possibilities, AI is helping mitigate test
regressions to come to terms with specific quality levels.

From a QE point of view in Manufacturing, sustainability canalso come through circularity-closing loops.
In IT we could resort to energy packetization for a similar net
result. It is an interesting concept because it consists of internet
connections and data, going back and forth, with packages thatpower an energy-driven world.
For instance – we could instruct HVACs in a manufacturing unit
to coordinate with a central system to manage energy demand
units by receiving energy packets for cooling, with flexible
timing. This could balance energy peaks and grid congestion
while integrating flexible timing for tasks thereby reducing
overall energy consumption effectively.
But again, to make a real difference, these use cases will haveto become mainstream, and organizations will have to adopt
an end-to-end approach to incorporate sustainable processes
throughout the chain.

SO, WHAT ARE OUR
RECOMMENDATIONS?
•Use QE throughout product lifecycles to eliminate
unnecessary quality checks, improve the test
environment, and optimize overall output.

•

Utilize DTs and hybrid setups strategically topreserve brand legacy while reaping the benefits oftech innovation.

•

Explore sustainability in terms of QE circularity and use a
top-down approach to boost net impact.

Parallelly, we are also seeing increased usage of hyperautomation which involves streamlining code delivery byautomating complex tasks.
But there’s a catch – while much can be automated, human input
is needed to create initial test cases. In certain fields like defense
and medicine, manual intervention remains essential.
This is yet another challenge that we will have to address inthe coming years.

CAN THESE TECHINTERVENTIONS CONTRIBUTETO SUSTAINABILITY IN
MANUFACTURING?
If you look at sustainability in physical products, you will noteit’s quickly resorting to calculating CO2 emissions, but we thinksustainability is much more than that.
78

World Quality Report I 2023-24

Survey Watch

66%

Organizations are using AI to
increase productivity

55%

Organizations are using automation
to improve customer experiences

63%

Organizations are using advanced
automation solutions like Low Code/No Code,AI Based and automation frameworks

67%

Organizations have adopted QE

WILL QUALITY FINALLYSTEAL THE SPOTLIGHT?
Deepdiving into the public sector’s QE
trajectory• GUNNAR MENZEL
Vice President, Head of Public Sector, Capgemini UK• BASTIAAN MINNEEHead of Sales Public Sector, Sogeti NL• PIERRE-ADRIEN HANANIA
Global Offer Leader, Data & AI in Public Sector,
Capgemini Germany

The public sector has always been the binding force to bringeconomies, administrative authorities, technologies, andthe people together. It is often critiqued and perceived as
defiantly snail-paced, yet remains one of the most innovativeand resilient sectors.
Over the last few years, public sector IT has had to deal withcloud migrations, the explosive growth in data, and the rise of AI.
It has to deliver its services faster and better - while also dealing
with unique geopolitical and economic events like COVID, andeconomic recession, among others.
This year we continue to see increased adoption of automation,
artificial intelligence (AI), and agile by public sector organizations
as they seek to identify and bridge the quality gaps to improve
the services they provide.
As adoption and the complexity of the use cases increase, welook at the current trends and understand the future of QE
across the whole of the public sector.

The data ecosystem in the public sector is extremely complex
because it has tax and customs, defense, welfare, security, justice,
healthcare, and public administration, all in different systems,
different technologies, and at different stages of maturity. The
public sector must constantly engage in a balancing act between
leveraging data for decision making, being cautious about thesovereignty of data, and being respectful of privacy concerns.
Data sovereignty remains a critical concern in the public sector,
as governments try to define who owns and controls data anddeliver trusted services to citizens while adhering to rules arounddata privacy.

LEVERAGING AGILE,
AUTOMATION, AI
While many public sector organizations have been using
the agile approach for years, the adoption of agile
practices varies from country to country and across different
government departments.
The introduction of agile has driven a shift towards moreproduct-based organizations, where cross-functional teams
take ownership of IT services and systems and break away frommassive, monolithic multi-year waterfall programs.
Quality engineering has also been increasingly seen as a strategic
asset and is being factored in right at the start of any project
or initiative paving the way for automation of processes andsystems.
The public sector has a diverse landscape of departments andagencies, each responsible for various aspects of governancewhich means difficulties in legislation, coordination, and ensuring
the end-to-end quality of implementations. Automation tools
have helped in streamlining processes, especially in areas suchas document processing, improving overall experiences thus
aiding government reliability.
We are seeing impressive results of automation in some of thesocial benefit calculations that involve tax agencies. Automationis also aiding the move towards interoperability and the “onceonly” principle, where citizens provide information only once
and it is shared across government departments, which cuts
down complexity and facilitates modernization efforts. We saw
this firsthand in migration processes in some Western European
countries where automation helped migrants, public servants,
and cross-border organizations work together to create aseamless, hassle-free experience for everyone involved.

A DATA LANDSCAPE LACEDWITH UNIQUE COMPLEXITIESThe combination of agile, automation, and AI tools is birthing
intelligent systems, that have healthy infrastructure andprocesses on which we can run analytics, detect anomalies, anddecrease costs while using the time of the workforce optimally.

One of the trends we see is the acknowledgment of the value
behind data. This sector clearly understands the power ofstoring, analyzing, and controlling the data with the right set ofpeople and organizations while also dealing with the increasing
complexity of data privacy.

We expected these transformations to drive fundamental
changes in the role of quality engineers in the public sector andthey have. Testers are now closer to the engineers and coding
processes, blurring the lines between testing, development,
and support of the systems.
79

80

World Quality Report I 2023-24

CREATING PRODUCTCHAMPIONS OUT OF QUALITY
ENGINEERS
While the changing dynamics in this QE ecosystem might lookexciting, a big challenge is the lack of IT champions who canrepresent IT at the highest levels in organizations – somethingthat can hold back innovation and introduce risk to programsand the wider organizations and their customers.
We are in serious need of peer-level engineers in the boardroom,making sure that the views of testing and the rest of the IT
functions are heard and understood. They need to have a say
and the authority to make decisions around core IT services.
What would help is to take technical experts and convert
them into product owners for the technology to have relevant
conversations and make meaningful contributions.
In fact, this approach can also help in building non-functional
characteristics like sustainability alongside reliability, scalability,
and availability.
When product champions are involved in the decision-making,they can integrate sustainability into the development processfrom the beginning, thereby reducing public sector organizations’environmental impact.
We know that gratifying the various agencies of the public sector
is not easy, but we have ample evidence to show that the sooner
we adapt to these principles and technologies, the better it isfor everyone.
With the rise of AI and the disruption that it is already bringing,
we see product champions who understand the technology
are making a difference and this is starting to be recognized in
the public sector, a trend we hope to continue to see in theyears to come.

Survey Watch

83%

Respondents denoted 25-50% of theirorganizations have adopted QE

60%Organizations said the top-most issue
in agile adoption was lack of time to test

43%

Respondents said that cloud andinfrastructure testing is not followed asa standard practice59%

Organizations think QE&T can play ahuge role in achieving sustainability goals81

TELECOM OFTOMORROW
How invested is media and telecom in QE?

82

World Quality Report I 2023-24

83• ASH PANDEY
Vice President, Head of Telco, Capgemini UK

In our opinion, even though telcos are cautiously investing inthis space, especially in Generative AI, we think it is poised to
become an indispensable part of their toolkit in the next three
to four years.

• ARUN SANTHANAM
Vice President, Head of Telco, Capgemini USA

But it is not a uniform industry-wide trickle-down effect. Whilesome regions such as the United Kingdom and Europe are early
adopters of technology in telco, others comparatively lag.

• SIDHARTH KAPILA
Vice President & COO Technology Market Unit,
Capgemini USA

As early adopters pave the way for innovation, we notice thatmost of their decisions revolve around migrating features andeven offshoring some roles to optimize processes, highlighting ashift from cost-focused decision-making. Today, the need seems
to lean towards a nuanced approach that combines automation
with process redesign.
Interestingly, this trend is being driven by telcos that are
increasingly collaborating with startups and incubators to findtechnology-driven solutions that can replace manual processes
to reinvent how the industry functions.

Did you know that there are more mobile subscribers today
than people on this planet? A recent report by the International
Telecommunication Union (ITU) revealed that there wereapproximately 7.9 billion mobile cellular subscriptions worldwide
and this number continues to grow rapidly each year.
The telecom sector is the backbone of the globally interconnectedecosystem we live in, and quality engineering (QE) plays a criticalrole in creating and maintaining the innovative technologies on
which the industry relies.
Through this year’s World Quality Report (WQR) survey wewanted to understand how artificial intelligence (AI) andautomation are increasingly becoming mainstream, and how
the industry is adopting newer trends in data management,
customer centricity, and sustainability.

ARE WE WITNESSING THE
GLORIOUS REIGN OF AI?
Given the fiercely competitive nature of the industry, telcos now
are hyper-focusing on things that really matter. Many market
leaders are seeking ways to eliminate unnecessary transactions
and reduce operational complexities through the strategic useof AI and automation.
Both these technologies are being used as tools to optimize
processes or even completely redesign and reimagine businessprocesses when required. We saw this coming, considering
how these technologies have delivered exemplary efficiency,
predictability, and cost savings.
84

World Quality Report I 2023-24

ARE BUREAUCRACY AND
RED TAPE IMPEDING THE
MOMENTUM?
Telecom is notoriously characterized by its caution and budget
constraints. Known for their conservative approach to spending,they are hesitant to disrupt existing systems that, while not
perfect, are functional. This makes perfect sense given theindustry’s complex decision-making processes that often involve
multiple stakeholders and consulting companies, result in much
longer timelines for implementing changes.
Yet, despite the cost-conscious environment, it is incrediblyreassuring to see telcos making significant investments insoftware and subscription-based services. This reflects a shift
toward delivering products and services faster to the market,
embracing high-tech product life cycles, and focusing on theentire value chain, including testing.

A WAVE OF TRANSFORMATION
Like many other sectors, testing functions in this space are now
being decentralized. Telcos are realizing the need to distributetesting functions across different regions, enabling them toharness local expertise and streamline processes.
The industry is also in the process of re-evaluating several quality
benchmarks and defining quality drivers by including test data,
data validation, cloud platforms, and orchestration.
Interestingly, we are also noticing a shift away from end-to-end
testing platforms towards domain-based testing. With a focus
on integration with development, large-scale, in-house testing
centers of excellence (Data Governance Center of Excellence) arediminishing, especially in the United States, in favor of integrated
domain-specific testing approaches.
Parallely, the decoupling of media and connectivity is emerging as
an important part of the conversation. While telecom previously
integrated media as a part of the business, we think it is now
time for platform players to focus on B2B, provide connectivity
services, and align with hyperscalers to create renewed supply
chains and marketplaces instead.
In contrast to these developments, we note that data security
and integration continue to be a major threat keeping telcos
from unlocking the true value of data for testing and achieving
broader business purposes.

IN CONCLUSION
We think in order to thrive, businesses must invest in Automation
and AI, cultivate innovation through collaborations, strategicallypartner to optimize supply chains, and harness data ecosystems
for enhanced testing and valuable business insights. But whilethese measures hold the possibility of a promising future, only
those who strategically apply them and transform with changing
times stand to reap rewards. After all, as the legendary Chinese
philosopher Kong Fuzi (Confucius) once said, “Only the wisestor the stupidest refuse to change.”

And as far as broader business purposes are concerned,
awareness regarding sustainability and efforts to make it anactive element of business models are coming to the forefront.
“Overall, we know that cost-consciousness is what drives telcos
to optimize their digital supply chains, but surprisingly, it is also
what is enhancing sustainability efforts along the way.”
There has been a growing emphasis on green quality engineeringand testing to contribute to sustainability goals in telecom. It isrefreshing to see how the industry acknowledges quality andtesting as essential components in achieving compliance andsustainability goals.
In fact, automation has been repeatedly highlighted as a key
factor directly contributing to cost savings and indirectly tosustainability by reducing time cycles for product releases inour client discussions.

Survey Watch

41%

Organizations cite sustainability
concerns as the biggest roadblockto Gen AI

80%

Organizations say they consistently
invest in AI to better quality

55%

Organizations are using automation
for risk reduction

45%

Organizations consider CI/CD
and Orchestration as the top skills for
quality engineers

85

86

World Quality Report I 2023-24

ABOUT
THE STUDYWorld Quality Report 2023-24
The World Quality Report 2023-24 is based on research ﬁndings from 1,750 interviews
carried out during June and July 2023 by Coleman Parkes Research. The average lengthof each interview was 30 minutes and the interviewees were all senior executives incorporate IT management functions, working for companies and Public Sector organizations
across 32 countries.
The interviews this year were based on a questionnaire of 46 questions, with the actualinterview consisting of a subset of these questions depending on the interviewee’s rolein the organization. The quantitative research study was complemented by additional
in-depth interviews to provide greater insight into certain subject areas and to informthe analysis and commentary. The main themes for all survey questions remained thesame, though a few objective responses were also added for the ﬁrst time this year.
Quality measures were put in place to ensure the questionnaire was understood, answered
accurately and completed in a timely manner by the interviewee.
Research participants were selected so as to ensure sufﬁcient coverage of different regions
and vertical markets to provide industry-speciﬁc insight into the quality assurance andtesting issues within each sector.
To ensure a robust and substantive market research study, the recruited sample must be
statistically representative of the population in terms of its size and demographic proﬁle.
The required sample size varies depending on the population it represents – usually
expressed as a ratio or incidence rate. In a business-to-business (B2B) market researchstudy, the average recommended sample size is 100 companies. This is lower than the
average sample size used for business-to-consumer (B2C) market research because wholeorganizations are being researched, rather than individuals.
As mentioned above, the B2B market research conducted for the World Quality Report2023-24 is based on a sample of 1,750 interviews from enterprises with more than 1,000
employees (23%), organizations with more than 5,000 employees (33%) and companies
with more than 10,000 employees (35%). The approach and sample size used for the
research this year enables direct comparisons of the current results to be made withprevious research studies conducted for the report, where the same question was asked.
During the interviews, the research questions asked of each participant were linked to
the respondent’s job title and the answers he/she provided to previous questions whereapplicable. For this reason, the base number of respondents for each survey question
shown in the graphs is not always the full 1,750 sample size.
The survey questionnaire was devised by Quality Engineering experts in Capgemini, Sogeti,
and OpenText (sponsors of the research study), in consultation with Coleman Parkes
Research. The 46-question survey covered a range of software quality engineering anddigital assurance subjects. The analysis of the survey results was enriched by qualitative
data obtained from the additional in-depth interviews.
87

INTERVIEWS BY SECTORSWQR 2023- Global Results

19%

15%

Hi-Tech including
start ups,
hardware vendors +Aerospace &DefenceFinancial Services
industry, including
Capital Markets, Banking
and Insurance

Healthcare &Life Sciences

8%

Automotive (Manufacturing)

7%

Public Sector/
Government

Energy, Utilities,
and Chemicals

7%

11%

Manufacturing,(incl. industrial product& excl. Automotive)

Telecommunications,Media & Entertainment

11%

10%

88

World Quality Report I 2023-24

7%

Consumer goods &retail/ Distribution& Logistics

Transportation

6%

INTERVIEWS BY JOB TITLEWQR 2023- Global Results

CIO

24%

QA Testing Manager or
Testing Manager or Quality
Engineers

18%

IT Directors

17%

VP Applications/Product Owner

15%

CTO/Product Head10%

VP or Director of Research &Development/Engineering

10%

CMO/CDO

6%

89

INTERVIEWS BY COUNTRY

350

North
AmericaSouthern
Europe66
179

Nordics

99

EasternEurope65

WesternEurope

138
LATAM

80
158

90

World Quality Report I 2023-24

UK andIreland

BeNeLux

345

MiddleEast Asia

WQR 2023- Global Results

32 COUNTRIES
USA

16%

France170

South
East Asia

Australia &New Zealand

100

9%

Germany

7%

UK

7%

Netherlands

6%

Australia

5%

Sweden5%

Brazil

5%

China

4%

Canada

4%

Japan

3%

Italy

3%

Spain

2%

Portugal2%

Belgium & Luxembourg

2%

Czech Republic2%

Hungary

2%

Poland2%Finland2%

Denmark

2%

Norway2%

Ireland2%

Singapore

1%

Hong Kong

1%

Switzerland

1%

UAE(excluding Dubai, Abu Dhabi)

1%

Jordan & Bahrain

1%

Saudi Arabia

1%

Abu Dhabi

1%

Dubai

1%

Qatar

1%

New Zealand

1%

91

ABOUT
THE SPONSORSWorld Quality Report 2023-24
About Capgemini and Sogeti

About OpenText

Capgemini is a global leader in partnering with companies totransform and manage their business by harnessing the powerof technology. The Group is guided every day by its purposeof unleashing human energy through technology for an
inclusive and sustainable future. It is a responsible and diverse
organization of nearly 350,000 team members in more than 50
countries. With its strong 55-year heritage and deep industry
expertise, Capgemini is trusted by its clients to address theentire breadth of their business needs, from strategy and design
to operations, fueled by the fast evolving and innovative worldof cloud, data, AI, connectivity, software, digital engineering,and platforms. The Group reported in 2022 global revenuesof €22 billion.

OpenText, The Information Company™, enables organizations togain insight through market leading information management
solutions, powered by OpenText Cloud Editions.

Part of the Capgemini Group, Sogeti makes business value
through technology for organizations that need to implement
innovation at speed and want a local partner with global scale.
With a hands-on culture and close proximity to its clients,
Sogeti implements solutions that will help organizations work
faster, better, and smarter. By combining its agility and speedof implementation through a DevOps approach, Sogeti delivers
innovative solutions in quality engineering, cloud and application
development, all driven by AI, data and automation.
Visit us at
Get The Future You Want | www.capgemini.com
www.sogeti.com

92

World Quality Report I 2023-24

For more information about OpenText(NASDAQ: OTEX, TSX: OTEX) visit www.opentext.com

THANK YOU
Capgemini, Sogeti, and OpenText
would like to thank*All research carried out by Coleman Parkes Researchis conducted in compliance with the Code of Conduct andguidelines set out by the MRS in the UK, as well as the legal
obligations under the Data Protection Act 1998.

The 1,750 IT executives who took part in the research study thisyear for their time and contribution to the report. In accordancewith the UK Market Research Society (MRS) Code of Conduct(under which this survey was carried out), the identity of theparticipants in the research study and their responses remain
confidential and are not available to the sponsors.
All the business leaders and subject matter experts who provided
valuable insight into their respective areas of expertise andmarket experience, including the authors of the country andindustry sections and subject-matter experts from Capgemini,
Sogeti and OpenText.

Main Report Authors
Ashwini Nadiger, Kavitha Velchal, Jeff Spevacek, Andrew Fullen,
Anish Behanan, Jeba Abraham, Jean-Baptiste Bonnet
Program Management
Dennis John
Creative Director
Sujoy Karmakar
Content Strategist
Shardul Shaligram
Design Team (MACS Agile)
Partha Pal, Sourav Karmakar

www.worldqualityreport.com©2023 Capgemini, Sogeti and OpenText.
All Rights Reserved.
Capgemini and OpenText, and their respective marks
and logos used herein, are trademarks or registered trademarksof their respective companies. All other company, productand servicenames mentioned are the trademarks of theirrespective owners and are used herein with no intention of
trademark infringement. Rightshore® is a trademark belonging
to Capgemini. TMap®,TMap NEXT®, TPI® and TPI NEXT® areregistered trademarks of Sogeti, part of the Capgemini Group
Rightshore® is a trademark belonging to Capgemini. TMap®,
TMap NEXT®, TPI® and TPI NEXT® are registered trademarksof Sogeti, part of the Capgemini Group.

Writing and Editorial
Sneha Christuraj, Arshbir Kaur, Divya Arya, Arijit Sen
Marketing & Comms. (Capgemini)
Dennis John
Marketing & Comms. (OpenText)
Regina Powell, Grace Bickerstaff
Project Management
Jay Kartha
Market Research
Coleman Parkes Research*93

PREVIOUS EDITIONS

94

2009-10

2010-11

2011-12

2012-13

1st edition

2nd edition

3rd edition

4th edition

2016-17

2015-16

2014-152013-14

8th edition

7thedition

6th edition

5th edition

2017-18

2018-192019-20

2020-21

9th edition

10th edition

11th edition

12th edition

2022-232021-2214th edition

13th edition

World Quality Report I 2023-24

OpenText
Regina Powell
ADM Product Marketing Leadrpowell@opentext.com

Grace Bickerstaﬀ
ADM Product Marketing Manager
gbickerstaﬀ@opentext.com

SogetiMark Buenen
Global Leader, Quality
Engineering & Testing
Capgemini & Sogeti
mark.buenen@sogeti.com

Sathish Natarajan
Group Vice President, Quality Engineering& Testing, Capgemini & Sogeti Americas
Sathish.n@capgemini.com

Parinita Patankar
Vice President,
Quality Engineering &Testing, Capgemini India
parinita.patankar@capgemini.com

Capgemini
Anand Moorthy
Vice President & Head of Quality
Engineering & Testing,
Financial Services, Capgemini
anand.moorthy@capgemini.com

Vice President, Financial Services,
Quality Engineering & Testing,
Americasshivakumar.balasubramaniyan@capgemini.com

Kirthy Chennaian
Vice President & Head of Quality Engineering& Testing, Capgemini Americas
kirthy.chennaian@capgemini.com
in association with:

MACS-ODS_07-11-2023_Partha Pal

Shivakumar Balasubramaniyan