# Human-AI Collaboration in Quality Control with Augmented Manufacturing Analytics

Alexandros Bousdekis, Stefan Wellsandt, Enrica Bosani, Katerina Lepenioti, Dimitris Apostolou, Karl Hribernik, Gregoris Mentzas

► To cite this version:

Alexandros Bousdekis, Stefan Wellsandt, Enrica Bosani, Katerina Lepenioti, Dimitris Apostolou, et al.: Human-AI Collaboration in Quality Control with Augmented Manufacturing Analytics. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.303-310, 10.1007/978-3-030-85910-7_32. hal-03806493

HAL Id: hal-03806493

https://inria.hal.science/hal-03806493

Submitted on 7 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Human-AI Collaboration in Quality Control with
Augmented Manufacturing Analytics

Alexandros Bousdekis¹, Stefan Wellsandt², Enrica Bosani³, Katerina Lepenioti¹,
Dimitris Apostolou¹, Karl Hribernik² and Gregoris Mentzas¹

¹ Information Management Unit (IMU), Institute of Communication and Computer Systems (ICCS), National Technical University of Athens (NTUA), Greece
{albous, klepenioti, dapost, gmentzas}@mail.ntua.gr

² BIBA - Bremer Institut für Produktion und Logistik GmbH
at the University of Bremen, Germany
{wel, hri}@biba.uni-bremen.de

³ Whirlpool EMEA, Benton Harbor, USA
enrica_bosani@whirlpool.com

**Abstract.** Augmented analytics is an emerging topic which deals with the enhancement of analytics with conversational interfaces as well as the exploitation of the human knowledge representation through intelligent digital assistants allowing users to easily interact with data and insights. The communication with the user by voice poses new challenges to the development and execution of data analytics services. In this paper, we outline a framework for implementing quality analytics for decision augmentation through optimized human-AI interaction. Our approach aims to reduce the number of quality issues through fast, mobile, and easy access to quality predictions for products and processes. An application case is the production of white goods is presented.

**Keywords:** Quality control, augmented analytics, human-AI collaboration

# 1 Introduction

Quality of products and processes concerns more and more the manufacturing firms because negative consequences do not show up until the product is actually produced or worse, until the customer returns it [1]. The acceleration of technological growth in the context of Industry 4.0 does not automatically result in improvements in manufacturing quality. The products' complexity (i.e. number of components and their relations) and variety is continuously increasing, while the traditional quality management practices, such as Material Requirements Planning, Lean Manufacturing, Theory of Constraints and Six Sigma, have lost their effectiveness [2]. The future of quality management is to be proactive: adverse effects of quality flaws have to be prevented before they become evident in the actual use of a product [1, 3] or even before they cause inefficiencies in the production process. This trend is significantly facilitated by the latest advancements in machine learning [4] which dictate that production is gauged based upon the product quality, the process quality, and the quality of the services provided surrounding the product. To this end, predictive quality is an ap-
proach that moves beyond traditional quality evaluation methods towards extracting
useful insights from various data sources by determining patterns, revealing correla-
tions between products and defects and predicting future outcomes (e.g. product de-
fects and fault localization) [1-3, 5].

Predictive quality approaches can be further enhanced by extending data analytics with augmented analytics. Augmented analytics aims at optimizing the use of data for decision making in order to augment human intelligence and contextual awareness [6]. It brings automation to the complete analytics cycle [7] by leveraging AI algorithms in order to transform how analytics content is developed, consumed and shared. Augmented analytics uses natural language processing and conversational interfaces, allowing all users to interact with AI through spoken and written language, without requiring advanced skills [6], something which has long been a goal of AI researchers [8]. On the other hand, prescriptive analytics moves beyond the provision of meaningful insights about the current or the anticipated states (e.g. prediction of future defects) and aims at prescribing the best decision options within time constraints in order to take advantage of the predicted future, e.g. to eliminate or mitigate the impact of a future undesired event [9, 10]. With augmented analytics, conversational interfaces can provide users fast access to non-intrusive recommendations and real-life feedback from the shop floor. Moreover, intelligent digital assistants allow users to interact with data and insights easily.

In this paper, we focus on Augmented Manufacturing Analytics (AMA). We present a voice-enabled Digital Intelligent Assistant (DIA), which interacts with a prescriptive quality analytics service. The assistant’s interface aims to allow workers to access and customize quality predictions and the prescribed mitigating measures. Our approach aims to reduce the number of quality issues through fast, mobile, and easy access to quality predictions for products and processes. The application case is the production of white goods. The next section briefly overviews the current state of the art in business analytics and digital intelligent assistants. Section 3 presents our conceptual framework for augmented quality analytics and highlights some of our innovation objectives. The fourth section outlines the case study in quality management in the home appliances industry, while the fifth section presents the concluding remarks.

## 2 State of the Art

### 2.1 Augmented Analytics

Companies have at their disposal staggering amounts of data, which, if analyzed and processed properly, can generate valuable insights and lead to better decisions. The use of advanced techniques that can analyze and process large and diverse data sets that include structured, semi-structured and unstructured data, from different sources, and in different sizes from terabytes to zettabytes, has led to the field of business analytics [11]. The data analytics lifecycle consists of three phases: description, prediction, and prescription. While descriptive analytics analyzes past events, predictive analytics predicts future events – both do not provide direct support for decision-
making [12, 13]. Prescriptive analytics, on the other hand, is a newer data analytics
type enabling data-driven optimization for decision support and planning [9]. Pre-
scriptive analytics has the potential to provide the greatest benefit for business by
providing insights about proactive mitigating actions for the predicted undesired
events [14]. Current solutions, however, require that users have higher skills in data
science and machine learning, which impairs wide adoption. To address this barrier,
combining prescription with augmented analytics can enable users with lower data
science and machine learning skills to find and surface the most important insights or
changes in the business by interacting through spoken and written language taking
advantage of natural language processing and conversational interfaces [15].

## 2.2 Digital Intelligent Assistants

Digital Intelligent Assistants (DIAs) emerged from fragile niche applications to eve-
ryday helpers. Consumers use them for home automation, to schedule appointments,
and to search for facts. There are various voice-based assistants such as Amazon
Alexa, and text-based assistants (chatbots) [16, 17]. Market researchers expect that
AI-based digital assistants will become a key element in the future of work [18, 19].
A voice-enabled DIA processing pipeline has four core components [20, 21]: Speech-
to-Text (STT) to transcribe voice inputs, Natural Language Understanding (NLU) to
extract intents and entities, Dialog Management (DM) to track dialog states and de-
cide the next actions, and Text-to-Speech (TTS) to generate a computer voice output.
Morana et al. classified DIA's along two dimensions: (1) the degree of interactivity
enabled by the user assistance, and (2) the degree of intelligence of user assistance
[22]. The former characterizes the DIA's capability to support humans in an ongoing
dialog and the latter describes its capability to assist the user considering, for instance
personal and task characteristics.

Applications in manufacturing include, for instance, smart voice-enabled digital
assistants for human-robot communication [23]. Although further research is still
needed for a more accurate voice recognition and conversational intelligence, the
technology is mature enough for running experiments [25, 26] and building commer-
cial applications in manufacturing, such as Spix<fnref n="1" />. Realizing DIA in this domain is
challenging though. Achieving robustness under industrial operating conditions, clari-
ty about an assistant's accountability, acceptance among employees, ethics issues, and
data security risks are some examples that slow down wider adoption [27]. Tools and
frameworks for building DIAs, however, prosper. Many global IT leaders, such as
Google, Amazon, Microsoft, Baidu, or IBM offer tools to design, develop, deploy,
and maintain assistants. A noteworthy effort in the Open Source domain is Mycroft²,
which offers a privacy-centred digital assistant framework. Their solution offers ex-
changeable natural language processing modules, speech-to-text, intent parsing, and
text-to-speech.

<footnote n="1">¹ https://www.simsoft-industry.com/en/intelligent-vocal-assistant-for-my-industry/
² https://mycroft.ai</footnote>
3 Augmented Manufacturing Analytics Framework for
Human-AI Collaboration in Quality Control

Our framework (Fig. 1) implements quality analytics for decision augmentation through optimized human-AI interaction. The framework covers the whole data analytics lifecycle (descriptive, predictive, and prescriptive analytics) aiming at extracting increased value from quality data and prescribing appropriate mitigating actions through a voice-enabled DIA. As such, it can support tasks related to product and process quality control, such as detecting abnormal behaviors and root causes of defects, predicting their consequences in terms of product and process quality and prescribing appropriate actions in the form of voice-first advices. Specifically, the framework foresees stream processing to perform real-time data processing for: (i) detecting potential sources of defects and revealing correlations between products and defect rates; (ii) predicting future quality issues and their consequences; and (iii) prescribing mitigating actions to optimize relevant manufacturing performance indicators, such as Overall Equipment Effectiveness (OEE), uptime and scrap rate.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Augmented Manufacturing Analytics framework</figcaption></figure>

We use the DIA to recognize worker intents, questions, and instructions in the in-
dustrial environment. The DIA has to manage typical interaction constraints in manu-
facturing, such as factory noise, multiple languages, jargon, and workers wearing
earplugs, masks, safety googles, and safety gloves. Its STT component has to be reli-
able under these conditions, which is a technical challenge. Reliable solutions ground
on thousands of hours of audio data and their transcription – each supported language
needs its own dataset. Therefore, STT model training is costly and companies, such as
Google, Amazon, Apple, and Baidu, invested millions of Euros in this field to devel-
op proprietary solutions. Their closed solutions are a cause for ethical concerns be-
cause the training data are not transparent and biases hard to identify. Open solutions
are not as reliable yet, but projects such as DeepSpeech<fnref n="3" /> and CommonVoice<fnref n="4" /> develop
transparent alternatives. The application of trustworthy STT mechanisms is challeng-
ing and it is not clear how successful open state of the art technology can be.

Furthermore, the DIA has to address: i) data security requirements of manufactur-
ing environments through data access management, traceability, and encryption; ii)
data privacy requirements derived from the legal frameworks through a privacy-by-
design approach with clear data ownership, data processing scope, and user consent
management. The assistant needs to operate on mobile devices to be usable wherever
and whenever the user needs it. Since time pressure is a typical work condition, the
DIA's dialogs must be fast, unambiguous, and easy to use. Voice interactions are
inherently fast, potentially hands-free, and we assume they are also easier to learn
than interactions via graphical interfaces. For multi-dimensional analytics results (e.g.
graphs or tables) voice can be inefficient because of the slower audio processing on
the human side. For these cases, the assistant needs access to graphical interfaces.
Relevant interaction options include voice-based drill-down, drill-up, and customiza-
tion options for the analytics processes. Wake words should activate the assistant for
hands-free interactions.

# 4 Case study on Quality Control

The case study of our approach concerns the quality control procedures of Whirlpool,
one of the leading companies in the home appliances industry, with around 92,000
employees and over 70 manufacturing and technology research centers worldwide.
The use case will address the end-of-line quality control with human-AI collaboration
with the aim to adopt a predictive quality strategy that will link the quality control of
the finished product with the design stage and the shop floor.

## 4.1 As-is situation

In the Whirlpool production model, the whole white goods production is tested from
quality and safety point of view in order to ensure a high standard level of product
quality to final customers. These tests are executed in all factories either through the
usage of automatic dedicated machines at the end of production line or through auto-
matic, semiautomatic or manual checks in some critical workstation along the produc-
tion flow (visual quality checks, quality gates). To these testing actions, Whirlpool
Production system adds also some statistical quality check actions that are applied
both on internal production parts on quality critical processes (statistical process con-
trol stations) and on finished goods, after the packaging process. In particular, this last
testing, called Zero Hour Testing (ZHT) is referring to the Statistical Quality Control
applied in a dedicated laboratory out of production flow on some finished products
retrieved from the quantities ready to be delivered to the markets; see Fig. 2.

<footnote n="3">³ https://github.com/mozilla/DeepSpeech/wiki</footnote>

<footnote n="4">⁴ https://commonvoice.mozilla.org/en</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 2. Zero Hour Testing (ZHT) laboratory at Whirlpool</figcaption></figure>

The main objectives of ZHT are to measure the quality level of the outgoing prod-
uct from an aesthetic, functional, and normative point and to measure the effective-
ness of process control. These tests are executed in dedicated laboratory environment,
created in each production site, and following a specific STD operating procedure.
This testing method is designed to replicate the customer approach to the product,
simulating the normal product usage conditions at final customer first usage. Current-
ly, the procedure is executed manually by a laboratory operator and it is fixed, stati-
cally defined during process design phase both for what concerns checklist and refer-
ence parameters, and for statistical product withdrawal rate.

## 4.2 To-be scenario

With our proposed framework we aim at adopting a predictive quality strategy that will link the quality control of the finished product with the design stage and the shop floor. By integrating all available information sources (e.g. sensor data, historical operation-al data, and expert knowledge), the factory will be able to a) predict low-quality products and b) to plan effective control actions to mitigate the impact. In this way, Whirlpool will be able to effectively capture defects risk and proactively initiate resolution process before it may impact on the final customer level. The support to decision making process in mitigation actions identification and implementation will ensure a prompt reaction of the overall Whirlpool quality network system, ensuring fast and effective reconfiguration with the modification of control points to capture un/conformities. A second important contribution is in the facilitation of root cause analysis that, due to the high complexity of white goods production process (high number of components, several quality-critical processes within production flow, high production pace, high variety of product range), is very often difficult to be executed at shop floor level. An interactive system collaborating with workers and leading into a deep dive analysis, will facilitate the identification of early signal of quality derail-ing effects and the possibility to consolidate knowledge to be used for the future. An augmented human-AI interaction is of outmost importance to enhance the shop floor workers' capabilities in identifying potential failures, investigating root-causes, and addressing the causes effectively, allowing them the opportunity to fully leverage on whole data availability to anticipate the events instead of reacting the events
# 5 Conclusions and further work

Augmented analytics is an emerging topic which deals with the enhancement of analytics with conversational interfaces as well as the exploitation of the human knowledge representation through intelligent digital assistants allowing users to easily interact with data and insights. The communication with the user by voice poses new challenges to the development and execution of data analytics services. Apart from visualization dashboards, the data analytics outcomes should be structured in a way that can be translated to speech. On the other way around, they should be able to take as input parameters that are derived from the human speech. In this paper, we outlined a framework for implementing quality analytics for decision augmentation through optimized human-AI interaction. The coupling of analytics with augmentation is in line with the overall trend towards human-AI osmosis for Operator 4.0 as a grand challenge [28].

**Acknowledgements.** This work is partly funded by the European Union's Horizon 2020 project COALA "COgnitive Assisted agile manufacturing for a LAbor force supported by trustworthy Artificial Intelligence" (Grant agreement No 957296). The work presented here reflects only the authors' view and the European Commission is not responsible for any use that may be made of the information it contains.

# References

1. Nalbach, Oliver, Christian Linn, Maximilian Derouet, and Dirk Werth. "Predictive quality: Towards a new understanding of quality assurance using machine learning tools." In International Conference on Business Information Systems, pp. 30-42. Springer, Cham (2018).

2. Bai, Yun, et al. "Manufacturing quality prediction using intelligent learning approaches: A comparative study." *Sustainability* 10.1, 85, (2018).

3. Berger, Dietrich, et al. "Predictive quality control of hybrid metal-CFRP components using information fusion." *Production Engineering* 12.2: 161-172 (2018).

4. Gunasekaran, A., N. Subramanian, and Wai Ting Eric Ngai. "Quality management in the 21st century enterprises: Research pathway towards Industry 4.0.", 125-129 (2019).

5. Gittler, Thomas, et al. "Towards predictive quality management in assembly systems with low quality low quantity data–a methodological approach." *Procedia CIRP* 79, 125-130 (2019).

6. Prat, Nicolas. "Augmented analytics." *Business & Information Systems Engineering* 61.3, 375-380 (2019).

7. Gartner Inc.: "When and How to Combine Predictive and Prescriptive Techniques to Solve Business Problems", Published: 25 October 2018 ID: G00368423 (2018)

8. Sangaiah, Arun Kumar, Arunkumar Thangavelu, and Venkatesan Meenakshi Sundaram. "Cognitive computing for big data systems over IoT." *Gewerbestrasse* 11, 6330 (2018)

9. Lepenioti, Katerina, Alexandros Bousdekis, Dimitris Apostolou, and Gregoris Mentzas. "Prescriptive analytics: Literature review and research challenges." *International Journal of Information Management* 50, 57-70 (2020).

10. Bertsimas, Dimitris, and Nathan Kallus. "From predictive to prescriptive analytics." *Management Science* 66, no. 3, 1025-1044 (2020).
11. Davenport, Thomas H. "Competing on analytics." Harvard business review 84, no. 1 (2006).

12. LaValle, Steve, Eric Lesser, Rebecca Shockley, Michael S. Hopkins, and Nina Kru schwitz. "Big data, analytics and the path from insights to value." MIT sloan management review 52, no. 2, 21-32 (2011).

13. Chen, Hsinchun, Roger HL Chiang, and Veda C. Storey. "Business intelligence and ana lytics: From big data to big impact." MIS quarterly, 1165-1188 (2012).

14. Frazzetto, Davide, Thomas Dyhre Nielsen, Torben Bach Pedersen, and Laurynas Šikšnys. "Prescriptive analytics: a survey of emerging trends and technologies." The VLDB Journal 28, no. 4, 575-595 (2019).

15. Gartner, "Augmented Analytics Is the Future of Data and Analytics", Published: 31 Octo ber 2018, ID: G00375087 (2018).

16. Maedche, A., Morana, S., Schacht, S., Werth, D. and Krumeich, J., "Advanced user assis tance systems", *Business & Information Systems Engineering*, 58(5), pp.367-370 (2016).

17. Maedche, A., C. Legner, A. Benlian, B. Berger, H. Gimpel, T. Hess, O. Hinz, S. Morana, and M. Söllner. "AI-based digital assistants: opportunities, threats, and research perspec tives. Business & Information Systems Engineering, 61: 535–544." (2019).

18. Gartner Newroom, "Gartner Predicts 25 Percent of Digital Workers Will Use Virtual Em ployee Assistants Daily by 2021", January 9, 2019, https://www.gartner.com/en/newsroom/press-releases/2019-01-09-gartner-predicts-25-percent-of-digital-workers-will-u, last accessed 2021/03/02.

19. Bradley, A. "Brace Yourself for an Explosion of Virtual Assistants", Gartner Blog Post, August 10 (2020), https://blogs.gartner.com/anthony_bradley/2020/08/10/brace-yourself-for-an-explosion-of-virtual-assistants/, last accessed 2021/03/02.

20. Deriu, Jan, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. "Survey on evaluation methods for dialogue systems." Artificial Intelligence Review 54, no. 1 (2021): 755-810.

21. Maedche, Alexander, Christine Legner, Alexander Benlian, Benedikt Berger, Henner Gimpel, Thomas Hess, Oliver Hinz, Stefan Morana, and Matthias Söllner. "AI-based digital assistants." Business & Information Systems Engineering 61, no. 4 (2019): 535-544.

22. Morana, Stefan, Jella Pfeiffer, and Marc TP Adam. "User assistance for intelligent sys tems.", Business Information Systems Engineering, 62(3):189–192 (2020).

23. Ghofrani, Javad, and Dirk Reichelt. "Using Voice Assistants as HMI for Robots in Smart Production Systems." *CEUR Workshop Proc.* Vol. 2339. 2019.

24. Longo, Francesco, and Antonio Padovano. "Voice-enabled Assistants of the Opera-tor 4.0 in the Social Smart Factory: Prospective role and challenges for an advanced human- machine interaction." Manufacturing Letters 26, 12-16, (2020).

25. Abner, Brunno, Ricardo J. Rabelo, Saulo P. Zambiasi, and David Romero. "Production Management as-a-Service: A Softbot Approach." In IFIP International Conference on Ad vances in Production Management Systems, pp. 19-30. Springer, Cham, 2020.

26. Rabelo, Ricardo J., Saulo Popov Zambiasi, and David Romero. "Collaborative softbots: enhancing operational excellence in systems of cyber-physical systems." In Working Con ference on Virtual Enterprises, pp. 55-68. Springer, Cham, 2019.

27. Wellsandt, Stefan, Mina Foosherian, and Klaus-Dieter Thoben. "Interacting with a Digital Twin using Amazon Alexa." Procedia Manufacturing 52 (2020): 4-8.

28. Bousdekis, Alexandros, Dimitris Apostolou, and Gregoris Mentzas. "A human cyber phys- ical system framework for operator 4.0–artificial intelligence symbiosis." Manufacturing Letters 25, 10-15 (2020).