# Towards Active Learning Based Smart Assistant for Manufacturing

Patrik Zajec, Jože M. Rožanec, Inna Novalija, Blaž Fortuna, Dunja Mladenić,
Klemen Kenda

► To cite this version:

Patrik Zajec, Jože M. Rožanec, Inna Novalija, Blaž Fortuna, Dunja Mladenić, et al. Towards Active Learning Based Smart Assistant for Manufacturing. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2021, Nantes, France. pp.295-302, 10.1007/978-3-030-85910-7_31. hal-03806482

HAL Id: hal-03806482

https://inria.hal.science/hal-03806482

Submitted on 7 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Towards Active Learning Based Smart Assistant for Manufacturing

Patrik Zajec¹,²*, Jože M. Rožanec¹,²,³*[0000-0002-3665-639X], Inna Novalija²[0000-0003-2598-0116], Blaž Fortuna²,³, Dunja Mladenić², and Klemen Kenda¹,²,³[0000-0002-4918-0650]

¹ Jožef Stefan International Postgraduate School, Jamova 39, 1000 Ljubljana, Slovenia,

joze.rozanec@ijs.si,

² Jožef Stefan Institute, Jamova 39, 1000 Ljubljana, Slovenia

³ Qlector d.o.o., Rovšnikova 7, 1000 Ljubljana, Slovenia

**Abstract.** Smart assistants in manufacturing can guide and aid on decision-making while also provide means to collect additional insights and information available to the users. A general approach for building a smart assistant that provides users with machine learning forecasts and a sequence of decision-making options is presented in this work. The system provides means for knowledge acquisition by gathering data from users. To minimize interactions and friction with users, we envision active learning can be used to get data labels for most data instances expected to be most informative. The system is demonstrated on a demand forecasting use case in manufacturing. The methodology can be extended to several use cases in manufacturing.

**Keywords:** smart assistant · artificial intelligence · machine learning · demand forecasting · knowledge acquisition · active learning

## 1 Introduction

The increasing digitalization of manufacturing has accelerated the information flow. Technologies such as Cyber-Physical Systems (CPS), Industrial Internet of Things (IIoT), and Artificial Intelligence (AI) are bringing an extensive added value into Industry 4.0 value chains [6]. In particular, AI has been successfully researched and applied on several manufacturing tasks (e.g., predictive maintenance, production simulation, and production planning). The advancement of AI and its applications in manufacturing is conditioned by data availability. Though much data is available from software, such as Enterprise Resource Planning (ERP) or Manufacturing Execution Systems (MES), many aspects are not captured by sensors or such software. An example of such information is the collective knowledge, which employees are aware of, but is not captured by existing software integrations. To mitigate this data gap, we propose a software solution

<footnote>* equal contribution</footnote>
with a user interface to collect locally observed collective knowledge [2]. An example of such data acquisition can be feedback collection regarding forecasts, forecast explanations, or decision-making options. Feedback can be provided in an implicit (e.g., by not editing an option) or explicit form (e.g., marking a forecast explanation to be improbable). Another example can be asking the user to input yet unknown data (e.g., a decision taken in a certain context that was not registered in the past). When recommending decision-making options to the user, many decisions must be made: which subset of available decision-making options to display, how to rank them, or how to enable the user to provide useful feedback. We envision active learning to weigh which decision-making options are most informative to the system when users' feedback is provided.

We demonstrate a conceptual design and a developed system that can acquire and encapsulate complex knowledge. The system is based on semantic technologies, considering ontology concepts that are generic and ported to multiple use cases. We demonstrate its usability on demand forecasting, providing recommendations for transport scheduling. The system integrates demand forecasting models, explainable AI (XAI), a decision-making recommender system, and a knowledge graph. The aforementioned components are used to develop decision-making workflows, which are displayed through an interactive user interface. Feedback is collected from users regarding forecasts, forecast explanations, and decision-making options displayed to the users.

## 2 Related Work

In smart manufacturing, several characteristics (such as context awareness, modularity, heterogeneity, and interoperability) and technologies (such as intelligent control, energy efficiency, cybersecurity, CPS, IoT, data analytics, and IT-based production management) have been identified to play a crucial role [9]. Variety and depth of technologies in Industry 4.0 represent a great potential, and micro-level local units' usage provides the best outcomes [3]. Several risks are associated with the implementation of smart technologies in manufacturing [8] such as i) the perceived risk of novel technologies, ii) the complexity of integration, and iii) the consideration of human factors. As several most accurate AI algorithms (such as gradient boosting or deep neural networks) are difficult to explain and justify, they represent a significant perceived risk of novel technologies in a shop-floor environment.

In the context of manufacturing, explainable AI (XAI) technologies [1] have been tested in several scenarios such as predictive maintenance [7], real-time process management [10] and quality monitoring [5]. One of our research goals is to highlight the explainability of the algorithms and methods used in smart manufacturing processes, aligning XAI technologies with human interaction. We also aim to collect feedback on the quality of such explanations, since there are few validated measurements for user evaluations on explanations' quality [16].

Active Learning (AL) is usually the natural approach to provide human-in-the-loop functionalities in advanced AI systems. Typically, AL attempts to
improve learners' performance by asking questions to an expert (e.g., query a human operator) to obtain labels for data instances [14]. Since users are usually reluctant to provide information and feedback, AL is used to identify a set of data instances on which the provided users' input conveys the most valuable information to the system [4]. In a decision-making process in manufacturing, AL can also be implemented in recommender systems. In such cases, it tackles obtaining high-quality data that better represents the user's preferences and improves the recommendation quality. The ultimate goal is to acquire additional feedback that enables the system to generate better recommendations [4]. Collecting feedback from forecast explanations can be realized with a framework of three components: a forecasting engine, an explanation engine, and a feedback loop to learn from the users [15]. We extend this approach to collect feedback from forecasts, forecast explanations and decision-making options we recommend to the users.

## 3 Proposed Methodology and System

To realize a system described in Section 1, we developed a methodology to identify relevant components, decision-making options, information and feedback of interest, and how to collect them. The methodology consists of ten steps:

1. create an AI model, to provide predictions that comply with a given use case;

2. provide local forecast explanations, either by querying a glass-box model or using complementary methods for black-box models;

3. list decision-making options available to the user;

4. create a flow of decision-making options available to the user;

5. list the kind of feedback expected for forecasts, forecast explanations, and decision-making options. Identify opportunities for implicit feedback;

6. create a list of relevant entities related with forecasts, explanations, and decision-making options;

7. extend the ontology [13] to include use-case specific entities and relate them to the entities that model the AI model, forecasts, forecast explanations, and feedback;

8. instantiate a knowledge graph based on the ontology entities;

9. develop a software application binding forecasts, forecast explanations, and decision-making options while enabling decision-making flows and feedback gathering;

10. develop an Active Learning module that receives input from the database and knowledge graph to suggest data instances that are expected to be most informative to the system. This input can be used by the decision-making recommender system;

The system requires at least eight components (see Fig. 1A):

- **Database**, stores operational data from the manufacturing plant. Data can be obtained from ERP, MES, or other manufacturing platforms;
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Fig. 1A displays a diagram of the system components and their interaction. Fig. 1B shows the main ontology concepts we considered, and their relationships.</figcaption></figure>

- **Knowledge Graph**, stores data ingested from a database or external sources and connects it, providing a semantic meaning. To map data from the database to the knowledge graph, virtual mapping procedures can be used, built considering ontology concepts and their relationships;

- **Active Learning module**, aims to select data instances whose labels are expected to be most informative to the system and thus help enhance AI model's (e.g., predictive model or recommender system model) performance. Obtained labels are persisted to the knowledge graph and database;

- **AI model**, aims to solve a specific task relevant to the use case, such as classification, regression, clustering, or ranking;

- **XAI Library**, provides some insight into the AI model's rationale used to produce the output for the input instance considered at the task at hand. E.g., in the case of a classification task, it may indicate the most relevant features for a given forecast;

- **Decision-Making Recommender System**, recommends decision-making options to the users. Recommended decision-making options can vary depending on the users' profile, specific use case context, and feedback provided in the past;

- **Feedback module**, collects feedback from the users and persists it into the knowledge graph;

- **User Interface**, provides relevant information to the user through a relevant information medium. The interface must enable user interactions to create two-way communication between the human and the system.

The knowledge graph is a central component of the system. Instantiated from an ontology (see Fig. 1B), it relates forecasts, forecast explanations, decision-making options, and feedback provided by the users. To ensure context regarding decision-making options and feedback provided is preserved, different relationships are established. The feedback entity directly relates to a forecast, forecast explanation, and decision-making option. While a chain of decisions can exist for
a given forecast, there is a need to model the decision-making options available at each stage and the sequence on which they are displayed. To that end, the decision-making snapshot entity aims to capture a list of decision-making options provided at a given point in time. A relationship between decision-making option snapshots (*followedBy*) provides information on such a sequence. For each decision-making snapshot, a *selectedOption* relationship is created to the user's selected decision-making option. To link the first decision-making options to the forecast, the *suggestsActionFor* relationship is created between the forecast entity and entities that correspond to the first decision-making options displayed for that forecast. Since the decision-making options are linked to decision-making option snapshot and preserve a sequential relationship, all decision-making options can be traced back to the forecast that originated them.

# 4 Use Case

Demand forecasting is a key component of supply chain management since it directly affects production planning and order fulfillment. Accurate forecasts enable operational and strategic decisions regarding manufacturing and logistics for deliveries. For the evaluation of our methodology, we developed a model to forecast demand on a material and client level daily. The model was trained on three years of data for 516 time-series corresponding to 279 materials and 149 clients of a European automotive original equipment manufacturer's daily demand. To validate the system we developed, a subset of demand forecasts was used. Forecast explanations were generated with the LIME library [11]. The decision-making options recommender system was implemented with a set of simple heuristics that allow to select a new transport or chose among existing ones that satisfy certain criteria (e.g., have enough capacity to satisfy the expected demand for a given client). Finally, a user interface was developed to display forecasts, forecast explanations, and decision-making options (see Fig. 2). In the user interface, we identify four distinct parts:

A **Forecast panel:** given date and material, it displays the forecasted demand for different clients. For each forecast, three options are available: edit the forecast (providing explicit feedback on the forecast value), display the forecast explanation, and display the decision-making options. The lack of editing on displayed forecasts is considered implicit feedback approving the forecasted demand quantities.

B **Forecast explanation panel:** displays the forecast explanation for a given forecast. Our implementation displays the top three features identified by the LIME algorithm as relevant to the selected forecast. If the users considers some of the features displayed do not explain the given forecast, they can provide feedback by removing it from the list.

C **Decision-making options panel:** displays possible decision-making options, that are tied together into a decision-making snapshot, for a given forecast or step in the decision-making process.
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** User interface, displaying forecasts, forecast explanations, and recommended decision-making options.</figcaption></figure>

**D Feedback panel:** gathers feedback from the user to understand the reasons behind the chosen decision-making option. While some pre-defined are shown to the user, we always include the user's possibility to add their own reasons and enrich the existing knowledge base. Such data can be used to expand feedback options displayed to the users in the future.

Though the current decision-making options recommender system is constrained to heuristics, we envision that in the future, more complex models can be developed leveraging data regarding user interactions with the recommendations we currently display.

# 5 Evaluation and Discussion

To evaluate the proposed methodology and system, empirical evaluations and the development of a concrete use case were utilized [6]. In particular, we applied
the methodology outlined in Section 3 (except for the Active Learning module implementation) to develop a system that enables displaying forecasts, forecast explanations, and decision-making options, provide a decision-making workflow, and means to collect feedback and knowledge from the users<fnref n="4" />. The system we developed for the particular use case of demand forecasting proves that a semantic approach enables effective and flexible means to solve complex knowledge acquisition and solve interoperability conflicts. Since the system was not deployed to production environments, we cannot assess the perceived quality of the forecast explanations, and the impact of the given forecast explanations and decision-making options provided. The impact of the forecasting models was assessed on data provided by EU H2020 FACTLOG and STAR project partners [12]. The development of an Active Learning module remains the subject of future work.

# 6 Conclusion and Future Work

The current work presents a system's conceptual design to acquire and encapsulate complex knowledge using semantic technologies and AI. The system was instantiated for the demand forecasting use case in the manufacturing domain, using real-world data from partners from the EU H2020 projects STAR and FACTLOG. In particular, the system provides forecasts, forecast explanations, decision-making options, and the capability to provide implicit and explicit feedback. The system enables the development of an active learning module that can enhance data collection by identifying promising data instances that, when labeled, are expected to be most informative to the system. Future work will focus on implementing an active learning module and explore recommender systems that learn from data to provide decision-making options to the users.

# Acknowledgements

This work was supported by the Slovenian Research Agency and the European Union's Horizon 2020 program projects FACTLOG under grant agreement H2020-869951 and STAR under grant agreement number H2020-956573.

This document is the property of the STAR consortium and shall not be distributed or reproduced without the formal approval of the STAR Management Committee. The content of this report reflects only the authors' view. The European Commission is not responsible for any use that may be made of the information it contains.

# References

1. Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al.: Explainable artifi-

<footnote n="4">⁴ A video demonstrating the application is available at https://youtu.be/Kx5UnE_yTMO</footnote>
cial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion **58**, 82–115 (2020)

2. Bradeško, L., Witbrock, M., Starc, J., Herga, Z., Grobelnik, M., Mladenić, D.: Curious cat–mobile, context-aware conversational crowdsourcing knowledge acquisition. ACM Transactions on Information Systems (TOIS) **35**(4), 1–46 (2017)

3. Büchi, G., Cugno, M., Castagnoli, R.: Smart factory performance and industry 4.0. Technological Forecasting and Social Change **150**, 119790 (2020). https://doi.org/https://doi.org/10.1016/j.techfore.2019.119790

4. Elahi, M., Ricci, F., Rubens, N.: A survey of active learning in collaborative filtering recommender systems. Computer Science Review **20**, 29–50 (2016)

5. Goldman, C.V., Baltaxe, M., Chakraborty, D., Arinez, J.: Explaining learning models in manufacturing processes. Procedia Computer Science **180**, 259–268 (2021)

6. Grangel-González, I.: A knowledge graph based integration approach for industry 4.0. Ph.D. thesis, Universitäts-und Landesbibliothek Bonn (2019)

7. Hrnjica, B., Softic, S.: Explainable ai in manufacturing: A predictive maintenance case study. In: Lalic, B., Majstorovic, V., Marjanovic, U., von Cieminski, G., Romero, D. (eds.) Advances in Production Management Systems. Towards Smart and Digital Manufacturing. pp. 66–73. Springer International Publishing, Cham (2020)

8. Micheler, S., Goh, Y.M., Lohse, N.: Innovation landscape and challenges of smart technologies and systems – a european perspective. Production & Manufacturing Research **7**(1), 503–528 (2019). https://doi.org/10.1080/21693277.2019.1687363

9. Mittal, S., Khan, M., Romero, D., Wuest, T.: Smart manufacturing: Characteristics, technologies and enabling factors. Proceedings of the Institution of Mechanical Engineers Part B Journal of Engineering Manufacture **233**, 1342–1361 (04 2019). https://doi.org/10.1177/0954405417736547

10. Rehse, J.R., Mehdiyev, N., Fettke, P.: Towards explainable process predictions for industry 4.0 in the dfki-smart-lego-factory. KI-Künstliche Intelligenz **33**(2), 181–187 (2019)

11. Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)

12. Rožanec, J.: Explainable demand forecasting: A data mining goldmine. Companion Proceedings of the Web Conference 2021 (WWW ’21 Companion), April 19–23, 2021, Ljubljana, Slovenia (2021). https://doi.org/10.1145/3442442.3453708

13. Rožanec, J.M., Zajec, P., Kenda, K., Koval, I., Fortuna, B., Mladenić, D.: Ontologies for XAI and decision-making feedback collection. (2021). https://doi.org/10.7910/DVN/UGYHLP, https://doi.org/10.7910/DVN/UGYHLP

14. Settles, B.: Active learning literature survey (2009), http://digital.library.wisc.edu/1793/60660

15. Tulli, S., Wallkötter, S., Paiva, A., Melo, F.S., Chetouani, M.: Learning from explanations and demonstrations: A pilot study. In: 2nd Workshop on Interactive Natural Language Technology for Explainable Artificial Intelligence. pp. 61–66 (2020)

16. van der Waa, J., Nieuwburg, E., Cremers, A., Neerincx, M.: Evaluating xai: A comparison of rule-based and example-based explanations. Artificial Intelligence **291**, 103404 (2021)