# Case-Study-Based Requirements Analysis of Manufacturing Companies for Auto-ML Solutions

Günther Schuh, Max-Ferdinand Stroh, Justus Benning

► To cite this version:

Günther Schuh, Max-Ferdinand Stroh, Justus Benning. Case-Study-Based Requirements Analysis of Manufacturing Companies for Auto-ML Solutions. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.43-50, 10.1007/978-3-031-16407-1_6. hal-05386005

HAL Id: hal-05386005

https://inria.hal.science/hal-05386005v1

Submitted on 27 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Case-Study-Based Requirements Analysis of Manufacturing Companies for Auto-ML Solutions

Günther Schuh, Max-Ferdinand Stroh, and Justus Benning(✉)

FIR, Institute for Industrial Management at RWTH Aachen University, Campus-Boulevard 55, 52074 Aachen, Germany

justus.benning@fir.rwth-aachen.de

**Abstract.** Methods of machine learning (ML) are difficult for manufacturing companies to employ productively. Data science is not their core skill, and acquiring talent is expensive. Automated machine learning (Auto-ML) aims to alleviate this, democratizing machine learning by introducing elements such as low-code or no-code functionalities into its model creation process. Due to the dynamic vendor market of Auto-ML, it is difficult for manufacturing companies to successfully implement this technology. Different solutions as well as constantly changing requirements and functional scopes make a correct software selection difficult. This paper aims to alleviate said challenge by providing a longlist of requirements that companies should pay attention to when selecting a solution for their use case. The paper is part of a larger research effort, in which a structured selection process for Auto-ML solutions in manufacturing companies is designed. The longlist itself is the result of six case studies of different manufacturing companies, following the method of case study research by Eisenhardt. A total of 75 distinct requirements were identified, spanning the entire machine learning and modeling pipeline.

**Keywords:** Machine learning · Auto-ML · Requirements · Manufacturing

## 1 Introduction

Manufacturing remains one of the most promising areas for the application of machine learning. However, since data science is not a core competency of manufacturing companies, they are severely affected by the lack of experts in this field [1]. Automated machine learning (Auto-ML) is defined as both the concept as well as the underlying technology to accelerate the development lifecycle of machine learning models through automation [2]. It therefore addresses this problem by democratizing and simplifying the machine learning value chain process, from data acquisition to model validation.

### 1.1 Background and Overall Research Goal

Software solutions that integrate Auto-ML functionalities have seen a surge in popularity over the last year, resulting in a dynamic vendor market, as software providers try to offer
unique solutions to creating machine learning models without data science expertise [3]. This market proves to be a challenge for manufacturing companies, as they still struggle to build knowledge about the basics of artificial intelligence [4], making the choice of an apt Auto-ML solution for their use cases a complex challenge. This paper aims to alleviate said challenge by providing a longlist of requirements that companies should pay attention to when selecting a solution for their use case. The paper is part of a larger research effort, in which a structured selection process for Auto-ML solutions in manufacturing companies is designed (see Fig. 1).

<figure><img src="image_3.png" /><figcaption>Fig. 1. Overall research goal and context of this paper</figcaption></figure>

Simultaneously in another research effort, a descriptive model is being developed which comprises the functions of current Auto-ML solutions in the form of a function tree. In successive endeavors, these two models will be combined to match the user and the product perspective. This will yield in an explanatory model that links requirements of manufacturing companies to functions of Auto-ML solutions. In a final step, this model will be integrated into a structured selection process that companies can use.

## 1.2 State of the Art and Current Research

Most of the current literature regarding Auto-ML focuses on benchmarking the performance of the underlying algorithms or applying the technology to existing optimization and prediction problems. Publications that focus on the requirements of users are limited. While no sources have been found that provide a longlist of requirements for manufacturing companies, there is research that evaluates the user perspective on the technology, which will be briefly introduced here.

Krauß et al. analyse the applicability of Auto-ML for a predictive use case in a production context. However, the requirements which they test against are purely focused on model performance [5]. Crisan and Fiore-Gartland conduct an experiment focusing on human-machine-interaction of user and Auto-ML systems. In their setup, they implicitly regard requirements such as degree of automation and interpretability of the models. However, their research is not focused on the manufacturing sector and leaves out requirements that are posed by non-human entities, such as the legal framework or the organizational structure of the company [6]. Elshawi et al. conduct a survey of current Auto-ML solutions and look at functions as well as functional requirements. Still, their focus lies on the technical description of the solutions themselves [7]. The same thing holds true for the benchmarking survey of Zöller and Huber [8]. In their paper, Xin et al. conducted interviews with experienced Auto-ML users to provide general guidelines
for applying these solutions in companies. A part of their results contains a subjective rating of solutions regarding criteria like explainability and ease of use. Thus, it provided a good reference point for the completeness of the longlist created in this paper, even though it lacks the focus on the manufacturing sector [9].

## 1.3 Delineation of the Term Requirements Used in the Context of this Paper

The term requirements used in the context of this paper follows the definition by the IEEE Standard Glossary of Software Engineering Terminology. Requirements are defined as “a condition or capability needed by a user to solve a problem or achieve an objective” [10]. This includes technical requirements as well as the economic objectives that the stakeholders of Auto-ML strive to achieve by employing the technology.

## 2 Methodology

In order to investigate which requirements are relevant when selecting an Auto-ML solution from the perspective of manufacturing companies, theory building according to Eisenhardt was applied [11]. The case study research method was selected due to its capability of describing and explaining novel phenomena with relatively sparse information [12]. Furthermore, as this paper deals with the descriptive part of the overall research goal (see Fig. 1), methods that focus on prescriptive knowledge, such as design science research, were excluded. The method used comprises of a complete process for building theory from case studies, which is especially suitable for investigating new areas of research like the one at hand. At the beginning of the case study research process, questions were formulated for the interviews. Table 1 shows the final questionnaire. It contains 16 open-ended questions and served as the basis for the case studies, which were conducted as semi-open interviews.

<table><caption>**Table 1.** Final questionnaire</caption><thead><tr><td>No</td><td>Question</td></tr></thead><tbody><tr><td>1</td><td>What is the current state of implementation of ML in your company?</td></tr><tr><td>2</td><td>What challenges do you see in the introduction and usage of ML?</td></tr><tr><td>3</td><td>What goals do you want to achieve by using an Auto-ML solution?</td></tr><tr><td>4</td><td>Which usage scenarios should be realized in your company with the help of an Auto-ML solution?</td></tr><tr><td>5</td><td>Which working steps are to be optimized using Auto-ML?</td></tr><tr><td>6</td><td>Who should be responsible for creating models? Who should use them?</td></tr><tr><td>7</td><td>Is Auto-ML intended to compensate for a specific expertise?</td></tr></tbody></table>

(continued)
<table><caption>**Table 1.** (continued)</caption><thead><tr><td>No</td><td>Question</td></tr></thead><tbody><tr><td>8</td><td>Which communication interfaces are essential for you?</td></tr><tr><td>9</td><td>Are there other requirements for software, such as guidelines?</td></tr><tr><td>10</td><td>What are your requirements for the performance of an Auto-ML model?</td></tr><tr><td>11</td><td>What are your requirements for the transparency and explainability of an Auto-ML model?</td></tr><tr><td>12</td><td>How do you envision optimal user interaction with an Auto-ML solution?</td></tr><tr><td>13</td><td>How should data visualization be implemented in your opinion?</td></tr><tr><td>14</td><td>In which cases would you like to be notified?</td></tr><tr><td>15</td><td>How should the system behave in the event of an error?</td></tr><tr><td>16</td><td>Do you have any other requirements for an Auto-ML solution that have not been mentioned yet?</td></tr></tbody></table>

<table><caption>**Table 2.** Case subject data</caption><thead><tr><td>Case number</td><td>Manufactured product/industry sector</td><td>AI use case</td></tr></thead><tbody><tr><td>1</td><td>Agricultural machinery</td><td>Demand forecasting</td></tr><tr><td>2</td><td>Cutting machinery</td><td>Predictive quality</td></tr><tr><td>3</td><td>Trailers and cargo equipment</td><td>Production control</td></tr><tr><td>4</td><td>Medical equipment</td><td>Predictive maintenance</td></tr><tr><td>5</td><td>Plant supplier for the metallurgical industry</td><td>Customer relations</td></tr><tr><td>6</td><td>Coating and surface solutions</td><td>Predictive maintenance</td></tr></tbody></table>

In total, 6 case studies with machine learning users and digitalization officers from manufacturing companies were conducted. Data about the enterprises and case studies is given in Table 2.

To reach closure, i.e., to determine that enough case studies have been conducted theoretical saturation must be reached. That means, that the incremental improvement to the results become minimal [11]. The sixth case study only yielded in three new requirements (compared to 72 total recorded requirements at that point, cf. Fig. 2). Due to the high redundancy at this point, the case number was capped at six.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Saturation of results according to Eisenhardt</figcaption></figure>

## 3 Results

In total, 75 unique requirements were synthesized in the case study research process. To aid readability of this paper while fully disclosing the research results, the longlist of requirements can be downloaded via hyperlink under [13]. In this section, general observations derived from the requirements will be shared.

First, the requirements were clustered according to the model creation process steps for automated machine learning which is roughly based on the analyses of Zöller and Huber [8] (cf. Fig. 3).

<figure><img src="image_5.png" /><figcaption>Fig. 3. Auto-ML process steps used for categorizing the requirements</figcaption></figure>

Although slight variations exist to this process, for the context of this analysis there are assumed to be seven distinct steps to creating and using models with Auto-ML-solutions. They start with handling the input data and end with the deployment and management of the finished models. A superordinate step (zero) was introduced, as a significant portion of the requirements did not match a distinct step but rather referred to Auto-ML solutions in general. In the following paragraph, the clusters and patterns of the requirements within them will be described.

The requirements of step zero (superordinate step) either described economic considerations that were expected of the application of Auto-ML or constraints, that the company had regarding the use of IT solutions, such as a cloud strategy or data protection guidelines. The requirements of cluster one (“handle input data”) consisted mainly of the various data sources (e.g., SQL databases) that companies need the solutions to integrate seamlessly. The second cluster (“create pipeline structure”) showed an interesting dichotomy. Although this step is a field of high concern for researchers, as there
are various mathematical challenges yet to be solved, the user perspective is not concerned with the algorithms and techniques that the software offers. They are interested in the varying degrees of automation vs. control that the underlying algorithms offer. The third and fourth cluster ("pre-process data" and "automate feature engineering") paint a similar picture, as users are mainly occupied with efficiency and degree of automation of these steps. The fifth cluster, "generate models" showed themes of user control and interaction during the model creation process as well as more technical requirements like GPU-support to reduce training times. The defining topic of the sixth cluster, "evaluate models", is transparency, both about model performance and the decisions they make. The requirements of the seventh cluster "deploy, manage and use models" show that the digital units want to distribute their finished models as efficiently as possible, while the model users require transparent, easy to use models.

To give an overview over the 75 distinct requirements and their clusters, the distribution of requirements is shown in Fig. 4.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Distribution of requirements over Auto-ML steps</figcaption></figure>

This graphic should not be interpreted as a scoreboard that indicates which steps are the most important in Auto-ML, because the requirements are not weighted in any way. However, it gives an impression which steps manufacturing companies are concerned with when they think about solving their AI use cases with Auto-ML-solutions.

In an additional analysis, the requirements were categorized according to the three goals of Auto-ML according to Crisan and Fiore-Gartland. In their work, which is concerned with the interaction of humans with Auto-ML-systems, they synthesize three objectives that companies can achieve with employing Auto-ML. The first is automating routine work, the second is rapid prototyping of models to solve business problems, and the third and final is democratizing machine learning and data science [6]. The distribution is shown in Fig. 5. Some of the requirements did not match the three goals (e.g., compliance or legal requirements) and are subsumed in the "other" category.

The graphic shows a rather balanced distribution of goals, with automation being the main goal of manufacturing companies, closely followed by rapid prototyping and democratization of machine learning.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Distribution of requirements over Auto-ML targets after Crisan and Fiore-Gartland [6]</figcaption></figure>

# 4 Discussion

The results of the paper are subject to the characteristic weaknesses of case study research, which tend to be overly complex or narrow models that do not generalize well [11]. To mitigate this risk, the a-priori variables have been selected from a wide range of sources, yielding in a diverse questionnaire. However, before the applicability of the results needs to be further validated once all partial models have been created and the Auto-ML selection process is used in the industry. Furthermore, the longlist focuses on specific requirements regarding Auto-ML-solutions. General procurement criteria, which virtually every enterprise has, are not included. Thus, an application of this longlist should always be supplemented by the usual economic considerations. General criteria for selecting and employing innovative technologies in enterprises are well researched and can be found in works such as [14] and [15].

# 5 Summary and Outlook

In this paper, the authors derived 75 distinct requirements of manufacturing companies for Auto-ML-solutions from six case studies using the research method of Eisenhardt. The requirements were clustered according to the model creation process and their distribution along these steps is shown. Furthermore, the requirements were assigned to the three goals of Auto-ML according to Crisan and Fiore-Gartland, showing a rather even distribution along the objectives. In successive research efforts, these user-centric results need to be matched to the solution provider perspective, i.e., the functions that current Auto-ML solutions offer. Furthermore, the synthesized stakeholder groups will be analysed. This will yield in a structured selection process for Auto-ML solutions in manufacturing companies.
## References

1. Reder, B.: Studie Machine Learning 2021 (2021). https://www.lufthansa-industry-solutions.com/de-de/studien/idg-studie-machine-learning-2021. Accessed 2 Sept 2021

2. Masood, A., Sherif, A.: Automated Machine Learning, 1st edn. Packt Publishing, Safari (2021)

3. Statista Research Department: Number of AI/ML service offerings at hyperscale CSPs worldwide 2020–2021, by provider (2022). https://www.statista.com/statistics/1268286/worldwide-ai-machine-learning-service-offerings-hyperscalers/. Accessed 7 Mar 2022

4. Kaul, A., Schieler, M., Hans, C.: Künstliche Intelligenz im europäischen Mittelstand. Status quo, Perspektiven und was jetzt zu tun ist (2019). https://www.uni-saarland.de/fileadmin/upload/lehrstuhl/kaul/Universita%CC%88t_des_Saarlandes_Ku%CC%88nstliche_IntelliGENZ-im_europa%CC%88ischen_Mittelstand_2019-10_digital.pdf. Accessed 7 Mar 2022

5. Krauß, J., Pacheco, B.M., Zang, H.M., Schmitt, R.H.: Automated machine learning for predictive quality in production. Procedia CIRP 93, 443–448 (2020). https://doi.org/10.1016/j.procir.2020.04.039

6. Crisan, A., Fiore-Gartland, B.: Fits and starts: enterprise use of AutoML and the role of humans in the loop. In: Kitamura, Y. (ed.) Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. CHI 2021: CHI Conference on Human Factors in Computing Systems, Yokohama Japan, 08–13 May 2021, pp. 1–15. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3411764.3445775

7. Elshawi, R., Maher, M., Sakr, S.: Automated machine learning: state-of-the-art and open challenges (2019). http://arxiv.org/pdf/1906.02287v2

8. Zöller, M.-A., Huber, M.F.: Benchmark and survey of automated machine learning frameworks. J. Artif. Intell. Res. 70, 409–474 (2021)

9. Xin, D., Wu, E.Y., Lee, D.J.-L., Salehi, N., Parameswaran, A.: Whither AutoML? understanding the role of automation in machine learning workflows (2021). http://arxiv.org/pdf/2101.04834v1

10. IEEE Standards Board: IEEE Standard Glossary of Software Engineering Terminology, New York (IEEE Std 610.12–1990) (1990). http://www.informatik.htw-dresden.de/~hauptman/SEI/IEEE_Standard_Glossary_of_Software_Engineering_Terminology%20.pdf. Accessed 25 June 2022

11. Eisenhardt, K.M.: Building theories from case studies. Acad. Manag. Rev. 14, 532–550 (1989)

12. Sprinz, D., Wolinsky, Y. (eds.): Models, Numbers, and Cases. Methods for studying international relations, 2nd edn. University of Michigan Press, Ann Arbor (2007)

13. Justus Benning: Case-Study-Based Requirements Analysis of Manufacturing Companies for Auto-ML solutions. Long List of Requirements (2022). https://github.com/BenningJustus/APMS-Longlist/raw/main/APMS_Requirements_Long-List_public_Bn.xlsx

14. Schuh, G.: Innovationsmanagement. Springer, Heidelberg (2012) https://doi.org/10.1007/978-3-642-25050-7

15. Schuh, G., Klappert, S.: Technologiemanagement. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-12530-0