<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# How Does the Implementation of AI Agents Affect Human Agents' Job Profiles? Insights from Two Industrial Cases

Antonia Schulte¹, Wilhelm Klat¹, Thomas Süße²

¹CircoVision UG, Elsener Str. 33, 33102 Paderborn, Germany
aschulte@mintkitchen.de, klat@circovision.com

²Bielefeld University of Applied Sciences, Faculty of Engineering and Mathematics, Gütersloh Campus, Langer Weg 9 a, 33332 Gütersloh, Germany
thomas.suesse@fh-bielefeld.de

**Abstract.** Traditionally, operators, system users as well as other non-management employees have been rarely involved in the design, planning, and construction of machines. With the recent dawn of deep learning ever more AI-based agents enter workplaces across industries. This trend is already affecting the way how employees interact with technology. In this paper we show that operative employees rather become co-creators while collaborating with AI agents. This affects traditional job profiles. The contribution of our research is twofold: Firstly, we discuss a novel approach of systematically evaluating how AI agents affect job profiles based on insights from two real-world industrial cases. Secondly, we provide new research perspectives for AI implementation in collaborative environments as well as for AI-related technological enhancements. These results can be fruitful for firms and decision makers as well as for academia, e.g., in fields like employee training, technology and innovation management, change management, applied machine learning and other.

**Keywords:** AI agents, Job profiles, Human-AI interaction, AI Support Processes

## 1 Introduction

Traditionally, operators, system users as well as other non-management employees have been rarely involved in the design, planning, and construction of machines. With the breakthrough of deep learning in 2012 ever more AI-based agents that built on deep neural networks enter workplaces as “open-ended” technologies across industries on a global scale [1]. As AI agents learn tasks from data and can increase their performance with additional training data, recent research in this emerging field identified the need for a closer collaboration between human and AI agents such as “human-AI partnerships” [2]. Although the existing body of research motivates human-AI collaboration with supervision of AI decisions or improving AI systems, it remains to be examined how individual job profiles become affected by the introduction of AI agents in collaborative work. In organizations with higher degrees of labor division and classical hierarchy, various employees with different roles and job profiles are involved in collaborative value creation processes. Thus, we argue that the introduction and implementation of AI agents within such collaborative processes affect job profiles of
the involved employees working together with AI. Based on two industrial use cases we seek to reveal that especially operators who are doing the most basic work are experiencing some significant changes in their job profiles.

The contribution of this paper is twofold: Firstly, we provide a first systematization on why and how AI agents affect individual job profiles in collaborative work contexts. Secondly, we derive research perspectives and advice for practice, e.g. in the popular field of AI implementation in organization where a more explicit focus on employees is considered as one success critical element [3, 4].

This paper is structured as follows: Section 2 gives an overview of the related research. Section 3 introduces the two industrial use cases and outlines individual job profiles of operators, supervisors and managers. Section 4 discusses the findings and gives for further research and practical relevance of our findings.

## 2 Related Research

For about three decades there has been a trend emerging in organizations that shows an increasing amount of technologies, particularly computer-based technologies, that are designed and implemented as so-called “open-ended” technologies. A main characteristic of these kinds of modern and more flexible technologies is seen in the fact that they are more adaptable in a sense that they can take up users’ input in order to change or improve in a context-specific manner. As a result, further development and adjustments of technologies in organizations is no longer just an exclusive task for developers or designers, but should involve users of such technologies as well [5].

However, in order to foster co-creative and collaborative processes during technical design, development and adjustment there are a number of prerequisites at both sides. the human actor as well as the technology. As mentioned above, on the one hand a critical antecedent for co-creative design and development is that technologies have to become more flexible and adaptable towards various use-cases and use-scenarios. Particularly in that field, the further dissemination of AI-based systems and applications provides promising opportunities as AI-based systems are more often designed in such a way that they can use human actors’ input for their own learning and improvement processes. Technically, these processes are often based on machine learning approaches building on the supervised learning concept which demands input from the human counterpart [6]. On the other hand, at the side of the human actor a critical antecedent for the success of supervised learning of AIs demands a shift in the understanding of the human actors role from a user or operator towards a co-creator and collaborator of AI technology. It is not surprising that this shift in the interactional logic between human actors and technologies which goes beyond usage towards a more collaborative form of interaction is of increasing relevance in work environments where AI-based technologies have already been introduced.

Researchers point out that new technologies at work can lead to changes of qualification requirements as well as more skill variety in production environments [7–9]. In addition, an increasing amount of literature is focusing on the readiness of organizations when implementing AI technologies and highlight the upskilling as well as changes of job profiles among employees as a critical success factor [3, 4, 10].
Regarding human employees' shifting role related to successful AI implementation, a number of newly emerging concepts are shaping and supporting the understanding towards a human-AI collaboration in organization. Some of these concepts are outlined in literature as human-AI hybrids [11], human-AI partnerships [2] or human-AI systems [12]. With regards to the changing role of humans required for human-AI collaboration Waefler and Schmid [2] point out that particularly in situations of collaborative decision making between human and AI the human's tasks of verifying AI's decisions, improving the AI system, learning from the AI system as well as taking responsibility for the final decision are gaining increasing importance. In addition, this transformative potential of AI regarding employees' roles and job profiles contributes to the growing importance of collaborative learning capacities as well as creativity and self-organization, to mention just a few [13]. However, these contributions are still on a rather conceptual level and are lacking further reflection in collaborative workplaces in the industry.

# 3 Changing Job Profiles: Insights from Two Industrial Use-cases

The introduction of open-ended AI agents into collaborative workplaces is accompanied by a set of new processes we call AI support processes. These new processes are required for performance supervision (Algorithm Supervision Process) and for improving AI performance with additional training data captured during operations. The latter can be divided into training data creation (Data Acquisition Process, Data Labeling Process) and the actual algorithm training (Algorithm Training Process). We illustrate the introduction of open-ended AI agents and AI support procedures with two industrial use cases in which we had a leading role as systems provider. In both cases AI agents took over an assisting role in collaboration with human operators who observe and verify AI decisions. AI agents are designed to acquire the majority of training data required for optimal performance predominantly during operations. Therefore data points automatically collected during operations are permanently stored, labeled and used for algorithm retraining.

## 3.1 Use Case A: Circular Economy in the Automotive Industry

The first use case is from the automotive industry and the task at hand is a visual inspection of used automotive parts for remanufacturing purposes. The parts need to be sorted into quality categories based on a visual inspection of wear and tear, damage and anomalies. The AI agent is a camera-based machine which uses image classifier algorithms to support the operator with the visual inspection. It judges and pre-sorts parts into categories. Operators can observe the AI decisions in real-time and are tasked to verify the pre-sorted parts. The supervisor is in charge for in-bound and out-bound logistics at this workplace. Finally, the line manager holds full responsibilities for all workplaces in the remanufacturing line including personnel planning and quality control.
## 3.2 Use Case B: Waste Management Services

The second use case is a communal service in the field of waste management. Garbage trucks collect the contents from garbage bins placed at roadsides. Bins with separated household waste (e.g. paper, organic, plastics) need to be visually checked for contamination by wrong materials to avoid problems in down-stream waste recycling procedures. Garbage truck drivers (operators) are instructed to refuse the collection of highly contaminated bins. An AI agent has been introduced to assist operators with visual inspections of bins. The AI agent is a camera-based system attached to the garbage truck observing the bin contents while being thrown into the truck. The AI agent logs all visual assessments and warns operators in cases of high contamination levels. Operators can observe AI decisions in real-time and have final decision power with regard to refusing garbage collection. Supervisors organize surrounding tasks such as tour planning and deal with customer inquiries. The manager has the authority to change, among others, workplace procedures, truck equipment and contractual agreements with municipalities.

## 3.3 Factors for Job Profile Changes

Ensuring operational reliability of AI agents and their continuous performance improvement is a crucial challenge for organizations. AI agents, or machine learning teams responsible for their development respectively, have limited means of completing AI support processes without extensive collaboration with operational staff. In the above mentioned use cases, we empirically observed three main reasons favoring the extensive inclusion of operative staff in the co-creation process we call: (1) Ground Truth Proximity, (2) Task Familiarity, and (3) Production Line Insights and Authority.

**Ground Truth Proximity.** In the field of machine learning, ground truth is often referred to as labels of a data point which are known to be true and of critical importance for algorithm training [14]. The ground truth can be obtained from empirical observation or measurement. In Use Case A, the operator can obtain the ground truth for a data point from physically inspecting the car part including mechanical measurements while it is being processed by the AI agent.

In Use Case B the operator can personally inspect the contents of a garbage bin while it has been photographed and move waste inside the bin to verify a suspected contamination. Access to the ground truth of data points is unequally distributed among human agents. If the ground truth is only accessible temporarily in the moment of data capture then often only operators can access it. In such cases, operators possess a higher degree of ground truth proximity compared to management-oriented colleagues without direct real-time access to the physical objects.
**Task Familiarity.** Task familiarity includes explicit knowledge about formal decision criteria but also implicit experience from observing vast amounts of data points including their corresponding ground truth. Management-oriented employees and machine learning teams usually have superior explicit knowledge about a task but lack the operator's implicit task familiarity from having personally seen many garbage bins or car parts respectively. This tacit knowledge is of great value for AI development as AI agents possess the ability of replicating this knowledge as abstract features in deep neural networks when trained with data points that have been labeled by humans in possession of tacit task knowledge.

**Production Line Insights and Authority.** The integration of AI agents into existing workplaces can require changes of established procedures and workplace setups to ensure successful execution of AI support processes. Redesigning processes requires production line authority. A typical change in procedures can be the collection of defective or rare items which previously have been discarded. In Use Case A defective car parts have immediately been discarded for material recycling during inspection. With AI agent introduction, a new process has been implemented to collect certain and rare defective car parts for algorithm retraining.

## 3.4 Job Profile Analysis

The execution of AI support processes can greatly benefit from co-creation with operational staff. We call this contribution from operational staff ‘AI Support Impact’. Due to variations in the degrees of ground truth proximity, task familiarity and production line authority, the potential AI support impact varies among human agents.

**Algorithm Support Process (ASP).** With regard to performance supervision of AI agents we empirically observed significant effects of the ASP on the operator's job profile. Operators have the highest degree of ground truth proximity and task familiarity which, in both use cases, rendered the inclusion of operators into the ASP mandatory. The job profiles of managers also have been extended to a weaker extent with regard to verifying edge cases. Supervisors can take over a weaker role of first-level support for operators when they request guidance on how to deal with edge-cases or similar.

**Data Acquisition Process (DAP).** Continuous data acquisition requires human support and supervision to ensure that data points are suitable for algorithm training and the available data is representative in terms of including sufficient and balanced representations of relevant patterns. Ideally the operator's job profile is extended to support the DAP by flagging unsuitable data points on the fly and increasing the availability of e.g. rare patterns in the dataset. Managers with production line authority also play an important role as continuous data acquisition processes usually require a redesign of workplace procedures. Managers need to redesign, communicate and enforce workplace procedures among human agents. Again, supervisors can take over a weaker role of first-level support for operators.
**Data Labeling Process (DLP).** All human agents with task familiarity are potential contributors in the DLP. While operators can use their implicit task familiarity, supervisors and managers bring knowledge on explicit task requirements and upstream/downstream requirements into the labeling process. A practice-proven division of labeling work is to use supervisors and managers as second-level labelers especially for edge cases.

**Algorithm Training Process (ATP).** AI agents have shades of being right or wrong as deep neural networks return inferences with almost cardinally scaled confidence levels. Lower confidence decisions, for example, can be the result of algorithm weaknesses. Ground truth proximity can help human agents to identify potential sources of algorithm weaknesses and is therefore of high importance for the AI agents as an open ended technology. The tacit knowledge of operators from collaborating with the AI agents on a regular basis is a critical success factor for retraining AI agents. Their job profiles should be adjusted to become sparring partners in the identification of AI weaknesses.

<figure><img src="image_2.png" /><figcaption>Fig. 1 AI support processes and AI support impact of human agents.</figcaption></figure>

# 4 Discussion

In collaborative networks, open-ended AI agents require a novel set of operational procedures that we argue can be called co-creative AI support processes. They are required to ensure supervision and continuous improvement of AI agents. When performing AI support processes, machine learning teams can greatly benefit from including operators, supervisors and managers as co-creators. Their inclusion, however, requires changes and extension of their job profiles in different ways. These findings confirm other research contributions in the field of human-AI interaction [2, 7, 8].

Traditionally, operators are rarely involved in the creation process of regular machines. With AI agents present, their relevance increases significantly. As operators
may be the only human agents with direct access to the ground truth of data points, we argue that operators should become mandatory co-creators as they may share some of their tacit knowledge with AI agents. As machine learning specialists responsible for the development of AI agents in both industrial use cases presented in this paper we empirically observed tremendous contributions of operators with supervision and improvement of AI agents alike. Operator feedback in both industrial use cases indicates that this new role as co-creators is also beneficial for operators themselves as it enriches the workplace with more responsibility, room for creative problem thinking and task variety.

Managers and supervisors play an important role in the workplace redesign to ensure optimal conditions to execute AI support processes. Especially managers with the authority to change production line procedures require a basic machine learning understanding to correctly redesign workplace procedures. Besides this, their job profile needs to be extended by communication and enforcement tasks as changes in operational procedures can induce human resistance if their purpose is unclear. Supervisor and manager feedback in both industrial use cases indicates the importance of basic machine learning understanding to motivate extensive workplace redesigns.

Our findings also revealed promising research perspectives in the fields of human factors, especially human resources and AI implementation in collaborative workplaces. Firstly, our estimations of the AI support impact of operators, supervisors and managers on AI support processes is derived from few empirical observations in which we introduced AI agents in real-world industrial workplaces. Our first systematization and estimations can be the foundation for extended field research. Secondly, the requirement for extending existing job profiles to accommodate AI support processes raises a variety of research questions on how human agents with different job profiles should be trained and prepared for the future. Thirdly, as ensuring successful implementation of AI support may also require redesigning workplaces, collaboration procedures, production lines and other organizational parts, it remains to be examined how, when and to what extent the various perspectives on collaborative networks should be further adjusted or extended. Finally, we argue that our result should also be recognized by designers of AI agents like digital engineers who have to take into account the important role of operators for learning and improvement of such AI agents in productive environments.

## References

1. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y.

2. Waefler, T., Schmid, U.: Explainability is not Enough: Requirements for Human-AI- Partnership in Complex Socio-Technical Systems. 11 (2021).

3. Hamm, P., Klesel, M.: Success Factors for the Adoption of Artificial Intelligence in Organizations: A Literature Review. (2021).

4. Pumplun, L., Tauchert, C., Heidt, M.: A New Organizational Chassis for Artificial
Intelligence - Exploring Organizational Readiness Factors. In: ECIS (2019).

5. Orlikowski, W.J.: The Duality of Technology: Rethinking the Concept of Technology in Organizations. Organ. Sci. 3, 398–427 (1992).
https://doi.org/10.1287/orsc.3.3.398.

6. Zhou, Z.-H.: A brief introduction to weakly supervised learning. Natl. Sci. Rev. 5, 44–53 (2018). https://doi.org/10.1093/nsr/nwx106.

7. Süße, T., Kobert, M., Kries, C.: Antecedents of Constructive Human-AI Collaboration: An Exploration of Human Actors’ Key Competencies. In: Camarinha-Matos, L.M., Boucher, X., and Afsarmanesh, H. (eds.) Smart and Sustainable Collaborative Networks 4.0. pp. 113–124. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-85969-5_10.

8. Pfeiffer, S.: The 'Future of Employment' on the Shop Floor: Why Production Jobs are Less Susceptible to Computerization than Assumed. Int. J. Res. Vocat. Educ. Train. 5, 208–225 (2018). https://doi.org/10.13152/IJRVET.5.3.4.

9. Wegman, L.A., Hoffman, B.J., Carter, N.T., Twenge, J.M., Guenole, N.: Placing Job Characteristics in Context: Cross-Temporal Meta-Analysis of Changes in Job Characteristics Since 1975. J. Manag. 44, 352–386 (2018).
https://doi.org/10.1177/0149206316654545.

10.Ansari, F., Hold, P., Khobreh, M.: A knowledge-based approach for representing jobholder profile toward optimal human-machine collaboration in cyber physical production systems. CIRP J. Manuf. Sci. Technol. 28, 87–106 (2020).
https://doi.org/10.1016/j.cirpj.2019.11.005.

11.Rai, A., Constantinides, P., Sarker, S.: Next generation digital platforms: toward human-AI hybrids. MISQ. iii–ix (2019).

12.Schelble, B., Flathmann, C., Canonico, L.-B., Mcneese, N.: Understanding Human-AI Cooperation Through Game-Theory and Reinforcement Learning Models. Presented at the Hawaii International Conference on System Sciences (2021). https://doi.org/10.24251/HICSS.2021.041.

13.Markauskaite, L., Marrone, R., Poquet, O., Knight, S., Martinez-Maldonado, R., Howard, S., Tondeur, J., De Laat, M., Buckingham Shum, S., Gašević, D., Siemens, G.: Rethinking the entwinement between artificial intelligence and human learning: What capabilities do learners need for a world with AI? Comput. Educ. Artif. Intell. 3, 100056 (2022). https://doi.org/10.1016/j.caeai.2022.100056.

14.Kondermann, D.: Ground truth design principles: an overview. In: Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications. pp. 1–4. ACM, St. Petersburg Russia (2013).
https://doi.org/10.1145/2501105.2501114.