# User-Centric Digital Assistance with Smart Tools for Manual Assembly Processes

Simon Piontek, Hermann Lödding

► To cite this version:

Simon Piontek, Hermann Lödding. User-Centric Digital Assistance with Smart Tools for Manual Assembly Processes. IFIP International Conference on Advances in Production Management Systems (APMS), Sep 2022, Gyeongju, South Korea. pp.101-109, 10.1007/978-3-031-16407-1_13 . hal-05386033

HAL Id: hal-05386033

https://inria.hal.science/hal-05386033v1

Submitted on 27 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# User-Centric Digital Assistance with Smart Tools for Manual Assembly Processes

Simon Piontek(✉) and Hermann Lödding(✉)

Hamburg University of Technology, Hamburg, Germany
{simon.piontek, loedding}@tuhh.de

**Abstract.** To efficiently leverage digitalization potentials for large proportions of manual processes, workers must be integrated in the digital world in a seamless manner. This paper contributes to the ongoing research on Digital Assistance Systems. In particular, manual assembly processes in aircraft production are scrutinized to identify specific requirements for such a system. Following the resulting analysis, we propose a concept for user-centric digital assistance that uses smart, localized hand tools and provides workers with information based on a digital twin of the product and the production process. The proposed digital assistance system accurately tracks the tool position and enables event-based and bidirectional data flow between workers and a digital twin. The tools are automatically configured with the correct process parameters based on their current position. Quality and progress are documented and ultimately operations and everyday efforts are simplified. The concept is examined for its potentials on productivity and quality in manual assembly processes. The results show that further development of the concept promises significant improvements in manual work.

**Keywords:** Digital assistance · Smart tools · Digital twin · User-Centric design

## 1 Introduction

### 1.1 Motivation

Looking at the aircraft industry, increasing overall demand as well as rising product variants and complexities [1, 2] amplify the necessity of assisting workers. Product complexity leads to error-prone manual labor and paper-based information accumulations on the shop floor. This has immediate effects on labor productivity and quality. Such deficits in terms of value-added work can be targeted to optimize manual operations. Worker states can be structured in a generic work cycle of work preparation, work execution and follow-up processes [3]. Necessary technology for assisting these parts is already on the market. Nevertheless, technologically enhanced hand tools are not used and most digital assistance systems focus on the work preparation elements of information and material procurement, and component as well as site preparation in one-off production (e.g., [4, 5]). Deficits prevail especially in the documentation of work (follow-up) and in error prevention and optimization of work execution.
However, the amount of time spent on work preparation in series production is significantly lower than in one-off production. Worker acceptance of assistance systems may suffer if increased handling of the digital document is necessary. This underlines the necessity of improving further elements of work cycles in series production. Hence, this paper proposes a user-centric, scalable, and dynamic solution that integrates digital information assistance with smart hand tools which are both connected to a digital twin to optimize the entire workflow of manual assembly in aircraft manufacturing.

## 1.2 Current State of Research

Digital assistance systems are developing rapidly in the industrial context, leading to an abundance of approaches to support assembly operations [2]. User-centered system development places the worker at the center to raise efficiency and usability and will regularly re-evaluate expectations and worker feedback. The goal is to assure that the system will fulfill its intended purpose and thereby leads to higher effectiveness [6] and is crucial in the development of digital assistance systems [7].

Figure 1 shows the user model in the context of assembly processes and hence the elements which can be digitally enhanced and optimized. It consists of the worker at the center, the information that is provided, the tools and means of production necessary for the work tasks, the product (or parts to assemble), and the environment in which the human works. Digital assistance systems should address and improve these elements to achieve a comprehensive digital integration over most of the generic work cycle.

<figure><img src="image_3.png" /><figcaption>Fig. 1. User model in the context of assembly processes</figcaption></figure>

By digitally enhancing these elements the performance requirements of work tasks and workers capabilities are aligned to reduce training times, search times and operating errors [9]. Looking at the current state in research, most approaches focus on the optimization of information provision. Functionalities and features aim to provide process information and instructions in relation to real world objects. Information is filtered and visualized in accordance with work progress and in reference to specific components and means of production to improve work preparation and to ensure correctness [8, 9]. While this influences work preparation, it does not influence work execution itself in the sense that the assistance system can intervene and support in the work process. An exception is
the use of smart tools for optimizing manual drilling processes in aircraft manufacturing developed by Hintze et al. [1]. Semi-automatic drilling units were developed, which can be monitored regarding position- and process-data and can be configured depending on their position in the process. Process and quality control of work tasks are feasible. However, this solution relies on a static database and server and therefore lacks integration with informational assistance.

To comprehensively support the worker over the entire work cycle, a concept that revolves around a digital twin and comprises smart tools, digital information assistance and other processing services in a unified namespace is needed. The idea is to use the digital twin to control hand-held tools such as drilling machines by leveraging bidirectional data transmission between physical and digital entities. Assembly guidance can be dynamically updated in real-time and process performance can be predicted and controlled [10]. Including workers and variable manual processes requires adaptive assistance systems that include context sensitive information from the environment, e.g., sensors and localization systems [7, 8]. A digital twin of a manual process which can integrate sensors dynamically through modular interfaces and that includes scalable services to react to new data streams is needed.

# 2 Manual Assembly Processes

The aircraft industry manufactures highly complex products. Due to bad accessibility and changing product variants, many of the work processes are still manual. This suits the frame of the proposed digital assistance concept and offers possibilities of transferring knowledge from one-off production to series production. To find relevant tasks and current deficits in the industry, a thorough analysis of fuselage assembly processes was conducted. Workers in a fuselage completion workstation were investigated and consulted regarding their everyday workflows and required assistance.

## 2.1 Aircraft Structure Assembly Processes

In the context of assembling aircraft structures and subsystems, operations can often be grouped into: Preparation of work, drilling of boreholes, and joining the components through riveting connections. Preparing the assembly consists of finding correct tools, production means and information of positioning parts to be assembled in the fuselage structure. Components must be fixated with production means and clamps as well as shims to compensate voids. Workers rely on complex working documents to find information in textual elements, which often describe work steps and assembly positions using the frames and stringers of the fuselage.

Drilling and riveting are characterized as work execution and have similar properties. Information about different areas and process parameters (e.g., drilling and tightening torque, feed rates, collar types, connecting joints) must be repeatedly procured and memorized, examining technical drawings and various execution instructions. Accordingly, machines have to be configured regarding their offsets, status of tool wear, clearance and process parameters. Quality and progress are often assessed and documented manually at the end of a shift.
## 2.2 Core Tasks and Deficits

Core tasks can be identified along the process. Figure 2 shows the relation of relevant tasks to the generic work cycle and correlating elements of the user model to assist the worker.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Core tasks and influencing elements of user model</figcaption></figure>

The way these tasks are usually carried out is inadequate in several aspects: Information must be gathered by skimming through several different documents and locations. The complexity of information leads to a trade-off between spending time on searching and memorizing information or working based on experience and being more prone to errors. Configuration of tools and equipment is thus prone to setting wrong process parameters or not adjusting them in time. Resulting drilling and riveting errors can lead to severe quality issues and high costs. Subsequent documentation of quality and progress usually is recorded on paper after visual assessment. This is especially problematic if offset heads are used to work on points with difficult accessibility and can lead to a lack of transparency in actual torque used for the processes. Additionally, current work

<figure><img src="image_5.png" /><figcaption>Fig. 3. State proportions in fuselage assembly processes (information proportion was increased by 10% after consulting with experts; Preparation & Follow-up was decreased accordingly)</figcaption></figure>
progress must be explained at shift change, at which point it is documented for control-
ling purposes. Figure 3 shows portions of methods-time measurement elements for the
process, underlining the necessity of assisting all elements of the work cycle.

3 Digital Assistance Concept

The proposed digital assistance concept contains three main elements:

*   **Digital twin.** A digital twin is the core component and is accessible as a platform, being the key to both scalability and functionality.

*   **Smart tools.** Smart tools with highly accurate position sensors receive the correct process parameters and send the actual values to the digital twin.

*   **Digital information assistance.** Information- and tool-based processes are assisted by creating means to efficiently provide information as well as interact with workers and write information back to the system.

Figure 4 provides a schematic overview of the concept elements and their connec-
tions.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Overview of user-centric digital assistance concept elements</figcaption></figure>

The concept is built upon the specifics of the analyzed manual processes but also
enables the seamless integration of further elements and sensors as well as logic modules
in the form of microservices. At this point in time, the concept is in the process of being
implemented. This means that a basic platform of the digital twin has been implemented
that can provide modular client interfaces and microservices. Furthermore, there are
prototypes of the smart hand tools that are being further developed in terms of their
industrial applicability and data interfaces for integration into the concept. Similarly,
existing software development kits for digital information assistance are being further
developed in terms of their synchronization with the digital twin and extensions for infor-
mation filtering on smartwatches. The following descriptions of the concept elements
can thus be regarded as a target state.
## 3.1 Digital Twin Platform

The digital twin is planned to act as core of the digitalization concept, as it will map the physical process and product to a digital entity using real-time data transmission from several sources. It is embedded in a platform that enables clients (software and hardware elements of the concept) to establish connections using different interfaces and transmission protocols and then bridges information to different modules using event-distribution software. The basic structural components and interfaces are already set up for testing purposes. Once the client connections are established, the event-distribution software allows microservices to subscribe to data messages and react according to workflow information. The platform enables and hosts the microservices that can access information from the centralized repository of data for various purposes, e.g., to optimize work execution in real-time or to feed machine-learning algorithms to learn from data correlation over time.

The envisioned purpose of the proposed modular structure is to allow new logic and software modules to be created and integrated seamlessly. Regarding the analyzed assembly processes, the intended use is to integrate and connect smart hand tools, position sensors and digital information assistance. On the other hand, the platform should integrate product data, workplans and process data from enterprise resource planning systems. 3D product models can be used to derive process parameters for the tools and work steps for the information assistance.

The applications and microservices will be running as containerized applications on a cloud-based cluster and be handled by an orchestrator. The idea is to ensure availability and to facilitate application scaling. Finally, a web-based user-interface should be included in the concept to allow planning engineers to view the quality and progress of the process in real time and to configure work packages, tools and connections as well as to interact with workers on the shopfloor.

## 3.2 Smart Tools

The key to the envisioned use of the smart tools is to equip them with highly accurate position sensors, enabling event-based data transmission to the digital twin. Given the requirements of assembly environments, the positioning accuracy needs to be approximately 10 mm to ensure accurate allocation to the correct drilling and riveting points. In addition, the tools will be equipped with an integrated chip and sensor technology that allows the effective recording of torque-force curves to enable conclusions to be drawn about the quality of boreholes and rivets. Determining and configuring which process parameters and tool attachments are being used, completes the system.

A bidirectional and low-latency connection to the digital twin facilitates the collected sensor data to be simultaneously evaluated in process by the platform's various logic modules, for example to take countermeasures in the event of mishandling. Via the communication of the position data, correct process parameters can be automatically adjusted. Moreover, the clearance can be denied if an incorrect attachment or a worn tool is about to be used. This avoids errors caused by using wrong machine configurations and simultaneously optimizes the assembly workflow by reducing necessary information gathering and configuration tasks to a minimum.
The comparison of data recording close to the active point and the torque fed in allows the calculation of wear in tool and attachment. Hence, there is no need to release tools only for a limited number of operations and for certain material pairings, which reduces maintenance measures. Besides the mentioned optimizations, the progress and quality of each machine operation is automatically documented in the digital twin.

## 3.3 Digital Information Assistance

To assist workers in information procurement combining a tablet as main information device and a smartwatch for crucial in-process information is recommended. The idea is to use a smartwatch to display information during work execution that reassures the worker of correctly operating. For example, the currently used torque can be displayed or visual aids can show directions to the next drilling or riveting point. Tactile feedback can be given when errors are about to be made and for more complex problems workers can be referred to the tablet. The tablet is used to display information in form of small work packages, e.g., preparatory steps of positioning components or overviews of drilling steps. Connecting it to the 3D product model allows to visualize complex information in reference to the product element of the user model. Role-based information layouts assure optimal filtering for each organizational entity.

A bidirectional connection to the digital twin will assure consistent up-to-date information about progress and quality so workers can visually confirm the current status if desired. This is likely to simplify both shift handover and collaboration as well as information procurement in general. By confirming work steps, the progress is automatically documented for tool-independent process steps. Errors and comments can be recorded in relation to these steps and collaboration with employees of indirect areas is possible. Combining these elements leads to an effective assistance of workers in preparation, execution and follow-up.

# 4 Potentials for Productivity and Quality

The idea of the digital assistance concept is to provide workers with an optimized and technologically augmented workflow as follows: After establishing the connection to the digital twin and the initial job synchronization, process guidance via the digital information assistant displays the current progress and next work step. With regard to drilling and riveting, the workers receive an optimized suggestion for the machining sequence and can then seamlessly carry out the steps with the smart hand tools while these are automatically configured or blocked in the event of mishandling. Progress and quality are automatically documented by the system and workers can concentrate on the execution without being exposed to the trade-off between information procurement and being prone to errors. Workflows can be optimized through data accumulation and machine-learning algorithms over time. Figure 5 shows a detail of the proposed workflow with the example of a seat rail assembly. The tablet will be used for processes like positioning the rail and preparing for the task. The smart tool used to drill holes and rivet the rail to the fuselage structure, will be automatically configured and the smart watch will display information such as current process parameters and potential errors.
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Information provision and augmented execution in seat rail assembly</figcaption></figure>

The described workflow holds various benefits compared to the current situation in assembly. Productivity and quality potentials are created, for example, in that core tasks are partly eliminated and partly simplified. The need for tool configuration and the respective documentation of progress or quality is eliminated. In addition, information provision is significantly improved and preparatory steps are simplified by visualizing the task. The execution of drilling and riveting becomes more efficient because constant checking of parameters is not needed and the machining sequence is suggested by the system in an optimized way. Collaboration between employees is enabled more intuitively and lengthy shift handovers can be better managed. All in all, the non-value-adding state proportions in fuselage assembly processes can be significantly reduced.

Finally, further potentials arise through the modular and seamless structure of the digital assistance concept, as it will enable future integration and utilization of benefits in the unified namespace. Dynamic flexibility is created through decoupling the client-interfaces from the event-distribution inside of the platform, allowing further sensors and data sources to be included. The long-term acceptance and implementation of the digital assistance concept in existing corporate structures thus becomes more likely.

# 5 Outlook

As of now, the digital assistance system is in the early stages of development. Nevertheless, the concept presented has been coordinated with the development partners and will be elaborated via feedback cycles with the industry in order to facilitate a transfer of the results. To verify the potential improvements of productivity and quality, a testable demonstrator will be developed and then tested and evaluated in the investigated assembly process. If the potentials can be realized, the concept will be transferred to industry to achieve long-term integration. The ultimate goal of our research is to improve quality and productivity in aircraft industry. The modular and platform-independent concept combining localized production means with bidirectional data flows also promises the potential to improve application scenarios in various industries.

By developing a systematic approach of configuring interfaces and new modules for the digital twin platform, the implementation of new data streams could then be designed in such a way that desired functionality is ensured for diverse applications. The approach should focus on matching the requirements of the digital twin use case with the characteristics of data transmission protocols and transmission technologies and propose a configuration accordingly.
**Acknowledgement.** The authors would like to thank Bundes-Ministerium für Wirtschaft und Klimaschutz (BMWK) for funding the project "Software architecture for a digital twin and connection of a digital assistance system to worker guidance, smart hand tools and location system". (Project No. 20W1922E).

## References

1. Hintze, W., et al.: Digital assistance systems for smart drillings units in aircraft structural assembly. In: 7th International Workshop of Aircraft System Technologies (AST 2019), pp. 255–266, Hamburg (2019)

2. Hinrichsen, S., Riediger, D., Unrau, A.: Assistance systems in manual assembly. In: Proceedings of 6th International Conference on Production Engineering and Management, pp. 3–14, Lemgo (2016)

3. Tietze, F.: Analyse und Verbesserung der Arbeitsproduktivität in der Unikatproduktion. Wissen schafft Innovation (32), Hamburg (2017)

4. Halata, P.: Augmented-Reality-gestützte Informationsbereitstellung für die Unikatproduktion. Wissen schafft Innovation (35), Hamburg (2018)

5. Mueller, R., Vette, M., Scholer, M. Ball, J.: Assembly assistance and position data feedback by means of projection lasers. In: SAE 2016 Aerospace Manufacturing and Automated Fastening Conference and Exhibition, Detroit (2016)

6. Abras, C., Maloney-Krichmar, D., Preece, J.: Encyclopedia of Human-Computer Interaction, 1st edn. Sage Publications, Bainbridge, Thousand Oaks (2004)

7. Apt, W., Schubert, M., Wischmann, S.: Digitale Assistenzsysteme – Perspektiven und Herausforderungen für den Einsatz in Industrie und Dienstleistungen. Autonomik Industrie 4.0, Berlin (2018)

8. Hold, P., Erol, S., Reisinger, G., Sihn, W.: Planning and evaluation of digital assistance systems. Procedia Manufact. 9, 143–150 (2017)

9. Rost, R.: Digitale Assistenzsysteme in der Industrie: Visualisierungskonzept. TUHH Universitätsbibliothek, Hamburg (2021)

10. Qiu, C., Zhou, S., Liu, Z., Gao, Q., Tan, J.: Digital assembly technology based on augmented reality and digital twins: a review. Virtual Reality Intell. Hardware 1(6), 597–610 (2019)