# Towards Human and Robot Collaborative Ergonomic Handling of Long Parts with a Loose Grip

Michael Hofmann, Markus Ganglbauer, Matthias Propst, Christian Wögerer, Andreas Pichler

► To cite this version:

Michael Hofmann, Markus Ganglbauer, Matthias Propst, Christian Wögerer, Andreas Pichler. Towards Human and Robot Collaborative Ergonomic Handling of Long Parts with a Loose Grip. 9th International Precision Assembly Seminar (IPAS), Dec 2020, Held virtually, Unknown Region. pp.249-259, 10.1007/978-3-030-72632-4_18. hal-03520387

HAL Id: hal-03520387

https://inria.hal.science/hal-03520387

Submitted on 11 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Human and Robot collaborative ergonomic handling of long parts with a loose grip

Michael Hofmann¹, Markus Ganglbauer¹, Matthias Propst¹, Christian Wögerer¹, and Andreas Pichler¹

Profactor Gmbh, Austria
`firstname.lastname@profactor.at`, 
WWW home page: http://profactor.at

**Abstract.** The concept of robots collaborating with humans has gained a lot of interest in the recent years. The pure use of the robot to follow the human's actions involves several important components, which have to fullfil economic, ergonomic, safety and usability aspects. This paper presents a setup build with recent hardware and taking economic, safety and usability aspects into account. A concept is introduced how the ergonomics of a human worker is actively influenced by the system.

**Keywords:** human robot interaction

## 1 Introduction

The concept of robots cooperating with humans has gained a lot of interest in recent years, in both domestic and industrial areas. Combining the cognitive strength of humans together with the physical strength of robots can lead to numerous applications [1]. For example, in industrial scenarios a certain assembly processes requires the worker's strenuous effort of lifting heavy objects, operating in non-ergonomic positions etc., which lead to negative long-term effects. This is becoming increasingly important also because of the aging work-force [1] and the fact that there is a trend to automate such work-places even if it does not lead to additional short-term profits [2]. The preferred solution for these work places is a robotic assistant to interact and aid a human operator rather than a fully automated system. Combining the flexibility of adapting in humans with the physical strength and efficiency of the robots/machines will potentially transform life and work practices, raise efficiency and safety levels and provide enhanced levels of service [1][3].

Human robot interaction (HRI) is a challenging field that combines robotics, artificial intelligence, cognitive science, computer science, engineering, human computer interaction, psychology and social science [4]. One of the primary goals of this research is to find an intuitive way in which humans can communicate and interact with a robot [5]. The essential components of HRI include evaluating the capabilities of humans and robots and designing the technologies and training
that produce desirable interactions between them [6]. Humans in general have perception and cognitive functions and are able to act and react with respect to a given situation. Characterizing and understanding a situation, to describe and detect a situation and often predict the next steps, comes naturally to humans. Whereas, to develop a robotic system with ‘Context Awareness’ or in other words to enable a robotic system to understand the circumstances under which they operate and react accordingly in a cooperative fashion is a challenging task [7].

Human robot interaction can be realized in various forms. A binary input (e.g. yes or no) from the human to the robot can be seen as an interaction in its simplest form. Depending on the kind of interaction [8], HRI in industrial scenarios can be partitioned into:

- human robot coexistence – where both agents (human and robot) operate in a close proximity on different tasks

- human robot assistance – where the robot passively aids the human in a task (helping in lifting heavy objects)

- human robot cooperation – where both agents simultaneously work on the same work piece (each agent has their own task to do on the work piece)

- human robot collaboration – where both agents perform coordinated actions on the same task (for e.g., robot handing over a work piece to the human operator, who then completes the coordination by taking the work piece)

## 2 Problem Formulation and Approach

The ability to use the robot to manipulate a part with a robot has been seen in many different ways. While the robot has to be designed to function with the applied task, the perception of the humans intend if of high importance. Latest implementations are using the position and actions of the body, to directly control the robot [9], with the drawback of accumulated errors. But also other, indirect control mechanisms like force measurement at the gripper has been shown [10] to work properly. The crucial part of the system is the measurement of the position deviation introduced from the human.

For part handling with force measurement, the measurement of forces strongly depends on the rigid grasp of the robot. Typically, this is achieved with a parallel gripper applying a high force to the part. There are several reasons, why a grasp with high forces is not possible for certain parts. For example if the part is not allowed to be deformed or the part, variation is too high. Another reason to not use an actual gripper at all is, if the costs for such a gripper is too high. Another problem is the high leverage of a long part, which is handled. The estimation of the actual deviation at the workers side is strongly effected by the noise produces by the force-torque sensor at the robots flange.

Therefore, this paper will try to avoid the measurement of forces and the need for rigid grasps. The approach taken is to measure the actual deviation at the workers side. Similar approaches have been seen in the domain of remote controlling robots. There have been several approaches to utilize low cost devices to control industrial robots. Approaches lead from skeleton tracking based on
RGBD sensor to inertial tracking of the human arm movement and vision based tracking technologies. For industrial solution, the stability of the tracking is of critical importance. Therefore, vision based tracking systems, which are affected by changing light conditions, are not suitable. Therefore a tracking system is used which uses modulated infrared light in combination with a special receiver.

There are several values of using a robot as the second side of a long part. One is the reduction of persons needed to actual handle a long part. The distribution of cognition and work force is divided between human a machine, where the human does the cognition. The revenue of this setup has to be evaluated by the industrial use case.

To increase the value of such a system, an additional perspective is intro-
duced, the ergonomic. The common ergonomy argument of a shared handling
approach with a robot is the high force capabilities of a robot. This is true, as
long as safety regulations are not considered. The problem with safety regula-
tions are, that a robot which is capable of lifting high loads is also capable of
hurting a human and therefore is not safe to use in a human robot collaboration
task. To overcome this issue, an other ergonomic benefit is discussed, to assist
the human to do the lifting in a more ergonomic way by analyzing the human
posture.

3 Handling of Long Parts

The basic setup of a human robot collaborative handling task is to control the robot via the human movements. The robot takes the role of mirroring the grasp position of the human to its own grasp position. The basic idea is to let the robot compensate the deviation made bad the human. The target control value is the angle of the part, relative to the floor. In Other words, the part has to be horizontally aligned.

There are four critical parts [11] to enable this setup to for human robot
collaboration. First, the control loop has to be closed. The control loop starts
with the tracking of the deviation of the human. The tracking system has to
calculate the correction action within a strict latency. Therefore, the correction
actions have to be transferable to the robot within a deterministic latency. The
next step is for the robot to apply the correction actions to the handled part in a
safe way for the human. The part itselfe has to be rigid. Otherwise, the applied
corrections by the robot are not transfered back to the human. The human closes
the loop by sensing the correction, and if desired, continous to manipulate the
part. The full setup is seen in Figure 1. These components will be discussed in
the following section.

3.1 Tracking system

The tracking system is the main interaction system for the human. The human
intend has to be measured by the tracking system. Intends which are lost, will
not be able to reconstruct in the later system. For vision-based system, this could
<figure><img src="image_2.png" /><figcaption>Fig. 1. PID Overview.</figcaption></figure>

happen if some occlusions happen between sender and receiver. Another issue is the latency and accuracy of the human measurement, if the latency increases too much, the human experience is lost. It has been shown in [12] that latencies above 300ms would significantly increase the error rate of a teleoperated robot. In addition, the accuracy of the measurement has to be accurate enough, to be not noticeable by the human.

To cope with these issues, a consumer tracking system is used, the HTC-Vive. Beside the fact, that it is compareable cheap, it also copes with the discussed issues. Accuracy and latency are within tolerable ranges, for a human robot collaboration [13] [14]. In addition, the stability of the system is high, because two measurements are used, first an optical measurement is done, to get an absolute positioning of the device, second an inertia measurement is done to scope with occlusions and to detect fast orientation changes.

## 3.2 Grasping the part

The connection of the robot to the part is a very important aspect of a human robot collaboration task. Depending on how rigid the connection is the possebilities of what the human can do changes. The scale goes from very rigid connections, where the robot controls the actual movement of the parts, to soft rigid connections, where the part can be moved without any movement of the robot.

In general connection with high rigidnes are desirable [15], if the part can be moved to much, it could be possible, that the part slips out of the robot gripper and a collaboration is not possible anymore. The payoff of a high rigid connection is the need for a good gripper, which is capable of performing a high rigid grasp.

As one of the scopes of the setup is to reduce the complexity, a pure mechanical connection of part and robot has been chosen. The outline of the connection can be seen in Figure 2. The robot only holds the part with a spine, which prevents slipping of the part in two directions. The human handles the third direction. This dramatically reduces the complexity and cost of the grasping
system, but it presumes that the part has an adequate counterpart to the spine.
The same gripper has been implemented for the worker, including the tracker,
shown in Figure 3

<figure><img src="image_3.png" /><figcaption>Fig. 2. Gripper construction.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Worker tool with tracker.</figcaption></figure>

The chosen construction introduces a low rigid connection between part and
robot. Therefore, the part can be easily moved around, without the need for
the robot to actual move. In this case, this is a desired behaviour, because the
deviation between part and robot is observed at the human side of the handling.
Due the size of the part a significant deviation of the human and robot position is introduced which allowes the tracking system to measure it accurately.

## 3.3 Realtime-Control of Robot

Similar to the tracking system, the real time control of the robot is a crucial part for the user experience. As described earlier, latencies above 300ms [12] can influence the task performance negatively if a human is involved.

The accumulated latency in the setup has to respect the tracking, pid-controller, robot-communication and robot-controller. Based on common implementation the latency of the system can be calculated. The summary of the latencies is shown in Table 1, the dominant latency is the network connection to the universal robot [16], which not a realtime connection.

Our implementation is based on the newer version of the universal robots controller, which including a realtime network interface [17] working at 125Hz. The usage of this network interface rapidly increases the latency of the system to latency below 50ms, our results are shown in Table 2. Additional latency measurements inside the system have been done and are shown in Figure 4

<table><caption>**Table 1.** Accumulated Latency of tracker to robot control.</caption><thead><tr><th>Component</th><th>Time</th><th>Reference</th></tr></thead><tbody><tr><td>HTC-Vive</td><td>22ms</td><td>[13]</td></tr><tr><td>PID-Controller</td><td>8ms</td><td></td></tr><tr><td>UR10-Communication</td><td>160ms</td><td>[16]</td></tr><tr><td>UR10-Controller</td><td>8ms</td><td>[16]</td></tr><tr><td>Sum</td><td>198ms</td><td></td></tr></tbody></table>

<table><caption>**Table 2.** Optimized Latency of tracker to robot control.</caption><thead><tr><th>Component</th><th>Time</th></tr></thead><tbody><tr><td>HTC-Vive</td><td>22ms</td></tr><tr><td>UR data capture and receive network data</td><td>8ms</td></tr><tr><td>Dominant role is HTC-Vive</td><td>22ms</td></tr><tr><td>PID-Controller</td><td>8ms</td></tr><tr><td>UR send network data</td><td>8ms</td></tr><tr><td>UR10-Controller</td><td>8ms</td></tr><tr><td>Sum</td><td>46ms</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Latency-Measurement in 8ms steps. Status values directly from RTDE(actual_q_0) and values read from URScript and transfered via RTDE(output_double_register).</figcaption></figure>

## 3.4 Safety-constraints

The collaboration scenario described in this paper is a collaboration of human robot in with a common task and workspace [18]. Therefore the safety requirements for have to be repected. The safety requirements for industrial robots are described in ISO 10218-1:2011 and ISO 10218-2:2011 [19][20], which also include standards for collaborative robots.

The assessment of the setup has shown, that the part extens the robot flange. The danger of the robot is reduced by limiting the robot to only X/Y-axis movement. This increases the safety of the system, because the robot cannot produce any force in the direction of the human. To ensure the safe behaviour of the robot, a safety plane has been configured inside the robot controller [21]. Figure 5 shows the available workspace of the robot, the visible plane cannot be left in either Z-axis direction.

## 3.5 Discussion

This section has shown the crucial parts of handling long parts in a human robot collaboration task. The described setup is designed to reduce costs as much as possible, while keeping the performance of the system. The operation of the system is designed to work within safety regulations.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Limitation of the robot movement to a plane.</figcaption></figure>

## 4 Ergonomic Handling of Long Parts

In this section, a concept to optimize the ergonomics of a human robot collaboration task is described. As described in the previous section, the human takes the control of the setup. The roles of the human and the robot are clear, the human known which task has to be done, while the robots helps the human to perform the task.

The proposed setup introduces a second role schema, where the robot tries to improve the ergonomics of the human. This can be done, by introducing limited autonomy in the collaboration setup. The improvement of the ergonomy should happen subliminal, which means, no display or other direct communication of the intend of the robot is communicated to the robot. The only way to influence the human is how the collaboration task is executed. This means, that the robot can influence the human by e.g. pushing the part higher, which should lead to a better attitude of the human. A graphical represenation of the setup is shown in Figure 6

The main part of the setup is the evaluation of the human in real time. A possible evaluation assesment is RULA (rapid upper limb assessment), which is a survey method for use ergonomics investigations of workplaces where work-related upper limb disorders are reported [22]. It has shown that RULA can be calculated in real time with a color and depth camera [23]. The RULA-score of the worker has to be optimized. A fitting algorithm for this task has to be found, or a fitting AI setup has to be defined [24].

## 5 Conclusions and Future Work

This paper presented a human robot collaboration setup using well-established consumer and industrial components. It has been shown that with using recent
<figure><img src="image_7.png" /><figcaption>Fig. 6. RULA Overview.</figcaption></figure>

hardware can drastically reduce the latency. The usage of consumer hardware
and the implementation of a simple but effective strategy to establish the han-
dling at the robots side has shown to reduce the overall finencial effort to be
taken to build the system.

As basis for the future work, a concept has been presented, to increase the
ergonomics of the human robot collaboration task.

6 Acknowledgements

This research is funded by the project MMAssist II (FFG, 858623).

References

1. eurobotics aisbl, robitcs 2020 strategic research agenda for robotics in europe 2014
   - 2020. Accessed: 2019-10-1.

2. LLP PricewaterhouseCoopers. The new hire: How a new generation of robots is transforming manufacturing. *Delaware limited liability partnership*, 2014.

3. JA Corrales, GJ Garcia Gomez, F Torres, and Véronique Perdereau. Cooperative tasks between humans and robots in industrial environments. *International Journal of Advanced Robotic Systems*, 9(3):94, 2012.

4. Kerstin Dautenhahn. Methodology & themes of human-robot interaction: A growing research field. *International Journal of Advanced Robotic Systems*, 4(1):15, 2007.

5. Brian Scassellati. Theory of mind for a humanoid robot. *Autonomous Robots*, 12(1):13–24, 2002.

6. Michael A Goodrich, Alan C Schultz, et al. Human-robot interaction: a survey. *Foundations and Trends in Human-Computer Interaction*, 1(3):203–275, 2008.
7. Albrecht Schmidt. Implicit human computer interaction through context. *Personal technologies*, 4(2-3):191–199, 2000.

8. Yi Shen. *System für die Mensch-Roboter-Koexistenz in der Fließmontage*, volume 305. Herbert Utz Verlag, 2015.

9. Wandelbots gmbh. http://www.wandelbots.com. Accessed: 2019-10-1.

10. Herman Bruyninckx. Orocos: design and implementation of a robot control software framework. In *Proceedings of IEEE International Conference on Robotics and Automation*. Citeseer, 2002.

11. Christian Wögerer, Harald Bauer, Martijn Rooker, Gerhard Ebenhofer, Alberto Rovetta, Neil Robertson, and Andreas Pichler. Locobot-low cost toolkit for building robot co-workers in assembly lines. In *International Conference on Intelligent Robotics and Applications*, pages 449–459. Springer, 2012.

12. Jessie YC Chen, Ellen C Haas, and Michael J Barnes. Human performance issues and user interface design for teleoperated robots. *IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)*, 37(6):1231–1245, 2007.

13. Diederick C Niehorster, Li Li, and Markus Lappe. The accuracy and precision of position and orientation tracking in the htc vive virtual reality system for scientific research. *i-Perception*, 8(3):2041669517708205, 2017.

14. Miguel Borges, Andrew Symington, Brian Coltin, Trey Smith, and Rodrigo Ventura. Htc vive: Analysis and accuracy improvement. In *2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pages 2610–2615. IEEE, 2018.

15. Martin Lawitzky, Alexander Mörtl, and Sandra Hirche. Load sharing in human-robot cooperative manipulation. In *19th International Symposium in Robot and Human Interactive Communication*, pages 185–191. IEEE, 2010.

16. Thomas Timm Andersen. Optimizing the universal robots ros driver. 2015.

17. Real-time data exchange (rtde) guide - 22229. https://www.universal-robots.com/how-tos-and-faqs/how-to/ur-how-tos/real-time-data-exchange-rtde-guide-22229/. Accessed: 2019-10-1.

18. George Michalos, Sotiris Makris, Panagiota Tsarouchi, Toni Guasch, Dimitris Kontovrakis, and George Chryssolouris. Design considerations for safe human-robot collaborative workplaces. *Procedia CIRP*, 37:248–253, 2015.

19. ISO ISO. 10218-1: 2011-robots and robotic devices-safety requirements for industrial robots-part 1: Robots. 2011. URL: http://www.iso.org/iso/iso catalogue/catalogue tc/catalogue detail.htm.

20. ISO ISO. 10218-2: 2011: Robots and robotic devices-safety requirements for industrial robots-part 2: Robot systems and integration. *Geneva, Switzerland: International Organization for Standardization*, 2011.

21. Naresh Chowdary Chitturi, Markus Ganglbauer, Matthias Plasch, Pavel Kulha, and Andreas Pichler. Holisafehrc: Holistic safety concepts in human-robot collaboration. *chest*, 25:40.

22. Lynn McAtamney and E Nigel Corlett. Rula: a survey method for the investigation of work-related upper limb disorders. *Applied ergonomics*, 24(2):91–99, 1993.

23. Vito Modesto Manghisi, Antonio Emmanuele Uva, Michele Fiorentino, Vitoantonio Bevilacqua, Gianpaolo Francesco Trotta, and Giuseppe Monno. Real time rula assessment using kinect v2 sensor. *Applied ergonomics*, 65:481–491, 2017.

24. Yujiang Xiang, Jasbir S Arora, Salam Rahmatalla, Timothy Marler, Rajankumar Bhatt, and Karim Abdel-Malek. Human lifting simulation using a multi-objective optimization approach. *Multibody System Dynamics*, 23(4):431–451, 2010.