# Design Guidelines for Collaborative Industrial Robot User Interfaces

Helena Anna Frijns, Christina Schmidbauer

► To cite this version:

Helena Anna Frijns, Christina Schmidbauer. Design Guidelines for Collaborative Industrial Robot User Interfaces. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.407-427, 10.1007/978-3-030-85613-7_28. hal-04292362

HAL Id: hal-04292362

https://inria.hal.science/hal-04292362v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Design Guidelines for Collaborative Industrial
Robot User Interfaces

Helena Anna Frijns¹[0000-0003-3866-7380] and
Christina Schmidbauer¹[0000-0002-9761-9058]

Institute of Management Science, TU Wien, Vienna, Austria
{helena.frijns,christina.schmidbauer}@tuwien.ac.at

**Abstract.** Collaborative industrial robot (cobot) systems are deployed to automate tasks or as a tool for Human-Robot Interaction (HRI) scenarios, especially for manufacturing applications. A large number of manufacturers of this technology have entered the cobot market in recent years. Manufacturers intend to offer easy control possibilities to make cobots suitable for different user groups, but there are few evaluation tools for assessing user interface (UI) design specifically for cobots. Therefore, we propose a set of design guidelines for cobots based on existing literature on heuristics and cobot UI design. The guidelines were further developed on the basis of modified heuristic evaluations by researchers with robotics expertise, as well as interviews with cobot UI/User Experience (UX) design experts. The resulting design guidelines are intended for identification of usability problems during heuristic evaluation of the UI design of cobot systems.

**Keywords:** Human-Robot Interaction · Human-Robot Collaboration · Collaborative Robots · Design Guidelines · Heuristic Evaluation · User Interface Design

# 1 Introduction

This paper is concerned with the design of robots and associated UIs that are part of collaborative industrial robot systems (cobots). Cobots are systems that are intended for collaborative operation, i.e. the concurrent execution of tasks by human(s) and robot(s) in the collaborative workspace (as defined in [14]). Such a system is not collaborative by itself; rather, the collaborative nature arises from the way the application and interaction are designed. A cobot system usually includes a robotic arm and a graphical user interface (GUI) with which, for instance, a factory worker can program the robot to do a specific task in a manufacturing context. The last decade has seen increased interest in the introduction of cobots to the shop floor [7]. Various manufacturers of automation solutions (e.g., ABB, COMAU, FANUC, KUKA, Stäubli, Yaskawa/Motoman, Doosan) have extended their product portfolio with cobots, and new suppliers such as Universal Robots, Techman Robot, and Franka Emika have entered the market. Examples of cobots are Universal Robots' UR5 [35], Kuka's LBR iiwa [17], and Franka Emika's Panda [10] (Fig. 1).
The design of current cobot UIs that are on the market can be improved. Researchers have evaluated several cobot systems and found them to have low usability scores<fnref n="1" />. High ease of use is especially important, as the promise of cobot systems is that these will enable reprogramming by factory workers with limited programming expertise [23]. The importance of making industrial robot systems easier to use is further exemplified by several recent research projects that aim to make cobot UIs easier to use or introduce different concepts for the design of programming environments on cobot UIs [12, 15, 27, 32]. Companies are entering the market offering alternative UIs for existing cobot hardware with the proposition that these are easier to use [6], which indicates that improvement of the usability of cobots is seen as a market opportunity. Other researchers have pointed to the importance of designing cobots to be easy to use and intuitive to interact with [23, 36] and that are positively evaluated by operators, in order to make the shift from traditional manufacturing robotics to user-friendly cobot systems [20]. Therefore, we claim that establishing factors that are important for cobot design and providing cobot UI designers with a tool for usability inspection is a relevant endeavor.

In this paper, we propose design guidelines for heuristic evaluation of cobot UIs. We argue that cobot UIs require a separate, more specific set of guidelines as the context these systems operate in is distinct from other HRI (Human-Robot Interaction) or HCI (Human-Computer Interaction) applications for which heuristics have been developed previously. One of the main tasks is the programming of the cobot's task, which is a rather specific HRI application for which factors such as reuse of previous work are important. This sets it apart from applications such as teleoperation in field robotics. Moreover, cobots have a specific use, as these systems are integrated in a work context in which they are subject to interaction with operators repeatedly over long periods of time, which increases the importance of human factors and accessibility [36]. Cobots also represent a challenge in terms of the variety of users, who differ insofar as their roles, preferences, and abilities to modify the cobot control program are concerned (levels of interaction [31]). The UI should support all these users in programming, maintaining, and monitoring the cobot.

We propose a set of 24 design guidelines that can be used during the development, design, and implementation phases of cobot systems to identify UI design problems by means of heuristic evaluation. The contribution of this paper is the identification and refinement of cobot design guidelines based on literature and expert evaluation, thereby providing a resource for design and evaluation of cobot UIs (a form of intermediate-level knowledge for HRI design [18]). In Sec. 2, a short literature review is provided. The procedure and database search to establish the initial guidelines are described in Sec. 3. The guidelines were further developed by means of an evaluation study as reported in Sec. 4, and interviews

<footnote n="1">¹ Several usability evaluations yielded SUS scores between 50-70 for different cobot systems [8, 31], which has been argued to indicate that a product is marginal in terms of usability and should be improved [3].</footnote>
with UI/UX design experts as described in Sec. 5. Finally, Sec. 6 contains the
proposed guidelines.

<figure><img src="image_3.png" /><figcaption>Fig. 1: Franka Emika Panda cobot</figcaption></figure>

# 2 Related Work

## 2.1 Interaction with cobot systems

With regard to design, several factors must be considered by a manufacturer to enter the market and be competitive. Besides the main factors functionality and safety, other factors such as risk and ergonomics assessment, risk reduction in the workspace, and specifications of use limits and transitions to other operations are required for the design of cobot applications [14]. The interaction is characterized by the levels of interaction based on safety implications of cobots. Cobots are partly completed machinery; these systems have to be designed in compliance with regulations (e.g. [33]) and include safety mechanisms [19]. Cobots have particular challenges, as they are intended to be operated in close proximity by end users with differing levels of programming experience, in the context of different types of tasks. For instance, users can take on the role of programmer, operator/supervisor, or maintainer [7, 20]. Programming and maintenance activities are usually of limited duration or even restricted to single events, while supervision requires more continuous levels of interaction and attention [20].

Another consideration is that of UI design. The cobot user interface is that part of the system that enables the user to interact with the cobot. The interaction takes place by means of system inputs and outputs, and can be realized using multiple different interaction modalities (multimodal interaction). The design of the interface determines how commands can be given to the machine or application and how information is represented to the user. Several different interaction modalities exist (e.g. visual, acoustic, haptic) and are realized in different ways (keyboard, mouse, buttons, sounds, signal lamps, projectors) (see [7, 31] for an overview). A common piece of hardware that is part of the UI is the teach pendant, which is a hand-held device that is usually equipped with physical
buttons and a GUI. This is often the main interface for programming and maintenance activities [20]. Examples of GUIs are Franka Emika's web-based Desk interface [10] and the GUI of UR5, which runs on a teach pendant [35]. Teach pendants and other associated screens enable the user to program and control the cobot and to observe status information. Programming environments often contain representations of the robotic arm and its configuration, and location and trajectories of the end effector.

In order to introduce cobots in industrial environments, research has been conducted into enabling human awareness and so-called intuitive programming of cobots [7]. The term intuitive is often used to characterize UIs and refers to an effective interaction between a user and a technical system without conscious use of previous knowledge [25]. Moreover, it refers to the use of design strategies that appeal to prior knowledge in a way that makes it easier to (learn to) use an interface and reduces user effort. Another relevant term is usability, which has been defined as the “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [13]. It has been argued that making technologies more intuitive and easier to learn may promote access to jobs that typically require higher skills [23]. Developments towards making cobot programming more intuitive include integration of teach pendants and other input devices, programming by demonstration, multimodal HRI, and virtual and augmented reality (VR, AR) [16]. A drawback of using communication modes such as speech and gesture is that they add additional uncertainty; GUIs may be better suited for use in industry. Ways of programming a cobot through a GUI include making use of cobot teach pendants, Icon-based programming, CAD-based programming, and task-based programming. Another way to control the cobot is by means of haptics and force [7]. Krot and Kutia distinguish between online, offline, and hybrid programming approaches [16]. For more information on cobot programming we refer to [7, 16].

## 2.2 Evaluating cobots: metrics, measures, methodologies

Several researchers have performed usability evaluations of cobot systems. Ferraguti et al. [8] propose a methodology for executing a comparative analysis of cobots, which they apply to the cobots KUKA LWR 4+, UR5, and Franka Emika Panda. They propose that technical data as provided by the manufacturer, experimental verification, usability evaluation and required physical and mental effort to program the robot should be taken into account for such a comparative analysis. For the usability score, they used the SUS (System Usability Scale) and the Questionnaire for the Evaluation of Physical Assistive Devices (QUEAD). Schmidbauer et al. [31] evaluated three cobots UIs (Franka Emika, UR, Fanuc) using the SUS. Weintrop et al. [37] evaluated the time on task, task success, ease of use, learnability and satisfaction of the CoBlox environment they developed using Blockly, ABB's FlexPendant, and Universal Robots' Polyscope. For an overview of metrics, we refer to Marvel et al.'s metrology framework for Human-Robot Collaboration in manufacturing [20]. We conclude that for the
evaluation of the usability of cobot systems, often qualitative and quantitative
measures are combined. Examples of quantitative metrics for assessing inter-
faces are learning time, expert use time, and error cost, while qualitative metrics
include the NASA-Task Load Index (NASA-TLX) and the SUS scale [20].

## 2.3 Design guidelines and heuristic evaluation

Heuristics are principles that can be used to identify usability problems with a particular UI. Examples of heuristics are: “Does the interface provide feedback?” [5] and “Provide indicators of robot health/state” [1]. Heuristic evaluation refers to a type of evaluation that is usually conducted by UI/UX experts, during which they use heuristics or design guidelines to identify usability problems. Quiñones and Rusu [29] describe the method of heuristic evaluation. First, each evaluator individually compiles a list of problems with the help of a set of heuristics. The evaluators combine the problems to form a list of unique problems, which are then rated for severity, criticality, and frequency by each individual evaluator. Advantages are (1) the low cost of heuristic evaluation, (2) that no extensive planning is required, (3) the broad applicability of the method, especially early in the software development process, and (4) that the method can help find many usability problems without involving end users. On the downside, heuristic evaluation requires experienced evaluators with task-specific knowledge. Additionally, the method helps identifying problems but does not offer pre-packaged solutions to these problems [29]. Heuristics and design guidelines can be considered as intermediate-level knowledge for HRI design [18].

Different sets of design guidelines and heuristics exist, many of which are
targeted at the design of computer software, such as the usability heuristics pro-
posed by Nielsen [26]. For robotics, alternative sets have been proposed for appli-
cations such as field robotics [24], robot teleoperation [1], assistive robotics [34],
and HRI in general [5] (these have also been applied in video-based heuristic eval-
uation [38]). Robots are embodied and HRI often requires different interfaces to
support multiple user roles, which sets HRI apart from HCI applications [24].
While these arguments apply to cobots as well, cobots present a different applica-
tion area compared to field robotics, teleoperation, and assistive robotics. When
interacting with a cobot, the human and the robot are co-located, which distin-
guishes this type of interaction from scenarios such as teleoperation, as opera-
tors have access to more contextual information [20] and users can, for instance,
switch between physical interaction and interaction through a GUI. Cobots can
be interacted with and observed via different means (e.g. programmed on a com-
puter or using a teach pendant, or hand-guided). Cobot status can be observed
on the teach pendant or by looking at the cobot itself (e.g. checking indicator
LEDs). Interaction modalities such as AR, VR, and a variety of handheld devices
are subjects of ongoing investigation [36]. This makes it important to guarantee
consistency with respect to the way information is presented to the user across
different modalities, as well as to support the user in managing their attention.
Teach pendants of cobots can be rather heavy, but existing sets of heuristics
(e.g. [5, 26]) do not refer to physical ergonomics, which is a relevant factor for
usability of cobot systems. This means that existing design guidelines for HCI or HRI in general are not sufficient, nor are sets that have been developed for other applications such as teleoperation. In the next section, the development of a set of guidelines for heuristic evaluation of cobot systems is described.

## 3 Development of the cobot UI design guidelines

With regards to the realization of a new set of heuristics, Quiñones and Rusu [29] recommend establishing which features of the target domain are application-specific, identifying existing heuristics that can be reused, specification of heuristics according to a template, and validation of the heuristics. Common methods include collecting design recommendations from industry, from the public domain [2], and from academia [1]. The collected guidelines can then be clustered using asynchronous affinity diagramming [2] or other methods, and sorted using methods such as open or closed card sorting [1]. Methods to evaluate heuristics include performing modified heuristic evaluations, user studies based on heuristic evaluation [2, 5, 34], and expert reviews [2]. Other practices include the use of templates for problem reporting [5], customizing heuristics to the application domain [5], and using an iterative process [2, 5, 29]. We make use of these methods and recommendations to establish heuristics for cobots, as described below.

### 3.1 Procedure for establishing cobot UI guidelines

The methodology for establishing the cobot UI guidelines was as follows (see Fig. 2). Prior to establishing the design guidelines, an informal heuristic evaluation was conducted using existing sets of heuristics, namely [1, 26], to determine if these were sufficient or if some issues were not appropriately covered by those heuristics. In order to establish which design guidelines had previously been proposed in the academic literature, papers were collected that propose guidelines for HRI design by searching academic research databases (ACM digital library, IEEE Xplore®, and SpringerLink, Sec. 3.2). Design recommendations that were listed in those papers were collected and categorized during an affinity diagramming session. This resulted in clusters of guidelines that were summarized into individual heuristics. Additional heuristics and clusters were proposed on the basis of literature on cobot systems in manufacturing (beyond literature focusing on design guidelines, e.g. [7]). The preliminary heuristics were used in an evaluation of two cobot systems, after which they were revised to make them more clear, actionable, and applicable. During the first empirical evaluations (Sec. 4), study participants (early-career researchers with HRI expertise) were asked to apply the guidelines in the context of a modified heuristic evaluation of a cobot system, and rate the guidelines for clarity. Based on participant feedback, the guidelines were revised. Experts in cobot UI/UX design (such as UX designers at companies developing cobot systems) were invited to participate in interviews to collect feedback on the guidelines (Sec. 5), which led to a final revision of the guidelines. For the final version of the guidelines, see Tables 1 and 2.
<figure><img src="image_4.png" /><figcaption>Fig. 2: Overview of the research process, in which different sources of information were used to propose design guidelines for heuristic evaluation.</figcaption></figure>

## 3.2 Literature review with database search

We performed a database search with the aim of finding design guidelines, recommendations, and heuristics that have previously been proposed in the academic literature, especially for HRI and cobots. The inclusion criteria were that the publication had to be in English, that it contained specific recommendations and heuristics, and that its topic should concern design guidelines, heuristics, or interaction design for an interactive technology or robotics. The research question was: *What are specific design guidelines, recommendations, and heuristics that have been proposed in the academic literature, especially for HRI?*

The databases ACM digital library, IEEE Xplore®, and SpringerLink were searched for combinations of keywords. The timeframe was restricted to 1999-2019. Databases were searched for combinations of one of the terms "Human Robot Interaction", "cobot", "collaborative robot", "robot", plus one of the following terms: "heuristics", "design guidelines", "design recommendations", "usability guidelines", "design principles". If possible, it was specified that these terms needed to appear in the abstract of the paper. After searching for combinations of keywords, duplicates were removed. Then, the paper's title, keywords, and abstract were reviewed and the paper was excluded if it did not meet the inclusion criteria. For the ACM database search (conducted 25-11-2019, 28-11-2019), 354 results were checked, resulting in 32 papers. For the IEEE Xplore® database search (on 19-11-2019), checking 2808 results led to selection of 53 papers. The Springer database search (on 3-12-2019 and 4-12-2019) yielded too many search results to check manually. For this database, we specified the search terms as a combination: ((Human AND Robot AND interaction) OR robot OR (collaborative AND robot)) AND (heuristics OR (design AND guidelines))
OR (design AND recommendations) OR (design AND principles) OR (usability AND guidelines)). This yielded a total of 35,851 results. Results were sorted with SpringerLink's "sort by relevance" feature and restricted to the first 500 items. Of these items, 51 papers met the inclusion criteria.

The database search yielded 131 unique papers after checking titles and abstracts for relevance. Each paper was checked for concrete guidelines and recommendations, which resulted in 42 papers containing 561 guidelines and recommendations. The main application domains that these papers were concerned with, were HRI in general (11 papers, e.g. [38]), rescue robotics, field robotics, safety-critical systems and tactical systems (11 papers, e.g. [1, 24]), assistive and service robotics (10 papers, e.g. [34]), telepresence systems (3 papers), and industrial robotics (2 papers, [9, 21]). After removing duplicate guidelines, this resulted in 480 guidelines, heuristics, and other recommendations.

## 3.3 Clustering of guidelines based on affinity diagramming

All 480 design recommendations were clustered into groups of similar topics by means of affinity diagramming (similar to [2]). This resulted in 38 clusters, each of which contained between 2 and 27 recommendations. Additional design issues specific to cobots were identified, mostly based on [7, 16], such as the use of multiple modalities and providing information on ways a user can teach the cobot. Finally, some clusters were merged, changed, or deleted (e.g. the cluster "cultural expectations" was deleted, as the recommendations collected in that section mostly applied to conversational agents). We summarized each cluster into a single design guideline, resulting in a total of 35 guidelines.

## 3.4 Revision based on modified heuristic evaluation

The first version of the guidelines was applied by the authors of this paper in an evaluation session that was structured as a modified heuristic evaluation modeled after [2]. The interface of Franka Emika Panda and UR5 were evaluated. The Panda cobot was controlled from the Desk environment, version 2.1.1 ced048bda3. The UR5/CB3 (version 3.9) cobot was equipped with software 3.9.1 64192 (April 1 2019), installed on a teach pendant (12 inch touchscreen) with the PolyScope GUI. A Robotiq gripper [30] plus custom 3D printed safety encasing was attached to the UR5 robot, and functionality of the gripper was embedded in the GUI with URCaps. During the evaluation, examples of applications and violations of the guidelines were collected. If no applications or violations of the guidelines were found, deleting the guideline was considered, as this was an indication that the guideline would be difficult to evaluate by means of heuristic evaluation. If there was overlap between applications and violations listed for different guidelines, we considered merging them. This resulted in 26 guidelines. Items regarding technical capabilities, human-oriented perception, and safety were removed, as these are basic requirements of robotic systems. Assessing safety or privacy is outside the scope of UI/UX design or heuristic evaluation, as safety of the cobot system must comply with (legal) standards [14].
# 4 Evaluation study based on modified heuristic evaluation

After the database search and subsequent refinement of the proposed guidelines, an evaluation study was conducted (N=9) that was based on modified heuristic evaluation of two cobot systems by participants with robotics and/or cobot systems expertise. The goal of this evaluation study was to find out if the formulations of the guidelines were clear and if they could be applied to existing cobot systems. We used feedback by participants, clarity scores given by participants to guidelines, and survey responses to revise the guidelines established in Sec. 3.

## 4.1 Procedure of the evaluation study

The evaluation study was structured as a modified heuristic evaluation. The evaluated cobots were described in Sec. 3.4. The study took place at the teaching and learning factory at TU Wien [28]. Participants were asked to sign an informed consent form and complete a personal information survey. They were explained what heuristic evaluation is, what it is used for, and they were given some examples of applications and violations of a heuristic in the context of HCI and HRI. Next, they received a safety briefing for the cobot they were going to work with. Participants were asked to read the guidelines and the survey form was explained to them (see next section). They were asked to identify applications and violations of each design guideline, to rate each guideline for clarity and to make notes if they had any other feedback. Participants were informed that their performance on the task would not be scrutinized and that constructive criticism was welcomed, as this would help to improve the guidelines. Then, they worked with the cobot by themselves and completed the survey form. Participants took about one hour to complete the evaluation study.

**Participant task in the evaluation study** Each participant interacted with a cobot for an hour. They were supplied with a programming task, but were allowed to deviate from it. The task contained instructions to program a movement trajectory with several intermediate points for the cobot and to open and close the gripper. The task for UR5 was as follows: 1) *Use the Move tab to program a trajectory.* Program at least 3 points. 2) *Change the second point of the trajectory you programmed.* 3) *Open the gripper.* 4) *Add another Move trajectory: Use the Freedrive button to move the robot and store some points for a trajectory.* 5) *Close the gripper.* The task for Panda was similar. We asked participants to complete a survey form while programming the task on which they 1) gave a rating for clarity of each guideline, 2) wrote examples of applications and violations of guidelines they encountered in the UI (for example, one participant noted "It is not clear what is the exact location of the end effector" as a violation regarding accessibility of information), and 3) gave feedback on guidelines. We asked for examples to check whether people could apply the guidelines.
**Survey forms** The personal information sheet included questions regarding participants' professional occupation, specialization, name of employer, and asked participants to estimate their experience working with cobots in hours and robotics expertise in years. The survey form contained every guideline in the set. The first question for every guideline was "*The formulation of this guideline is very clear / clear / neutral / confusing / very confusing*". Participants were asked to circle the answer they felt was appropriate. There were open fields where participants could list applications and violations of each guideline. They were informed that they did not need to list any examples if they did not find any, but that they could also list multiple examples. The final question on the survey form asked participants for feedback on the guidelines. Half of the participants received a survey form on which the guidelines were sorted alphabetically and the other half in reverse alphabetical order.

## 4.2 Participants

A total of 9 participants (5 male, 4 female) took part in the first evaluation sessions. Participants were recruited internally, thus 8 of 9 participants reported being employed by TU Wien. The participants were mostly early-career researchers (average age M=29.3, SD=2.2 years), with specializations in the areas of (industrial) robotics, mechanical engineering, assistive systems, HRI, and social robotics. Regarding their robotics expertise, one participant reported that their robotics experience was limited, 3 reported between 18-24 months of experience, 3 reported 4-5 years experience, and 2 reported 10 years experience. All participants had experience directly working with cobots (e.g. programming cobots).

## 4.3 Evaluation results

The explanation phase of the experiment, including reading the guidelines, took 20 minutes on average and the average working time was 63.3 minutes. Of the participants, 4 evaluated the UR5 UI and 5 the Franka Emika Panda UI. See Fig. 3a for an overview of the clarity scores for each of the guidelines. Then, the listed applications and violations were analyzed. Items were excluded if they were not specific enough, did not apply to the guideline, or were simply stated as a confirmation of the guideline. For instance, one participant listed "*Panda is very intuitive*" as a response to *Support user learning*, which was removed on the grounds of not being specific enough. Another wrote "*indicating system state*" in response to *System state awareness*, which was removed as it does not specify how the guideline is applied or violated in the UI. See Fig. 3b for an overview of the unique number of applications and violations for each guideline (if multiple participants listed the same application/violation, these would be counted as one unique application/violation). As multiple examples were listed for each guideline, we conclude that the guidelines could be applied to the cobot systems and that guidelines could be interpreted by the evaluation participants.
<figure><img src="image_5.png" /><figcaption>(b) This figure shows the number of unique applications and violations of each guideline were listed for both the Panda cobot and the UR cobot.</figcaption></figure>

<caption>Fig. 3: Results of the evaluation study.</caption>

## 4.4 Revision of the guidelines

Guidelines were revised on the basis of the participants' feedback and if a guideline's clarity score was higher than 1.5 (guidelines with a score of 1 were very clear, guidelines with a score of 5 very unclear). During the revision process, the applications and violations listed by the participants on the survey form were compared to the meaning the researchers intended to capture with the guideline. Guidelines were revised by changing terms that were not understood, changing complex words to simpler formulations, removing word repetitions, and ensuring guidelines contained between 5-12 words as recommended in [2]. For example, item 12 had a clarity score of 1.67, thus 12. Errors: Accommodate troubleshooting when errors occur was changed to 12. Errors: Give clear explanations and steps to recover when errors occur. Terminology was made consistent (e.g. referring to the person interacting with the system as "user"). Guidelines were grouped into categories and initial category headings were added.
# 5 UI/UX Design Expert Interviews

In order to get a practitioner perspective on the guidelines and on the practice of developing cobot UIs in industry, cobot UI/UX design experts were invited to participate in interview sessions. The main aims of the interviews were to evaluate if participants could recognize and relate to the guidelines, as well as to gather feedback to the guidelines and check whether they needed revision. In other words, both the relevance of individual guidelines to participants’ work practices and the formulation of the guidelines were evaluated. Six online interviews were conducted, which resulted in 41-74 min. of recorded material per interview (average 52 min.). The interviews were transcribed and coded using a thematic analysis approach [4]. On the basis of interviewees’ feedback, minor changes were made to the guidelines.

## 5.1 Participants

We contacted 9 companies and company branches via email and contact forms. Additionally, 16 individuals were contacted who had cobot UI/UX design experience and/or had a leading design or innovation role at companies that developed cobots, who were found via company LinkedIn pages, referrals by university colleagues, and referrals by interview participants. We found 6 suitable participants who agreed to an interview, four of whom worked at two different companies producing cobot systems in the role of (senior or junior) UX designer or engineer. One participant worked at a university as a researcher and had prior experience in software development for cobot systems. The final participant worked at an industrial automation company and was responsible for usability testing of industrial robotic systems. Five were based in Europe and one in China.

## 5.2 Procedure and Interview Protocol

Prior to the interview, participants received the guidelines for reference, information regarding storage of data in accordance with GDPR, and an informed consent form. At the start of the interview, the interviewer (the first author of this paper) thanked the interviewee for participating, introduced the goal of the research study, and gave the participant an overview of the topics that would be discussed. Participants were informed that constructive criticism was welcome, that they could withdraw at any time, and that they could notify the researcher if they did not want to answer a particular question. The participant was informed that the recording was started and was first asked to answer introductory questions regarding their job description, the main (design) activities and evaluation methods they used in their job, and their familiarity with cobots. Subsequently, participants were asked for feedback regarding the individual guidelines, their response to the whole set, and the categorization. Finally, participants were asked what challenges they encountered in the cobot UI design process. Participants were not informed regarding previous steps of the development of the guidelines.
## 5.3 Interview themes

A thematic analysis approach [4] was used to analyze the interviews using the software MAXQDA [22]. First, transcripts were read by both authors of this paper. One of the interview transcripts was analyzed and coded separately by both researchers (generating initial codes), who then agreed on a coding scheme, which was used to (re)code all six transcripts. The coding scheme included responses to guidelines and other categories such as design methods used. Both researchers were involved in coding the transcripts and subsequent analysis. Based on the analysis of the interviews, recurring themes were identified regarding goals, methods, and design constraints. Job-related goals mentioned by interviewees were focusing on and solving problems for the end user (explicitly mentioned by 3 different participants), improving ease of use and the user experience (2x), improving efficiency (2x), changing the minds of engineers on the team, improving system understanding, conducting research, and solving problems in the company product (1x). Methods that were mentioned most were getting feedback from team members or customers (5x), interviews with end users/customers, persona development or definition of target end users (4x), getting familiar with or explicitly evaluating competitor products (3x), comparing interfaces using metrics (such as time to complete certain tasks), identifying dependencies and requirements, drawing, and prototyping (2x).

**Design constraints** Interviewees indicated that the implementation of the guidelines not only depended on the design of the cobot, but also on the application that the cobot was used for, regulations, safety measures, company culture, and differences between users. Different users need different information (mentioned 3x), have different backgrounds and levels of expertise (3x), and have different expectations. Human factors is influenced by the application and the way the work environment is organized, by regulation, and different users have different abilities (e.g. some users are color blind). Cultural effects and country-specific regulations will also influence how the guidelines can be implemented. One person mentioned that autonomy given to workers by the companies varies greatly from one country to another. While we would propose that information that is necessary to (better) achieve the user's task should not be hidden by the system manufacturer, we note that company culture, the culture of the region, and commercial interests will influence how guideline number 3 (in Table 1), accessibility of information, is implemented in practice.

**Diversity of end users as an effect on guideline implementation** Interviewees emphasized that there are many different end users for whom they are designing interfaces. People who interact with cobots have different roles in the company (such as operator or programmer). The type of interface that is desirable depends on the previous knowledge and expertise of the end user, but also on what kind of application the user wants to implement on the cobot. A software developer with several years of coding experience, but no previous
interaction experience with any robot needs different user guidance than a shop
floor worker who has several years of experience in working with machines and
industrial robots but has never programmed or controlled them by themselves.
Novices were often contrasted to expert users during the interviews. In addition,
different stakeholders of the cobot development process were mentioned, such as
company salespeople, system integrators, customers, end users, engineers, and
programmers. We connect the fact that different users have different needs to
the frequent use of methods such as developing personas. While many of the
interviewees emphasized that their focus is on the end user, they also noted
that it can be difficult to get access to end users. One participant mentioned
that in specific industries, exchange of information is not desired by companies,
although this information is much needed to improve cobot UI/UX design.

**Response to design guidelines and revisions** Some participants responded to almost every guideline, either by stating a confirmation, supplying a practical example, asking for clarification, or voicing disagreement. Guidelines 13. *Human factors* and 17. *Adaptable system architecture* were responded to by most participants (5 out of 6). Participants also remarked on the whole set, its potential uses, and on the categorization of the guidelines. One participant noted that *"it is something I can relate to, even though I never had this structured list before (...) it's (...) putting a word on most items I do here at work"*. Two participants explicitly stated they thought the guidelines were quite complete. A few extra topics were suggested, such as safety (which we had previously decided to exclude) or learning about safety (which is implicitly included in item 9). Possible alternative uses of the guidelines were mentioned, such as using the guidelines as a checklist, using them for discussion with top management to indicate which items need to be worked on, or applying the guidelines to other machine interfaces in a manufacturing context. Two participants remarked that similar UI/UX guidelines exist, which is understandable as the guidelines are based on literature search. Participants made comments that many of the issues touched on by the guidelines were large research projects in themselves. One participant also emphasized that cobots are part of larger systems, and that programming cobots is just one piece of the puzzle when it comes to solving automation problems.

Interviewee feedback indicated that the guidelines did not need substantial
revision. Details in the wording were adjusted, for instance "operator attention"
was adapted to "user attention" as this guideline did not apply only to oper-
ators on the shop floor. Several guidelines were adapted to reflect remarks by
participants. For instance, the item accessibility of information was adapted,
as participants pointed out that some information is not understandable to end
users (such as raw data), and information requirements and accessibility also de-
pend on the user, culture of the region and company, and user access. Guidelines
3, 8, 9, 11, 13, 15, 17, and 18 were adapted slightly, 20 was adapted substantially
for clarification and merged with a different guideline, and several initial cate-
gory headings were revised based on participant feedback (the numbering refers
to the guidelines as presented in Table 1 and 2).
# 6 Proposed set of guidelines

The proposed set of cobot UI design guidelines consists of 24 items, see Table 1 and 2. An explanation and an example are included for each of the guidelines. The guidelines cover the main topics of situation awareness, system understanding, task efficiency, human factors, configurability, and interaction design of the UI, with a focus on the usability and user experience of the cobot system. The guidelines should be considered from the perspective that the user group is diverse and that different users will have different needs, for instance in terms of the information they need to achieve their task, that they might need different levels of support and complexity, and have different abilities, all of which have consequences for the required interface design. Regarding intuitive use (guideline 22), mental models (guideline 7), and user learning (guideline 9), we remark that the differences in users' backgrounds will also have an influence on how these guidelines are operationalized; for different people, different things are familiar, and (thus) different UI features will be easy to use or easy to learn.

## 6.1 Comparison with existing heuristics and usability guidelines

After the final revision based on the expert interviews, we compared our proposed guidelines with heuristics for robot teleoperation [1], assistive robotics [34], and HRI [5]. Guidelines 2, 5, 7, 9, 11, 12, 15-18, and 20, as proposed in the current paper are not included in the set for assistive robotics [34]. While the compact form of the HRI heuristics [5] is useful for heuristic evaluation, not all issues indicated by the guidelines proposed in the present paper are covered (guidelines 5, 9-13, 16, 18, 24, and to some extent 23 are not covered by [5]). The set that was the most similar to the guidelines proposed in the current paper was the one for teleoperation [1]. We note that while there is some overlap of many guidelines on a surface level regarding the topics, the formulation and meaning of the guidelines is different. Several guidelines we propose for cobots are not included in the teleoperation guidelines, namely guidelines 12 (reuse of previous work), 13 (human factors/ergonomics), and 19 (consistent behavior), due to the differences between the domains of robot teleoperation and cobots for manufacturing. Several guidelines had overlaps, but with different practical implications. For example, guideline 20 (Multimodal UI) is in a sense similar to the item "Complement video stream with feedback information from other sensors" [1, pp. 257-258], but the presented information is very different (video stream from remote teleoperated robot versus a programming environment on a GUI in a manufacturing context). Some items relevant for teleoperation are not relevant in the context of cobot systems (e.g. "Ability to self-inspect the robot's body for damages or entangled obstacles"[1]).

## 6.2 Validity and generalizability of guidelines

In this section, potential threats to validity are considered. In the evaluation study (Sec. 4), participants worked in a factory-like setting with equipment and
## Situation awareness

1. **System state awareness: Inform the user on the cobot's state.** The interface should support the user in maintaining appropriate awareness of the system's state. Example: The cobot informs the user via lights on the robotic arm about the current state, for instance a green light means a program is running.

2. **Situation awareness: Inform the user regarding the cobot's environment and configuration.** Help the user in understanding the configuration of the cobot in its environment, as well as other sensor inputs. Example: The GUI shows a 3D model of the robot, which indicates the cobot's configuration and end effector position.

3. **Accessibility of information: Allow users to access information required for the task.** Make sure the information the user needs for the task is available and accessible, considering possible restrictions. Example: When editing a trajectories, a user can access information about the exact end effector location on the GUI.

## System understanding

4. **Feedback: The UI is responsive to user actions.** Respond to user actions so the user can follow task progress and understand the effects of their actions. Example: The UI is responsive when buttons are clicked, when the user navigates to a different menu, or when values are updated.

5. **Affordances: Signify how the user can interact with the cobot.** The interface should indicate which actions are currently possible and which ones are not. Example: An icon of a trash bin next to a stored point indicates the point can be deleted by clicking on the icon.

6. **Errors: Give clear explanations and steps to recover when errors occur.** Tolerate minor user errors, prevent critical system errors, support undo and redo. Example: When a trajectory cannot be executed, a popup appears with an explanation why the error occurred and steps to recover from the error.

7. **Mental model: Support the user in understanding the way the system works.** Support the user in understanding the connection between user actions and system response, for instance by providing feedback and using appropriate terminology. Example: The user can play a programmed trajectory as an animation on the GUI before execution by the cobot.

8. **Help and documentation: Provide contextual help and documentation.** Give users clear explanations of functionality and errors. Example: When the help icon is clicked, the help menu displays help items that relate to the functions that are currently on the display.

9. **Support user learning: Help the user solve their (automation) problem.** Support trial-and-error behavior and provide templates, contextual instructions or other clues that indicate how the cobot can be interacted with. Example: Templates for robot tasks are provided, so the user has an idea what a program should look like.

## Task efficiency

10. **Efficiency: Avoid unnecessary work on the user's side.** Minimize the number of steps required to achieve goals and provide shortcuts. Example: The user does not have to set the speed and acceleration for each point in a trajectory, but can specify these values for the whole trajectory.

11. **Task progress: Communicate to the user which task is being executed.** The GUI should make it easy for the user to follow task execution by indicating previous, current and next steps. Example: When the robot is executing a series of actions, the current action that is being executed is highlighted on the GUI.

12. **Reuse: Enable reuse of previous work.** Support users in reusing their work or the work of others. Example: Previous programs can be copied and edited.

<caption>Table 1: Guidelines for heuristic evaluation of cobot UIs, continued in Table 2.</caption>
## Human factors

**13. Human factors: Design cobot and UI with ergonomics and accessibility in mind.** Ensure the cobot UI is comfortable to work with for the necessary duration. Example: The teach pendant is light to carry or can be placed on a table, so it will be comfortable to work with it for a few hours.

**14. Avoid cognitive overload: Reduce mental strain.** Support recognition instead of requiring users to recall information, and limit the number of options that are presented. Example: A function name such as "Wait" is easy to remember and indicates its function.

**15. User attention: Support the user in directing their attention.** Make menu items that need attention visually salient. Do not attract attention unnecessarily. Example: When the user is in the submenu for editing a trajectory, the menu items for editing points are the largest items.

## Configurability

**16. Level of automation: Let the user determine the level of human input.** The user can decide to integrate human input or to make the program fully automatic. Example: There is a function for integrating human input, which requires the operator to press a button before the robot continues its task.

**17. Adaptable system architecture: Enable easy software integration after hardware exchange.** The system architecture should allow for adapting the system to different types of tasks and application scenarios. Example: It is easy to exchange the gripper and add sensors to the system.

**18. Adaptable tasks: Support easy editing of robot programs.** Robot programs, trajectories, configurations should be editable by the user. Example: Points in a previously stored trajectory can be deleted or changed.

## Interaction design of the UI

**19. Consistent behavior: Make sure cobot and UI behave in a consistent way.** Cobot behaviors, movement, and responses are predictable. Example: The cobot always executes the same motion trajectory the same way.

**20. Multimodal UI: Consider the relation between different interaction modalities.** Manage user attention across modalities and ensure the way information is presented via different modalities is consistent. Example: The system provides feedback with LED lights on the cobot, which matches specific events on the UI.

**21. Graphic design: Design GUI items with usability, accessibility, and aesthetics in mind.** Make sure information is presented in a clear and structured way, and use color, contrast and salience appropriately. Example: Fonts are legible and the interface has appropriate contrast.

**22. Clarity of interface: Ensure the UI is easy and intuitive to use.** Avoid a complex UI design; make use of simple graphics and icons. Example: When selecting an action for the cobot, a sub menu for editing this action opens automatically.

**23. High vs. low complexity: Display programming functions at different levels of detail.** Allow users to switch between simple and more complex ways of programming the cobot. Example: There is the possibility to change between a simple version of the UI and a more complex version that provides more options.

**24. Customizability: Support user preferences.** Enable users to change the interface according to their wishes and needs. Example: It is possible to adapt different features based on user preference, such as the size of windows on the UI.

<caption>Table 2: Guidelines for heuristic evaluation of cobot user interfaces</caption>
robots [28]. This gives the experiment some level of ecological validity (find-
ings generalize to the real world [11]). The first evaluation study indicates that
guidelines can be applied to cobot systems. The external validity of the study
is threatened to some level as the participants did not have specific expertise in
applying design guidelines. Ideally, participants would have known how to work
safely with cobots, would have previously worked with heuristics, and would
have design-related expertise, besides having time to participate. As the most
important aspect was being able to safely program the cobot, we chose robotics
expertise as most relevant criterion. We chose to conduct the interviews with
participants who had expertise in cobot UI/UX design to balance this initial
focus with people with design-related expertise. Participants in the expert inter-
views had experience with other cobots besides those used in our experimental
evaluation. This provides indications that the findings generalize to other cobots
(external validity). Feedback by participants given in the studies may be biased
by their desire to cooperate, which may have led to feedback that was skewed
on the positive side. We aimed to lessen this effect by specifically asking for
(constructive) criticism and by using the clarity scores in the evaluation study
as an indication of those items that were most unclear and that thus needed
revision. The sample size was small for the both the evaluation study and the
interviews. In both studies, the main aim was to obtain detailed feedback to be
able to improve the proposed guidelines, which was achieved. Next, quantitative
evaluation can be used to validate the guidelines. Such an evaluation study could
also evaluate the use of heuristics that comprise a shortened, summarized version
of the presented guidelines, which could lead to possible additional refinement.

7 Conclusion

In this paper, a set of 24 design guidelines for the heuristic evaluation of cobot UIs is proposed. Three different sources of information were used, namely a database search, modified heuristic evaluations with participants with robotics expertise (N=9), and interviews with cobot UI/UX design experts (N=6). On the basis of a comparison with existing design guidelines for HRI, we note that the proposed guidelines are specific to cobot UI design in a manufacturing context and that they are distinct from existing guidelines in the literature.

Visions of future manufacturing in which human workers can reprogram
cobots in collaborative interaction settings [31] can only be realised if cobot
technologies are sufficiently usable. The design guidelines that were proposed
in this paper indicate which design features are important in such interaction
scenarios. However, more research into cobot design and design proposals to
improve current cobot UIs are necessary.

Acknowledgements

Supported by Doctoral College TrustRobots, TU Wien. The first author of this
publication was responsible for the research project and writing of the pub-
lication, and was supported by the second author. We thank Hans Küffner-
McCauley, Tanja Zigaert, Michael Hornáček, all paper reviewers, evaluation
study participants, and interviewees.

References

1. Adamides, G., Christou, G., Katsanos, C., Xenos, M., Hadzilacos, T.: Usability guidelines for the design of robot teleoperation: A taxonomy. IEEE Transactions on Human-Machine Systems 45(2), 256–262 (2015). https://doi.org/10.1109/THMS.2014.2371048

2. Amershi, S., Inkpen, K., Teevan, J., Kikin-Gil, R., Horvitz, E., Weld, D., Vorvoren-anu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P.N.: Guidelines for Human-AI Interaction. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. pp. 1–13. ACM Press (2019). https://doi.org/10.1145/3290605.3300233

3. Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 24(6), 574–594 (2008). https://doi.org/10.1080/10447310802205776

4. Bladford, A.: Semi-structured qualitative studies. In: The Encyclopedia of Human-Computer Interaction. The Interaction Design Foundation, 2 edn. (2013), https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/semi-structured-qualitative-studies

5. Clarkson, E., Arkin, R.C.: Applying heuristic evaluation to human-robot interaction systems (2007), American Association for Artificial Intelligence

6. drag&bot: Industrieroboter wie ein smartphone bedienen (2020), https://www.dragandbot.com/de/, Last visited on 2020-09-10

7. El Zaatari, S., Marei, M., Li, W., Usman, Z.: Cobot programming for collaborative industrial tasks: An overview. Robotics and Autonomous Systems 116, 162–180 (2019). https://doi.org/10.1016/j.robot.2019.03.003

8. Ferraguti, F., Pertosa, A., Secchi, C., Fantuzzi, C., Bonfè, M.: A methodology for comparative analysis of collaborative robots for industry 4.0. In: 2019 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1070–1075 (2019). https://doi.org/10.23919/DATE.2019.8714830

9. Fletcher, S.R., Johnson, T.L., Larreina, J.: Putting People and Robots Together in Manufacturing: Are We Ready? In: Aldinhas Ferreira, M.I., Silva Sequeira, J., Singh Virk, G., Tokhi, M.O., E. Kadar, E. (eds.) Robotics and Well-Being, vol. 95, pp. 135–147. Springer International Publishing (2019). https://doi.org/10.1007/978-3-030-12524-0-12

10. Franka Emika GmbH: Franka Emika Panda (2020), https://www.franka.de/technology

11. Hoffman, G., Zhao, X.: A Primer for Conducting Experiments in Human-Robot Interaction. ACM Transactions on Human-Robot Interaction 10(1), 1–31 (2020). https://doi.org/10.1145/3412374

12. Ionescu, T.B., Schlund, S.: A participatory programming model for democratizing cobot technology in public and industrial fablabs. Procedia CIRP 81, 93–98 (2019). https://doi.org/10.1016/j.procir.2019.03.017

13. ISO: ISO 9241-210 Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems (2010), https://www.sis.se/api/document/preview/912053/
14. ISO: ISO/TS 15066:2016(en) Robots and robotic devices — Collaborative robots (2016)

15. Komenda, T.: SAMY - semi-automatische modifikation (2020), https://www.fraunhofer.at/de/forschung/forschungsfelder/SAMY.html, Visited on 2020-09-10

16. Krot, K., Kutia, V.: Intuitive methods of industrial robot programming in advanced manufacturing systems. In: Burduk, A., Chlebus, E., Nowakowski, T., Tubis, A. (eds.) Intelligent Systems in Production Engineering and Maintenance, vol. 835, pp. 205–214. Springer International Publishing (2019). https://doi.org/10.1007/978-3-319-97490-3_20

17. KUKA AG: LBR iiwa (2020), https://www.kuka.com/en-de/products/robot-systems/industrial-robots/lbr-iiwa

18. Lupetti, M.L., Zaga, C., Cila, N.: Designerly Ways of Knowing in HRI: Broadening the Scope of Design-oriented HRI Through the Concept of Intermediate-level Knowledge. In: Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction. pp. 389–398. ACM, Boulder CO USA (2021). https://doi.org/10.1145/3434073.3444668, https://dl.acm.org/doi/10.1145/3434073.3444668

19. Malik, A.A., Bilberg, A.: Developing a reference model for human-robot interaction. International Journal on Interactive Design and Manufacturing (IJIDeM) (2019). https://doi.org/10.1007/s12008-019-00591-6

20. Marvel, J.A., Bagchi, S., Zimmerman, M., Antonishek, B.: Towards effective interface designs for collaborative HRI in manufacturing: Metrics and measures. ACM Transactions on Human-Robot Interaction 9(4), 1–55 (2020). https://doi.org/10.1145/3385009

21. Maurice, P., Allienne, L., Malaise, A., Ivaldi, S.: Ethical and Social Considerations for the Introduction of Human-Centered Technologies at Work. In: 2018 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO). pp. 131–138. IEEE, Genova, Italy (2018). https://doi.org/10.1109/ARSO.2018.8625830

22. MAXQDA, VERBI GmbH: MAXQDA | All-In-One Qualitative & Mixed Methods Data Analysis Tool (2020), https://www.maxqda.com/

23. Michaelis, J.E., Siebert-Evenstone, A., Shaffer, D.W., Mutlu, B.: Collaborative or simply uncaged? understanding human-cobot interactions in automation. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. pp. 1–12. ACM (2020). https://doi.org/10.1145/3313831.3376547

24. Murphy, R.R., Tadokoro, S.: User interfaces for human-robot interaction in field robotics. In: Tadokoro, S. (ed.) Disaster Robotics, vol. 128, pp. 507–528. Springer International Publishing (2019). https://doi.org/10.1007/978-3-030-05321-5_11

25. Naumann, A., Hurtienne, J., Israel, J.H., Mohs, C., Kindsmüller, M.C., Meyer, H.A., Hußlein, S.: Intuitive use of user interfaces: Defining a vague concept. In: Harris, D. (ed.) Engineering Psychology and Cognitive Ergonomics. Lecture Notes in Computer Science, vol. 4562, pp. 128–136. Springer Berlin Heidelberg, Berlin, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73331-7_14

26. Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: CHI '94: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 152–158 (1994). https://doi.org/10.1145/191666.191729

27. Paxton, C., Hundt, A., Jonathan, F., Guerin, K., Hager, G.D.: CoSTAR: Instructing collaborative robots with behavior trees and vision. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 564–571 (2017). https://doi.org/10.1109/ICRA.2017.7989070
28. Pilotfabrik TU Wien: Pilot Factory TU Vienna – Industry 4.0 (2021), https://www.pilotfabrik.at/?lang=en

29. Quiñones, D., Rusu, C.: How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces **53**, 89–122 (2017). https://doi.org/10.1016/j.csi.2017.03.009

30. Robotiq: Products: Grippers, Camera and Force Torque Sensors (2021), https://robotiq.com/products

31. Schmidbauer, C., Komenda, T., Schlund, S.: Teaching cobots in learning factories – user and usability-driven implications. Procedia Manufacturing pp. 398–404 (2020). https://doi.org/10.1016/j.promfg.2020.04.043

32. Steinmetz, F., Wollschläger, A., Weitschat, R.: RAZER—a HRI for visual task-level programming and intuitive skill parameterization. IEEE Robotics and Automation Letters **3**(3), 1362–1369 (2018). https://doi.org/10.1109/LRA.2018.2798300

33. The European Parliament and the Council of the European Union: Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on machinery, and amending Directive 95/16/EC (recast). Official Journal of the European Union (2006), https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2006:157:0024:0086:EN:PDF

34. Tsui, K.M., Abu-Zahra, K., Casipe, R., M'Sadoques, J., Drury, J.L.: Developing heuristics for assistive robotics. In: 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). pp. 193–194 (2010). https://doi.org/10.1109/HRI.2010.5453198

35. Universal Robots: UR5 collaborative robot arm | flexible and lightweight robot arm (2019), https://www.universal-robots.com/products/ur5-robot/

36. Villani, V., Pini, F., Leali, F., Secchi, C.: Survey on human-robot collaboration in industrial settings: Safety, intuitive interfaces and applications. Mechatronics **55**, 248–266 (2018). https://doi.org/10.1016/j.mechatronics.2018.02.009

37. Weintrop, D., Afzal, A., Salac, J., Francis, P., Li, B., Shepherd, D.C., Franklin, D.: Evaluating CoBlox: A Comparative Study of Robotics Programming Environments for Adult Novices. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. pp. 1–12. ACM, Montreal QC Canada (2018). https://doi.org/10.1145/3173574.3173940

38. Weiss, A., Wurhofer, D., Bernhaupt, R., Altmaninger, M., Tscheligi, M.: A methodological adaptation for heuristic evaluation of HRI. In: 19th International Symposium in Robot and Human Interactive Communication. pp. 1–6. IEEE (2010). https://doi.org/10.1109/ROMAN.2010.5598735