# Implementation and Evaluation of Dynamic Task Allocation for Human-Robot Collaboration in Assembly
## 2 Materials and Methods
### Study Design and Evaluation Methodology
#### 2.3.2 Metrics and Data Collection  
For the evaluation study, the independent and dependent variables were defined as shown in Figure 15. The study aims to examine how the different modes of task allocation explained above affect the various qualitative and quantitative aspects relevant in collaborative assembly. Moreover, the influence of a parallel assembly of multiple components (process flexibility) as well as the influence of process delays are investigated. In addition to these aspects, the proposed overall system is evaluated with respect to usability, trust in robot, information quality, and qualitative feedback. Figure 14: Assembly sequences for the implemented task allocation modes in human-robot collaboration. In accordance with [52], the relevant variables in the context of collaborative human-robot assembly were classified into the aspects of effectiveness, efficiency, and satisfaction listed in ISO 9241-11 [53]. Beyond the variables proposed by [52], we have added a number of metrics for the aspect of efficiency that quantitatively evaluate HRC performance (HRC fluency [54]), as well as various investigated variables with respect to user satisfaction (or human factors, cf. [55]). Table 2 provides an overview of the variables examined with the utilized metrics and the respective approach for data collection. The required data are collected from three sources: First, notes are taken by the study coordinator regarding successful process completions as well as errors during assembly. Second, the timestamps of the start and end times of all sub-processes including the final assignment to human or robot are saved in an automatically generated process log file. This can be used for detailed process evaluations regarding the analysis of the efficiency aspects. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Evaluation** & **Variable** & **Metrics** & **Symbol** & **Data Collection** \\ \hline Effectiveness & Process effectiveness & Task completion rate & TCR & notes during study \\  & Process efficiency & Number of errors & \(n_{\text{err}}\) & notes during study \\ \hline  & Process efficiency & Process time & \(t_{\text{process}}\) & process data logging \\ \cline{2-5}  & HRC performance & Number of human tasks & \(n_{\text{H}}\) & process data logging \\  & efficiency & Human wait time & \(t_{\text{H,wait}}\) & process data logging \\  & & Robot wait time & \(t_{\text{R,wait}}\) & process data logging \\  & & Concurrent activity time & \(t_{\text{HR,concurrent}}\) & process data logging \\ \hline  & Workload & NASA Raw TLX & RTLX & questionnaire \\ \cline{2-5}  & Fluency and satisfaction & Fluency score & Fluency & questionnaire \\  & & Satisfaction score & Satisfaction & questionnaire \\ \cline{2-5} User & User preference & Preferred task allocation mode & - & questionnaire \\ \cline{2-5}  & & System usability scale score & SUS & questionnaire \\  & & Trust evaluation score & Trust & questionnaire \\  & & Quality of provided information & QInfo & questionnaire \\  & & Qualitative feedback & - & free text input \\  & & Suggestions for improvement & - & free text input \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Overview of investigated variables, utilized metrics, and the approach for data collection. Figure 15: Dependent and independent variables of the evaluation study. Third, questionnaires with radio buttons and free text input fields are used to investigate the evaluation aspect of user satisfaction. Two questionnaires were prepared for the metrics to be collected qualitatively by user survey. The first questionnaire contains the dependent variables, which are depending on each task allocation mode (NASA raw TLX [56; 57], fluency [54], and satisfaction [58]). The second questionnaire contains the general system evaluation variables (trust [59], usability [60; 61], information quality [62; 63], preferred task allocation mode, and qualitative feedback as well as demographic information and technical affinity [64]), and is surveyed at the end of the study session.