## 3 Tools and Methods
### Data Collection and Methods of Data Analysis  
Data were collected throughout the study using a concurrent think-aloud protocol and de-brief interviews. The think-aloud protocol [67] encouraged our participants to verbalize their assumptions, thoughts, and problems encountered during the composition process. This protocol, thus, enabled capturing the participants' mental models and reasoning for their service-design strategies. Moreover, self-report data and interaction behaviour were recorded using a screen capturing software, and later transcribed, permitting noteworthy actions to be inserted into a spreadsheet for subsequent analysis.  
The quantitative data included objective measures of service-development performance and subjective measures of user satisfaction for each user group. Objective measures were calculated from the interaction videos, and included task completion time (in seconds), completion rate, and correctness of responses. Subjective measures were collected using post-study questionnaires and included perceived usability ratings (on a 5-point Likert scale with 1 = disagree, 3 = neutral, and 5 = agree) of core functionalities of SOA4 All tools and development tasks. The questions used for those satisfaction questionnaires are included in Figures 5 and 6.  
The qualitative data included each participant's conceptual understanding, development challenges experienced, and feedback about the service-design strategies. Initially, a data analysis scheme was devised whereby data items were divided into three main categories: actions, comments, and problems. Actions refereed to user activities performed to achieve the development tasks; for example navigation, execution, etc. Comments referred to verbal statements and remarks about the development tasks, and included \"positive feedback and improvement suggestions.\" Problems refer to design challenges users encounter during the development tasks and include \"conceptual and usability\" problems. Conceptual problems signified weaknesses in conceptual understanding of composition concepts and in design strategies to solve composition tasks, whereas usability problems signified user interface related difficulties that hindered user interaction with the tools. The comments and problems were further coded and classified into finer-grained themes using the thematic analysis  
Figure 4: Types of service-composition tasks tested.  
technique [66]. The thematic analysis technique is a coding methodology whereby patterns and similarities emerge from the data. The resultant themes are presented in tables, in Sections 4 and 5, along with the percentage of occurrence of each theme.  
Two researchers coded user comments and problems independently to split the workload and reduce bias. In order to assess rating agreement, both researchers initially coded 10% [68] of the overall sample data achieving 80% inter-coder reliability [69]. Coding disagreements were discussed and reconciled. Inter-coder reliability is an important step in thematic analysis and allows researchers to divide work among coders. Ensuing this, each researcher coded 50% of the remaining qualitative data using the same coding procedure. On completion, the researchers integrated their results by revisiting the emerging themes and resolving any conflicts.