# Speech Act Theory as an Evaluation Tool for Human-Agent Communication
## 6 Results
### Evaluating Agent Communication
#### 6.2.3. How Can an Agent's Perlocutionary Act Be Evaluated? SAT states that the effect of the illocutionary act appears in the perlocutionary act. This means that to check the effect of the verbal messages of the agent, we need to determine the influence of these messages on the current collaborative situation. To evaluate this influence, objective and subjective analysis methods were used. In the objective evaluation of the agent's perlocutionary act, the goal was to measure how successful each of the agent's speech acts were in achieving the agent's intention, i.e., illocutionary act. To conduct this objective evaluation, the data collected in the automatic log files that track the human's actions must be analysed to check if the user demonstrates understanding and positively responds to the verbal message. The idea behind this technique is to determine if, after the agent has expressed his intention, the human then takes a decision based on the agent's intention. The result showed the percentage of the humans' responses to the agent's verbal requests in each cycle. The results show that the humans' behaviour reflects 64.41%, 67.80%, and 70.69% human acceptance of the agent's requests in cycle1, cycle2 and cycle3, respectively (see Figure 4). 2.4. Could the Evaluated Agent's Illocutionary and Perlocutionary Acts Impact Collaboration Performance? In Section 6.2, SAT was used as a tool to evaluate the agent's locutionary, illocutionary, and perlocutionary acts. The fourth research question proposes a further utilisation of SAT concepts to evaluate the impact of the agent's verbal communication on the performance of the human-agent collaboration. The objective of this research question is to investigate if the humans' perception  
Figure 4. Evaluating the agent’s perlocutionary acts. Figure 3. Evaluating the agent’s illocutionary acts. of the agent's illocutionary and perlocutionary acts is associated with human-agent performance. To objectively evaluate the performance of human-agent teamwork, the time to complete each cycle was logged during participants' usage of the virtual system. The result of analysing the log files showed that the average time to complete consecutive cycles decreased from 51.1 s in the first cycle to 40.66 s in the second cycle and 33.78 s in the last cycle. To measure the performance of human-agent collaboration, human perception of performance was surveyed. Participants were asked two questions to estimate the final performance of the collaboration. The first question asked the participants to estimate how appropriate they found the flow of collaboration from the agent's side. The results showed that 64.06% and 14.06% of the participants agreed and strongly agreed, respectively, that the agent's flow of actions was appropriate. Using a t-test to determine if there were any significant differences between participants' responses, it was found that that there was a significant difference between participants in their responses to survey questions about the perception of the agent's role in collaboration performance (t(65) = 27.54, \(p<0.01\), see Table 2). The second question asked the participants how appropriate they found the agent's reaction to the human's role in the collaboration. The results showed that 59.38% and 9.38% of the participants agreed and strongly agreed, respectively, that the agent's reactions toward the human's role in the collaboration was appropriate. Using a t-test to evaluate any significant differences in the participants' responses, it was found that there was a significant difference between participants in their perception of the agent's reactions to the human's role in the collaboration (t(65) = 24.39, \(p<0.01\)). To measure the strength and direction of association between human perceptions of the agent's illocutionary act, perlocutionary act, and collaboration performance, [PERSON]'s rho correlation was used. [PERSON]'s rho correlation was selected as it is more appropriate for small sample sizes or non-normally distributed responses.