# Speech Act Theory as an Evaluation Tool for Human-Agent Communication
## 5 Evaluating Human-Agent Communication Using SAT
### Scenario from Omosa Virtual World  
The communication model is illustrated in a scenario where the user and the virtual agent have a shared goal to capture a Yernt by surrounding the animal with an octagon-shaped fence. In this task, both the human and the agent should collaborate. After the human or the agent draws the first line near the animal, they will continue to take alternating turns to form a geometric shape to trap the animal inside by drawing a line beginning from the two ends of the existing drawn shape. For practicality, the Yernt is tranquilised before the collaborative activity commences so that it remains calm for a specific pre-determined amount of time before it runs away. The process of capturing the animal should therefore be done before the time deadline is reached. Figure 1 shows a snapshot from Omosa Virtual World.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Locutionary Acts** & **Illocutionary Acts** & **Perlocutionary Acts** \\ \hline \multirow{2}{*}{**Meaning**} & The structure of the utterances. & The intention of the speaker. & What is achieved by saying something. \\ \hline \multirow{6}{*}{**Evaluation Method**} & Subjective analysis of participants’ perception of the structural characteristics of the agent’s speech. & Objective analysis of the agent’s intended meaning. & Objective analysis of the agent’s achievement due to verbal communication. \\  & & Objective analysis of the interlocutor’s perception of the intention in the agent’s messages. & Subjective analysis of participants’ perception of achievement due to the agent’s verbal communication. \\ \hline \multirow{6}{*}{**Evaluation Data**} & Surveying participants’ perception of the structure of the agent’s utterance. & Logging the ratio of participants’ acceptance of agent’s requests. & Logging the completion time of each cycle in the collaborative task. Surveying if the participants perceive the intention behind the agent’s speech to direct agent’s locutionary acts. & \\ \hline \hline \end{tabular}
\end{table}
Table 1: The speech acts, their meaning, and evaluation.  
The design of the scenario is noteworthy in that it is:  
* Collaborative, with an explicit goal where the user can observe the changes in the situation.
* A two-way multimodal interaction: The agent (virtual biologist) interacts with the human participant through speech and visual actions, and vice versa.
* SAT is used as an evaluation tool to measure the effectiveness of communication in achieving a designated goal.
* Real-time: The agent's plan is generated in real-time as the human participant's actions may vary and lead to unexpected changes in the environment.
* The task itself shows the effect of both verbal and non-verbal communication through the human-agent social interaction.  
As an example of agent messages, an agent may ask a human teammate using a directive illocutionary act \"_Why don't you go to region x and I will go to region y_\". Before the human replies to the agent's request, there is an option to ask for explanation from the agent about the proposed request. The agent replied using a directive illocutionary act \"_Why don't you go to region x and I will go to region y_\". The human participant can either accept the agent's proposition or reject it using a representative illocutionary act, saying \"_I like this idea_\" or \"_I do not like this idea_\". In Appendix A, a sample of human-agent messages is listed in addition to the illocutionary classification of each message. The illocutionary classification is performed and reviewed by the authors of this study. To evaluate the perlocutionary act of an agent request and explanation, human replies and actual actions are recorded to see if the human's reply to either accept or reject the agent's proposal is honoured or not.