## 5 Experiment
### Metrics for Open-Ended Player Responses  
Our final survey results consist of both English and Chinese languages due to the diverse origins of our participants, spanning six different countries, including non-English native speakers. As a result, we tasked three suitable human evaluators who are proficient in both English and Chinese with the analysis. With limited qualified human evaluators, we ask an LLM, ChatGPT (GPT-4), to help us with our classification task, described below, for open-ended player responses. In previous research [36], an endeavor was made to employ ChatGPT to classify bullet comment types in games, achieving success. This shows \begin{table}
\begin{tabular}{c c c} \hline \hline  & **FFS Ability** & \\  & Phase One & Phase Two \\ \hline Mean value & 1.18 & 1.24 \\ Standard deviation & 0.68 & 0.73 \\ Shapiro-Wilk Test (\(p\)-value) 1  & \(5.2\times 10^{-6}\) & \(8.7\times 10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Players’ general information for both phases. that ChatGPT can be used for classification tasks. In addition, ChatGPT, as a powerful LLM, has the ability of translation tasks and reading comprehension tasks, which can be used in multilingual semantic analysis for our classification task. Therefore, ChatGPT and human evaluators are employed to classify all player responses from the final survey. The objective of this analysis is to uncover the reasons behind the objective results. However, the same previous research also mentioned that both human evaluators and ChatGPT will have inconsistent answers in some complex sentences. To mitigate this issue, as well as the effect of hallucinations present in ChatGPT, we have designed the following procedure:  
* For ChatGPT, we use it with an originally crafted prompt for classification, running it three times for each classification instance. We consider a result of interest reliable if and only if it is consistent across all three runs; otherwise, it is deemed unreliable and labeled as 0. * For our human evaluators, we compare their responses. For each classification instance, if they all give the same answer, we consider the answer reliable; otherwise, it is deemed unreliable and labeled as 0. Afterwards, we combine the results of ChatGPT and the human evaluators as follows:  
* If the result from the human evaluators is labeled as 0 but that from ChatGPT is not, we select ChatGPT's result. * If both results from ChatGPT and the human evaluators are labeled as 0, the result is categorized as 0. * Otherwise, we select the result from the human evaluators. In cases where both results from ChatGPT and the human evaluators are not labeled as 0, we prioritize the response of the human evaluators because we consider them to be more reliable than LLMs. To gain clarity on our players' feelings about the audio cue design in each phase, we categorize all open-ended player responses from the final survey into three categories:  
1. Not mentioned
2. The paragraph directly mentions audio cues in the game and complains about them
3. The paragraph directly mentions audio cues in the game and praises them  
Prompt 1 shows the prompt. It is used by ChatGPT (via OpenAI API) and our human evaluators to analyze the open-ended player responses, individually shown under @paragraphs. This ensures that both ChatGPT and the human evaluators performed this classification task based on the same instruction. Please classify the following @paragraphs into three categories based on their content related to audio cues in the game. If the paragraph doesn't clearly and directly express a positive or negative sentiment towards the game's audio cues, please choose A. If the paragraph clearly and directly expresses a negative sentiment towards the game's audio cues, please choose B. If the paragraph clearly and directly expresses a positive sentiment towards the game's audio cues, please choose C.  
If the text doesn't mention audio cues, it means there are no complaints or compliments about audio cues. Please avoid overinterpreting.