Q3 Effect of Majority Voting
In previous work (Troiano et al., 2023 ) , the accuracy of human reader-annotators guesses was considerably improved when their ratings were aggregated using the majority voting over the guesses of five annotators. Encouraged by this, we used GPT-4 to generate five completions for each text in the reader-annotator dataset, and tested whether the majority voting has an impact on prediction accuracy.
If there was a clear majority vote, that rating was selected as the final prediction.
In case of ties (e.g., either two ratings were predicted twice or all five ratings were different), we adopted the following essentially random procedure. While each text had five independent predictions from the GPT-4 model, we pretended to have done just a single completion and chose the rating from the first of the five runs.
In total, we generated predictions for 1200 \times 5 data samples using a prompt I1 with added question "How confident is the model about chosen ratings?".

The application of the majority voting algorithm improved the RMSE on average by about 30% from 1.61 to 1.12 (columns GPT-4 {}_{\text{avg}} and GPT-4 {}_{\text{maj}} in Table 2 ). However, compared to the results shown in previous Section 4.2 (also shown in Table 2 ), generating five runs and taking the average increased the RMSE considerably, which is now in the same range to the average over the human reader-annotator ratings (1.61 vs 1.57; columns GPT-4 {}_{\text{avg}} and Human {}_{\text{avg}} in Table 2 ).
At the same time, the RMSE obtained using the majority voting is also considerably lower than the results shown in the previous subsection from just one run (1.12 vs 1.45; columns GPT-4 {}_{\text{maj}} and GPT-4 in Table 2 ). Similar or even better results were obtained by applying majority voting to the guesses of the human reader-annotators, showing an improvement from 1.57 to 0.99 (columns Human {}_{\text{avg}} and Human {}_{\text{maj}} in Table 2 ).

Next, we analyzed how often a tie needs to be resolved in the ratings of human reader-annotators and GPT-4. We found that on average, 22% of human reader-annotators guesses required tie-breaking, while for GPT-4, only 9% required that. Since the number of ties was not negligible, we considered using a method similar to the one adopted by Troiano et al. ( 2023 ) based on rating the confidence. Specifically, we added to the prompt’s questions list the question ”How confident is the model about chosen ratings?” as an alternative to the question posed to the human reader-annotator’s by Troiano et al. ( 2023 ) "How confident are you about your answer?”. 5 5 5 “Answer” refers to “emotion” used in the questionnaire Similar to other questions, this question was rated on the 5-point Likert scale with values ranging from 1-5. To break a tie, the value with a highest average confidence rating was chosen.

The results of this experiment are shown in Table 2 columns GPT-4 {}_{\text{conf}} and Human {}_{\text{conf}} . We can see that using the confidence rating to break the ties did not improve the RSME neither for the model nor for the human reader-annotators compared to the random choice. The mean RMSE for GPT-4 from 1.12 to 1.15 (columns GPT-4 {}_{\text{maj}} and GPT-4 {}_{\text{conf}} in Table 2 ), and the mean RMSE for human reader-annotators increased from 0.99 to 1.06 (columns Human {}_{\text{maj}} and Human {}_{\text{conf}} in Table 2 ). When looking at individual appraisal dimensions, we can see that the RMSE’s are in most cases lower in the random tie-breaking setting compared to using the confidence rating, and that applies to both the model and humans. Thus, we conclude that using the model-generated confidence rating is a not a useful cue for breaking the ties when aggregating several ratings via majority voting.