The neural principles of behavior modification using socioemotional facial feedback cues in economic decision-making

The present dissertation aims to shed light on different mechanisms of socio-emotional feedback in social decision-making situations. The objective is to evaluate emotional facial expressions as feedback stimuli, i.e., responses of interaction partners to certain social decisions. In addition to human faces, artificial emojis are also examined due to their relevance for modern digital communication. Previous research on the influence of emotional feedback suggests that a person's behavior can be effectively reinforced by rewarding stimuli. In the context of this dissertation, the differences in the feedback processing of human photographs and emojis, but also the evaluation of socially expected versus socially unexpected feedback were examined in detail in four studies. In addition to behavioral data, we used the electroencephalogram (EEG) in all studies to investigate neural correlates of social decision-making and emotional feedback. As the central paradigm, all studies were based on a modified ultimatum game. The game is structured as follows: there is a so-called proposer who holds a specific amount of money (e.g., 10 cents) and offers the responder a certain amount (e.g., 3 cents). The responder then decides whether to accept or reject the offer. In the version of the ultimatum game presented here, different types of proposers are introduced. After the participants have accepted or rejected in the role of the responder, the different proposers react to the participant’s decision with specific emotional facial expressions. Different feedback patterns are used for the individual experiments conducted in the course of this dissertation.In the first study, we investigated the influence of emotional feedback on decision-making in the modified version of the ultimatum game. We were able to show that a proposer who responds to the acceptance of an offer with a smiling face achieves more accepted offers overall than a control proposer who responds to both accepted and rejected offers with a neutral facial expression. Consequently, the smile served as a positive reinforcement. Similarly, a sad expression in response to a rejected offer also resulted in higher acceptance rates as compared to the control identity, which could be considered an expression of compassion for that proposer. On a neuronal level, we could show that there are differences between simply looking at negative emotional stimuli (i.e., sad and angry faces) and their appearance as feedback stimuli after rejected offers in the modified ultimatum game. The so-called feedback-related negativity was reduced (i.e., more positive) when negative emotions appeared as feedback from the proposers. We argued that these findings might show that the participants wanted to punish the proposers by rejecting an offer for its unfairness and therefore the negative feedback met their expectations. The altered processing of negative emotional facial expressions in the ultimatum game could therefore indicate that the punishment is interpreted as successful. This includes the expectation that the interaction partner will change his behavior in the future and eventually make fairer offers. In the second study we wanted to show that smiling and sad emojis as feedback stimuli in the modified ultimatum game can also lead to increased acceptance rates. Contrary to our assumptions, this effect could not be observed. At the neural level as well, the findings did not correspond to our assumptions and differed strongly from those of the first study. One finding, however, was that the neural P3 component showed how the use of emojis as feedback stimuli particularly characterizes certain types of proposers. This is supported by the fact that the P3 is increased for the proposer who rewards an acceptance with a smile as well as for the proposer who reacts to rejection with a sad emoji compared to the neutral control proposer.The third study examined the discrepancy between the findings of the first and second study. Accordingly, both humans and emojis representing the different proposers were presented in the ultimatum game. In addition, emojis were selected that showed a higher similarity to known emojis from common messenger services compared to the second study. We were able to replicate that the proposers in the ultimatum game, who reward an acceptance of the offer with a smile, led to an increased acceptance rate compared to the neutral control proposers. This difference is independent of whether the proposers are represented by emojis or human faces. With regard to the neural correlates, we were able to demonstrate that emojis and human faces differ strongly in their neural processing. Emojis showed stronger activation than human faces in the face-processing N170 component, the feedback-related negativity and the P3 component. We concluded that the results of the N170 and feedback-related negativity could indicate a signal for missing social information of emojis compared to faces. The increased P3 amplitude for emojis might imply that emojis appear unexpectedly as reward stimuli in a social decision task compared to human faces.The last study of this project dealt with socially unexpected feedback. In comparison to the first three studies, new proposer identities were implemented. In particular, the focus was on a proposer who reacted to the rejection of an offer unexpectedly with a smile and to the acceptance with a neutral facial expression. According to the results, participants approach this unexpected smile through increased rejection, although it is accompanied by financial loss. In addition, as reported in studies one and three, we were able to show that proposers who respond to the acceptance of an offer with a smiling face and thus meet the expectations of the participants have higher offer acceptance rates than the control proposer. At the neuronal level, especially the feedback from the socially unexpected proposer led to an increased P3 amplitude, which indicates that smiling after rejection is attributed a special subjective importance. The experiments provide new insights into the social influence through emotional feedback and the processing of relevant social cues. Due to the conceptual similarity of the studies, it was possible to differentiate between stable findings and potentially stimulus-dependent deviations, thus creating a well-founded contribution to the current research. Therefore, the novel paradigm presented here, and the knowledge gained from it could also play an important role in the future for clinical questions dealing with limited social competencies.