Measure Of Belief Change as an Evaluation of Persuasion

In the field of natural argumentation and computer persuasion, there has not been any clear definition of the persua siveness of a system trying to influence the user. In this paper, we desc ribe a general evaluation task that can be instantiated on a number of domains to evaluate the beliefs change of participants. Throu gh the use of a ranking task, we can measure the participant’s change of b liefs related to a behaviour or an attitude. This general metric al lows a better comparison of state of the art persuasive systems. 1 Motivation and Related Work In novel fields of research, researchers often want to compar e their approaches and the efficiency of their research. Thus, along side the new field a research movement is created to develop robust eva luation frameworks that can provide comparative results and fa ir ev luations of research output for the field. For example, in the Inf ormation Retrieval field, the researchers have long studied differen t techniques of evaluation and selected the precision/recall measures, thu creating a framework of measures that can be used by all researcher s and create evaluation campaigns such as the Text REtrieval Conf erences (TREChttp://www.trec.nist.gov). The field of automated persuasion is attracting a growing int erest in the research community, with new conferences and worksho ps every year [16, 19]. However, there has yet not been an agreed me thod for evaluating and comparing persuasive systems’ output. Existing research already provides examples of evaluation techniques for persuasion. For instance [4] uses a long term eval uation procedure to follow the change of students’ behaviour when t ryi g to persuade them to walk more. The measure of persuasiveness in troduced by the authors is computed from the evolution of steps c ount for each participant, showing the change in walking behavio ur of the students over one month. In this experimental setup, the res ea chers need a large amount of resources and time to provide pedomete rs to students, motivate them to use the system on a long term basis and wait for results; this amount of resources are not always ava ilable to all researchers. In addition, following a long term behavio ur change is not an atomic setup and it is difficult to control for every e xternal factors that can influence the user’s behaviour. [21] describes a smoking cessation program that tries to per suade participants to stop smoking through tailored letters. The us rs are asked if they think they will stop smoking in the month or six m onths following the reading of the letter. Participants were also asked if they had actually quit six month after the intervention. In t his experiment, the authors show that there is no difference in the cha nge of behaviour between the control group and the group that reads the tailored letters. The authors acknowledge that their experime nt and trial was too small to show any statistical evidence. It is in fact d ifficult 1 University of York, United Kingdom, email: pierre.andrews @gmail.com; suresh@cs.york. ac.uk with such binary observation to extract enough data and it is a general problem to be able to find enough participants to follow o n such a long term experiment. In behavioural medicine, many measures have been developpe d to evaluate the changes in different mental constructs associ ated to behaviour change. [11], for example, proposes a questionnair e to evaluate the stage of change (see [20]) of participants within th e pain treatment domain, while [15] develops a scale to measure the volution of self efficacy in the domain of arthritis treatment. Other persuasive system researches take a more concise appr oach by evaluating a change during the persuasive session of an ex ternal representation of the user’s intentions towards a behaviou r. For instance [8] evaluates an embodied conversational agent simu lat ng a real estate agent by comparing the amount of money that clien ts are prepared to spend on a house before and after the interaction with the estate agent. The estate agent tries to convince users to buy a house fifty percent more expensive than what they are actuall y ready to spend. The persuasiveness of the system is evaluated by lo oking at the increase in the user’s budget after the interaction. The measure is between zero percent to 100% increase relative to the target incr ase chosen by the system. [18] tries to evaluate the effect of distance over persuasio n for computer mediated communication. The author uses a setup fo llowing the desert survival scenario [14] where participants ha ve to rank a set of items relating to their survival in the desert. After having given an initial ranking, the participants are then faced wi th persuasive messages relevant to these items and finally give a ranki ng of the same items after the persuasive session. The author uses as a me ure of persuasion the distance between the participant’s final r anking and the ranking of the persuader. [7] introduces a variation of t he ranking task in the domain of house buying; instead of having to reran k a full list of items (houses in this case), the participants are per suaded to insert a new item in their initial ranking. This evaluation m easures how many users actually chose the new alternative and where t y ranked it in the initial ranking. These measures allow the au thors to evaluate the persuasion and the effectiveness of the tailor ing f the arguments. We believe that a ranking task such as the one used by [18] can a pply to different domains and be used as a common evaluation me tric to compare persuasive systems. In this paper, we ground the v alidity of this ranking task in theory of persuasion and describe a fo rmalisation of the ranking task that provides an evaluation metri c fo controlled experiments that can be more robust to external fact or. It also provides a standard measure available in many domains and th t can be compared between researches. We also conclude that there is a need for more research in persuasion evaluation frameworks t help the development of the automatic persuasion and natural arg umentation field. 2 Behaviour and Belief Change When thinking of persuasion, the immediate indicator of suc cess is a change in the behaviour of the persuaded subject; in fact, [ 17] proposed the following definition of Persuasive Communication : “Any message that is intended to shape, reinforce or change t he responses of another or others. ” from [22], p. 4 It is generally accepted that the “changed responses”refers to either a change in behaviour or a change in the attitude towards the behaviour (see [22]). However, behaviours can take many for ms and the method of evaluating of a change in behaviour will be diff erent for every application domain. For instance [4] tries to e valuate a change in walking behaviour and uses the number of steps a us er performs as a measure of behaviour change. In another health advice domain, [21] tries to convince participants to stop smoking , the evaluation output is thus the number of participants that stoppe d smoking. [4] describes a continuous evaluation value for each pa rticipant that is hard to port to other domains whereas [21] describes a binary value that does not provide powerful data for analysis but is easy to understand. Both evaluation methods consider a change of be haviour and provide the authors with a tool to demonstrate the persua siveness of their system. However, it is difficult for the reader t o make a comparison between the approaches’ performances. However, research in sociology and persuasive communicati on shows that intentions towards a behaviour can be modelled as a function of the user’s beliefs about such behaviour and the socia l norms influencing the user. For instance, [1] presents the Theory o f Reasoned Action that is designed to predict one’s intention(IB) to perform a particularbehaviour(B) as a functionf of one’sattitudetoward this behaviour ( AB) and of thesubjective normsthe behaviour is exposed to ( SNB). Equation (1) represents this influence, where W1 andW2 are the personal importance attributed to each component: The attitude is defined by (2) wherebi is the “belief value” and ei is the “evaluation” of that belief, The subjective norms is defined by (3) wherebi is the “normative belief value”. i.e. the reference belief of the group the receiver considers himself in – and mi the “motivation” to follow the group beliefs. IB = f(W1 × AB + W2 × SNB) (1)