An Attempt to Model Human Trust with Reinforcement Learning

1 INTRODUCTION . Trust seems to be one of the most overused words in the Information Technology world . Yet the concept is struggling to find a universal definition : “ the term is never defined . The usage is much clearer than the concept itself ” ( Hunyadi , 2020 ) . At the end of the day , the question of trust in the digital world is rarely raised ; scientists and industrialists prefer to focus on the effects of its erosion - most often - or its improvement - more rarely - on related issues : digital identity , privacy , personal datas , security , customer relationship , artificial intelligence , financial services digitalisation . On the other hand , from a practical point of view - e.g . for industries and institutions - we observed that there is a lack of tools to forecast the evolution of trust among stakeholders ( customers , suppliers , partners ) according to the strategic choices of digital transformation . New job positions , such as that of chief trust officer , could be interested in this type of tool . This paper presents the preliminary research that allows us to envision the building of such future applications . On the understanding of trust as a structure of a social system , we choose to overcome the very first question concerning its definition on the basis of scientific knowledge in sociology , psychology and philosophy . According to social scientists , trust functions as a way to reduce complexity in a social system , while allowing it to evolve towards richer interactions . It acts as a stabilising structure for societies ( Gambetta et al. , 2000 ) . Trust makes it possible to act in contexts where information is lacking , or even unavailable ( the relations between present and future events being indeterminable ) . Luhmann ( 2000 ) has provided semantic distinction between trust and confidence . Trust or interpersonal trust is an assessment of risk by an individual who makes a rational choice . Trust allows the binding of groups of people whose number does not exceed 100 to 150 ( Aral , 2020 ; Harari , 2016 ; Dunbar , 1992 ) Confidence allows an individual to act in spite of an inherent danger in the social system because he has faith in institutionalised factors of the society , his culture , his knowledge . Confidence allows a social system to be unified at scale despite the oppositions of the groups within it . In his recent work , the philosopher Hunyadi proposes a new unified theory of trust , which leads to the following definition : “ Trust is a bet on the behavioural expectations of things ( e.g . that the ground will support me when I walk ) , people ( that the driver of the car I pass will obey the rules of the road ) , and institutions ( that the money I use for transactions will have some value ) . ” He argues that former theories - such as Luhmann ’ s - are discontinuous because the intrinsic mechanisms of trust are not the same depending on the scale at which they are observed . Hunyadi ’ s theory seems both to overlap with Luhmann ’ s definition of interpersonal trust and to be practical enough - e.g . the notion of a bet leads quite naturally to defining a reward in the future - to be simulated with reinforcement learning . We want to develop a numerical model based on this new definition of trust - the contribution presented in this paper - and hypothesise that such a model would allow the emergence of other types of trust ( e.g . Luhmann trust ) on a large scale , or even validate the trust continuum hypothesis proposed by Hunyadi - our future works . The main contribution of this paper is a trust learning algorithm based on reinforcement learning . We called this algorithm SelfQ-Trust because the trust level an agent has in itself is evaluated by a Q-value calculated by the algorithm . Leveraging on recent findings about the reward circuit in the human brain , we propose a dynamic reinforcement of the trust reward ; a kind of distributional reinforcement learning applied on a vector of trust levels ; a method for explore and exploit the trust space . We then evaluate the model using a recent dataset of psychological research on the Dunning-Kruger effect on self-confidence . 2 RELATED WORK . The economics of internet platforms heavily depend on the quality of information and the security of users ( Barbosa et al. , 2020 ) . Trust models based on multi-agent systems ( MAS ) are used to help users choose their interactions by avoiding those who are allegedly incompetent or malicious ( Ramchurn et al. , 2004 ; Sabater & Sierra , 2005 ) . Many algorithms assign each agent a score based on the result of its past interactions with others ( Jøsang et al. , 2007 ) . Both implicit ( e.g . frequency , duration , closeness of relationships ) and explicit ( e.g . comments and ratings from other agents , payment informations ) measures are used and the score is considered as a measure of the trust the community can place in the agent ( Ben-Naim et al. , 2020 ) . The applications cover almost all uses of the Internet . Examples include : e-commerce ( e.g . ( Xiong & Liu , 2003 ) ) , collaborative economy platforms ( e.g . Airbnb ( Alsheikh et al. , 2019 ) ) , messaging ( e.g . ( Lien & Cao , 2014 ) ) . In such a model , the concept of trust is often confused with that of reputation ( Jøsang , 2007 ) . To the best of our knowledge , the theoretical frameworks are based on a cognitive model ( Cho et al. , 2015 ) or on game theory ( Wang et al. , 2016 ) . In cognitive models , an agent evaluates the trust it places in another based on the latter ’ s underlying degree of belief . But as it does not have direct access to the mental states of the latter , the trust decision is based on the observation of the outcome of the interaction . The observation model takes into account the uncertainty that arises from the - at least initial - unpredictability of the other agent . In this context , Bayesian networks are a logical choice of knowledge representation ( Esfandiari & Chandrasekharan , 2001 ) . In game-theoretic work , trust is the result of a game that involves a utility calculation based on knowledge of the history of relationships . A well-known trust game used in econmics is that of Berg et al . ( Berg et al. , 1995 ) . The assessment of these models can be done theoretically by establishing a set of properties to be achieved by the system ( see e.g . ( Cox , 2004 ) ) . The experimental approach consists in developing a test bed where different trust systems compete ( Sabater & Sierra , 2005 ) . In economics , Zak & Knack ( 2001 ) links trust and economic growth based on game theory . The hypothesis of the rational economic agent is to be reviewed . Indeed , if an agent in an economic system were purely rational , then the trust game of Berg et al . would respect the Nash equilibrium , which is not the case . Several attempts have been made in this direction , the most successful of which use Bayesian models ( e.g . ( Tribus , 2016 ) ) . The stakes are very high . It is a question of taking into account the realistic behaviour of social relations in a society , in order to define - in the long term - theoretical frameworks and indicators capable of making economic decisions in uncertain contexts . Previous work attempting to measure trust in MAS with Q-learning ( see e.g . Vijaya Kumar & Jeyapal ( 2014 ) ; Aref & Tran ( 2018 ) ) has relied on explicit measures of the relationship - which introduces a modelling bias . Meanwhile , the Berg ’ s hypothesis is being confirmed by recent work in neuroscience . Meyniel et al . ( 2015 ) argue that in humans , trust is not derived from a heuristic process but rather is formed during their learning process . For those reasons , we have taken the problem to be that of modelling trust as a human brain learning process by exploiting the latest knowledge on the functioning of the reward circuit . 3 BACKGROUND . 3.1 TERMINOLOGY . We consider a social system S defined by an environment and a finite set of predefined actions A . A population I of individuals represented by a MAS whose trust evolution relative to S is to be modelled . As defined by Russell & Norvig ( 2020 ) , “ an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors ” . Any agent of I is defined mathematically as an application f which maps every possible percepts sequence ( o1 , ... , on ) ∈ P∗ ( O ) , where O is the set of possible observations of the environment by the agent , to an action a ∈ A that the agent can perform or to any other function ( e.g. , each subfunction of an algorithm intrinsic to the agent ) that affects its possible actions : f : P∗ ( O ) → A . In his doctoral work , Marsh ( 1992 ) was the first to formalise trust as a computational concept . He introduced a measure of trust Tx ( y , α ) ∈ [ −1 , 1 [ as the probability that an agent y will act if Tx ( y , α ) ≥ 0 ( respectively minus the probability it will not act , if Tx ( y , α ) < 0 ) in the situation α to reach an outcome that another agent x presupposes to expect if it had a full trust in y , before y acts . If Tx ( y , α ) > threshold ( y , α ) the agents cooperate . Research in social sciences and information technology has extended the English terminology from property law 1 2 . Thus , according to Hardin ( 2002 ) , a ” trustor ” is an entity that trusts another entity , the ” trustee ” . According to Cofta ( 2007 ) , a trustor can be a social actor ( such as a person or an institution ) or a technical actor ( such as a computer or a software ) , which acts on behalf of a social actor . In the following , we will note f ∈ I to refer to a trustor who is placing its trust in a trustee g ∈ I when it wants to perform an action chosen from a part of the available actions P ( A ) . 3.2 Q-LEARNING . A single-agent reinforcement learning problem is commonly modelled by a Markov decision process ( MDP ) . A MDP is usually formalized by a tuple { S , A , T , R } where s ∈ S , a ∈ A and r ∈ R stands respectively for state , action and reward . T is a transition function defined as a probability distribution over the states . For any s ∈ S , agent ’ s choice to perfom an action a ∈ A occurs with the probablility T ( s , a , s′ ) ∈ [ 0 , 1 ] where s′ is the next state following s. It will result in the environment entering the new state s′ and give a reward r = R ( s , a ) . A policy π describes which action the agent takes in each state . Formally , it is therefore a function S → A in the case of a deterministic policy or S × A → [ 0 ; 1 ] in the stochastic case . The goal for the agent is to learn a policy that maximizes the cumulative rewards received over its learning process . A common way to achieve the goal is to solve the Bellman equation applied to the total sum of discounted rewards Qπ ( s , a ) reaped by the agent if it starts in state s and first takes the action a , before then applying the policy π ad infinitum ( γ states for the discount factor ) : q∗ ( s , a ) = ∑ s′ , r p ( s′ , r|s , a ) [ r + γmax a′∈A q∗ ( s ′ , a′ ) ] ( 1 ) Introduced by Watkins ( 1989 ) , Q-learning is a numerical method of solving Equation 1 . At each time step t of the algorithm , agent chooses action At from state St using policy derived from Q ( e.g . 1trustor ( Collins ) : ( in property law ) a person who sets up a trust transferring property to another person 2trustee ( Collins ) : someone with legal control of money or property that is kept or invested for another person , company , or organization -greedy ) . Then it makes a step in environment , observe the new state St+1 and the reward Rt+1 . Q-value is updated by applying a temporal-difference method ( TD ) ( Sutton , 1988 ) : Q ( St , At ) ← Q ( St , At ) + α [ Rt+1 + γmax a∈A Q ( St+1 , a ) −Q ( St , At ) ] ( 2 ) where Q is the learned action-value function which approximates the optimal action-value function q∗ ( Watkins & Dayan , 1992 ) , independent of the policy being followed and α ∈ [ 0 , 1 ] is a learning rate setting the updating speed of Q-values at each time step . The part of the equation in brackets is the difference between the reward estimated by the model and the reward actually received .

Summary: The paper presents a computational formulation of trust. For this, it introduces a Q-learning trust algorithm; where the agent's trust levels (at self and when dealing with another agent) can be evaluated using a Q-value function. This is achieved by: i) varying the trust reward received, ii) introducing a distributional TD learning over the possible trust levels, and iii) using an epsilon greedy strategy to learn the Q-value function for the trust model. The framework is face-validated in a grid-world where the agent must navigate nxn maze to reach a particular end location.   Interestingly, the paper proposes that this trust learning function proposed can be introduced in other model-free RL algorithms.