An Attempt to Model Human Trust with Reinforcement Learning
Authors: Vincent Frey
Simon Bécot
Venue: ICLR 2022 Submitted (2022)
Keywords: Trust
Confidence
Q-learning
Reward Circuit

Abstract: Existing works to compute trust as a numerical value mainly rely on ranking, rating or assessments of agents by other agents. However, the concept of trust is manifold, and should not be limited to reputation. Recent research in neuroscience converges with Berg's hypothesis in economics that trust is an encoded function in the human brain. Based on this new assumption, we propose an approach where a trust level is learned by an overlay of any model-free off-policy reinforcement learning algorithm. The main issues were i) to use recent findings on dopaminergic system and reward circuit to simulate trust, ii) to assess our model with reliable and unbiased real life models. In this work, we address these problems by extending Q-Learning to trust evaluation, and comparing our results to a social science case study. Our main contributions are threefold. (1) We model the trust-decision making process with a reinforcement learning algorithm. (2) We propose a dynamic reinforcement of the trust reward inspired by recent findings of neuroscience. (3) We propose a method to explore and exploit the trust space. The experiments reveal that it is possible to find a set of hyperparameters of our algorithm to reproduce recent findings on overconfidence effect in social psychology research.

Under review as a conference paper at ICLR 2022
AN ATTEMPT TO MODEL HUMAN TRUST WITH REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Existing works compute trust a numerical value mainly rely ranking, rating or assessments of agents by other agents. However, the concept of trust is manifold, and should not be limited to reputation. Recent research in neuro- science converges with Berg's hypothesis in economics that trust is an encoded function in the human brain. Based on this new assumption, we propose an ap- proach where trust level is learned by an overlay of any model-free off- off-policy reinforcement learning algorithm. The main issues were i) to use recent find- ings on dopaminergic system and reward circuit to simulate trust, ii) to assess our model with reliable and unbiased real life models. In this work, we address these problems by extending Q-Learning trust evaluation, and comparing our results to a social science case study. Our main contributions are threefold. (1) We model the trust-decision making process with reinforcement learning algorithm. (2) We propose a dynamic reinforcement of the trust reward inspired by recent findings neuroscience. (3) We propose method to explore and exploit the trust space. Ex- periments show that it is possible to find a set of hyperparameters of our algorithm that reproduces the Dunning- Kruger effect, whereby beginners are overconfident while experts tend to assess their confidence level more accurately.
INTRODUCTION
Trust seems to be one of the most overused words in the Information Technology world. Yet the concept struggling to find a universal definition: "the term is never defined. The usage is much clearer than the concept itself" (Hunyadi) 2020). At the end of the day, the question of trust in the digital world is rarely raised; scientists and industrialists prefer to focus on the effects of its crosion most often or its improvement more rarely- related issues: digital identity, privacy, personal datas, security, customer relationship, artificial intelligence, financial services digitalisation.
On the other hand, from a practical point of view e.g. for industries and institutions observed that there is a lack of tools to forecast the evolution of trust among stakeholders (customers, suppli- ers, partners) according the strategic choices of digital transformation. New job positions, such as that of chief trust officer, could be interested in this type of tool. This paper presents the preliminary research that allows us envision the building of such future applications.
On the understanding of trust as a structure of a social system, we choose to overcome the very first question concerning its definition on the basis of scientific knowledge in sociology, psychology and philosophy.
According social scientists, trust functions as a way to reduce complexity in a social system, while allowing it to evolve towards richer interactions. It acts as a stabilising structure for societies (Gambetta et al. 2000). Trust makes it possible to act in contexts where information lacking, or even unavailable (the relations between present and future events being indeterminable). Luhmann (2000) has provided semantic distinction between trust and confidence. Trust or interpersonal trust an assessment of risk by an individual who makes rational choice. Trust allows the binding of groups of people whose number does not exceed 100 to 150 Aral 2020 Harari 2016 Dunbar 1992) Confidence allows an individual to act in spite of inherent danger in the social system because he has faith in institutionalised factors of the society, his culture, his knowledge. Confidence allows social system to be unified at scale despite the oppositions of the groups within it.

---

Under review as a conference paper at ICLR 2022
In his recent work, the philosopher Hunyadi proposes new unified theory of trust, which leads to the following definition: "Trust is a bet on the behavioural expectations of things (e.g. that the ground will support me when I walk), people (that the driver of the car I pass will obey the rules of the road), and institutions (that the money use for transactions will have some value)." He argues that former theories- such Luhmann's are discontinuous because the intrinsic mechanisms of trust are not the same depending on the scale at which they are observed.
Hunyadi's 's theory seems both overlap with Luhmann's definition of interpersonal trust and to be practical enough- e.g. the notion of a bet leads quite naturally to defining reward in the future to be simulated with reinforcement learning.
We want to develop a numerical model based on this new definition of trust the contribution pre- sented in this paper- - and hypothesise that such model would allow the emergence of other types of trust (e.g. Luhmann trust) on a large scale, or even validate the trust continuum hypothesis proposed by Hunyadi our future works.
The main contribution of this paper is a trust learning algorithm based on reinforcement learning. We called this algorithm SelfQ-Trust because the trust level an agent has in itself is evaluated by Q-value calculated by the algorithm. Leveraging on recent findings about the reward circuit in the human brain, we propose a dynamic reinforcement of the trust reward a kind of distributional reinforcement learning applied trust space. We then evaluate the model using a recent dataset of psychological research on the Dunning- Kruger effect self-confidence.
vector of trust levels method for explore and exploit the
2 RELATED WORK
The economics of internet platforms heavily depend on the quality of information and the security of users (Barbosa et al. 2020). Trust models based on multi-agent systems (MAS) are used to help users choose their interactions by avoiding those who are allegedly incompetent malicious (Ramchurn et al., 2004 Sabater & Sierra 2005). Many algorithms assign each agent a score based on the result of past interactions with others (Jøsang et al. 2007). Both implicit (e.g. frequency, duration, closeness of relationships) and explicit (e.g. comments and ratings from other agents, payment informations) measures are used and the score is considered as a measure of the trust the community can place in the agent (Ben-Naim et al.) 2020). The applications cover almost all uses of the Internet. Examples include: e-commerce (e.g. (Xiong Liu 2003), collaborative economy platforms (e.g. Airbnb (Alsheikh et al. )019), ), messaging (e.g. (Lien & Cao 2014)).
In such a model, the concept of trust is often confused with that of reputation (osang 2007 ). To the best of our knowledge, the theoretical frameworks are based on cognitive model Cho et al.) 2015) or on game theory (Wang et al.) 2016). In cognitive models, an agent evaluates the trust places in another based on the latter's 's underlying degree of belief. But it does not have direct access to the mental states of the latter, the trust decision is based on the observation of the outcome of the interaction. The observation model takes into account the uncertainty that arises from the at least initial unpredictability of the other agent. In this context, Bayesian networks are a logical choice of knowledge representation (Esfandiari & Chandrasekharan 20001). In game-theoretic work, trust is the result of a game that involves a utility calculation based on knowledge of the history of relationships. A well-known trust game used in econmies is that of Berg et al. (Berg et al.) 1995). The assessment of these models can be done theoretically by establishing a set of properties to be achieved by the system (see e.g. (Cox) 2004)). The experimental approach consists in developing a test bed where different trust systems compete (Sabater & Sierra, 2005).
In economies, Zak & Knack (200T) links trust and economic growth based on game theory. The hypothesis of the rational economic agent is to be reviewed. Indeed, if an agent in an economic system were purely rational, then the trust game of Berg et al. would respect the Nash equilibrium, which is not the case. Several attempts have been made in this direction, the most successful of which use Bayesian models (e.g. (Tribus] 2016)). The stakes are very high. It is a question of taking into account the realistic behaviour of social relations in a society, in order to define in the long term theoretical frameworks and indicators capable of making economic decisions in uncertain contexts.
2

---

Under review as a conference paper at ICLR 2022
Previous work attempting to measure trust in MAS with Q-learning (see e.g. Vijaya Kumar & Jeyapal (2014); Aref & Tran (2018)) has relied on explicit measures of the relationship which introduces modelling bias. Mcanwhile, the Berg's 's hypothesis is being confirmed by recent work in neuroscience. Meyniel et al. (015) argue that in humans, trust is not derived from a heuristic process but rather is formed during their learning process. For those reasons, have taken the problem to be that of modelling trust as a human brain learning process by exploiting the latest knowledge on the functioning of the reward circuit.
BACKGROUND
3.1 TERMINOLOGY
We consider social system S defined by environment and finite set of predefined actions A. A population I of individuals represented by a MAS whose trust evolution relative to S is to be modelled. As defined by Russell & Norvig (2020), "an agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors". Any agent of I defined mathematically as an application f which maps every possible percepts sequence (01, 0n)) E (O)) where O is the set of possible observations of the environment by the agent, an action E A that the agent can perform or to any other function (e.g., each sub- function of an algorithm intrinsic to the agent) that affects its possible actions: f (O) - A.
In his doctoral work, Marsh (1992) was the first to formalise trust as a computational concept. He introduced a measure of trust Ta (n, a) [-1, 1[ as the probability that an agent y will act if To (y: ar) 2 0 (respectively minus the probability it will not act, if T, (v, a)) < 0) in the situation a to reach outcome that another agent presupposes to expect if it had full trust in y. before y acts. If (y, a) > threshold (y, a') the agents cooperate.
Research in social sciences and information technology has extended the English terminology from property law P Thus, according to Hardin (2002), a "trustor" is an entity that trusts another entity, the "trustee". According Cofta (007)) trustor can be social actor (such as person or an institution) or a technical actor (such as a computer or a software), which acts on behalf of a social actor. In the following, will note / E I to refer trustor who is placing its trust in trustee E I when wants to perform an action chosen from a part of the available actions ((A).
3.2 Q-LEARNING
A single-agent reinforcement learning problem is commonly modelled by a Markov decision process (MDP). A MDP is usually formalized by tuple A, T, R) where S, a A and r R. stands respectively for state, action and reward. T is a transition function defined as a probability distribution over the states. For any s E S, agent's 's choice perfom an action a E A occurs with the probablility ['ss, a, s') E [0. 1] where s' is the next state following It will result in the environment entering the new state s' and give reward r R(s,, a). A policy и describes which action the agent takes in each state. Formally, it is therefore a function S + AA in the case of a deterministic policy or S + [0: 1] the stochastic case. The goal for the agent is to learn policy that maximizes the cumulative rewards received over its learning process.
common way to achieve the goal is solve the Bellman equation applied to the total sum of discounted rewards (=, a) reaped by the agent if it starts in state s and first takes the action o, before then applying the policy π ad infinitum (7 states for the discount factor):
(=(s,a) Ep(s' r(s,, r+ ymax 9 (o)) a'EA
5'.
Introduced by Watkins (1989). Q-learning is a numerical method of solving Equation ! At each time step of the algorithm, agent chooses action A, from state S, using policy derived from Q (c.g.
'trustor (Collins): (in property law) a person who sets up a trust transferring property to another person "trustee (Collins): someone with legal control of money property that is kept or invested for another person, company, organization
3

---

Under review as a conference paper at ICLR 2022
e-greedy). Then it makes step in environment, observe the new state S1+1 and the reward R4+1- Q-value updated by applying a semporal-differene method (TD) (Sutton 1988)
Q(S. Q(S,, R1+1 +
m Q(S. aEA
where Q the learned action-value function which approximates the optimal action-valus function q+ (Watkins & Dayan 1992). independent of the policy being followed and a E [0, is learning rate setting the updating speed of Q-values at each time step. The part of the equation in brackets is the difference between the reward estimated by the model and the reward actually received.
3.3 MODELLING THE DOPAMINERGIC SYSTEM
Berg hypothesised that trust is a function that has been encoded in the human brain. Cognitive neu- roscientists linked the functioning of the neurotransmitter dopamine to the theoretical computational framework of reinforcement learning in the early 2000s (Holroyd & Coles 2002). It is now estab- lished that the release of dopamine the reward eircuit reveals a prediction error by the individual (llimceer 2011). This surprise signal favours the learning of reward predictions and shapes the individual's future behaviour.
Predictions of reward were previously represented as single scalar quantity. However, recent artifi- cial intelligence research on distributional reinforcement learning has inspired new model which the brain represents possible future rewards not single mean, but rather as a probability distri- bution, effectively representing multiple future outcomes simultaneously and parallel (Bellemare) al. 2017). Recordings from the mouse ventral tegmental area provided strong evidence for the neural realisation of distributional reinforcement learning (Dabney et al. 2020). From a functional point of view, interesting recent hypothesis suggests that dopaminergie neurons encode both dif- ferences between rewards and expectations in a goal-directed system. These prediction errors trigger reward learning (Bogac7 2020)
METHODS
We propose the following definition based on Hunyadi's work:
Definition 1 (Trust) Trust is a bet made by an intelligent agent on the behavioural expectations it has of another agent or of itself. Considering that an agent f. named trustor, trusts another agent g, named trustee, we define a level of trust Ty (g) E [-1,1] the trustor f that its bet will come true if the level positive, and false if the level negative.
measure of the subjective certainty of
4.1 PROBLEM STATEMENT
We consider population P of agents whose behaviour in given environment is modelled by MDP A,, R}. The agents learn an optimal policy by means of any underlying model-free off- policy reinforcement learning algorithm.
The goal is to estimate the trust level Ty E [-1, that a trustor f E P can have in any agent g P which advises it to take the action E A in state E S following the introspection of its MDP model. We require that: (1) the trust measurement algorithm runs in parallel with the MDP learning algorithm and (2) it can be implemented with any model-free off-policy RL algorithm.
4.2 LEARNING PRINCIPLESS
The general explanations concern the multi-agent case, in which each agent assumes both the role of trustor and that of trustee. For better understanding of the basics of the method, simplifying assumptions are made: the trust environment perceived by the agent is deterministic and the trust actions defined hereafter are discretised.
The trust level [g, s) is learned by trust model formally described by MDP {S, A, R Righ), where À [0, 2n] is the of trust actions corresponding trust levels ( Josagen and R

---

Under review as a conference paper at ICLR 2022
S x A + R is reward function. By applying the Definition we define the trust action à as follows: f trusts g to take action a chosen by w.I.t its underlying MDP model with trust level T = A_ means that the closer T is to 1, the greater f is certain that action is optimal regarding the underlying MDP and the closer T is to 1,, the greater f is certain that action a isn't optimal regarding the underlying MDP.
7.
Overall operation The trustor f asks any agent for the action a it would have taken it had been in the same state s as This advice allows choose its action. then learns its own model of trust towards thanks to trust reward f. may be a trustee found in a queue of trusted agents or f itself. In the latter case, the algorithm calculates the trust that f has in itself.
To do so, the algorithm learns Q-function Q 8 x Å R that calculates the quality of action combination. At all times, the trust that f places in in state can be derived from the Q-function. Equation[]iis the application of Q-learning the state-trust actions space. Applying the Q-learning principle, the trust level estimated by the model is therefore the one obtained thanks to the index of the trust action for the max of the Q-value. To report in the interval [-1, 1]. we divide by n and subtract 1.
arg max QE -
Tylg, 8) with (1)g()) (54)154 the Q-value trust vector
(3)
However, our initial attempts to make simple instantiation of Q-learning in the state-trust action space yielded poor results. We faced two main challenges: (1) a fixed reward did not work for learning realistic trust model. (2) when reinforcement is applied to only one trust level, the model quickly locks on the initial level. We used the following approach to answer these two issues.
Using the neural realization of distributional reinforcement learning hypothesis.
Firstly, we hypothesised that trust is a process that is built up in the human brain. Secondly, we define this process as a bet therefore fundamentally reward-based. We therefore require that model takes into account the latest neuroscientific knowledge about the reward circuit in the human brain. The reward prediction error (RPE) hypothesis of dopamine the discrepancy between ob- served and expected reward - has become central to research at the intersection of neuroscience and computer science because RPE is precisely the signal that RL system would need update reward expectations (Montague et al. 1996, Schultz et al., 1997). Dabney et al. findings provide strong evidence to support the hypothesis that the brain represents future rewards not as a single scalar quantity, but rather as a probability distribution.
We apply the hypothesis to our model of trust learning. We therefore require that (1) the reward magnitude must be updated according to an observation of the result of the bet ("Dynamic rein- forcement" section below) and (2) the bet made by the trustor in trust relationship with a trustee must give rise to several simultaneous rewards over all possible trust levels ("Distributional rein- forcement learning" section below).
Dynamic reinforcement. A way vary the reward according to the experience of trust relationships were to introduce a kind of attention mechanism. The general idea is (1) to increase the reward at the end of an episode when a positive trust action leads to an improved score and conversely (2) to make the reward proportional to the trust level of the trust action. For this purpose, we assume that the agent is able to measure its performance and that of other agents in solving the underlying MDP. The trustor uses this measure and the prior trust level it has towards its advisor to calculate the magnitude of the trust reward. The magnitude of the trust reward fig.g is regularly updated according to the observed performance of the trustee in finding the optimal policy of the underlying MDP. Each time the trustor f chooses an action advised by the prior trust level T it places in and the magnitude of the trust reward m..gg
trustee g. receives a trust reward depending on
F m.
(4)
Distributional reinforcement learning. We tried to apply Equation to the Q-value of trust level T prior to the relationship and observed that the learned trust level quickly locks onto its initial level. We chose to use distributed learning design to overcome this problem, while taking advantage of the
5

---

Under review as a conference paper at ICLR 2022
distributed reinforcement brain hypothesis. We therefore apply the learning all possible Q-values of trust actions at the given state-action pair. The 2n trust levels T2 are learned by 2n Q-functions Qi simultaneously and in parallel, as shown in Figure
In the following, T is the a priori level of trust that the trustee ff has in the trustee g. At each time step of the algorithm, difference 6(à, 7, F) between the reward es- timated by the model and the reward actually received is computed in the same way as in Equation 2 A Gaussian random variable X centered on I' is modulated by this difference. Equation 5 used to update the 2n functions Q: corresponding to the 2n discrete trust levels:
4:0 8(kk, 2 FXX
8(a,
(5)
with X EER g4nt1 ~ (T, 1)
Exploration and exploitation of trust model. The trustor has manage trade-off between trusting other agents without prior knowledge (trust exploration) and using the trust model it has learned to take into account other agents' advices (trust exploitation). We use for that an epsilon-gredd
Figure 1: Distributional trust learning. In the example, before the trust action, the trust level of the trustor towards the trustee is 0.6, as given by the model in gray. The trustor chooses an a priori trust level in range [0.6 ε, 0.6 + é say 0.2 (see Equation in next sec- tion). A Gaussian random variable X centered on 0.2 is modulated by the dif- ference between actual reward and ex- pectation (in green). Added to the Q- value, the result gives the model after the trust action in gray, indicating that the trust level drops from 0.6 to 0.4.
method where: (1) Trust exploration amounts to the trustor choosing to have full trust in the trustee; we hypothetize that trusting someone or oneself in a first relational experience should it be self-oriented is choosing to systematically take into account the ad- vice he/she gives. (2) Rather than just using the model being learned, the exploitation involves an uncertainty on the level of trust that the trustor grants to the trustee. In fact, as it is stated in the survey of Cho et al., the uncer- tainty on the priori trust that one grants in interper- sonal relationship tends to fade with time. In the single- agent case, only the phase of exploitation played.
4.3 SINGLE-AGENT CASE
SelfQ-Trust Algorithm (see Appendix A focuses the single-agent case, with agent of assuming both roles of trustor and trustee. From a technical point of view, the trust evaluation should be seen introspection function of the agent that evaluates in real time the relevance of its learning algorithm's 's prediction of an underlying MDP with respect to its current state. In the single-agent case, the evaluation of trust amounts to the evaluation of self-confidence. For better clarity, we instantiate below Definition T on the single-agent case.
Definition 2 (Self-confidencce) Self-confidence is a bet made by an intelligent agent on its future action. We define level of self-confidence T3 1- as measure of the subjective certainty of the agent f that future action it will take is optimal with respect to the task it is currently solving if the level is positive, and non optimal if the level is negative.
For the dynamic reinforcement, we update fj I at the end of each episode w.r.t the score e the agent achieved in the episode i. Equation i a heuristic that weights the reward of the trust model according to the score that the agent measures at the end of learning episode of the underlying MDP. The idea is as follows: we consider that the agent stores its scores in memory list of fixed- size. If the agent beats the highest score has stored the last card(memory) episodes, one can consider that it was right to trust itself over the past episode, driving to positive reward for the next episode. Conversely, if the agent does less well than its highest score, one can consider that was wrong to trust itself on the past episode, driving negative reward for the next episode. The magnitude of the reward will be proportional to the difference with the highest score observed the memory.

---

Under review as a conference paper at ICLR 2022
U
MOL Ey Ememory
(0z)
MJ
(6)
For the exploitation of trust model, Equation (aciionn/llle allows for uncertainty in the level of trust T that the agent grants to itself. The uncertainty decreasing with time, i.e. it is depending on the probability é of choosing to explore the trust model.
X n
T
with X ~ U (max(-,, -)), min(n, ((y((,,s)) + E)))
5 EXPERIMENTS
There are public quantitative surveys dealing with trust (e.g. interpersonal trust, confidence in pub- lie institutions (Fitzgerald et al., 2016; Ortiz-Ospina & Roser 2016)). Nevertheless, the available datasets are likely to be extremely biased the term is never defined. Moreover, the interpretation of general questions about trust can be very different from one individual to another.
However, there are studies in the social sciences and economics that give results observable properties of known systems. These studies can be used as test cases to assess the model. Among them, have chosen the so-called Soaniing-Kugge effect (DKE) assess the trust of agent in itself. Specifically, in their seminal paper, Kruger & Dunning 1999) put forward as an explanation a metacognitive difficulty of the unqualified people which prevents them from accurately recognising their incompetence and evaluating their real abilities. Conversely, the most qualified people would tend to underestimate their level of competence and would wrongly think that tasks that are easy for them are also easy for others. Dunning (2011) suggests that the effect is universal, as it has been observed in very different areas of learning.
Experimental Setup. We implemented the SelfQ-Trust algorithm in Python; the code is available at /tttps://git thub. com/ semm/selfee ftt.ss//tttttttte/eeeeessts code Environment a randomly generated maze of dimension (n, nj. The goal of the agent is to start from the upper left corner and find the exit at the lower right corner. The maze generation algorithm ensures that at least one path exists between the entrance and the exit.
The agent learns to trust itself select the optimal path in each state of the environment. In the maze case, the state is the cell where the agent is located. Thus, during an episode, agent learns a sequence of trust levels, each of which is linked to a cell he visited. Using this sequence, we calculate three metrics aggregate the trust measures during the episode: the mean trust all states of the environment at the end of the episode (M1); the mean trust over all states the agent passed through during the episode (M2); the max trust the agent had during the episode in the states it passed through (M3).
Simulation of the Dunning- Kruger Effect (DKE). In our first experiments, we were looking whether the algorithm is able to reproduce the DKE effect. We trained a population of 1000 agents over 3 mazes of dimension (6, 6). (10, 10), (30, 30). Computations of the three metries M1, M2, M3 are shown in Figure 2
The results show that the SelfQ-Trust algorithm reproduces the DKE trend: the agent's self- confidence goes through peak systematically present at the beginning of the training, then de- creases abruptly a more less pronounced local minimum, to end up rising and then stabilising. Moreover, the effect strength seems to increase with the complexity of the environment. The fol- lowing experiment raises the question of the dependence of DKE on hyperparameters.
Sensitivity to learning hyperparameters. The learning model uses two pairs of learning hyper- parameters: the underlying MDP that the agent uses to solve the maze problem uses an a learning rate and y discount factor; the trust model uses rate of trust learning å and a discount factor of trust learning ÿ. The environment is the same maze of dimension (5, 5) for the whole experiment. We trained a population of 50 agents 60 episodes of MDP learning well on 60 episodes of trust learning we choosed those settings accordingly to Sanchez & Dunning (2018) experiments on Overconfidence Effect. The second metric M2 is used show the results Figure
7

---

Under review as a conference paper at ICLR 2022
M1
M2
M3
100
(6x)
80
(Mb25)
(3030)
Self-Confieene
60
40
Marw (Ui))
20
Mane (1Nv)))
Kiaor (3AdS)
10P
10 Episode (lng
10a
Fgisnde clog
Episode (log scale)
Figure 2: DKE simulation. 1000 agents are trained on 500 synchronised episodes of the underlying MDP and the trust model each of the 3 mazes. Hyperparameters are set to: (ax, 7, Ot, 7) (0.1, 0.99, 0.01, 0.99) for the mazes. Estimate of the central tendency (mean estimator) and 95% confidence interval of self-confidence computed by SelfQ-Trust vs. learning episode.
40
Cif--Conieeneee
20
501
7101
-70
D/C
-00
10
20
30
40
50
10
20
40
50
10
20
50
60
10
20
30
50
60
Episade
Episode
Eiiisade
Episode
7, â and 5. We set (a, 7, â, (0.1, 0.9,0.1,,11 0.9) then for each of the figures we vary the hyperparameter studied the set (0.01, 0.02, 0.05, 0.1, 0.2) for and á and in the set {0.8, 0.9, 0.99, 0.999) for and 9. Estimate of the central tendency (mean estimator) and 95% confidence interval of self-confidence computed by SelfQ- Trust vs. learning episode.
Figure 3:
Sensitivity to learning hyperparameters
0.1,
The experiment confirms the DKE trend, regardless of the hyperparameters choosen. Moreover, the magnitude of the trend appears be correlated with a, â and the experience. does not seem to have much effect on the trust model (all curves are in the same statistical confidence interval). We wonder why the peak of confidence which we will later call the peak of overconfidence almost always occurs during the same short window of learning episodes?
Sensitivity to the complexity of the environment. The complexity of the environment should be narrowly assessed from the agent's 's point of view. We thus calculate it by training population of agents with the underlying MDP learning algorithm- -i i.e. a simple Q-learning on maze environment in the case of the experiment. We randomly generate 1000 mazes of dimension (10, 10). We train an agent on each over 1000 episodes with = 0.1 and 7 0.99. We deñne complexity as the smallest number of steps the agent has taken in the environment during the training episodes.
We then train 600 agents with SelfQ-Trus Trust on subset of mazes according to the range of the measured complexity values. The number of episodes of the underlying MDP training and that of the trust model are equal to 60. Hyperparameters are set to: a 0.1 = 0.99, á 0.1 {0.9, 0.99]. Results are in Figure
Figure ()) show that the beginner's overall self-confidence tends to increase with the complexity of the task being learned. It may be hypothesised that, the more complicated the problem is to solve, the more difficult it will be to leam self-confidence and the more biases in that learning such as overeonfidence will manifest themselves. Moreover, the closer the trust discount factor is to the greater the overconfidence effect. Shall the y factor SelfQ- -russt algorithm be able to measure the overconfidence effect studied by Sanchez & Dunning?
8

---

Under review as a conference paper at ICLR 2022
(a) (2- 0.9)
(b) (9 0.99)
complexity
complexity
15
Self-
20
20
69
10
20
30
50
18 28
Episode
Fpisode
Complexity
Figure 4: Sensitivity to environment complexity measured with the metric M2. 600 agents are trained on 8 mazes of dimension (10, 10) and complexity (18, 28, 38, 48, 58, 68, 78, 88). (a.b) Self- confidence vs. episode with two different discount factors of trust. = 0.9 in (a) and y = 0.99 in (b). (c) Self-confidence tendency per agent vs. complexity in both cases. All curves show an estimate of the central tendency (mean estimator) and 95% confidence interval
We also observe on Figure 4a) and (b) that the overconfidence peak occurring at the outset of learning is not exclusive. Depending on the trust discount factor and the level of complexity, one two peaks appear afterwards. This repetition of overconfidence peaks at the beginning of learning has not been reported in any of the social science research we have studied, and may constitute hypothesis.
Simulation of the Overconfidence Effect. One most recent sci- entific result the DKE is the measure of the Overconfidence (OC) effect by Sanchez & Dunning (Sanchez & Dunning 2018). We ex- tracted dataset of the measure of the self-confidence from their paper and tuned the SelfQ-Trust hyperparameters to reproduce the results of their Study Using the hypermarameter sensitivity study, we have indicated the trend. We then used a brute force approach find suitable hyperparameters. Parameters are: Maze (10, 10) of complexity = 88, 60 agents trained, 60 episodes of learning MDP as trust model, are shown in Figure
Overvonfidence Effect
80
60
Sel:Contfidnce
40
Experiments
0.1, 7 = 0.99, à 0.05, &- 0.99. Results
20
SellQ-TTue
(b) Expermeen with humans
10
50
Episode
CONCLUSION
Figure 5: SelfQ-Trust repro- duces the results of an experi- ment involving human volun- teers.
this paper we provide new definition of trust computer sci- ence. is praxis-oriented allowing for the development of numeri- cal models. We introduce SelfQ-Trust, a RL algorithm for measur- ing self-confidence of any human like entity that be modelled by an intelligent agent trying to solve a MDP. Our method is inspired by the latest findings neurosciences and RL and shows convinc- ing results, in line with research on the Dunning-Krugeh Effect in social psychology. Moreover, we show that numerical simulations of our model allows formulate new hypotheses; e.g. the intensity of the overconfidence effect depends on the complexity of the task be learned.
Further experiments can be carried out to strengthen the evidence: (1) Justify the use of " "distribu- tional reinforcement learning" with an experiment showing that model with a fixed trust reward do not work. However, we need to study our empirical design in more theoretical way, notably using the Bellemare operator of distributional reinforcement learning theory. (2) An ablation study to assess the need for dynamic reinforcement. (3) An ablation study on the exploitation of the trust model, with and without time-decaying uncertainty on the prior trust level. (4) Experiments with different base algorithms e.g., DQN and A2C would evaluate the usefulness of SelfQ-Trus. in more complex RL formulations, and whether the results hold. (5) Experiments involving human experi- menters and serious games to further investigate the overall design hypothesis.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sinan S Alsheikh, Khaled Shaalan, and Farid Meziane. Exploring the effects of consumers' trust: predictive model for satisfying buyers' expectations based on sellers' behavior in the marketplace. IEEE Access, 7.73357-73372, 2019.
Sinan Aral. The hype machine. Currency, 2020.
Abdullah Aref and Thomas Tran. A hybrid trust model using reinforcement learning and fuzzy logic. Computational Intelligence, 34(2)55155541 2018.
Nată M Barbosa, Emily Sun, Judd Antin, and Paolo Parigi. Designing for trust: A behavioral framework for sharing economy platforms. In Proceedings of The Web Conference 2020, pp. 2020.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. International Conference on Machine Learning, pp. 449-458, PMLR, 2017.
Jonathan Ben-Naim, Dominique Longin, and Emiliano Lorini. Formalization of cognitive-agent systems, trust, and emotions, 2020.
Joyce Berg, John Dickhaut, and Kevin McCabe. Trust, reciprocity, and social history. Games and economic behavior, 10(1):122-142, 1995.
Rafal Bogacz. Dopamine role in learning and action inference. Elife, 9te53262, 2020.
Jin-Hee Cho, Kevin Chan, and Sibel Adali. A survey trust modeling. ACM Computing Surveys (CSUR), 48(2):1-40, 2015.
Piotr Cofta. Trust, complexity and control: confidence in a convergent world. John Wiley & Sons, 2007.
James C Cox. How to identify trust and reciprocity. Games and economic behavior, 46(2):260-281, 2004.
Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value in dopamine- based rein- forcement learning. Nature, 577(7792)6611-6755 202.. 2020.
Robin IM Dunbar. Neocortex size as constraint on group size in primates. Journal of human evolution, (26)):69-4999 1992.
David Dunning. The saunning-kruger effect: On being ignorant of one's own ignorance. In Advances in experimental social psychology, volume 44, pp. 247-296. Elsevier, 2011.
Babak Esfandiari and Sanjay Chandrasekharan. On how agents make friends: Mechanisms for trust acquisition. In Proceedings of the fourth workshop on deception, fraud and trust in agent societies, Montreal, Canada, pp. 27-34, 2001.
Rory Fitzgerald et al. Ess round 8: European social survey round 8 data (2016). data file edition 2.1., 2016. https://www.kagg/ www.kagll..co//ascaalllieee com/pascalbliem/eunppann sociann--oil-suus
wwwessoocccccee--eesessee-e -urvey-ess---eeeeee 1ess---ed21-221617
Diego Gambetta et Can we trust trust. Trust: Making and breaking cooperative relations, 213-237, 2000.
Paul W Glimcher. Understanding dopamine and reinforcement learning: the dopamine reward pre- diction error hypothesis. Proceedings of the National Academy of Sciences, 108(Sppeeme 3): 15647-15654, 2011.
Yuval Noah Harari. Homo Deus: A brief history of tomorrow. Random House, 2016.
Clay B Holroyd and Michael GH Coles. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological review, 109(4);679, 2002.
Mark Hunyadi. Au début est la confiance. Lectures, Publications reçues, 2020.
10

---

Under review as a conference paper at ICLR 2022
Audun Josang. Trust and reputation systems. In Foundations of security analysis and design IV, pp- 209-245..00 Springer, 2007.
Audun Jøsang, Roslan Ismail, and Colin Boyd. A survey of trust and reputation systems for online service provision. Decision support systems, 43(2):618-644, 2007.
Justin Kruger and David Dunning. Unskilled and unaware it: how difficulties in recognizing one's own incompetence lead to inflated self- aeses-assssmesss.... Journal of personality and social psychology, 77(6): 121, 1999.
Che Hui Lien and Yang Cao. Examining wechat users' motivations, trust, attitudes, and positive word-of-mouth: Evidence from china. Computers in human behavior, 41:104-111, 2014.
Niklas Luhmann. Familiarity, confidence, trust: Problems and alternatives. Trust: Making and breaking cooperative relations, 6:94-107, 2000.
Stephen Marsh. Trust in distributed artificial intelligence. In European Workshop on Modelling Autonomous Agents in a Mulli-Agent World, pp. 94-112. Springer, 1992.
Florent Meyniel, Daniel Schlunegger, and Stanislas Dehaene. The sense confidence during prob- abilistic learning: A normative account. PLoS computational biology, 11(6);e 1004305, 2015.
P Read Montague, Peter Dayan, and Terrence Sejnowski. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of neuroscience, 16(5):1936 1947, 1996.
Esteban Ortiz-Ospina and Max Roser.
Trust.
Our
World Data, 2016.
https//ouwwordiidat..oogguus
Sarvapali D Ramchurn, Dong Huynh, and Nicholas R Jennings. Trust in multi-agent systems. The knowledge engineering review, 19(1):1-25, 2004.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. 2020.
Jordi Sabater and Carles Sierra. Review on computational trust and reputation models. Artificial intelligence review, 24(1):33-66 60, 2005.
Carmen Sanchez and David Dunning. Overconfidence among beginners: little learning a dan- gerous thing? Journal Personality and Social Psychology, 114(1):10, 2018.
Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275(5300)) 593--1599 1997.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Myron Tribus. Rational Descriptions, Decisions and Designs: Pergamon Unified Engineering Se- ries. Elsevier, 2016.
Anitha Vijaya Kumar and Akilandeswari Jeyapal. Self-adaptive trust based abr protocol for manets using q-learning. The Scientific World Journal, 2014, 2014.
Yingjie Wang, Zhipeng Cai, Guisheng Yin, Yang Gao, Xiangrong Tong, and Qilong Han. A game theory-based trust measurement model for social networks. Computational social networks, 3(1): 1-16, 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-22)) 1992.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Li Xiong and Ling Liu. A reputation-based trust model for peer-to-peer e-commerce communities. In EEE International Conference on E-Commerce, 2003. CEC 2003., pp. 275-284. IEEE, 2003.
Paul Zak and Stephen Knack. Trust and growth. The economic journal. 11 (11(470):25-33) -21, 2001.
11

---

Under review as a conference paper at ICLR 2022
A ALGORITHM
Algorithm 1: SelfQ-Trust estimating self-trust of agent learning to solve MDP.
Data: trust learning rate á 1], =
Initialize (y(s) ~ (u, o²) - 1 for all s E S with e.g. = 0 and o² cardiQ 2n++1
Initialize intensity of trust reward R
Initialize empty list of scores for each past episode (Uy)ıgssi
foreach episode do
8 - So: steps +- 0 foreach step of episode until sT do
Get action w.r.t. MDP model f and state
n 5 uniform random number E [0, if n < then â <- 2n else
10
T uniform random number
E { - n [max(-1,77(,.* - mi), + +)]]
12
Get trust action à from T: <- n(1 + T)
13
Take action in enw; learn MDP
14
Learn trust: LearnTrustir 8,)
15
s'
16
steps + steps +
maz (1y) 18
i & (0,1) then Ry 18
84
Append ssepps (v:)
19
Decay غ
20
Function LearnTrust(a,, (, 8,
8 à a 1) + max ( (
21
w'E
Get E RRn++ ~(0, 1):Y Y (X2n-a, Xan-a)
22
(5(s) + (5(5)
23
12


Reviews:
[{"review_id": "n5v6K_dPSrs", "reviewer": "Reviewer_fiz1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles a novel trust problem with several (economics, psychological and engineering) implications for designing agents with appropriate metacognition for decision-making. However, there were a few conceptual concerns that need to be addressed for me:\n\nMain comments:\n- There are conceptual gaps between the definition of trust introduced in Section 1 ‘an assessment of risk by an individual who makes a rational choice’, and Section 4 ‘is both a decision-making process in a relationship between two intelligent agents and the measure by which the decision is made’. Particularly regarding the measure by which the decision is made – this speaks to the metacognition and not an assessment of risk. It would be helpful to explain exactly how trust differs from meta-cognition. Additionally, to avoid confusing the reader to introduce the definition right at the beginning. \n- The introduction was too long winded and not enough description about the research question e.g., what is a digital model? What are the evolution drivers? The paper speaks to it briefly in Section 3.3 regarding genetic fitness – is this the complexity? However, links are tenuous at best given that it jumps straight to learning in the brain, and not evolution. This would be interesting point to pursue, without jumping to dopamine, and reward circuitry. Additionally, the related literature needs to focus on economics, neuroeconomics, and social psychology literature e.g., Mazor, M., Fleming, S.M. The Dunning-Kruger effect revisited. Nat Hum Behav 5, 677–678 (2021). , Cox, J.C. (2004) How to Identify Trust and Reciprocity. Games and Economic Behavior, 46: 260-281.\n- The paper is premised on Berg’s hypothesis. Therefore, it was quite surprising to see that the simulations used a grid-world instead a more appropriate algorithm where the agent had to make a pay-off decision. This would allow the simulation results to be evaluated in comparison to the  ‘real-life world’ scenarios that were mentioned in the abstract. Additionally, it would be interesting to evaluate the DK effect in more realistic scenarios using this Q-value trust function. Additionally, it would be interesting, and useful, to evaluate under what conditions this isn’t replicated. Currently, this effect is barely visible when the environment set-up is simple (6x6).   \n- The paper mentions that this trust function can be added to other model-free RL algorithms. This claim should be supported by additional experiment with different base algorithms e.g., DQN and A2C. This would help the reader evaluate the usefulness of this in more complex RL formulations, and whether the results hold. \n- In the introduction, there was a discussion around the different types of actions available e.g., assured and chosen. The simulations don’t prescribe this separation in action type or ability to control the environment. Perhaps, as mentioned in the introduction this might be relevant for multi-agent systems. However, I would like to understand how the DKE, or the learning differs depending on the type of actions an agent makes. \n\nMinor comments:\n- Graphical representation of the algorithm, and exactly how the trust learning function can be included in other model-free RL algorithms. It is not spelt out clearly exactly how the trust learning function would benefit the other algorithms, and what the additional complexity costs associated with this are. \n- Please review and update typing errors and grammar throughout. \n- Please explain all variable / parameters being introduced.\n- There is repetition in the paper, e.g., some quotations are introduced twice, that should be removed. \n", "summary_of_the_paper": "The paper presents a computational formulation of trust. For this, it introduces a Q-learning trust algorithm; where the agent's trust levels (at self and when dealing with another agent) can be evaluated using a Q-value function. This is achieved by: i) varying the trust reward received, ii) introducing a distributional TD learning over the possible trust levels, and iii) using an epsilon greedy strategy to learn the Q-value function for the trust model. The framework is face-validated in a grid-world where the agent must navigate nxn maze to reach a particular end location. \n\nInterestingly, the paper proposes that this trust learning function proposed can be introduced in other model-free RL algorithms. ", "summary_of_the_review": "The paper nicely presents a simple function for defining trust. However, the conceptual and experimental shortcomings need to be addressed. Specifically, appropriate experiments need to be run to evaluate the observed effects and their implications on understanding human behaviour. Ideally, the paper would simulate previously evaluated human experiments and compare those results with their simulation analysis. The metrics used would also need to be comparable. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rzki_K3Qx6s", "reviewer": "Reviewer_owda", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of modeling self-confidence is indeed very interesting and fascinating. Having said that, unfortunately the paper is not completely convincing. The authors try to map a series of different concepts from other disciplines into reinforcement learning concepts, but the actual rationale at the basis of the composition of the different resulting component is unclear.\n\nThe key formulas in the paper (Formulae 3 and 5-7) are not really justified. They appear as given \"modeling assumptions\" in a sense. Moreover, the derivation of the trust vector (Formula 3) is not sufficiently discussed in this work. Why can we call it a trust vector? The introduction of distributional learning is also not clearly introduced in the paper. The authors say: \"instead of applying learning only on the trust level t prior to the relationship, it is applied on all the possible trust levels\". Why is this needed?\n\nThe reviewer really struggles to understand how the model of the dopaminergic system (Section 3.3) fits with the rest of the paper. At the end, it seems that this is not actually used in the rest of the paper, except for a potential link with the idea of reinforcement learning, but this \"analogy\" is not new per se.\n\nThe mathematical foundation of the \"dynamic learning\" presented in Section 4.2 is rather unclear. Its introduction is again not fully justified. Why do you need to introduce it? What does it really add? This should have been evaluated in the paper in my opinion.\n\nIn the formulation, the authors refer to the \"multi-agent\" case, but it is difficult to understand how this would be implemented in practice (also the notation for multiple agents is not completely clear). The reviewer does not understand why you need a different notation for the single agent as well. Should not be the same in a sense?\n\nThe authors also do not discuss the selection of the values of the models, which are difficult to set a priori.\n\nThe evaluation is based on experiment about self-confidence, including an experiment concerning the selection of the Dunning-Kruger effect. It seems to me that this is a specific case and, in any case, it is difficult to understand the actual effect on the results of the model. Are you really model self-confidence? That is not clear from the mathematical model - for this reason, the interpretation of the results provided by the authors might be questionable.\n\nMore detailed comments:\n\nSection 1.1: these are only two possible definitions of trust. It seems to me that this is an oversimplification in a sense.\n\nSection 1.2: it is unclear how this section is related to the rest of the work. These concepts are not really explored in this work.\n\nSection 1.3: the three questions listed here are not really unanswered in the paper in my opinion.\n\nSection 5: this section is essentially about self-trust. The reviewer wonders if it would be better to rewrite the paper with a focus on self-confidence?\n", "summary_of_the_paper": "The paper describes an approach to model \"self-confidence\". The authors use the term trust, but the work focusses mainly on confidence on a certain personal decision. The paper is not about trust in other agents. The model is evaluated using a simulation of a maze where the agent has different degrees of self-confidence. In general, I would say that the contribution is rather limited since the authors use a mathematical model, which is an integration of different components without a clear set of underlying principles.", "summary_of_the_review": "The problem of trust is interested, but the paper actually focuses on \"self-confidence\". The assumptions and formulation of the mathematical model are not justified. Several concepts are introduced in the paper, but they are not really used in the model. The findings of the simulations are not insightful, since the model is based on assumptions that are not based on a clear rationale.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FcniKfO3Wxc", "reviewer": "Reviewer_GxNj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think understanding trust is of obvious importance, so from that perspective the work is well-motivated. However, I think the paper as it currently stands spends too much time on motivation and setup and not nearly enough either explaining its specific contribution in detail, or providing a strong payoff to all the setup. \n\nSpecifically, the paper spends nearly a page and a half on a near-philosophical digression on trust and confidence -- what is the relevance / importance of this to the contribution? Similarly with the high-level claims on the importance of trust to society -- it's fairly obvious that understanding trust is important, and an extended discussion of this is not needed. The paper spends another half-page reviewing connections from RL to the modeling the dopaminergic system -- this connection is worth mentioning and citing but multiple paragraphs of discussion seem out of place). It spends a fair bit of space motivating the modeling of trust in a population of agents, even though the actual contribution is only a self-trust model. Additional space is spent on using a scrum master and agile development as a worked example -- I'm not sure how familiar this would be to the ICLR audience. At the same time, the specific algorithm description is left to the appendix, and from reading it I'm still not sure how the trust-learning algorithm interfaces with the regular RL algorithm? It seems like the trust algorithm has its own action space and reward space -- is it a sort of off-policy contextual bandit where the regular RL algorithm manages state transitions and SelfQTrust learns to choose trust actions given a state it doesn't get to control? I'm still not sure what exactly was done and confused about the distinction between what the general multi-agent framework is here, vs what was actually implemented and evaluated. \n\nIn addition, I have some technical puzzles and concerns: \n- If T_x() leads to one of two outcomes (i.e. it is a Bernoulli random variable) why the parameterization of p(act) - p(not_act)? Wouldn't p(act)+p(not_act)=1, implying that T_x(...) could just be p(act) w.l.o.g.? \n- Why does the agent's choice to perform an action *occur* with a probability that's a function of s'? Considering s' is not known, transition probability should be defined for each action given state s'. T(s, a, s') seems like the action-conditional transition probability, i.e. a property of the environment. \n- Expr 1 and 2 notation is underspecified: what is the expectation over? I believe it is the next action given some policy. More importantly, if the tickmark `'` means \"next\" (action/state), then shouldn't the notation have R' and max Q(s', a)? I.e. next reward and Q value of current action next state. \n\nFinally, there are some typos, including: \n- \"choosen action\" -> chosen.\n- \"agent choose action\" -> chooses. \n- \"whith the probability\". ", "summary_of_the_paper": "The submission proposes a model of the dynamics of trust from a reinforcement learning perspective. It provides a simulation of the Dunning-Kruger effect in a simple grid world using this model. \n", "summary_of_the_review": "I am not an expert in trust modeling, but even so I think the paper as framed is mistargeted for the conference. It needs to have much shorter / pithier motivation and review, and much more detailed and precise explanation of what was actually done. In addition, there need to be some sort of payoff (i.e. new results / insights) from all this framing -- as-is, there is the DK and OC effects, but to the extent there's a prediction of new effects (e.g. the location of the first confidence peak) the paper dismisses it as a potential artifact of the setup, so I'm not sure what to make of it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2GvT-CGtw5t", "reviewer": "Reviewer_dEMd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1.\tThe narrative perspective is interesting and the analogy from trustor-trustee advisory is intuitive, and easy to relate to the technical formulations.\n2.\tMore contemporary concept like distributional reinforcement learning is employed for learning multiple trust levels, and the authors proposed a novel dynamic trust reward based on the change in trust level during the learning process.\n3.\tThis paper provides comprehensive experiments on evaluating hyperparameter sensitivity as well as simulating the DKE overconfidence effect and reproduce the results from other works, adding confidence to the experiment findings.\n\nCons: \n1.\tAlthough it is clear that the goal is to construct a trust model in multi-agent scenario, but the ultimate motivation in the bigger picture perspective is not addressed clearly. For example, what is the benefit or application of constructing a trust model?\n2.\tFrom a policy performance standpoint, there is no comparison between the performance when the trust model is present/absent (ablation study). Understand that this may not be the primary goal of this paper (also somehow related to the first question), but this would add to the completeness and perhaps spark future interests among readers.\n\nQuestions for clarification:\n1.\tIn the background and introduction section, the definitions and terminologies of trust, confidence and available actions are explained as an integral part of the social systems structure. What role does the “confidence” play in the proposed modelling of trust-decision making process? It seems the proposed method is based primarily on “trust” and “confidence” is an \n2.\tIn section 1.3, “… shared narratives tend to become more liquid. As a consequence, confidence is decreasing to the benefit of trust.” This sounds interesting, but is not exactly clear. Can you elaborate more on this? High self-trust level seems to imply higher confidence in your discussion, but why does this statement observe an inverse relationship between the two?\n3.\tIs there an explanation or hypothesis on why overconfident is more pronounce in higher complexity problems (especially when the paper has a heavy psychological-inspired tone)? I feel it is better to include at least a line of hypothetical explanation rather than leaving it completely as an open question.\n\nSome typos: \n1.\tSection 1.2 paragraph 3 last sentence: condidence -> confidence\n2.\tSection 3.1 paragraph 2 second sentence: [-1, 1[ -> [-1, 1]\n\n", "summary_of_the_paper": " \nThe paper proposed an algorithm to model trust numerically in parallel with the policy learning process in multi-agent scenario. The algorithm resembles an overlay on any reinforcement learning algorithm with modification using a dynamic trust reward, depending on trust levels. The proposal is inspired by dopaminergic system and reaffirms the Dunning-Kruger effect (DKE) in societal studies.\n", "summary_of_the_review": "The paper takes on an interesting perspective in trust modelling while connecting the biological, humanities, psychological and societal studies inspirations to the proposed method. The technical formulations are easy to follow and  sound.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a methodology for modeling, and learning, trust in a multi-agent reinforcement learning system. The reviewers considered this to be an interesting and important question to answer. Nevertheless, they maintained concerns on multiple fronts. The paper could benefit from being more focused. Authors are strongly encouraged to further scale down the claims in the introduction, and ensure that claims made there and later in the paper are matched with experiments that quantify, and validate, the notions introduced. Model choices made, as well assumptions introduced should be clearly motivated/mapped to reality, in light of their strength. Extending experiments to broader example settings, as outlined in the reviews, would also strengthen the work.
Decision: Reject