Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  
The paper proposed an algorithm to model trust numerically in parallel with the policy learning process in multi-agent scenario. The algorithm resembles an overlay on any reinforcement learning algorithm with modification using a dynamic trust reward, depending on trust levels. The proposal is inspired by dopaminergic system and reaffirms the Dunning-Kruger effect (DKE) in societal studies.

main_review: Pros: 
1.	The narrative perspective is interesting and the analogy from trustor-trustee advisory is intuitive, and easy to relate to the technical formulations.
2.	More contemporary concept like distributional reinforcement learning is employed for learning multiple trust levels, and the authors proposed a novel dynamic trust reward based on the change in trust level during the learning process.
3.	This paper provides comprehensive experiments on evaluating hyperparameter sensitivity as well as simulating the DKE overconfidence effect and reproduce the results from other works, adding confidence to the experiment findings.

Cons: 
1.	Although it is clear that the goal is to construct a trust model in multi-agent scenario, but the ultimate motivation in the bigger picture perspective is not addressed clearly. For example, what is the benefit or application of constructing a trust model?
2.	From a policy performance standpoint, there is no comparison between the performance when the trust model is present/absent (ablation study). Understand that this may not be the primary goal of this paper (also somehow related to the first question), but this would add to the completeness and perhaps spark future interests among readers.

Questions for clarification:
1.	In the background and introduction section, the definitions and terminologies of trust, confidence and available actions are explained as an integral part of the social systems structure. What role does the “confidence” play in the proposed modelling of trust-decision making process? It seems the proposed method is based primarily on “trust” and “confidence” is an 
2.	In section 1.3, “… shared narratives tend to become more liquid. As a consequence, confidence is decreasing to the benefit of trust.” This sounds interesting, but is not exactly clear. Can you elaborate more on this? High self-trust level seems to imply higher confidence in your discussion, but why does this statement observe an inverse relationship between the two?
3.	Is there an explanation or hypothesis on why overconfident is more pronounce in higher complexity problems (especially when the paper has a heavy psychological-inspired tone)? I feel it is better to include at least a line of hypothetical explanation rather than leaving it completely as an open question.

Some typos: 
1.	Section 1.2 paragraph 3 last sentence: condidence -> confidence
2.	Section 3.1 paragraph 2 second sentence: [-1, 1[ -> [-1, 1]


summary_of_the_review: The paper takes on an interesting perspective in trust modelling while connecting the biological, humanities, psychological and societal studies inspirations to the proposed method. The technical formulations are easy to follow and  sound.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submission proposes a model of the dynamics of trust from a reinforcement learning perspective. It provides a simulation of the Dunning-Kruger effect in a simple grid world using this model. 

main_review: I think understanding trust is of obvious importance, so from that perspective the work is well-motivated. However, I think the paper as it currently stands spends too much time on motivation and setup and not nearly enough either explaining its specific contribution in detail, or providing a strong payoff to all the setup. 

Specifically, the paper spends nearly a page and a half on a near-philosophical digression on trust and confidence -- what is the relevance / importance of this to the contribution? Similarly with the high-level claims on the importance of trust to society -- it's fairly obvious that understanding trust is important, and an extended discussion of this is not needed. The paper spends another half-page reviewing connections from RL to the modeling the dopaminergic system -- this connection is worth mentioning and citing but multiple paragraphs of discussion seem out of place). It spends a fair bit of space motivating the modeling of trust in a population of agents, even though the actual contribution is only a self-trust model. Additional space is spent on using a scrum master and agile development as a worked example -- I'm not sure how familiar this would be to the ICLR audience. At the same time, the specific algorithm description is left to the appendix, and from reading it I'm still not sure how the trust-learning algorithm interfaces with the regular RL algorithm? It seems like the trust algorithm has its own action space and reward space -- is it a sort of off-policy contextual bandit where the regular RL algorithm manages state transitions and SelfQTrust learns to choose trust actions given a state it doesn't get to control? I'm still not sure what exactly was done and confused about the distinction between what the general multi-agent framework is here, vs what was actually implemented and evaluated. 

In addition, I have some technical puzzles and concerns: 
- If T_x() leads to one of two outcomes (i.e. it is a Bernoulli random variable) why the parameterization of p(act) - p(not_act)? Wouldn't p(act)+p(not_act)=1, implying that T_x(...) could just be p(act) w.l.o.g.? 
- Why does the agent's choice to perform an action *occur* with a probability that's a function of s'? Considering s' is not known, transition probability should be defined for each action given state s'. T(s, a, s') seems like the action-conditional transition probability, i.e. a property of the environment. 
- Expr 1 and 2 notation is underspecified: what is the expectation over? I believe it is the next action given some policy. More importantly, if the tickmark `'` means "next" (action/state), then shouldn't the notation have R' and max Q(s', a)? I.e. next reward and Q value of current action next state. 

Finally, there are some typos, including: 
- "choosen action" -> chosen.
- "agent choose action" -> chooses. 
- "whith the probability". 
summary_of_the_review: I am not an expert in trust modeling, but even so I think the paper as framed is mistargeted for the conference. It needs to have much shorter / pithier motivation and review, and much more detailed and precise explanation of what was actually done. In addition, there need to be some sort of payoff (i.e. new results / insights) from all this framing -- as-is, there is the DK and OC effects, but to the extent there's a prediction of new effects (e.g. the location of the first confidence peak) the paper dismisses it as a potential artifact of the setup, so I'm not sure what to make of it.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a computational formulation of trust. For this, it introduces a Q-learning trust algorithm; where the agent's trust levels (at self and when dealing with another agent) can be evaluated using a Q-value function. This is achieved by: i) varying the trust reward received, ii) introducing a distributional TD learning over the possible trust levels, and iii) using an epsilon greedy strategy to learn the Q-value function for the trust model. The framework is face-validated in a grid-world where the agent must navigate nxn maze to reach a particular end location. 

Interestingly, the paper proposes that this trust learning function proposed can be introduced in other model-free RL algorithms. 
main_review: The paper tackles a novel trust problem with several (economics, psychological and engineering) implications for designing agents with appropriate metacognition for decision-making. However, there were a few conceptual concerns that need to be addressed for me:

Main comments:
- There are conceptual gaps between the definition of trust introduced in Section 1 ‘an assessment of risk by an individual who makes a rational choice’, and Section 4 ‘is both a decision-making process in a relationship between two intelligent agents and the measure by which the decision is made’. Particularly regarding the measure by which the decision is made – this speaks to the metacognition and not an assessment of risk. It would be helpful to explain exactly how trust differs from meta-cognition. Additionally, to avoid confusing the reader to introduce the definition right at the beginning. 
- The introduction was too long winded and not enough description about the research question e.g., what is a digital model? What are the evolution drivers? The paper speaks to it briefly in Section 3.3 regarding genetic fitness – is this the complexity? However, links are tenuous at best given that it jumps straight to learning in the brain, and not evolution. This would be interesting point to pursue, without jumping to dopamine, and reward circuitry. Additionally, the related literature needs to focus on economics, neuroeconomics, and social psychology literature e.g., Mazor, M., Fleming, S.M. The Dunning-Kruger effect revisited. Nat Hum Behav 5, 677–678 (2021). , Cox, J.C. (2004) How to Identify Trust and Reciprocity. Games and Economic Behavior, 46: 260-281.
- The paper is premised on Berg’s hypothesis. Therefore, it was quite surprising to see that the simulations used a grid-world instead a more appropriate algorithm where the agent had to make a pay-off decision. This would allow the simulation results to be evaluated in comparison to the  ‘real-life world’ scenarios that were mentioned in the abstract. Additionally, it would be interesting to evaluate the DK effect in more realistic scenarios using this Q-value trust function. Additionally, it would be interesting, and useful, to evaluate under what conditions this isn’t replicated. Currently, this effect is barely visible when the environment set-up is simple (6x6).   
- The paper mentions that this trust function can be added to other model-free RL algorithms. This claim should be supported by additional experiment with different base algorithms e.g., DQN and A2C. This would help the reader evaluate the usefulness of this in more complex RL formulations, and whether the results hold. 
- In the introduction, there was a discussion around the different types of actions available e.g., assured and chosen. The simulations don’t prescribe this separation in action type or ability to control the environment. Perhaps, as mentioned in the introduction this might be relevant for multi-agent systems. However, I would like to understand how the DKE, or the learning differs depending on the type of actions an agent makes. 

Minor comments:
- Graphical representation of the algorithm, and exactly how the trust learning function can be included in other model-free RL algorithms. It is not spelt out clearly exactly how the trust learning function would benefit the other algorithms, and what the additional complexity costs associated with this are. 
- Please review and update typing errors and grammar throughout. 
- Please explain all variable / parameters being introduced.
- There is repetition in the paper, e.g., some quotations are introduced twice, that should be removed. 

summary_of_the_review: The paper nicely presents a simple function for defining trust. However, the conceptual and experimental shortcomings need to be addressed. Specifically, appropriate experiments need to be run to evaluate the observed effects and their implications on understanding human behaviour. Ideally, the paper would simulate previously evaluated human experiments and compare those results with their simulation analysis. The metrics used would also need to be comparable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes an approach to model "self-confidence". The authors use the term trust, but the work focusses mainly on confidence on a certain personal decision. The paper is not about trust in other agents. The model is evaluated using a simulation of a maze where the agent has different degrees of self-confidence. In general, I would say that the contribution is rather limited since the authors use a mathematical model, which is an integration of different components without a clear set of underlying principles.
main_review: The idea of modeling self-confidence is indeed very interesting and fascinating. Having said that, unfortunately the paper is not completely convincing. The authors try to map a series of different concepts from other disciplines into reinforcement learning concepts, but the actual rationale at the basis of the composition of the different resulting component is unclear.

The key formulas in the paper (Formulae 3 and 5-7) are not really justified. They appear as given "modeling assumptions" in a sense. Moreover, the derivation of the trust vector (Formula 3) is not sufficiently discussed in this work. Why can we call it a trust vector? The introduction of distributional learning is also not clearly introduced in the paper. The authors say: "instead of applying learning only on the trust level t prior to the relationship, it is applied on all the possible trust levels". Why is this needed?

The reviewer really struggles to understand how the model of the dopaminergic system (Section 3.3) fits with the rest of the paper. At the end, it seems that this is not actually used in the rest of the paper, except for a potential link with the idea of reinforcement learning, but this "analogy" is not new per se.

The mathematical foundation of the "dynamic learning" presented in Section 4.2 is rather unclear. Its introduction is again not fully justified. Why do you need to introduce it? What does it really add? This should have been evaluated in the paper in my opinion.

In the formulation, the authors refer to the "multi-agent" case, but it is difficult to understand how this would be implemented in practice (also the notation for multiple agents is not completely clear). The reviewer does not understand why you need a different notation for the single agent as well. Should not be the same in a sense?

The authors also do not discuss the selection of the values of the models, which are difficult to set a priori.

The evaluation is based on experiment about self-confidence, including an experiment concerning the selection of the Dunning-Kruger effect. It seems to me that this is a specific case and, in any case, it is difficult to understand the actual effect on the results of the model. Are you really model self-confidence? That is not clear from the mathematical model - for this reason, the interpretation of the results provided by the authors might be questionable.

More detailed comments:

Section 1.1: these are only two possible definitions of trust. It seems to me that this is an oversimplification in a sense.

Section 1.2: it is unclear how this section is related to the rest of the work. These concepts are not really explored in this work.

Section 1.3: the three questions listed here are not really unanswered in the paper in my opinion.

Section 5: this section is essentially about self-trust. The reviewer wonders if it would be better to rewrite the paper with a focus on self-confidence?

summary_of_the_review: The problem of trust is interested, but the paper actually focuses on "self-confidence". The assumptions and formulation of the mathematical model are not justified. Several concepts are introduced in the paper, but they are not really used in the model. The findings of the simulations are not insightful, since the model is based on assumptions that are not based on a clear rationale.
review_title: null

Meta-review: The paper presents a methodology for modeling, and learning, trust in a multi-agent reinforcement learning system. The reviewers considered this to be an interesting and important question to answer. Nevertheless, they maintained concerns on multiple fronts. The paper could benefit from being more focused. Authors are strongly encouraged to further scale down the claims in the introduction, and ensure that claims made there and later in the paper are matched with experiments that quantify, and validate, the notions introduced. Model choices made, as well assumptions introduced should be clearly motivated/mapped to reality, in light of their strength. Extending experiments to broader example settings, as outlined in the reviews, would also strengthen the work.
Decision: Reject