## 2 Materials and Methods
### Overview, Design Principles, Details
#### 2.1.4 Design Concepts  
Basic Principles  
In this ABM the governing agent does not enforce its authority over user agents. Rather, it offers them'responsible user' labels in return for cooperation with the governing agent. User agents see improved social status as the benefit of being recognized as a'responsible user'. The basis for this idea is the assumption that individuals have a motivation for better social status, and that they may take actions that cost them money if their peers and neighbors do so [54; 55; 56; 57]. Emergence  
Decisions of user agents are made individually. Emergence of a pattern of such decisions that is costly to the individuals will be an unexpected phenomenon. Adaptation  
The governing agent adapts its \(Q\) based on results of each step, and accordingly calculates a new _policy_ for its actions. Objectives  
The objective of the governing agent is to increase the ratio of cooperating users when it makes signals of 1. The target cooperation ratio is 0.5 in this model. The governing agent aims to reach a state with target cooperation ratio as soon as possible. The reward for the target state is defined as 0, and the reward for all other states is defined as unity minus the ratio of cooperating users. Therefore, the governing agent's reward at each step is between \(-1\) and 0. User agents react to their perceived conditions by comparing the benefit of the said cooperation against their _thresholds_, which represent the cost to each user of cooperation with the governing agent. Figure 1: Interactions between three classes of agents in one time-step. Class names are highlighted. Variable names are shown in italics. Learning  
The governing agent learns to adjust its behavior based on responses that it observes in the user agents. To this end, the governing agent uses RL algorithms. For each RL algorithm there is a separate model. In the RL algorithms, the governing agent stores the value of each action taken at each _state_ in its table, \(Q\). From \(Q\) it extracts _policy_. _Policy_ recommends an action at each _state_. Actions of the governing agent are the _signals_ it produces. In the next time-step, based on the outcome of its action, which is the observed ratio of cooperation of user agents, the governing agent updates its \(Q\) and repeats this loop. Prediction  
In each time-step, the governing agent predicts the present value of the sequence of future rewards of the actions that it may take. This prediction is made based on the results of previous time-steps, and it is stored in \(Q\). As such, \(Q\) is the basis for both learning and prediction in the governing agent. Sensing  
The governing agent and user agents read _nLast_ and _nSum_ from the registrar. The registrar reads _signal_ from the governing agent and _decisions_ from user agents. Interaction  
The interaction of the governing agent with user agents is through _signal_. If the governing agent produces a _signal_ of 0, it is asking for a task that has no cost to the users, hence giving'responsible user' labels to all users at no cost. If it produces a _signal_ of 1, it is asking the users to cooperate in a task that is costly to them. In this case, the response of each user is its _decision_. If the user produces a _decision_ of 0, it is not cooperating with the governing agent. If the user produces a _decision_ of 1, it is cooperating with the governing agent despite the costly demand of the governing agent. Stochasticity  
The governing agent produces its _signals_ using its _policy_, which is stochastic. In the governing agent's _policy_, the probability of recommendation of each action is the ratio of its estimated value to the sum of estimated values of all possible actions. In addition, _thresholds_ of user agents are defined at the beginning of the simulation as random numbers with given mean and standard deviation. Collectives  
Individual _decisions_ of each user agent affect future _decisions_ of itself and other user agents. Other than that, there is no connection between the user agents.