# Volunteers in the Smart City: Comparison of Contribution Strategies on Human-Centered Measures
## 3 Evaluation of Contribution Strategies
### Algorithms
#### 3.1.2 Localized Algorithms  
Decentralized algorithms distribute decision-making at the local level and allow communication between agents for coordination [25] or learning [26]. In this paper, we focus on _localized_ algorithms, a type of decentralized algorithms that operate only on local knowledge, without assuming the availability of specialized hardware for communication [27].  
Aspiration learning is a learning algorithm that is specifically tailored for coordination games [28]. Agents contribution does not consider the current context, e.g., current value or cost; it is instead based on the agent's _aspiration value_, which determines how satisfied the agent is with the status quo, given its previous experiences in the game.  
At every turn, the agent updates its aspiration level \(\rho_{i}(t)\) and chooses an action \(\alpha(t)\). The action for the current turn will be the same action executed in the past turn if the current reward exceeds the aspiration level; otherwise, a random action will be selected with probability  
\[p_{i}(t)=1-\max(h,1+c*[u_{i}(\alpha(t))-\rho_{i}(t)]),\]  
where \(u_{i}(\alpha(t))\) is the reward obtained for executing action \(\alpha(t)\). The aspiration level is updated according to the following formula:  
\[\rho_{i}(t+1)=\rho_{i}(t)+\varepsilon[u_{i}(\alpha(t))-\rho_{i}(t)]+r_{i}(t),\]  
where \(r_{i}(t)\) is a noise component and the value of \(\rho_{i}(t+1)\) is bound between \(\tilde{\rho}\) and \(\underline{\rho}\).  
The algorithm is implemented following the details contained in the original paper, with the following choice of parameters: \(\lambda=10^{-4};c=0.05;h=0.01;\zeta=0.05;\epsilon=10^{-4};\tilde{\rho}=1.1; \underline{\rho}=-1.1\).  
Q-Learning  
Q-Learning is a model-free unsupervised reinforcement learning approach that considers both the history and the current context in the decision [29]. The reward in our implementation is defined as \(R_{q}=S(q,\tau)-c_{i}\times a_{i}\), i.e., a component related to the success of the public good, driven by the collective action, and a negative component driven by the individual cost and individual action. The implementation of Q-Learning relies on TensorFlow, the schema of the network is detailed in Figure 2. Q-Learning is influenced by the choice of parameters \(\alpha\), the learning rate, and \(\gamma\), the learning discount. In our simulations, \(\gamma\) is set to \(0\), as the future states are independent from the choice of action: the scenario can be classified as contextual bandits, where the reward of an action depends on the current state, but the chosen action does not reflect the next state. A parameter sweep on the value of \(\alpha\) showed no impact of the learning rate on the performance of the algorithm; hence, the default value of \(\alpha=0.001\) has been chosen for the experiments.  
A disadvantage of reinforcement learning is its sensitivity to initial conditions--for example, a multi-agent learning process might converge to an inefficient equilibrium where nobody contributes. In order to make this outcome less likely, agents are pre-trained to prefer contribution in order to bias the initial exploration period. Pre-training is a reasonable solution as it can be performed during device manufacturing and its effect on the behavior of agents fades off quickly as agents start learning.