# Anticipatory Classifier System with Average Reward Criterion in Discretized Multi-Step Environments
## 2 Materials and Methods
### Integrating Reward Criterions in ACS2  
Despite the ACS's _latent-learning_ capabilities, the RL is realized using two classifier metrics-reward \(cl.r\) and immediate reward \(cl.ir\). The latter stores the immediate reward predicted to be received after acting in a particular situation and is used mainly for model exploitation where the reinforcement might be propagated internally. The reward parameter \(cl.r\) stores the reward predicted to be obtained in the long run.  
For the first version of ACS, [PERSON] proposed a _bucket-brigade_ algorithm to update the classifier's reward \(r_{c}\)[20; 29]. Let \(c_{t}\) be the active classifier at time \(t\) and \(c_{t+1}\) the active classifier at time \(t+1\):  
\[r_{c_{t}}(t+1)=\begin{cases}(1-b_{r})\cdot r_{c_{t}}(t)+b_{r}\cdot r(t+1),& \text{if }r(t+1)\
eq 0\\ (1-b_{r})\cdot r_{c_{t}}(t)+b_{r}\cdot r_{c_{t+1}}(t),&\text{if }r(t+1)=0 \end{cases} \tag{5}\]  
where \(b_{r}\in[0,1]\) is the _bid-ratio_. The idea is that if there is no environmental reward at time \(t+1\), then the currently active classifier \(c_{t+1}\) gives a payment of \(b_{r}\cdot r_{c_{t+1}}(t)\) to the previous active classifier \(c_{t}\). If there is an environmental reward \(r(t+1)\), then \(b_{r}\cdot r(t+1)\) is given to the previous active classifier \(c_{t}\).  
Later, [PERSON] adopted the Q-learning idea in ACS2 alongside other modifications [30]. For the agent to learn the optimal behavioral policy, both the reward \(cl.r\) and intermediate reward \(cl.ir\) are continuously updated. To assure maximal Q-value, the quality of a classifier is also considered assuming that the reward converges in common with the anticipation's accuracy. The following updates are applied to each classifier \(cl\) in action set \([A]\) during every trial:\[cl.r = cl.r+\beta\left(\phi(t)+\gamma\max_{cl^{\prime}\in[M]((t+1))\wedge cl ^{\prime}.E\
eq\{\phi\}^{L}}(cl^{\prime}.q\cdot cl^{\prime}.r)-cl.r\right) \tag{6}\] \[cl.ir = cl.ir+\beta(\phi(t)-cl.ir)\]  
The parameter \(\beta\in[0,1]\) denotes the learning rate and \(\gamma\in[0,1)\) is the discount factor. With a higher \(\beta\) value, the algorithm takes less care of past encountered cases. On the other hand, \(\gamma\) determines to what extent the reward prediction measure depends on future reward.  
Thus, in the original ACS2, the calculation of the discounted reward estimation at a specific time \(t\) is described as \(Q(t)\), which is part of Equation (6):  
\[Q(t)\leftarrow\phi(t)+\gamma\max_{cl^{\prime}\in[M]((t+1))\wedge cl^{\prime}. E\
eq\{\phi\}^{L}}(cl^{\prime}.q\cdot cl^{\prime}.r) \tag{7}\]  
The modified ACS2 implementation replacing the discounted reward with the averaged version with the formula \(R(t)\) is defined below (Equation (8)):  
\[R(t)=\phi(t)-\rho+\max_{cl^{\prime}\in[M]((t+1))\wedge cl^{\prime}.E\
eq\{ \phi\}^{L}}(cl^{\prime}.q\cdot cl^{\prime}.r) \tag{8}\]  
The definition above requires an estimate of the average reward \(\rho\). Equation (4) showed that the maximization of the average reward is achieved by maximizing the relative value. The next sections will propose two variants of setting it to use the average reward criterion for internal reward distribution. The altered version is named AACS2, which stands for _Averaged ACS2_.  
As the next operation in both cases, the reward parameter of all classifiers in the current action set \([A]\) is updated using the following formula:  
\[cl.r\gets cl.r+\beta(R-cl.r) \tag{9}\]  
where \(\beta\) is the learning rate and \(R\) was defined in Equation (8).