# Anticipatory Classifier System with Average Reward Criterion in Discretized Multi-Step Environments
## 4 Discussion  
Experiments performed indicated that anticipatory classifier systems with the averaged reward criterion can be used in multi-step environments. The new system AACS2 varies only in a way the classifier reward \(cl.r\) metric is calculated. The clear difference between the discounted criterion is visible on the payoff landscapes generated from the testing environments. The AACS2 can produce a distinct payoff-landscape with uniformly spaced payoff levels, which is very similar to the one generated by the R-learning algorithm. When taking a closer look, all algorithms generate step-like payoff-landscape plots, but each particular state-action pairs are more distinguishable when the reward-criterion is used. The explanation of why the agent moves toward the goal at all can be found in Equation (8)--it is able to find the next best action by using the best classifiers' fitness from the next match set.  
In addition, the rate at which the average estimate value \(\rho\) converges is different for AACS2-v1 and AACS2-v2. Figures 6, 8, and 10 demonstrate that the AACS2-v2 settles to the final value faster, but also has greater fluctuations. That is caused by the fact that both match sets' maximum fitness is considered when updating the values. [PERSON] also observed this and proposed that the learning rate \(\zeta\) in Equation (11) could decay over time [12]:  
\[\zeta\leftarrow\zeta-\frac{\zeta^{max}-\zeta^{min}}{NumOfTrials} \tag{12}\]  
where \(\zeta^{max}\) is the initial value of \(\zeta\), and \(\zeta^{min}\) is the minimum learning rate required. The update should take place at the beginning of each exploration trial.  
In addition, the fact that the optimal \(\rho\) value was not optimal value might be caused by the exploration strategy adopted. The selected policy was \(\epsilon\)-greedy. Because the estimated average reward is updated only when the greedy action is executed, the number of greedy  
Figure 12: Payoff-landscape in the Woods1 environment. Three threshold levels are visible for the Q-Learning and R-Learning algorithms representing states in the environment with a different number of steps to the reward state.  
actions to be performed during the exploration trial is uncertain. In addition, the probability distribution when the agent observes the rewarding state might be too low in order to enable the estimated average reward to reach optimal value. This was observed during the experimentation--the \(\rho\) value was very dependent on the \(\epsilon\) parameter used.  
To conclude, additional research would be beneficial paying extra attention to:  
* the performance on longer and more complicated environments (like containing irrelevant perception bits),
* the impact of different action selection policies, especially those used in ALCSs like the Action-Delay, Knowledge-Array or Optimistic Initial Qualities [18; 37],
* fine-tuning \(\epsilon\) parameter for optimal average reward estimation,
* difference between two versions of AACS2 in terms of using fitness from the match-set. The estimation is calculated differently in both cases, especially in the early phase of learning.  
Conceptualization, O.U. and N.K.; data curation, N.K.; formal analysis, N.K. and O.U.; funding acquisition, O.U.; investigation, N.K.; methodology, N.K. and O.U.; project administration, O.U.; resources, N.K. and O.U.; software, N.K.; supervision, O.U.; validation, O.U. and N.K., visualization, N.K.; writing--original draft preparation, N.K. and O.U.; writing--review and editing, N.K. and O.U. Both authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Not applicable.  
The authors declare no conflict of interest.  
The following abbreviations are used in this manuscript:  
AACS2 Averaged Anticipatory Classifier System 2 ACS Anticipatory Classifier System ALCS Anticipatory Learning Classifier System ALP Anticipatory Learning Process FSW Finite State World LCS Learning Classifier System MDP Markov Decision Problem RL Reinforcement Learning