## 5 Performance Evaluation
It describes the number of hope traveled between the consumer and the producer. It manifests that all the initial stages of \(\alpha\) value at 0.1, 0.5, and 0.9, the average number of hope count between consumer and producer varies between 3 and 4. Therefore, it means that at the initial stage, the agent is learning. After some experiences, because of the exploration/exploitation factor, the average number of hop count decreases gradually between consumer and producer. Moreover, at the alpha value of 0.5, the system is much more converged and optimal as compared to the rest of alpha 0.1 and 0.9. Moreover, The results clearly shows that the proposed strategy clearly converges towards the optimal stretch values and lesser number of hopes as compared to random and HARP strategy. Thus our proposed strategy outperforms in comparison to both routing strategies i.e., random and HARP. Moreover, Figure 6c illustrates the instant reward incurred in each episode the agent has taken. We have illustrated the reward gained by the agent or the cost value reward attained by the agent when the agent has followed a certain routing path. It is observed that even after the agent has convergence to the optimal path, there is still some point where the agent has shown the reward of 2, 5, and 7. This is due to the fact that we have kept the \(\varepsilon\) value at 0.5, which leads the agent to 50% explore and 50% exploit. Figure 6d illustrates the total hops covered by an agent from consumer to producer in one episode by an agent. Thus, Figure 6b illustrates the average stretch incurred while the path has been opted by the agent from consumer to producer. Figure 6d shows the hop count incurred in one episode. Similar to Figure 6c, it can be seen that even after the convergence to  
Figure 6: Stretch reduction and routing optimization progress of Q-learning agent at \(\gamma\) = 0.1: (**a**) Accumulated reward gain; (**b**) Average stretch gain; (**c**) Reward earned per episode; (**d**) Total hops/routers covered per episode. optimality, there are still some spikes at 4, 3 because of the reinforcement learning nature of exploration and exploitation. Figure 7a-d has been evaluated and illustrated by changing \(\gamma\) value from 0.1 to 0.5. The performance of the agent is also evaluated \(\gamma\) value of 0.5 with different variants of learning rate i.e., 0.1, 0.5 and 0.9. Figure 7a computes and illustrates the behavior of agent with performance metric of the accumulated reward of routing path between agent and final content router. The \(\gamma\) value is increased to 0.5 and accumulated reward is evaluated at \(\alpha\) values 0.1, 0.5, and 0.9. in Figure 7a more optimal and better trend at \(\alpha\) value 0.5 is observed. Similarly, Figure 7b illustrates the average stretch between the consumer and publisher when the content has been identified at content router. Similarly, we have computed and evaluated the performance of our model with other routing strategies i.e., random and HARP. In our proposed algorithm the stretch incurred at one episode when the agent has completed one episode is illustrated in Figure 7c. It is also illustrated at different \(\alpha\) values of 0.1, 0.5, and 0.9 when \(\gamma\) value is kept constant. The average steps show the corresponding hop count while st in Figure 7a, it illustrates the average stretch. Therefore, Figure 7a,b are describing each other for example, when the average reward is eight at episode 200, then the number of hops taken by the agent from consumer to the producer is 2. Therefore, the agent has been passed two routers while choosing the optimal path. Similarly, Figure 7c illustrates instant reward incurred at each episode the consumer has requested the content and Figure 7d illustrates the total hops the interest packet has traveled. At the same time, the discounted factor is increased to 0.5 and observing the agent's behavior at different \(\alpha\) values.