## 4 Result and Discussion
### Learning Convergence  
This section presents the study of the Temporal Difference (TD) error made by the proposed system in the learning process over time to explore the different parametric settings to get faster convergence of the system. The main goal is to attain a better convergence with a smaller number of training epochs to reduce the computation overhead. The TD is defined as follows:  
\[\Delta Q=TD(s,a)=r_{t}+\gamma\times\max_{a}Q^{\prime}(s_{t+1},a)-Q^{o}(s_{t},a _{t}), \tag{5}\]  
Figure 8: Packet latency histogram for 3 Dof haptic data transmission; (**a**) master and slave direct communication (no external host are involved), (**b**) External Host = 10, (**c**) External Host = 15, (**d**) External Host = 20.  
where \(r_{t}+\gamma\times\max_{a}Q^{\prime}(s_{t+1},a)\) show TD for newly computed Q-value, while \(Q(s_{t},a_{t})\) refers the previous Q-value for the state \(s_{t}\). The \(\Delta Q\) refers to the change in the Q-value and is obtained by subtracting the prior value from the calculated target value. To study the effect of the learning rate \(\alpha\), on the TD error the \(\alpha\) values from 0.1 to 0.5 is adopted. Practically, the values of the learning rate coefficient are selected in the range of 0.1 to 0.9, where \(\alpha\) value 0 implies no learning of the agent and \(\alpha\) = 1 mean the agent just focused on current knowledge and ignored the previous knowledge. In the proposed ITE we focus on the previous and current level of congestion in the network, so we investigate the ITE on learning rate values \(\alpha\) = 0.1\(\sim\)0.5. The \(\gamma\) highlights the importance of the future reward, and the value of discounted factor \(\gamma\) = 0.8 is adopted for experimentation. The system explored the hidden pattern to recommend the optimal action within fine time episodes \((T=40)\). The analysis of the TD error values for the ITE framework is illustrated in Figures 10 and 11 with values of \(\alpha\) 0.1 and 0.5, respectively. Figure 10a-d depicts the TD for the number of training epochs = 1, 2, 5 for the number of external host = 5, 10, 15, 20 during haptic communication between master and slave. Figure 10a with 5 external host depicts more TD error than Figure 10c,d, where the value of TD error lies between 0 to 0.05. Similarly, in Figure 10a, the ITE system with a value of learning rate 0.1 and epochs = 2-5 shows less error. However, in Figure 10d, with external host = 20 the shows convergence with number of training epoch = 1. The ITE is also simulated with a higher value of \(\alpha\) = 0.5 to explain the better convergence as depicted in Figure 11. Figure 11a-d clearly sgows that the system with \(\alpha\) = 0.5 reveals a significant decrease in TD error. As compared to Figure 10a with 11a from episodes = 1-10 show less error with epochs = 2 and 5. Similarly, Figure 11c for 15 hosts with epochs 20 to 40 shows fewer errors than Figure 10c. The experiment observation denotes that, with learning rate value \(\alpha\) = 0.5 and the number of epochs = 2-5, the proposed ITE shows less error than other parameters settings and shows fast convergence. So, this proves our attention to adopt the \(\alpha\) = 0.5, \(\gamma\) = 0.8 and 2-5 number of training epochs. Moreover, show the better convergence of the proposed system with the selected parametric values throughput, RTT, and power function graphs are also presented. Figure 9: Interarrival packet time for 3 DoF haptic teleoperation traces: (**a**) command/control signals and (**b**) feedback signals. Figure 11: Temporal difference error comparison with parametric value \(\alpha=0.5\) and the number of training epochs ranging 1–5; (**a**) External Host = 5, (**b**) External Host = 10, (**c**) External Host = 15, (**d**) External Host = 20. Figure 10: Temporal difference error comparison with parametric value \(\alpha=0.1\) and the number of training epochs ranging 1–5; (**a**) External Host = 5, (**b**) External Host = 10, (**c**) External Host = 15, (**d**) External Host = 20.