## 3 Proposed Framework
### Q-Learning Technique  
In this paper, we formulate the dynamic section of the QoS, and QoE provision approaches as the Markov decision process (MDP), where the ITE algorithm selects optimal policy as an agent and the communication network behaves as an environment. The intelligent agent communicate with the environment in sequential decision-making process to observe the current state \(s(t)\), perform an action \(a(t)\) at time \(t\), and move it into the next state \(s(t+1)\). The state of the system, action, and reward function for the proposed system can be summarized as follows:  
* _Agent_: An agent is an entity, which takes out learning tasks in the system. In the proposed framework the ITE that recommends the haptic codec at the application layer acts as an agent. * _State_: A state of the understudy system provides the observation of the environment that can be examined by the agent. In our proposed framework, we define the discrete level of network congestion as the states \(S=s(1),s(2),s(3),\ldots,s(T)\). * _Action_: An action reveals how an intelligent agent responds to the environment based on the observed state. In ITE, the action \(A\) is the set of the haptic codecs schemes, which reduce and compress the packet size or packet rate of haptic traffic during communication to ensure QoS and QoE provisioning. The agent performs an action from the action space of databases of haptic codecs approaches and is denoted as \(A=\{a_{0},a_{1},a_{2},\ldots,a_{n}\}\), where \(n\) refers to the total available haptic codecs as listed in Table 1. In our case, we have a total of seven codecs, so \(n=7\) and action space is \(A=\{a_{0},a_{1},a_{2},\ldots,a_{7}\}\). * _Reward_: The reward to the system is a scalar value. At each time, when an agent executes an action, the environment returns a reward as a response to the agent. The reward characterizes the behavior (good/bad) of the environment to action and the agent adjusts its policies based on the reward. In this system, our aim is to increase the network performance and provide QoS/QoE by minimizing the congestion that leads to delay and packet loss. The ITE uses the power function of the connection as a reward function and defined it as follows: \[R(s_{t},a_{t})=Tput/RTT,\]  
Figure 3: Indepth overview of the proposed ITS. where the \(Tput\) is the throughput and Round Trip Time (RTT) total E2E delay that a packet experiences. So, the reward is explained as the ratio of throughput to RTT. In the MDP, the successor state \(s_{t+1}\) of the system merely relies on the current state \(s_{t}\) of the system instead of the entire prior knowledge. Figure 4 depicts the overview of the optimal haptic codec scheme selection process of the proposed ITE based on the level of network congestion. As it can be seen from Figure 4, in this randomized procedure, when agents interact with the system during learning episodes, the records of the agent's interaction with systems are recorded as a sequence of state, action, and rewards \(\{s(1),a(1),r(1),s(2),a(2),r(2),\ldots,s(T-1),a(T-1),r(T-1)\}\). In every episode, the ITE performs the action \(a(t)\) to select the haptic codecs relating to the policy \(\pi(a_{t}|s_{t})=P[A=a_{t}|S=s_{t}]\). The agent selects the corresponding to the received observation from the environment and here we called it personalized learning action (PLA). After that agent receive a reward of \(r(t)\) and transfer to the next state. The goal is to find the optimal policy value, \(V^{\pi}(s)=\mathbb{E}_{\pi}[\Sigma_{t=1}^{T-1}r(t)]\) for each state after actions select and maximize it using the optimal value function value \(V^{\pi^{\pi}}(s)=\max\limits_{\pi}V^{\pi}\). To achieve this, we utilize the Q-learning algorithm to solve the MDP problem and maximize the state value to select the optimal action. In the literature, Q-learning and its variant algorithms have shown robust results to solve MDP problems. It is an off-policy approach that can be employed for any solution in the MDP framework.