## 4 System Model
### Reinforcement Learning
#### 4.2.2 Q-Learning  
Q-learning is an off-policy RL algorithm to solve the MDP of the proposed strategy. We modeled a stretch reduction problem as an MDP shown in Figure 4 and proposed a Q-learning-based stretch reduction strategy.  
First, we construct an MDP based on the stretch for mesh network topology. The MDP model can be defined by \(H,A,S\). The state space \(H_{n}\) is the set of possible states in our network topology from router \(H_{1}\) to next router. \(A_{i}\) is the set of actions that denotes which what are the required actions in our topology. \(S_{ij}\) is the reward function that has the feedback of reward of what action has been taken by the agent. The goal is to minimize the stretch, and reward shows the quality of step taken by the agent. To estimate the optimal value function, the optimal Q-value is selected for router \(H_{i}\) is defined as:  
\[Q_{i+1}(s_{i},a_{i})=(1-\alpha)Q_{i}(s_{i},a_{i})+\alpha\Delta Q_{i}(s_{i},a_{ i}) \tag{2}\]  
\[\Delta Q(s_{i},a_{i})=r_{i}+\gamma(\max_{a}Q(s_{i+1},a)-Q(s_{i},a_{i})) \tag{3}\]  
We have used the Q-learning-based RL approach; therefore, it is mandatory to declare the environment with some states, actions, policy, and reward. Therefore, \(H_{n}=[H_{1},H_{2},H_{3},\dots,H_{n}]\) is state space in our infrastructure. Where \(n=9\) in our scenario. Similarly, the action space is defined as \(A=[A_{1},\,A_{2},\,A_{3},\,A_{4},\,A_{5}]\). The action space defines the possible hop the content should follow to reach the consumer based on taking the policy. As discussed, the higher the reward minimum will be our stretch. Thus the higher reward declares the minimum stretch in the whole topology.  
\[S_{ij}=1/G_{ij} \tag{4}\]  
where \(G_{ij}\) is the number of routers or hops traveled by the interest packet. Therefore, the inverse of \(G_{ij}\) help to compute stretch \(S_{ij}\) between consumer and publisher.  
For routing decisions in given network topology, an agent searches for an optimal routing path with less number of routers from the BS to final router, where the content is identified with the help of PIT and FIB. The proposed ISO algorithm is discussed and illustrated in Figure 5 and Algorithm 1.  
Figure 4: Framework for the stretch reduction strategy in ICN: (1) BSs estimates the state based on received observations over time t, \(H_{i}\); (2) following the policy the system select the optimal action \(A_{i}\); (3) and finally system receives reward \(S_{ij}\) correspond to the selected action.  
```
Procedure: RL-BASED STRETCH REDUCTION \(H_{i}{\leftarrow}\) Set of all routers \(A_{i}{\leftarrow}\) Action space for routing \(S_{ij}{\leftarrow}\) Reward space \(Q(s,a){\leftarrow}\) Q-table record for every episode \(Q^{*}(s,a){\leftarrow}\) Optimal Q-value function \(\alpha{\leftarrow}\) Learning rate \(\gamma{\leftarrow}\) Discounted factor \(\epsilon{\leftarrow}\) Exploration/exploitation factor Input: Initialization of \(H_{i}\), \(A_{i}\), \(Q(s,a)\), \(\alpha\), \(\gamma\), \(\epsilon\) Output: \(Q^{*}(s,a)\) for shortest path selection  Initialization of \((Q(s,a))\) with zeros for\(i=0,\dots,T-1\)do  Observe current intersection state \(s_{t}=H_{i}\);  The agent selects action: \(a_{t}\gets argmax\ q(s_{t},a_{t})\) with probability \(1-\epsilon\)  The agent executes action \(A_{n}\)  The agent calculates reward; \(r(t)=S_{ij}\) and  Next state \(s_{t+1}\);  Choose nearest \(s_{t+1}\) router based on \((a_{t+1})\)  Calculate reward \(r(t){:}1/G_{i,j}\)  Calculate Q-value and save it into Q-table \(Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha(r_{t}+\gamma\times\max\limits_{a}Q(s_{t +1},a)-Q(s_{t},a_{t}))\)  Update routing table based on Q-table  end
```  
**Algorithm 1**Pseudo-code for proposed ISO strategy.  
Figure 5: Q-learning algorithm for ISO strategy.