## 4 Framework
### Probe-Scheduling Module  
While identifying device fingerprints, we need to send N probe requests one by one to obtain the device information. The optimal probe sequence for IoT devices can obtain device responses containing more fingerprint information while sending as few probe requests as possible. Different probe methods have different benefits for device identification, and the order in which probe requests are sent can affect the identification benefits dynamically.  
Figure 4: The process of data analysis.  
We model the scheduling problem as a Markov decision process denoted as \((S,A,r,\gamma,T)\)[8]. The goal is to maximize the expected discounted return:  
\[J=E_{\tau}\left[\sum_{t=0}^{T-1}\gamma^{t}r_{t}\right] \tag{1}\]  
where \(\tau\) is the trajectory \((s_{0},a_{0},r_{0},s_{1},\ldots,s_{T-1},a_{T-1},r_{T-1})\) and \(r_{t}=r(s_{t},a_{t})\). The core idea behind the PG algorithm is to obtain the policy gradient \(\
abla_{\theta}J\) of the expected discounted return with respect to the policy parameter \(\theta\).  
\[g_{policy}=\mathbb{E}_{\tau}[\
abla_{\theta}\log\pi_{\theta}(a_{\tau}|s_{\tau })G_{\tau}]=\mathbb{E}_{\tau}\bigg{[}\
abla_{\theta}\sum_{t=0}^{T-1}\log\pi_{ \theta}(a_{t}|s_{t})G_{t}\bigg{]} \tag{2}\]  
where \(G_{t}=\Sigma_{k=0}^{\infty}\gamma^{k}r_{t+k}\) denotes the discounted return following time \(t\).  
The scheduling algorithm is detailed in Algorithm 2. The state set \(S\) refers to the identification state of the device, including the brand, model, and firmware version. The executable action set \(A\) includes probe actions such as probing SNMP responses and probing special URLs.  
```
Input:\(Obj:\) IoT device to be identified; \(S:\) State Space; \(A:\) Action Space; \(r_{t}:\) reward function; Variables:\(P:\) running protocols set for \(Obj\); \(A:\) probe action set; Output:\(\pi_{\theta}:\) probe policy; \(\tau:\) optional probe sequence;
1 Initialise \(\theta\) arbitrarily;
2 for each episodedo
3\(P\,,s_{1}=\) PortScan(obj);
4\(A_{valid}=\) Mask(\(A\),\(P\));
5\(\{s_{1},a_{1},r_{2},...,s_{T-1},a_{T-1},r_{T}\}\sim\pi_{\theta},a_{i}eA_{valid}\);
6 for\(t=1\) to T - 1 do
7\(\theta\leftarrow\theta+\pi\
abla_{\theta}\log\pi_{\theta}(s_{t},a_{t})G_{t}\)
8 end for
9
10 end for return\(\theta\);
```  
**Algorithm 2**Scheduling Algorithm.  
The symbol \(r\) indicates the immediate reward provided by the change in identification status after sending the probe request. According to the identification granularity and the model convergence in the experiment, we set the reward function \(r\). If the device identification state remains unchanged, \(r=-5\); if the brand information of the device is added, \(r=10\); if the model information is added, \(r=50\); if the firmware version information is added, \(r=100\).  
The agent's task is to learn a strategy \(\pi:s\to a\) to choose the next action \(a_{t}\) based on the current state \(s_{t}\), i.e., \(\pi(s_{t})=a_{t}\). At each discrete time \(t\), the agent perceives the current state \(s_{t}\) and chooses the current action \(a_{t}\) according to \(s_{t}\). After obtaining the reward \(r_{t}=R(s_{t},a_{t})\), the agent generates the subsequent state \(s_{t+1}=\delta(s_{t},a_{t})\), which is related to only the current state. Moreover, the optional probe actions differ among devices due to the different open ports. Therefore, we use the MASK [37] to obtain the available action set \(A_{valid}\) from the action set \(A\) for valid policy gradient updates.